feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 14:08:19 -07:00
parent b9434967ed
commit 8e89f793d5
58 changed files with 3594 additions and 348 deletions
@@ -0,0 +1,50 @@
+# Lucky California weekly-ad spike (R2-A)
+
+| Field | Value |
+| --- | --- |
+| URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
+| Final URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
+| Fetched (UTC) | 2026-05-04 (single live fetch via scripts/spike_lucky_scrape.py) |
+| HTTP status | 200 |
+| HTML bytes | 201224 |
+| Items parsed (current parser) | 11 |
+| Captcha/block signal | benign — page contains an empty `<span id="recaptcha-element">` placeholder, no actual challenge served. Title: "Featured in Ad \| Luckys Supermarket". 13 real `coupon-card-wrapper` cards rendered. |
+| Error | none |
+| User-Agent | `Mozilla/5.0 (compatible; MealPlannerSpike/0.1; +https://github.com/MealPlanner; spike=R2-A)` |
+
+## Selectors used (verified against this fixture)
+
+- Card root: `div.coupon-card-wrapper` (fallback: `div.coupon-card-container`)
+- Price + short name: `.coupon-card-value-text` (e.g. `"$13.97 Pepsi 24 packs"`)
+- Long description: `.coupon-card-short-description`
+- Image: `img` inside `.coupon-card-img-container` (CDN URL: `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`)
+- No per-card link is present in the rendered DOM (offers are non-navigable tiles).
+
+## First parsed item (sample)
+
+```json
+{
+  "name": "Pepsi 24 packs",
+  "description": "$13.97 Pepsi Products 24 pack, Poppi 8 pack, Gatorade 18 pack, Rockstar 10 pack, or Pure Leaf 12 pack, select varieties +CRV in CA. While supplies last.",
+  "current_price": 13.97,
+  "image_url": "https://cdn.luckysupermarkets.com/loyalty/offer/205213.jpg",
+  "product_url": null,
+  "is_on_sale": true,
+  "scraped_at": "2026-05-04T21:42:56.952936",
+  "scraped_url": "https://luckysupermarkets.com"
+}
+```
+
+## Notes
+
+- ONE live fetch performed by `scripts/spike_lucky_scrape.py`. Do not rerun without reason.
+- HTML and PNG saved alongside this file (`weekly_ad.html`, `weekly_ad.png`).
+- Initial parser (regex-based on `h2/h3/a` text) produced only 1 item; selectors were
+  stale. A minimal additive fix landed in `backend/app/scraper/lucky_ca_scraper.py`:
+  new `_parse_coupon_card` + `parse_featured_coupons_html` methods that target
+  Swiftly-style `.coupon-card-wrapper` cards. Old `_parse_coupon_item` retained
+  as a fallback path.
+- Schema impact: `grocery_item` columns (name, current_price, image_url, is_on_sale,
+  scraped_at, scraped_url) are all populated. `product_url` is None for every
+  card (no per-offer link in DOM) — keep nullable. New optional `description`
+  field is produced; either add a `description TEXT` column or drop it.