feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-05 14:08:19 -07:00
co-authored by Claude Opus 4.7
parent b9434967ed
commit 8e89f793d5
58 changed files with 3594 additions and 348 deletions
+50
View File
@@ -0,0 +1,50 @@
# Lucky California weekly-ad spike (R2-A)
| Field | Value |
| --- | --- |
| URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
| Final URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
| Fetched (UTC) | 2026-05-04 (single live fetch via scripts/spike_lucky_scrape.py) |
| HTTP status | 200 |
| HTML bytes | 201224 |
| Items parsed (current parser) | 11 |
| Captcha/block signal | benign — page contains an empty `<span id="recaptcha-element">` placeholder, no actual challenge served. Title: "Featured in Ad \| Luckys Supermarket". 13 real `coupon-card-wrapper` cards rendered. |
| Error | none |
| User-Agent | `Mozilla/5.0 (compatible; MealPlannerSpike/0.1; +https://github.com/MealPlanner; spike=R2-A)` |
## Selectors used (verified against this fixture)
- Card root: `div.coupon-card-wrapper` (fallback: `div.coupon-card-container`)
- Price + short name: `.coupon-card-value-text` (e.g. `"$13.97 Pepsi 24 packs"`)
- Long description: `.coupon-card-short-description`
- Image: `img` inside `.coupon-card-img-container` (CDN URL: `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`)
- No per-card link is present in the rendered DOM (offers are non-navigable tiles).
## First parsed item (sample)
```json
{
"name": "Pepsi 24 packs",
"description": "$13.97 Pepsi Products 24 pack, Poppi 8 pack, Gatorade 18 pack, Rockstar 10 pack, or Pure Leaf 12 pack, select varieties +CRV in CA. While supplies last.",
"current_price": 13.97,
"image_url": "https://cdn.luckysupermarkets.com/loyalty/offer/205213.jpg",
"product_url": null,
"is_on_sale": true,
"scraped_at": "2026-05-04T21:42:56.952936",
"scraped_url": "https://luckysupermarkets.com"
}
```
## Notes
- ONE live fetch performed by `scripts/spike_lucky_scrape.py`. Do not rerun without reason.
- HTML and PNG saved alongside this file (`weekly_ad.html`, `weekly_ad.png`).
- Initial parser (regex-based on `h2/h3/a` text) produced only 1 item; selectors were
stale. A minimal additive fix landed in `backend/app/scraper/lucky_ca_scraper.py`:
new `_parse_coupon_card` + `parse_featured_coupons_html` methods that target
Swiftly-style `.coupon-card-wrapper` cards. Old `_parse_coupon_item` retained
as a fallback path.
- Schema impact: `grocery_item` columns (name, current_price, image_url, is_on_sale,
scraped_at, scraped_url) are all populated. `product_url` is None for every
card (no per-offer link in DOM) — keep nullable. New optional `description`
field is produced; either add a `description TEXT` column or drop it.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Binary file not shown.

After

Width:  |  Height:  |  Size: 582 KiB