Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
51 lines
2.5 KiB
Markdown
51 lines
2.5 KiB
Markdown
# Lucky California weekly-ad spike (R2-A)
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
|
|
| Final URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
|
|
| Fetched (UTC) | 2026-05-04 (single live fetch via scripts/spike_lucky_scrape.py) |
|
|
| HTTP status | 200 |
|
|
| HTML bytes | 201224 |
|
|
| Items parsed (current parser) | 11 |
|
|
| Captcha/block signal | benign — page contains an empty `<span id="recaptcha-element">` placeholder, no actual challenge served. Title: "Featured in Ad \| Luckys Supermarket". 13 real `coupon-card-wrapper` cards rendered. |
|
|
| Error | none |
|
|
| User-Agent | `Mozilla/5.0 (compatible; MealPlannerSpike/0.1; +https://github.com/MealPlanner; spike=R2-A)` |
|
|
|
|
## Selectors used (verified against this fixture)
|
|
|
|
- Card root: `div.coupon-card-wrapper` (fallback: `div.coupon-card-container`)
|
|
- Price + short name: `.coupon-card-value-text` (e.g. `"$13.97 Pepsi 24 packs"`)
|
|
- Long description: `.coupon-card-short-description`
|
|
- Image: `img` inside `.coupon-card-img-container` (CDN URL: `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`)
|
|
- No per-card link is present in the rendered DOM (offers are non-navigable tiles).
|
|
|
|
## First parsed item (sample)
|
|
|
|
```json
|
|
{
|
|
"name": "Pepsi 24 packs",
|
|
"description": "$13.97 Pepsi Products 24 pack, Poppi 8 pack, Gatorade 18 pack, Rockstar 10 pack, or Pure Leaf 12 pack, select varieties +CRV in CA. While supplies last.",
|
|
"current_price": 13.97,
|
|
"image_url": "https://cdn.luckysupermarkets.com/loyalty/offer/205213.jpg",
|
|
"product_url": null,
|
|
"is_on_sale": true,
|
|
"scraped_at": "2026-05-04T21:42:56.952936",
|
|
"scraped_url": "https://luckysupermarkets.com"
|
|
}
|
|
```
|
|
|
|
## Notes
|
|
|
|
- ONE live fetch performed by `scripts/spike_lucky_scrape.py`. Do not rerun without reason.
|
|
- HTML and PNG saved alongside this file (`weekly_ad.html`, `weekly_ad.png`).
|
|
- Initial parser (regex-based on `h2/h3/a` text) produced only 1 item; selectors were
|
|
stale. A minimal additive fix landed in `backend/app/scraper/lucky_ca_scraper.py`:
|
|
new `_parse_coupon_card` + `parse_featured_coupons_html` methods that target
|
|
Swiftly-style `.coupon-card-wrapper` cards. Old `_parse_coupon_item` retained
|
|
as a fallback path.
|
|
- Schema impact: `grocery_item` columns (name, current_price, image_url, is_on_sale,
|
|
scraped_at, scraped_url) are all populated. `product_url` is None for every
|
|
card (no per-offer link in DOM) — keep nullable. New optional `description`
|
|
field is produced; either add a `description TEXT` column or drop it.
|