Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2.5 KiB
2.5 KiB
Lucky California weekly-ad spike (R2-A)
| Field | Value |
|---|---|
| URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
| Final URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
| Fetched (UTC) | 2026-05-04 (single live fetch via scripts/spike_lucky_scrape.py) |
| HTTP status | 200 |
| HTML bytes | 201224 |
| Items parsed (current parser) | 11 |
| Captcha/block signal | benign — page contains an empty <span id="recaptcha-element"> placeholder, no actual challenge served. Title: "Featured in Ad | Luckys Supermarket". 13 real coupon-card-wrapper cards rendered. |
| Error | none |
| User-Agent | Mozilla/5.0 (compatible; MealPlannerSpike/0.1; +https://github.com/MealPlanner; spike=R2-A) |
Selectors used (verified against this fixture)
- Card root:
div.coupon-card-wrapper(fallback:div.coupon-card-container) - Price + short name:
.coupon-card-value-text(e.g."$13.97 Pepsi 24 packs") - Long description:
.coupon-card-short-description - Image:
imginside.coupon-card-img-container(CDN URL:cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg) - No per-card link is present in the rendered DOM (offers are non-navigable tiles).
First parsed item (sample)
{
"name": "Pepsi 24 packs",
"description": "$13.97 Pepsi Products 24 pack, Poppi 8 pack, Gatorade 18 pack, Rockstar 10 pack, or Pure Leaf 12 pack, select varieties +CRV in CA. While supplies last.",
"current_price": 13.97,
"image_url": "https://cdn.luckysupermarkets.com/loyalty/offer/205213.jpg",
"product_url": null,
"is_on_sale": true,
"scraped_at": "2026-05-04T21:42:56.952936",
"scraped_url": "https://luckysupermarkets.com"
}
Notes
- ONE live fetch performed by
scripts/spike_lucky_scrape.py. Do not rerun without reason. - HTML and PNG saved alongside this file (
weekly_ad.html,weekly_ad.png). - Initial parser (regex-based on
h2/h3/atext) produced only 1 item; selectors were stale. A minimal additive fix landed inbackend/app/scraper/lucky_ca_scraper.py: new_parse_coupon_card+parse_featured_coupons_htmlmethods that target Swiftly-style.coupon-card-wrappercards. Old_parse_coupon_itemretained as a fallback path. - Schema impact:
grocery_itemcolumns (name, current_price, image_url, is_on_sale, scraped_at, scraped_url) are all populated.product_urlis None for every card (no per-offer link in DOM) — keep nullable. New optionaldescriptionfield is produced; either add adescription TEXTcolumn or drop it.