Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
36 lines
2.3 KiB
Markdown
36 lines
2.3 KiB
Markdown
# R2-A — Lucky California live-scrape spike (DONE)
|
|
|
|
## Outcome
|
|
|
|
- ONE live fetch performed: `https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad`, HTTP 200, 201 KB rendered HTML.
|
|
- Fixture saved: `backend/tests/fixtures/lucky_ca/weekly_ad.html` + `weekly_ad.png` + `META.md`.
|
|
- Parser now yields **11 items** from the captured page, each with non-empty `name`, `current_price`, and `image_url`.
|
|
- Test `backend/tests/test_lucky_ca_scraper.py::test_parse_fixture` passes (`pytest` exit 0, 0.09 s, no network, no Playwright).
|
|
|
|
## Selectors (verified 2026-05-04)
|
|
|
|
- Card root: `div.coupon-card-wrapper` (Swiftly-rendered tiles).
|
|
- Price + short name: `.coupon-card-value-text` ("$13.97 Pepsi 24 packs").
|
|
- Long description: `.coupon-card-short-description`.
|
|
- Image: nested `<img>` from `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`.
|
|
- No per-card hyperlink in DOM; offers are non-navigable tiles.
|
|
|
|
## Scraper diff (minimal, additive)
|
|
|
|
`backend/app/scraper/lucky_ca_scraper.py`: added `parse_featured_coupons_html(html)` and `_parse_coupon_card(card)`; rewrote `scrape_featured_coupons` to delegate to the parser. Old `_parse_coupon_item` retained as fallback. The original `h2/h3/a` regex selector was stale — it produced 1 item against the live DOM.
|
|
|
|
## Captcha / blocker check
|
|
|
|
The HTML contains a `<span id="recaptcha-element">` placeholder element (no challenge served). Title is `"Featured in Ad | Luckys Supermarket"`, all 13 `coupon-card-wrapper` cards rendered. **Not a wall.** The spike script's heuristic flagged the keyword; META.md documents the false positive.
|
|
|
|
## Schema impact (grocery_item)
|
|
|
|
- Survives contact with reality. Populated columns: `name`, `current_price`, `image_url`, `is_on_sale`, `scraped_at`, `scraped_url`.
|
|
- `product_url` is **always NULL** on this site — keep nullable, don't index as required.
|
|
- Parser also produces `description` (long offer text). Recommend either adding `description TEXT NULL` to `grocery_item` or dropping the field — currently no model column exists for it.
|
|
- 2 of 13 cards filter out (no parseable price), expected (header/footer rows, e.g. "$35 minimum order"). Not a regression.
|
|
|
|
## Blockers
|
|
|
|
None. Phase R2-A clears the gate. Recommend a tiny migration adding `grocery_item.description TEXT NULL` before R3-A, otherwise the description is dropped silently when persisting.
|