Public Access

Files

T

adminandClaude Opus 4.7 8e89f793d5 feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-05 14:08:19 -07:00

2.3 KiB

Raw Permalink Blame History

R2-A — Lucky California live-scrape spike (DONE)

Outcome

ONE live fetch performed: https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad, HTTP 200, 201 KB rendered HTML.
Fixture saved: backend/tests/fixtures/lucky_ca/weekly_ad.html + weekly_ad.png + META.md.
Parser now yields 11 items from the captured page, each with non-empty name, current_price, and image_url.
Test backend/tests/test_lucky_ca_scraper.py::test_parse_fixture passes (pytest exit 0, 0.09 s, no network, no Playwright).

Selectors (verified 2026-05-04)

Card root: div.coupon-card-wrapper (Swiftly-rendered tiles).
Price + short name: .coupon-card-value-text ("$13.97 Pepsi 24 packs").
Long description: .coupon-card-short-description.
Image: nested <img> from cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg.
No per-card hyperlink in DOM; offers are non-navigable tiles.

Scraper diff (minimal, additive)

backend/app/scraper/lucky_ca_scraper.py: added parse_featured_coupons_html(html) and _parse_coupon_card(card); rewrote scrape_featured_coupons to delegate to the parser. Old _parse_coupon_item retained as fallback. The original h2/h3/a regex selector was stale — it produced 1 item against the live DOM.

Captcha / blocker check

The HTML contains a <span id="recaptcha-element"> placeholder element (no challenge served). Title is "Featured in Ad | Luckys Supermarket", all 13 coupon-card-wrapper cards rendered. Not a wall. The spike script's heuristic flagged the keyword; META.md documents the false positive.

Schema impact (grocery_item)

Survives contact with reality. Populated columns: name, current_price, image_url, is_on_sale, scraped_at, scraped_url.
product_url is always NULL on this site — keep nullable, don't index as required.
Parser also produces description (long offer text). Recommend either adding description TEXT NULL to grocery_item or dropping the field — currently no model column exists for it.
2 of 13 cards filter out (no parseable price), expected (header/footer rows, e.g. "$35 minimum order"). Not a regression.

Blockers

None. Phase R2-A clears the gate. Recommend a tiny migration adding grocery_item.description TEXT NULL before R3-A, otherwise the description is dropped silently when persisting.

2.3 KiB Raw Permalink Blame History