Meal-Planner/.agent/phase-summaries/r3-0-summary.md

R3-0 — Swiftly product API client (replaces Playwright path)

Schema migration

Added 0005_grocery_item_external_id.py (down_revision='0004'):

• grocery_item.external_id (String(100), nullable, indexed)
• grocery_item.source (String(50), nullable)
• composite index ix_grocery_item_source_external_id

Idempotency key for upserts is (source, external_id). Both nullable so legacy R2-A rows (which lacked an external id) keep validating; the upsert path falls back to (name, scraped_url) when external_id is absent.

Discovery + sample counts

• Categories page: 17 distinct slugs (e.g. Product/meat_seafood, Product/produce, Product/dairy_eggs_cheese, ...). Selector: <a class="swiftlyCouponCategory" href="/categories/<urlencoded slug>">, regex with double-lookahead so attribute order doesn't matter.
• Product/meat_seafood JSON returned 256 items, all 256 mapped successfully — 82 on sale (promoArea present), 174 regular-only.

Field mapping

Swiftly JSON	grocery_item column
id	external_id (new)
name	name
description	description
brand	brand
primaryImage.url	image_url
price.ok.regPriceText (e.g. "$3.49 /lb")	regular_price, unit
price.ok.promoArea.promoText	sale_price, is_on_sale=True
(queried slug → tail)	aisle (e.g. meat_seafood)
(constant)	source = "lucky_california"
(none)	product_url = NULL (Swiftly exposes none)
validityText	not parsed — sale_start_date/sale_end_date left null

current_price = sale_price when on sale, else regular_price. Prices stored as Decimal.

401 handling

SwiftlyAuthError is a custom exception raised:

1. Up-front when SWIFTLY_BEARER_TOKEN is empty (no HTTP call).
2. On response.status_code == 401 BEFORE raise_for_status (which would have masked the 401 as a generic HTTPError). The new client uses requests.Session.get directly — BaseScraper._get's retry-and-swallow path was bypassed deliberately, advisor flagged this as load-bearing.

_run_scrape_in_background catches all exceptions (existing behavior), writes status=FAILED + error_message="SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)". Admin sees it via GET /api/admin/logs/<id>.

Auth scoping

Two requests.Session objects: public_session (no auth, hits luckysupermarkets.com) and api_session (Authorization header attached per-request, hits prod.swiftlyapi.net). Bearer is never sent to the public host.

Verification

• pytest -q tests/test_swiftly_api.py → 5 passed, 1 skipped (Postgres-only).
• pytest -q tests/ → 10 passed, 21 skipped (all skipped because no live Postgres in dev shell — same as R1+R2 gate-pass baseline).
• python scripts/spike_swiftly_ingest.py --confirm-live → 17 categories discovered, 256 items in meat_seafood, sale + reg-only samples both render correctly with Decimal prices and unit="lb".
• Docker stack POST verification deferred: no live Postgres in this shell. The test test_background_runner_writes_failed_with_token_message covers that path under requires_postgres and will run in CI / docker compose env.

Caveats

• Token in .env.example is a real (expiring) credential, per task spec. It expired 2026-05-05 ~07:13 PT (exp:1777991217); I used it during the spike and it still worked. The user explicitly authorized this. Recommend gitignoring .env.example or rotating to a placeholder in a future cleanup task.
• Old test tests/test_lucky_ca_scraper.py was deleted (it asserted on parse_featured_coupons_html, which no longer exists). R2-A's fixtures weekly_ad.html, weekly_ad.png, META.md retained per task spec.
• BaseScraper and SeleniumScraper classes in app/scraper/base.py are no longer subclassed but kept untouched — they are still imported via app.scraper.__init__ and may be useful for a future second store. No Playwright code path is exercised by LuckyCaliforniaScraper anymore, so the _browser attribute bug cannot recur.
• Rate limit set to 0.75s/req in the new client (between the spec's 1–2 req/sec). Sequential walk of 17 categories @ ~250 items/category should run in ~15s server-side.
• _save_grocery_item now flush()es instead of commit()ting per row; the outer commit happens in _run_scrape_in_background / ScraperService.run_scrape. Trade-off: a single bad row aborts the whole scrape's transaction. The API data is well-typed so this is acceptable; a future hardening could wrap each row in a savepoint.
• Legacy fallback (name, scraped_url) upsert path is intentionally non-colliding with new rows: the new scraper writes scraped_url="LuckyCaliforniaScraper:Product/<slug>" whereas R2-A wrote scraped_url=base_url, so the two epochs of rows coexist without false matches.
• Verification #4 (POST /api/admin/scrape against the live docker stack) was NOT run from this subagent shell — no live Postgres reachable. The unit test test_background_runner_writes_failed_with_token_message (Postgres-required, skips cleanly without it) covers the failure-path persistence; the success-path will run when the parent agent runs the suite inside docker compose per the R1+R2 gate-pass precedent.