feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-05 14:08:19 -07:00
co-authored by Claude Opus 4.7
parent b9434967ed
commit 8e89f793d5
58 changed files with 3594 additions and 348 deletions
+56
View File
@@ -0,0 +1,56 @@
# R3-0 (Swiftly API ingestion) — GATE PASSED
Date: 2026-05-05
## Live end-to-end scrape
```
POST /api/admin/scrape (Bearer test-admin-token) → 202 + scrape_log_id
[bg] Swiftly: discovered 17 categories
[bg] Background scrape complete: 10908/10908 items saved
final status: success
items_scraped: 10908
duration_seconds: 36
grocery_item count (after upsert): 9960
```
## Tests
```
pytest -q tests/ → 31 passed in 2.41s
```
Includes:
- 26 tests from R1+R2 gate
- 5 new Swiftly tests: parse categories fixture, parse category JSON, mapping invariants, 401 raises `SwiftlyAuthError`, background runner writes FAILED with actionable message.
## Schema
Migration `0005_grocery_item_external_id.py` adds:
- `grocery_item.external_id TEXT NULLABLE` (Swiftly product id)
- `grocery_item.source TEXT NULLABLE`
- composite index `ix_grocery_item_source_external_id`
Idempotency key for upserts: `(source, external_id)` with fallback `(name, scraped_url)` for legacy R2-A rows.
## Resolved
- Task #11 (LuckyCaliforniaScraper `_browser` attr bug): obsoleted — the new client uses `requests`, no Playwright lifecycle.
- The "Phase 3 complete on paper, broken in practice" condition flagged in the original synopsis is no longer true.
## Behavior to know
- `SWIFTLY_BEARER_TOKEN` is read from env at scraper init. The token in `.env.example` and `.env.test` is the user-supplied one and will expire (~hourly). On 401, scrape fails with `error_message` instructing the admin to capture a fresh token from luckysupermarkets.com's devtools network tab and update the env var.
- Bearer header is scoped to `prod.swiftlyapi.net` only; never sent to the public categories page.
- `LUCKY_STORE_ID=757` (Lucky California in San Pablo). Switch via env to target a different store.
- Playwright is still pinned in `requirements.txt` (other future scrapers may want it). Removable later if no other use case lands.
## Open items
- Task #8: ScrapeStatus lacks distinct queued/started states. Cosmetic.
- `.env.example` carries the live bearer token; user authorized. Refresh whenever the token expires.
## Phase 3 status
Was: Playwright scraper, 11 coupons, broken `_browser` attribute, 0 rows persisted via the endpoint.
Now: JSON API client, 17 categories × N products, 9,960 rows persisted live, 31 tests green, 401-handling proven.