Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
57 lines
2.3 KiB
Markdown
57 lines
2.3 KiB
Markdown
# R3-0 (Swiftly API ingestion) — GATE PASSED
|
||
|
||
Date: 2026-05-05
|
||
|
||
## Live end-to-end scrape
|
||
|
||
```
|
||
POST /api/admin/scrape (Bearer test-admin-token) → 202 + scrape_log_id
|
||
[bg] Swiftly: discovered 17 categories
|
||
[bg] Background scrape complete: 10908/10908 items saved
|
||
final status: success
|
||
items_scraped: 10908
|
||
duration_seconds: 36
|
||
grocery_item count (after upsert): 9960
|
||
```
|
||
|
||
## Tests
|
||
|
||
```
|
||
pytest -q tests/ → 31 passed in 2.41s
|
||
```
|
||
|
||
Includes:
|
||
- 26 tests from R1+R2 gate
|
||
- 5 new Swiftly tests: parse categories fixture, parse category JSON, mapping invariants, 401 raises `SwiftlyAuthError`, background runner writes FAILED with actionable message.
|
||
|
||
## Schema
|
||
|
||
Migration `0005_grocery_item_external_id.py` adds:
|
||
- `grocery_item.external_id TEXT NULLABLE` (Swiftly product id)
|
||
- `grocery_item.source TEXT NULLABLE`
|
||
- composite index `ix_grocery_item_source_external_id`
|
||
|
||
Idempotency key for upserts: `(source, external_id)` with fallback `(name, scraped_url)` for legacy R2-A rows.
|
||
|
||
## Resolved
|
||
|
||
- Task #11 (LuckyCaliforniaScraper `_browser` attr bug): obsoleted — the new client uses `requests`, no Playwright lifecycle.
|
||
- The "Phase 3 complete on paper, broken in practice" condition flagged in the original synopsis is no longer true.
|
||
|
||
## Behavior to know
|
||
|
||
- `SWIFTLY_BEARER_TOKEN` is read from env at scraper init. The token in `.env.example` and `.env.test` is the user-supplied one and will expire (~hourly). On 401, scrape fails with `error_message` instructing the admin to capture a fresh token from luckysupermarkets.com's devtools network tab and update the env var.
|
||
- Bearer header is scoped to `prod.swiftlyapi.net` only; never sent to the public categories page.
|
||
- `LUCKY_STORE_ID=757` (Lucky California in San Pablo). Switch via env to target a different store.
|
||
- Playwright is still pinned in `requirements.txt` (other future scrapers may want it). Removable later if no other use case lands.
|
||
|
||
## Open items
|
||
|
||
- Task #8: ScrapeStatus lacks distinct queued/started states. Cosmetic.
|
||
- `.env.example` carries the live bearer token; user authorized. Refresh whenever the token expires.
|
||
|
||
## Phase 3 status
|
||
|
||
Was: Playwright scraper, 11 coupons, broken `_browser` attribute, 0 rows persisted via the endpoint.
|
||
Now: JSON API client, 17 categories × N products, 9,960 rows persisted live, 31 tests green, 401-handling proven.
|