Public Access

Files

T

adminandClaude Opus 4.7 8e89f793d5 feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-05 14:08:19 -07:00

2.3 KiB

Raw Permalink Blame History

R3-0 (Swiftly API ingestion) — GATE PASSED

Date: 2026-05-05

Live end-to-end scrape

POST /api/admin/scrape (Bearer test-admin-token) → 202 + scrape_log_id
[bg] Swiftly: discovered 17 categories
[bg] Background scrape complete: 10908/10908 items saved
final status: success
items_scraped: 10908
duration_seconds: 36
grocery_item count (after upsert): 9960

Tests

pytest -q tests/  →  31 passed in 2.41s

Includes:

26 tests from R1+R2 gate
5 new Swiftly tests: parse categories fixture, parse category JSON, mapping invariants, 401 raises SwiftlyAuthError, background runner writes FAILED with actionable message.

Schema

Migration 0005_grocery_item_external_id.py adds:

grocery_item.external_id TEXT NULLABLE (Swiftly product id)
grocery_item.source TEXT NULLABLE
composite index ix_grocery_item_source_external_id

Idempotency key for upserts: (source, external_id) with fallback (name, scraped_url) for legacy R2-A rows.

Resolved

Task #11 (LuckyCaliforniaScraper _browser attr bug): obsoleted — the new client uses requests, no Playwright lifecycle.
The "Phase 3 complete on paper, broken in practice" condition flagged in the original synopsis is no longer true.

Behavior to know

SWIFTLY_BEARER_TOKEN is read from env at scraper init. The token in .env.example and .env.test is the user-supplied one and will expire (~hourly). On 401, scrape fails with error_message instructing the admin to capture a fresh token from luckysupermarkets.com's devtools network tab and update the env var.
Bearer header is scoped to prod.swiftlyapi.net only; never sent to the public categories page.
LUCKY_STORE_ID=757 (Lucky California in San Pablo). Switch via env to target a different store.
Playwright is still pinned in requirements.txt (other future scrapers may want it). Removable later if no other use case lands.

Open items

Task #8: ScrapeStatus lacks distinct queued/started states. Cosmetic.
.env.example carries the live bearer token; user authorized. Refresh whenever the token expires.

Phase 3 status

Was: Playwright scraper, 11 coupons, broken _browser attribute, 0 rows persisted via the endpoint. Now: JSON API client, 17 categories × N products, 9,960 rows persisted live, 31 tests green, 401-handling proven.

2.3 KiB Raw Permalink Blame History Unescape Escape