Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
61 lines
5.5 KiB
Markdown
61 lines
5.5 KiB
Markdown
# Context — Recovery Takeover
|
|
|
|
## Why this plan exists
|
|
|
|
Prior agent marked Phases 1, 2, 3, 7 complete and consensus blockers "addressed" in docs, but verification of the repo shows:
|
|
|
|
1. Auth blocker (review §1.2) closed in docs only — no auth dependency on any router; `/api/admin/scrape` is open.
|
|
2. No tests, no CI; verification matrix from `Review/reviewconcensus.md §6` was never run.
|
|
3. Review §2.4 explicitly warned: spike scrape + email-approval BEFORE schema/UI commits. Prior agent did the opposite — schema, full API surface, and UI shell first; scrape unverified, email-approval not started.
|
|
4. `/api/admin/scrape` runs Playwright synchronously inside the request handler; will time out in production.
|
|
5. Phase 7 UI ships above engines (4/5/9) that don't exist — Dashboard renders meal plans the system can't generate.
|
|
|
|
## Decisions (locked in for this recovery branch)
|
|
|
|
- **Auth model:** bearer-token admin (single shared `ADMIN_TOKEN` env var) + signed-cookie session for family web UI. Matches what was claimed in ORIENTATION.md "Adversarial Review" section. No public-internet exposure assumed; nginx is sole entrypoint, already correct in `docker-compose.yml`.
|
|
- **Path canonicalization (R1-B+D):** dropped `/list` and `/planned` suffixes; routers use `@router.get("")` (no trailing slash) so the canonical paths are `/api/profile`, `/api/recipes`, `/api/recipes/ingredients`, `/api/meals`, `/api/pantry`, `/api/shopping-list`. Frontend `frontend/src/api/index.ts` and smoke tests updated to enforce.
|
|
- **Login bootstrap:** `/api/auth/login` signs the family-profile id; if no profile row exists yet, signs literal "bootstrap" so first-run isn't blocked. Cookie validates regardless; downstream code that needs a real id should re-issue after profile creation.
|
|
- **Recipe-ingredient:** stay JSONB-only (already chosen). Do not reopen.
|
|
- **Household model:** keep `family_member` table (already chosen). Do not reopen.
|
|
- **Day-of-week:** ISO (1=Mon). Already chosen.
|
|
- **Migrations:** Alembic only. Never `Base.metadata.create_all()` at runtime.
|
|
- **Background work:** FastAPI `BackgroundTasks` for the scrape now; APScheduler container with `--workers 1` later (R3-E).
|
|
|
|
## Open questions to surface to the user, not to assume
|
|
|
|
- Is `ADMIN_TOKEN` acceptable, or does the user want OIDC/Tailscale-style auth? Default for now: bearer token, easy to swap.
|
|
- Email backend for the spike: real SendGrid (needs key) or a console/file backend? Default for spike: console backend, swap to SendGrid in R3-C.
|
|
|
|
## Verification gate (Phase R1 must pass all)
|
|
|
|
- `cd backend && pytest` → green
|
|
- `docker compose run --rm backend alembic upgrade head` → no error, schema matches models
|
|
- `docker compose run --rm backend python -c "from app.main import app; print(app.title)"` → "MealPlanner"
|
|
- `docker compose run --rm frontend npm run build` → no error
|
|
- `curl -X POST http://localhost/api/admin/scrape` (no token) → 401
|
|
- `curl http://localhost/api/profile` (no session) → 200 (read), POST/PUT → 401
|
|
- CI workflow runs all of the above on push.
|
|
|
|
## Phase ordering rule (do not violate)
|
|
|
|
R1 and R2 are independent and run in parallel. R3 cannot start until BOTH R1 verification and R2 spikes pass. If R2 reveals schema impact, schema changes happen on this branch BEFORE R3-A.
|
|
|
|
## Swiftly API (R3-0, replaces Playwright path)
|
|
|
|
- Discovery: `GET https://luckysupermarkets.com/categories` (HTML, no auth). Selector: `<a class="swiftlyCouponCategory" href="/categories/<urlencoded slug>">`. Slug regex: `/categories/(.+)$` then `urllib.parse.unquote`. Fixture (2026-05-05) yielded 17 distinct slugs (e.g. `Product/meat_seafood`, `Product/produce`, ...).
|
|
- Products: `GET https://prod.swiftlyapi.net/search/api/v1/products/categories?cat=<slug>&store=757&limit=10000` with `Authorization: Bearer <SWIFTLY_BEARER_TOKEN>`. Response shape: `{"products": {"info": {"count": N}, "items": [...], "facets": [...]}}`. `meat_seafood` returned 256 items.
|
|
- Field mapping (item dict → grocery_item):
|
|
- `id` (string) → new `external_id` column (migration 0005)
|
|
- `name` → `name`
|
|
- `description` → `description`
|
|
- `brand` → `brand`
|
|
- `primaryImage.url` → `image_url`
|
|
- `price.ok.regPriceText` (e.g. `"$3.49 /lb"`) → parsed `regular_price` (Decimal) + `unit` (e.g. `"lb"`, may be NULL when no `/unit` suffix)
|
|
- `price.ok.promoArea.promoText` (e.g. `"$2.49 /lb"`) → parsed `sale_price` (Decimal); when present `is_on_sale=True`, else `is_on_sale=False`
|
|
- `price.ok.promoArea.validityText` (e.g. `"Valid 04/29/26 - 05/05/26"`) → ignored for v1 (no migration to add date columns; existing `sale_start_date` / `sale_end_date` left null)
|
|
- aisle: extracted from the queried category slug (`Product/meat_seafood` → `meat_seafood`)
|
|
- `product_url` → NULL (site has no public product page; per R2-A note kept nullable)
|
|
- Auth scoping: bearer header is attached ONLY to `prod.swiftlyapi.net` requests, NOT to the public `luckysupermarkets.com` HTML page. Two `requests.Session` objects (one with default UA, one with the bearer header).
|
|
- 401 detection: cannot use `BaseScraper._get` because it swallows HTTPError into a `None` return. The new client calls `session.get(...)` directly and checks `resp.status_code == 401` BEFORE `raise_for_status` to raise `SwiftlyAuthError`. Token in `.env.example` expires hourly per spec; on 401 the scraper aborts with a fixed error_message instructing the admin to refresh the token.
|
|
- Idempotency key: `(source, external_id)` upserts. Migration 0005 adds `grocery_item.external_id` (nullable text, indexed; not unique because legacy R2-A rows lack one).
|