Public Access

Files

T

adminandClaude Opus 4.7 8e89f793d5 feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-05 14:08:19 -07:00

4.2 KiB

Raw Blame History

Phase R1 + R2 Verification Gate — PASSED

Date: 2026-05-04

Test results

docker compose run pytest tests/ -q
26 passed in 2.54s

Suite: test_smoke, test_alembic (forward + roundtrip), test_config, test_auth, test_scrape_endpoint, test_approval, test_lucky_ca_scraper.

Verification matrix

Item	Result
`from app.main import app; app.title`	"MealPlanner", 22 routes
`alembic upgrade head` (fresh DB)	0001 → 0002 → 0003 → 0004 clean
`alembic upgrade → downgrade base → upgrade head`	clean (in-test)
`pytest -q`	26 passed, 0 failed
`npm run build` (frontend)	clean, 264 KB
`POST /api/admin/scrape` no token	401
`POST /api/admin/scrape` bad bearer	401
`POST /api/admin/scrape` good bearer	202 + scrape_log_id
`GET /api/profile` no session (read)	200
`PUT /api/profile` no session (mutation)	401
`POST /api/auth/login` good password	204 + Set-Cookie
`POST /api/auth/login` bad password	401
Email approval round-trip (R2-B)	OK: send → GET 200 → POST 200 → replay 409, item_status=approved
Live Lucky CA scrape (R2-A)	11 coupons parsed, schema survives

Pre-existing defects discovered and fixed during the recovery pass

backend/app/api/shopping_list.py:59 — Ingredient.id.in_ all_ingredient_ids syntax error blocking app import. One-char fix.
backend/app/models/__init__.py:201 — MealPlan.votes relationship had no FK target. Removed (votes are reachable via MealPlan.items[*].votes).
backend/alembic/versions/0001_initial_migration.py:101 — JSONB(astext=True) invalid kwarg. Dropped.
backend/alembic/versions/0001_initial_migration.py:288 — downgrade() was pass. Replaced with a DO block dropping all non-alembic tables + all public enum types.
backend/alembic/versions/0002_seed_data.py — duplicate Chickpeas + Black Beans seed rows; INSERT not idempotent. Removed dupes, added ON CONFLICT (name_lower) DO NOTHING.
backend/requirements.txt — scraper imports requests but it wasn't pinned. Added requests==2.31.0.
backend/app/models/__init__.py — every SQLEnum(...) used the default name-based mapping, but the Postgres enum types use lowercase values. All occurrences now use values_callable=lambda obj: [e.value for e in obj].
Schema drift: FamilyProfile.calorie_target existed in the model but not in the migration. New migration 0004_family_profile_calorie_target.py adds it.
docker-compose.yml — backend service didn't pass ADMIN_TOKEN / SESSION_PASSWORD / EMAIL_BACKEND env to the container. Added.
backend/tests/test_config.py — importlib.reload fired the RuntimeError outside pytest.raises. Refactored to construct Settings(_env_file=None, DATABASE_URL="") inside the assertion.
backend/app/api/admin.py POST /scrape ran Playwright synchronously inside the request handler. Now enqueues via BackgroundTasks and returns 202.
grocery_item.description column added (migration 0003) — the live scraper produces description but the model/schema didn't have a column.

Decisions captured in `.agent/context.md`

Auth model: bearer ADMIN_TOKEN for admin, signed-cookie session (itsdangerous, key=SECRET_KEY) for family UI mutations. Reads stay open inside the trusted network.
Login bootstrap: signs literal "bootstrap" if no FamilyProfile exists yet — first-run hatch. Worth flagging to the user explicitly.
Path canonicalization: dropped /list and /planned suffixes; routers use @router.get(""). Frontend updated.
Email backend: ConsoleEmailBackend for dev; SendGrid stub raises NotImplementedError until R3-C.

Open issues tracked but not blocking the gate

Task #8: ScrapeStatus enum lacks distinct queued/started states (R1-C reused STARTED).
Task #10: scraper_service._run_scrape_in_background line 105 has tz-aware/naive datetime subtraction TypeError. Affects bg task happy-path post-202; admin gate verified independently.

Phase R1 + R2 — DONE

Per .agent/plan.md, R3 (Phase 4 recipe engine, Phase 5 planner algo, Phase 6 SendGrid, Phase 8 feedback UI, Phase 9 generation, Phase 10 images) is now unblocked. Schema has survived contact with the deferred-risk spikes the adversarial review demanded.

4.2 KiB Raw Blame History