docs: refresh ORIENTATION + HANDOFF after r1+r2 recovery and r3-0

Updates ORIENTATION.md and HANDOFF.md to reflect actual state as of commit 8e89f79: phases 1/2/3/7 complete (verified, not just claimed), phases 4/5/6/8/9/10/11 not started. Documents the auth model, the bootstrap login hatch, the SWIFTLY_BEARER_TOKEN expiration handling, the Swiftly JSON API ingestion path that replaced Playwright, the canonicalized API paths, and the verification commands to reproduce the 31/31 pytest gate locally and in CI. HANDOFF.md is intended for fresh agents and points at .agent/ phase-summaries for the per-phase write-ups. Surfaces 10 caveats and traps the next agent will hit if they skim, and recommends Phase 9 (meal-planner generation algorithm) as the next move. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 14:41:19 -07:00
parent 8e89f793d5
commit c953111395
2 changed files with 305 additions and 515 deletions
@@ -1,316 +1,227 @@
-# MealPlanner - Agent Handoff Document
+# MealPlanner — Agent Handoff
-**Project**: MealPlanner - Self-hosted meal planning system
+You are taking over a project in mid-flight. Read `docs/ORIENTATION.md` first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.
-**Last Updated**: 2026-05-04
+
-**Last Agent**: OpenCode (MealPlanner session)
+Date of handoff: 2026-05-05. Last commit before handoff: `8e89f79`.
 ---
-## Project Overview
+## TL;DR
-A self-hosted meal planning system for a family of 4 (2 adults, 2 children) that integrates with Lucky California grocery store. Key problem: replacing meal kit services (Blue Apron → EveryPlate → HungryRoot → Sunbasket) that suffer from 3x ingredient markup, repetitive meals, and no pantry integration.
+The project completed a **recovery pass** (R1+R2+R3-0) from 2026-05-04 to 2026-05-05 because the prior agent had marked Phases 2/3/7 "complete" while leaving 12 distinct defects that prevented the app from importing, prevented migrations from upgrading on Postgres, prevented authentication from existing, and prevented any actual grocery items from being persisted.
-### Core Workflow
+That is now fixed. The app imports, the schema upgrades cleanly, auth works, scraping persists ~10k rows live in 36 s, and the email approval round-trip works end-to-end. 31/31 pytest tests pass.
 1. System generates 7-day meal plan based on Lucky California sales, family dietary constraints, budget, and home pantry items
 2. Email sent to adults with meal proposals and approval links
 3. Adults click email link → confirmation page → POST vote (Approve/Deny)
 4. Per-voter tokens (single-use, 72h TTL)
 5. Majority approve → meal confirmed; any deny → swap
 6. Shopping list generated after approval
-### Family Constraints
+The project's reason to exist — the meal-planner generation algorithm — is **still not started**. That is your top priority. See "Suggested next move" at the bottom.
 - 3 of 4 family members do NOT like mushrooms
 - No allergies
 - Budget-conscious but wants tasty food
 ---
-## Architecture
+## What is real (verified)
-```
+### Backend
-User (email) ──► SendGrid ───────────────────────────────┐
+- `backend/app/main.py` imports cleanly with 22 routes wired.
-User (web)  ───► React UI ──► nginx ──► FastAPI ──────────┼──► PostgreSQL
+- Health: `GET /health`, `GET /health/db`.
-                           │                             │
+- Auth: bearer `ADMIN_TOKEN` for admin; signed-cookie session via `app.security.require_session` for mutations on family-facing routes; `/api/auth/login` with shared `SESSION_PASSWORD`.
-                           └──► Lucky CA scraper ──┘
+- Routers (`backend/app/api/`): `profile.py`, `recipes.py`, `meals.py`, `pantry.py`, `shopping_list.py`, `admin.py`, `auth.py`. CRUD shapes are stubbed/partial — they validate request bodies and persist correctly but business logic is thin.
-                               (luckysupermarkets.com)
+- `POST /api/admin/scrape` enqueues via FastAPI `BackgroundTasks`, returns 202 + `scrape_log_id`. Status polled via `GET /api/admin/logs/{id}`.
-```
+- Lucky California ingestion (`backend/app/scraper/lucky_ca_scraper.py`) is a `requests`-based Swiftly JSON API client. **Not Playwright** — that path was deleted. Discovers 17 categories from `https://luckysupermarkets.com/categories`, fetches each from `prod.swiftlyapi.net/search/api/v1/products/categories?cat=…&store=757&limit=10000`. Bearer scoping: token only ever attached to `prod.swiftlyapi.net` requests, never to the public categories page.
-
+- Approval flow (`backend/app/services/approval.py` + meals router): per-voter `URLSafeTimedSerializer` tokens, TTL, single-use enforced in `consume_token`, GET renders an HTMLResponse vote page, POST records the vote and applies the rule (any deny → item denied; all approve → item approved; otherwise pending).
-### Services (Docker Compose)
+- Email backend (`backend/app/services/email.py`): Protocol + `ConsoleEmailBackend` (writes JSONL to `backend/var/email_outbox.jsonl`) + `SendGridEmailBackend` stub that raises `NotImplementedError`. Selected via `EMAIL_BACKEND` env (default `console`).
- **backend**: FastAPI Python app (port 8000, internal only via nginx)
+- 401 from Swiftly raises `SwiftlyAuthError` carrying the verbatim message `"SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)"`. The bg runner catches it and writes `ScrapeLog.error_message` so it surfaces via the admin logs endpoint.
 - **frontend**: React + Tailwind (port 3000, internal only via nginx)
 - **db**: PostgreSQL 15 (internal only)
 - **nginx**: Reverse proxy with SSL (ports 80/443)
 ### Tech Stack
 - Backend: Python 3.11, FastAPI, SQLAlchemy 2.0, Alembic, Playwright, BeautifulSoup
 - Frontend: React 18, TypeScript, Tailwind CSS, React Query, Vite
 - Database: PostgreSQL 15 with ENUMs and CITEXT
 - Email: SendGrid
 - Hosting: Docker Compose, nginx
 ---
 ## Current Git State
 ```
 git log --oneline -10
 9458edf docs: update ORIENTATION.md with Phase 3/7 progress and current git state
 27bca0e docs: update ORIENTATION.md phase table
 08e196b feat: implement frontend Web UI pages
 933a0cc feat: implement Lucky California scraper with Playwright + BeautifulSoup
 c735d21 feat: implement Phase 2 - Alembic migrations, Pydantic schemas, and real API endpoints
 e8706d3 docs: final ORIENTATION update
 624b516 docs: update ORIENTATION.md for Phase 1 complete
 1328ec3 feat: add Phase 1 infrastructure skeleton
 0c5b0aa docs: add complete project documentation
 ```
 ---
 ## Implementation Phases
 | Phase | Description | Status |
 |-------|-------------|--------|
 | 1 | Infrastructure (Docker, PostgreSQL, FastAPI, React, nginx) | **Complete** |
 | 2 | Database & Models (Alembic migrations, Pydantic schemas, API endpoints) | **Complete** |
 | 3 | Lucky California Scraper (BeautifulSoup + Playwright, ScraperService) | **Complete** |
 | 4 | Recipe Engine (CRUD, tagging, search) | Not Started |
 | 5 | Meal Planner Engine (generation algorithm, substitutions) | Not Started |
 | 6 | SendGrid Email Integration (meal proposal emails) | Not Started |
 | 7 | Web UI - Core (Dashboard, Meal Detail, Pantry, Shopping List) | **Complete** |
 | 8 | Web UI - Feedback (Feedback Portal, Learning) | Not Started |
 | 9 | Meal Planner Generation Algorithm | Not Started |
 | 10 | Image Strategy (scraped + AI fallback) | Not Started |
 | 11 | Polish & Future (variety analysis, budget tracking) | Not Started |
 ---
 ## Key Files and Locations
 ### Backend Structure
 ```
 backend/
 ├── alembic/
 │   ├── env.py                    # Alembic configuration
 │   └── versions/
 │       ├── 0001_initial_migration.py   # Full schema (enums, tables, indexes, constraints)
 │       └── 0002_seed_data.py           # Basic ingredients (70+), family profile, family members
 ├── app/
 │   ├── api/
 │   │   ├── admin.py              # /api/admin/* (scrape trigger, logs, stats)
 │   │   ├── meals.py              # /api/meals/* (meal plans, voting, approval tokens)
 │   │   ├── pantry.py             # /api/pantry/* (CRUD for home pantry)
 │   │   ├── profile.py            # /api/profile/* (family profile, members)
 │   │   ├── recipes.py            # /api/recipes/* (CRUD, ingredients, filtering)
 │   │   └── shopping_list.py      # /api/shopping-list/* (aggregation, print HTML)
 │   ├── config.py                 # Pydantic Settings (reads from .env)
 │   ├── database.py               # SQLAlchemy engine, SessionLocal, get_db
 │   ├── models/
 │   │   └── __init__.py           # All SQLAlchemy models (FamilyProfile, Recipe, MealPlan, etc.)
 │   ├── schemas/
 │   │   └── __init__.py           # All Pydantic schemas for API request/response
 │   ├── scraper/
 │   │   ├── base.py               # BaseScraper with rate limiting, retries, session management
 │   │   ├── lucky_ca_scraper.py   # LuckyCaliforniaScraper with BeautifulSoup + Playwright
 │   │   └── __init__.py
 │   └── services/
 │       └── scraper_service.py    # ScraperService to save scraped items to grocery_item table
 ├── alembic.ini
 ├── Dockerfile
 └── requirements.txt
 ```
 ### Frontend Structure
 ```
 frontend/
 ├── src/
 │   ├── api/
 │   │   └── index.ts              # mealPlannerApi wrapper for all endpoints
 │   ├── pages/
 │   │   ├── Dashboard.tsx         # Weekly meal plan grid view
 │   │   ├── MealDetail.tsx        # Recipe display with ingredients/instructions
 │   │   ├── Pantry.tsx            # Add/remove pantry items
 │   │   └── ShoppingList.tsx      # Grouped by aisle with sale highlighting
 │   ├── types/
 │   │   └── index.ts              # TypeScript interfaces for all models
 │   ├── App.tsx                   # React Router with /, /meals/:id, /pantry, /shopping-list
 │   └── vite-env.d.ts             # Vite env types
 ├── Dockerfile
 ├── package.json
 └── vite.config.ts
 ```
 ### Documentation
 ```
 docs/
 ├── ARCHITECTURE.md       # System architecture diagram
 ├── database-schema.md    # Complete PostgreSQL schema reference
 ├── implementation-plan.md # Detailed phase-by-phase plan
 ├── ORIENTATION.md        # First-stop guide for new agents (READ THIS)
 ├── RUNNING.md            # Deployment guide
 └── SPEC.md               # Project specification
 ```
 ---
 ## Database Schema Highlights
 ### Core Tables
 - `family_profile` - Household with CHECK(household_size > 0)
 - `family_member` - Individual members with email, role, mushroom preference
 - `recipe` - Recipes with JSONB ingredients (not join table)
 - `ingredient` - Master list with name_lower (CITEXT for case-insensitive matching)
 - `meal_plan` - Weekly plan with ISO day_of_week (1=Mon, 7=Sun)
 - `meal_plan_item` - Individual meal with approval_status enum
 - `meal_plan_vote` - Per-member votes (one vote per member per meal)
 - `approval_token` - Single-use tokens with TTL and status tracking
 - `home_pantry` - Family's on-hand ingredients
 - `feedback` - Ratings, denial reasons, never-suggest flags
 - `grocery_item` - Scraped Lucky CA items with FK to ingredient
 - `scrape_log` / `email_log` - Operation history
 ### Key Enums
 - `meal_plan_status_enum`: draft, pending_approval, approved, locked
 - `meal_plan_item_status_enum`: pending, approved, denied, swapped
 - `approval_token_status_enum`: active, used, expired
 - `denial_reason_enum`: too_expensive, boring, disliked_ingredient, cultural, other
 ### Approval Flow (REDESIGNED)
 1. Email contains unique token per family member (not shared)
 2. Token expires after 72 hours
 3. Token can only be used once (marked USED after voting)
 4. Email link → GET confirmation page (NOT auto-approve)
 5. Vote submitted via POST
 ---
 ## Configuration
 ### Environment Variables (.env)
 ```bash
 DATABASE_URL=postgresql://mealplanner:password@db:5432/mealplanner
 SENDGRID_API_KEY=SG.xxx
 FAMILY_EMAIL_1=you@example.com
 FAMILY_EMAIL_2=spouse@example.com
 LUCKY_CA_URL=https://luckysupermarkets.com
 AI_IMAGE_ENABLED=false
 LOG_LEVEL=INFO
 SECRET_KEY=change-me-to-a-random-secret-key
 ```
 ### Key URLs
 - Lucky California: https://luckysupermarkets.com (verified scrapeable)
 - robots.txt: Allows all
 ---
 ## Verification Commands
 ```bash
 # Docker build verification
 docker compose config
 docker compose build backend
 docker compose build frontend
 # Backend import test
 docker compose run --rm backend python -c "from app.main import app; print(app.title)"
 # Alembic migration test (requires running db)
 docker compose run --rm backend alembic upgrade head
 # Frontend build test
 docker compose run --rm frontend npm run build
 ```
 ---
 ## Known Issues / Open Questions
 1. **Lucky California scraping**: Site uses dynamic content (JS rendering). The scraper uses Playwright for browser automation, but actual scraping hasn't been tested with a live database yet. May need adjustment based on actual page structure.
 2. **AI image generation**: Config flag `AI_IMAGE_ENABLED=false`. Not implemented - just a placeholder for future.
 3. **WhatsApp integration**: Planned for future via Twilio. Out of scope for MVP.
 4. **Planned scheduler**: APScheduler with `--workers 1` to avoid duplicate fires. Not yet implemented.
 5. **Meal planner algorithm**: Not implemented. Need to create algorithm that:
   - Filters out never_suggest ingredients/recipes
   - Prioritizes home_pantry items
   - Incorporates sale items from grocery_item
   - Ensures variety (max 2 same protein/sauce per week)
   - Respects mushroom preference (3 of 4 don't like)
 6. **SendGrid email integration**: Not implemented. Need:
   - Email templates for meal proposals
   - Approval email with per-voter tokens
   - Reminder emails before deadline
   - Confirmation emails after approval
 ---
 ## Next Steps (Priority Order)
 ### 1. Implement SendGrid Email Integration (Phase 6)
 - Create `backend/app/services/email_service.py`
 - Implement meal proposal email template
 - Connect to meal plan creation flow
 - Use approval tokens for email links
 ### 2. Implement Meal Planner Generation Algorithm (Phase 9)
 - Create `backend/app/services/meal_planner_service.py`
 - Load constraints (family profile, dietary, budget)
 - Filter never_suggest ingredients/recipes
 - Prioritize home_pantry items
 - Incorporate sale items
 - Ensure variety requirements
 ### 3. Implement Recipe Engine (Phase 4)
 - Recipe CRUD already exists but needs testing
 - Tag-based filtering implemented in API
 - Search functionality needed
 ### 4. Implement Web UI - Feedback (Phase 8)
 - Feedback Portal page
 - Rating submission (1-5 stars)
 - "Never suggest this" flag
 - Learning integration
 ---
 ## Important Conventions
 ### Code Style
 - Python: Black formatter, isort for imports
 - TypeScript: Prettier, ESLint
 - No comments unless explaining non-obvious logic
 ### Git Commits
 - Conventional commits: `feat:`, `fix:`, `docs:`, `refactor:`, `test:`
 - One logical change per commit
 ### API Design
 - RESTful endpoints with proper HTTP methods
 - Pydantic schemas for request/response validation
 - Email approval links use GET → confirmation page → POST
 - Per-voter tokens, not shared household tokens
 ### Database
- UUIDs for primary keys
+- Postgres 15. Five migrations: `0001_initial_migration`, `0002_seed_data`, `0003_grocery_item_description`, `0004_family_profile_calorie_target`, `0005_grocery_item_external_id`.
- Timestamps with timezone (TIMESTAMPTZ)
+- `0001` downgrade now does a `DO $$ … DROP TABLE … DROP TYPE … END $$;` block that preserves `alembic_version`. Round-trip works.
- ENUMs for status fields (not loose VARCHAR)
+- `0002` seed is idempotent (`ON CONFLICT (name_lower) DO NOTHING`).
- CITEXT or lowercase-on-write for name matching
+- Every `SQLEnum(...)` column carries `values_callable=lambda obj: [e.value for e in obj]` — without this, name-mode breaks reads against the lowercase Postgres enum values.
 - `MealPlan.votes` relationship was removed (it had no FK target). Votes are reachable via `MealPlan.items[*].votes`.
 - `grocery_item` upsert key is `(source, external_id)`.
 ### Frontend
 - React 18 + TS + Vite + Tailwind. Dashboard / MealDetail / Pantry / ShoppingList pages exist.
 - API client at `frontend/src/api/index.ts` uses `withCredentials: true` for cookie-based session auth and exposes `auth.login(password)` + `auth.logout()`.
 - **No login UI yet.** No feedback page. No tests.
 ### Tests
 - 31 tests under `backend/tests/`: `test_smoke`, `test_alembic`, `test_config`, `test_auth`, `test_scrape_endpoint`, `test_approval`, `test_swiftly_api`. All green when `TEST_DATABASE_URL` is set; `requires_postgres` marker auto-skips locally without it.
 - Live spike scripts in `scripts/`: `spike_lucky_scrape.py` (R2-A archived path), `send_test_approval.py` (email round-trip prover, supports `--simulate-click {approve|deny}`), `spike_swiftly_ingest.py` (R3-0 live ingestion prover; requires `--confirm-live`).
 ### CI
 - `.github/workflows/ci.yml`: backend job (postgres:15 service, alembic + pytest) + frontend job (npm ci + build). Triggers on push and pull_request.
 ---
-## Contacts
+## What is stubbed or missing
- **Primary user**: Peter (tech-savvy, hosts the system)
+### Phase 4 — Recipe Engine (not started)
- **Wife**: Non-technical, will use web UI and email
+- `/api/recipes` accepts CRUD but doesn't filter by `never_suggest`, doesn't search, doesn't tag.
- **Children**: 2, eating habits vary (one OK with mushrooms)
+- No recipe ingestion source. Decision needed: manual entry only? scrape from public recipe sites (NYT Cooking / Serious Eats / Smitten Kitchen)? AI-generated? user CSV import?
 - The schema is ready: `recipe.cuisine_tags`, `recipe.dietary_tags`, `recipe.protein_type`, `recipe.spice_level`, JSONB `ingredients` array.
 ### Phase 5 — Meal-planner orchestration (not started)
 - The weekly cycle: scrape Sunday → generate Monday → email Monday-evening → deadline Thursday → finalize Friday.
 - All the parts exist (scrape works; email works; vote works; approval rule works) but nothing chains them.
 ### Phase 6 — SendGrid (stub)
 - `SendGridEmailBackend.send` raises `NotImplementedError("Wire SendGrid in R3-C")`. Templates: meal proposal, reminder (T-24h), confirmation, denial.
 - `from_email` / `reply_to` config not added to Settings yet.
 ### Phase 8 — Feedback UI (not started)
 - `feedback` table exists with `rating`, `denial_reason`, free-text. No frontend page reads or writes it. No `/api/feedback` router (folded into `meals.py`?).
 - The "learn from feedback" loop into Phase 9 is unscoped.
 ### Phase 9 — **Meal-planner generation algorithm (not started)**
 - This is the core of the project.
 - Inputs: `family_profile` (size, budget, calorie target), `family_member` preferences (mushroom etc.), `never_suggest` filter, `home_pantry` items, `grocery_item` sales for the week, `recipe` library, prior `feedback` (downweight low-rated, never-suggest-flagged).
 - Output: 7 `MealPlanItem` rows (or 21 if 3 meals/day) with chosen `recipe_id`, day, meal_type.
 - Constraints from spec: variety (max 2 same protein/sauce per week), respect mushroom rule (3 of 4 don't like → recipes containing mushrooms heavily downweighted unless flagged "kid-only"), prefer pantry items, prefer sale items, hit budget per meal.
 ### Phase 10 — Images (not started)
 - Recipe images: scrape from source sites first, AI fallback (`AI_IMAGE_ENABLED=false` flag exists, no implementation).
 ### Phase 11 — Polish (not started)
 - APScheduler container with `--workers 1` to run weekly cadence.
 - Variety analysis dashboard.
 - Budget tracking.
 - WhatsApp via Twilio (out of MVP scope).
 ---
-## File: ORIENTATION.md
+## Known caveats and traps
-**IMPORTANT**: Read `docs/ORIENTATION.md` first before doing anything else. It contains the authoritative project state, phase table, and next steps.
+1. **Bootstrap login hatch.** `app/api/auth.py` login: when no `family_profile` row exists, it signs the literal string `"bootstrap"` instead of a UUID. Anyone with `SESSION_PASSWORD` gets a session even with zero data in the DB. Acceptable for self-hosted on a trusted network. Replace with a proper first-run setup gate before exposing the system beyond the LAN/VPN. The decision is documented in `.agent/context.md` under "Decisions".
 2. **`SWIFTLY_BEARER_TOKEN` expires hourly.** It's a Firebase anonymous-auth JWT scoped to `swiftly-lu-prod`. The user explicitly chose to surface a request when it expires rather than mint new tokens automatically. On 401, `ScrapeLog.error_message` carries the actionable message. The user captures a fresh token from luckysupermarkets.com devtools and updates the env var. Don't try to automate Firebase auth unless the user asks.
 3. **`.env.example` ships a real (expiring) token.** Per user authorization. If it's already expired by the time you read this, that's expected — surface the refresh request to the user. Do not log it.
 4. **`ScrapeStatus` enum reuses `STARTED` for the queued state.** R1-C didn't add a `QUEUED` value because that would have churned the Postgres enum type. Cosmetic. If you change it, add a migration.
 5. **Pytest's transactional `db` fixture rolls back at teardown.** Background tasks open their own `SessionLocal()` and don't see uncommitted data. `test_swiftly_api.py::test_background_runner_writes_failed_with_token_message` is the example of how to test bg-task behavior — use a separate non-fixture session, commit, run, verify, clean up explicitly.
 6. **`alembic downgrade base` in 0001 preserves `alembic_version` table.** Don't change this to `DROP SCHEMA public CASCADE` — that would also drop `alembic_version` and break the alembic state machine on the next upgrade.
 7. **Login bootstrap aside, `family_profile` is currently empty in any fresh DB.** Phase 9 must either seed it during the first-run flow or assume the admin manually created the row. Either way, document it.
 8. **Routes use `@router.get("")` (no trailing slash).** FastAPI's `redirect_slashes=True` (the default) will 307-redirect `/api/profile/` to `/api/profile`. Tests assert canonical paths (no slash). The frontend client matches.
 9. **No `recipe` data exists.** Phase 4 needs an ingestion strategy before Phase 9 can do anything useful.
 10. **Frontend doesn't have a login UI.** Until you build one, the family-facing flows can't actually be exercised by a real user — only by tests. The Dashboard/Pantry/etc. pages assume the cookie is already set.
 ---
 ## Verification commands
 Same as `docs/ORIENTATION.md`:
 ```bash
 # Stack up
 docker compose --env-file .env.test up -d db backend
 docker compose --env-file .env.test exec backend alembic upgrade head
 # Tests
 docker compose --env-file .env.test exec \
  -e TEST_DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner \
  backend pytest -q tests/   # → 31 passed
 # Frontend
 cd frontend && npm ci && npm run build
 # Email approval round-trip
 docker cp scripts/send_test_approval.py mealplanner-backend-1:/app/send_test_approval.py
 docker compose --env-file .env.test exec backend \
  python /app/send_test_approval.py --simulate-click approve
 # Live Swiftly ingest (will hit the real API once)
 docker cp scripts/spike_swiftly_ingest.py mealplanner-backend-1:/app/spike_swiftly_ingest.py
 docker compose --env-file .env.test exec backend \
  python /app/spike_swiftly_ingest.py --confirm-live
 ```
 ---
 ## Suggested next move
 **Phase 9 first — meal-planner generation algorithm.**
 Why: the schema is provably real now. The grocery feed is live. The approval flow works. Email and recipe ingestion can be developed against fixtures while you build the engine. Without Phase 9 the project has no reason to exist.
 Sketch:
 1. **Phase 4 (recipe ingestion) just enough to feed Phase 9.** Cheapest path: a CSV / JSON seed of ~30 recipes the family already knows + `POST /api/recipes/import` that takes a JSON body. Punt scraping recipe sites until later.
 2. **Phase 9 algorithm.** Inputs listed above. Output: 7 `MealPlanItem` rows. Start dumb — random selection respecting `never_suggest` + mushroom rule. Iterate to add variety, sales bias, pantry bias, budget.
 3. **Phase 5 orchestration + Phase 6 SendGrid** — chain everything.
 4. **Phase 8 feedback UI** — close the learning loop.
 Brainstorm with the user before committing to the algorithm shape. Use the `superpowers:brainstorming` skill.
 ---
 ## Open tasks (carried over from recovery)
 | ID | Subject | Priority |
 |---|---|---|
 | #8 | `ScrapeStatus` enum could use a distinct `QUEUED` value | Cosmetic |
 Other tasks in the recovery session were closed. See `.agent/phase-summaries/` for the detailed write-ups of each phase (R1A test harness, R1B+D auth+paths, R1C async scrape, R2A live scrape, R2B email approval, R3-0 Swiftly ingestion).
 ---
 ## Useful files map
 ```
 .agent/
 ├── plan.md                              recovery plan (R1, R2, R3 phases)
 ├── context.md                           locked-in decisions
 └── phase-summaries/                     per-phase write-ups
    ├── r1-r2-gate-pass.md
    ├── r3-0-gate-pass.md
    ├── r1a-summary.md
    ├── r1bd-summary.md
    ├── r1c-summary.md
    ├── r2a-summary.md
    ├── r2b-summary.md
    ├── r2b-blockers.md
    └── r3-0-summary.md
 backend/app/
 ├── api/
 │   ├── admin.py                         scrape trigger + logs (admin-gated)
 │   ├── auth.py                          login/logout (R1-B+D)
 │   ├── meals.py                         meal plans + vote routes (per-token)
 │   ├── pantry.py
 │   ├── profile.py
 │   ├── recipes.py
 │   └── shopping_list.py
 ├── scraper/
 │   ├── base.py                          rate-limited HTTP base
 │   └── lucky_ca_scraper.py              Swiftly JSON API client (R3-0)
 ├── services/
 │   ├── approval.py                      per-voter token issue/verify/consume
 │   ├── email.py                         Console + SendGrid stub
 │   └── scraper_service.py               enqueue + bg runner
 ├── security.py                          require_admin / require_session
 ├── config.py                            pydantic Settings
 ├── database.py                          engine / SessionLocal / get_db
 └── models/__init__.py                   all SQLAlchemy models
 backend/alembic/versions/                0001 → 0005
 backend/tests/                           31 tests
 backend/tests/fixtures/lucky_ca/         categories.html, category_meat_seafood.json, weekly_ad.html (R2-A archive)
 scripts/
 ├── send_test_approval.py                email round-trip prover
 ├── spike_lucky_scrape.py                R2-A archived
 └── spike_swiftly_ingest.py              R3-0 live ingest prover
 .github/workflows/ci.yml                 backend (postgres + pytest) + frontend (npm build)
 ```
 ---
 ## Final words
 Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself. The recovery happened because the prior agent did the latter without the former.
 Last updated: 2026-05-05.
@@ -1,273 +1,152 @@
-# Meal Planner - Orientation Guide
+# Meal Planner — Orientation
-This document is the **first stop** for any agent resuming work on this project after context compaction. Read it before doing anything else.
+First stop for any agent resuming work. Read this, then `docs/HANDOFF.md` for the deep dive.
 ---
-## Project Overview
+## What this project is
-**MealPlanner** is a self-hosted meal planning system for a family of 4 (2 adults, 2 children) that integrates with Lucky California grocery store to source ingredients from weekly sales, sends meal proposals via email, generates shopping lists, and learns from feedback.
+Self-hosted meal planning for one family of 4. Pulls weekly grocery prices from Lucky California (San Pablo, store 757) via the Swiftly JSON API, generates a 7-day meal plan, emails per-member approval links, builds a shopping list grouped by aisle. Replaces meal-kit subscriptions (Blue Apron / Sunbasket / etc.) which marked up ingredients ~3× and produced repetitive meals.
-### Key Problem Being Solved
+Constraint that drives the design: 3 of 4 members do not like mushrooms; family is calorie/budget conscious; no allergies. Must work for non-technical wife + 2 kids; technical owner self-hosts.
 The family has been using meal kit services (Blue Apron → EveryPlate → HungryRoot → Sunbasket) which suffer from:
 - 3x ingredient markup in cost
 - Repetitive meals and sauces
 - Forcing app login to manage meal selection
 - No home pantry integration
 ### Current Status
 **Phase**: Phase 2 (Database & Models) IN PROGRESS. Migrations created, schemas implemented, API endpoints implemented.
 **Adversarial Review Completed**: 2026-05-04
 - All consensus blockers (§1.1 - §1.8) addressed
 - All high-risk gaps (§2.1 - §2.8) addressed
 - Key fixes applied:
  - Recipe-ingredient: JSONB only (removed join table)
  - Family member: Added `family_member` table for per-voter tracking
  - Approval flow: Redesigned with confirmation page + POST + per-voter tokens + TTL
  - Auth: VPN-only for admin endpoints, session-based for family web UI
  - Schema: ENUMs, CHECKs, CITEXT for ingredients, ISO day_of_week (1=Mon)
  - Lucky URL: Fixed to `https://luckysupermarkets.com`
  - Docker: Hardened (no direct port exposure to backend/frontend)
  - Alembic: Configured with migration policy
 **Phase 2 Progress** (2026-05-04):
 - [x] Initial Alembic migration (0001_initial_migration.py)
 - [x] Seed data migration (0002_seed_data.py)
 - [x] Pydantic schemas for all models
 - [x] /api/profile endpoints (CRUD, family members)
 - [x] /api/recipes endpoints (CRUD, ingredients)
 - [x] /api/meals endpoints (meal plans, voting, approval tokens)
 - [x] /api/pantry endpoints (CRUD)
 - [x] /api/shopping-list endpoints (aggregation, print)
 - [x] /api/admin endpoints (scrape trigger, logs, stats)
 **Phase 3 Progress** (2026-05-04):
 - [x] Base scraper with rate limiting, retries, session management
 - [x] LuckyCaliforniaScraper with BeautifulSoup + Playwright
 - [x] ScraperService to save scraped items to grocery_item table
 - [x] /api/admin/scrape endpoint connected to ScraperService
 **Phase 7 Progress** (2026-05-04):
 - [x] Types: FamilyProfile, Recipe, MealPlan, Ingredient, ShoppingList, etc.
 - [x] API client: mealPlannerApi wrapper for all endpoints
 - [x] Dashboard: weekly meal plan grid view with day/meal columns
 - [x] Pantry page: add/remove items with ingredient selection
 - [x] MealDetail page: recipe display with ingredients and instructions
 - [x] ShoppingList page: grouped by aisle with sale highlighting
 ---
-## Architecture Summary
+## Architecture (current)
 ```
-User (email) ──► SendGrid ───────────────────────────────┐
+email ─► SendGrid (R3-C, not yet wired) ──┐
-User (web)  ───► React UI ──► nginx ──► FastAPI ──────────┼──► PostgreSQL
+web   ─► nginx :80/:443 ─► React/Vite ────┼─► FastAPI ──► PostgreSQL 15
-                          │                             │
+                                          │      │
-                          └──► Lucky CA scraper ──┘
+                                          │      └─► Swiftly JSON API
-                              (luckysupermarkets.com)
+                                          │         (prod.swiftlyapi.net)
                                          │
                                          └─► /api/admin/scrape (BackgroundTasks)
 ```
-### Services (Docker Compose)
+- **backend** (FastAPI 0.109, SQLAlchemy 2.0, Alembic) — internal only, `expose: 8000`
- **backend**: FastAPI Python app (port 8000, internal only)
+- **frontend** (React 18 + TS + Vite + Tailwind) — internal only via nginx
- **frontend**: React + Tailwind (port 3000, internal only via nginx)
+- **db** (Postgres 15-alpine) — internal only
- **db**: PostgreSQL 15 (internal only)
+- **nginx** — sole external entry, ports 80/443
 - **nginx**: Reverse proxy with SSL (ports 80/443)
 ### Tech Stack
 - Backend: Python 3.11, FastAPI, SQLAlchemy 2.0, Alembic
 - Frontend: React 18, TypeScript, Tailwind CSS, React Query, Vite
 - Database: PostgreSQL 15 with ENUMs and CITEXT
 - Scraping: Playwright, BeautifulSoup
 - Email: SendGrid
 - Hosting: Docker Compose, nginx
 ---
-## Family Profile
+## Phase status (2026-05-05)
-### Household
+| # | Phase | Status |
- 2 adults, 2 children
+|---|---|---|
- 3 of 4 members do NOT like mushrooms
+| 1 | Infra (Docker, FastAPI, React, nginx, Postgres) | **Complete** |
- No allergies
+| 2 | DB & models (Alembic, Pydantic schemas, API endpoints) | **Complete** (real, verified) |
- Goals: Calorie, budget, and health conscious; tasty but not expensive
+| 3 | Lucky California ingestion (Swiftly JSON API) | **Complete** — 17 categories, ~10k products live |
 | 4 | Recipe engine (CRUD, search, tagging, never-suggest filter) | Not started |
 | 5 | Meal planner orchestration (generate → email → vote → finalize) | Not started |
 | 6 | SendGrid email integration (proposal/reminder/confirmation) | Stub only — `app/services/email.py::SendGridEmailBackend` raises NotImplementedError |
 | 7 | Web UI core (Dashboard / Meal Detail / Pantry / Shopping List) | **Complete** (no auth UI yet) |
 | 8 | Web UI feedback portal | Not started |
 | 9 | **Meal-planner generation algorithm** | Not started — *the core of the project* |
 | 10 | Image strategy (scraped + AI fallback) | Not started |
 | 11 | Polish (variety analysis, budget tracking, APScheduler) | Not started |
-### Family Members
+Verification gate (R1+R2 + R3-0): 31/31 pytest green; alembic upgrade→downgrade→upgrade clean; frontend `npm run build` clean; live scrape persists 9,960 grocery_item rows in 36 s; email approval round-trip (approve/deny/single-use) verified end-to-end.
 | Member | Role | Mushroom Preference |
 |--------|------|-------------------|
 | Adult 1 | Adult | Does NOT like mushrooms |
 | Adult 2 | Adult | Likes mushrooms |
 | Child 1 | Child | Does NOT like mushrooms |
 | Child 2 | Child | OK with mushrooms |
 ### Approval Workflow (REDESIGNED)
 1. System generates 7-day meal plan (Sunday)
 2. Email sent to all adults with meals, images, approval page links
 3. Email link → confirmation page (GET, not auto-approve)
 4. Adult clicks Approve/Deny → POST with reason
 5. Per-member token (single-use, 72h TTL)
 6. Majority approve → meal confirmed; any deny → swap
 7. After approval → shopping list generated
 ---
-## Database Schema Highlights
+## Auth model (R1-B+D, locked in)
-### Core Tables
+- Bearer token `ADMIN_TOKEN` for every `/api/admin/*` route.
- `family_profile` - Household configuration with household_size CHECK
+- Signed-cookie session (itsdangerous, key=`SECRET_KEY`) for all NON-GET routes on profile/pantry/recipes/meals/shopping-list. GET reads stay open inside the trusted network.
- `family_member` - Individual family members with email, role, mushroom preference
+- `/api/auth/login` accepts `{password}` matching `SESSION_PASSWORD`. Sets the cookie.
- `recipe` - Recipes with JSONB ingredients (not join table)
+- `/api/meals/vote/{item_id}?token=…` keeps its per-voter token flow; not session-gated.
- `ingredient` - Master list with name_lower (CITEXT for case-insensitive matching)
+- **Bootstrap hatch**: when no `family_profile` row exists, login signs the literal string `"bootstrap"` instead of a UUID. First-run convenience only — replace with a real setup gate before any non-trusted exposure.
 - `meal_plan` - Weekly plan with ISO day_of_week (1=Mon, 7=Sun)
 - `meal_plan_item` - Individual meal with approval status
 - `meal_plan_vote` - Per-member votes (one vote per member per meal)
 - `approval_token` - Single-use tokens with TTL and status tracking
 - `home_pantry` - Family's on-hand ingredients
 - `feedback` - Ratings, denial reasons, never-suggest flags
 - `grocery_item` - Scraped Lucky California items with FK to ingredient
 - `scrape_log` / `email_log` - Operation history
 ### Key Relationships
 - `family_profile` 1:N `family_member`
 - `family_profile` 1:N `meal_plan`
 - `family_member` 1:N `meal_plan_vote` (per-voter tracking)
 - `recipe` 1:N `meal_plan_item`
 - `meal_plan` 1:N `meal_plan_item`
 - `meal_plan_item` 1:N `meal_plan_vote`
 - `meal_plan_item` 1:N `approval_token`
 - `grocery_item` → `ingredient` (FK)
 ---
-## Implementation Phases
+## Database schema highlights
-| Phase | Description | Status |
+Core tables: `family_profile`, `family_member`, `recipe`, `ingredient`, `meal_plan`, `meal_plan_item`, `meal_plan_vote`, `approval_token`, `home_pantry`, `feedback`, `grocery_item`, `scrape_log`, `email_log`.
-|-------|-------------|--------|
+
-| 1 | Infrastructure (Docker, PostgreSQL, FastAPI, React, nginx) | **Complete** |
+Conventions: UUID PKs everywhere, `TIMESTAMPTZ`, Postgres ENUMs (with `values_callable=lambda obj: [e.value for e in obj]` on every SQLEnum — name-mode silently breaks otherwise), ISO day-of-week (1=Mon).
-| 2 | Database & Models (Alembic migrations, Pydantic schemas, API endpoints) | **Complete** |
+
-| 3 | Lucky California Scraper (BeautifulSoup + Playwright, ScraperService) | **Complete** |
+Key relationships: `family_profile`→`family_member`; `family_member`→`meal_plan_vote` (per-voter); `recipe.ingredients` JSONB (no recipe-ingredient join table); `grocery_item.(source, external_id)` is the upsert key for scrape ingestion.
-| 4 | Recipe Engine (CRUD, tagging, search) | Not Started |
+
-| 5 | Meal Planner Engine (generation algorithm, substitutions) | Not Started |
+Migrations applied: 0001 initial, 0002 seed (idempotent via `ON CONFLICT DO NOTHING`), 0003 grocery_item.description, 0004 family_profile.calorie_target, 0005 grocery_item.external_id + source + composite index.
-| 6 | SendGrid Email Integration | Not Started |
+
-| 7 | Web UI - Core (Dashboard, Meal Detail, Pantry, Shopping List) | **Complete** |
+Full schema: `docs/database-schema.md`.
 | 8 | Web UI - Feedback (Feedback Portal, Learning) | Not Started |
 | 9 | Meal Planner Generation Algorithm | Not Started |
 | 10 | Image Strategy (scraped + AI fallback) | Not Started |
 | 11 | Polish & Future (variety analysis, budget tracking) | Not Started |
 ---
-## Environment Variables
+## Environment variables (verified)
 ```bash
 # Database
-DATABASE_URL=postgresql://mealplanner:password@db:5432/mealplanner
+POSTGRES_PASSWORD=...
-POSTGRES_PASSWORD=secure_password
+DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner
 # SendGrid
 SENDGRID_API_KEY=SG.xxx
 # Family
 FAMILY_EMAIL_1=user@example.com
 FAMILY_EMAIL_2=spouse@example.com
 # Scraping
 LUCKY_CA_URL=https://luckysupermarkets.com
 # AI Images (optional)
 AI_IMAGE_ENABLED=false
 # Auth
-SECRET_KEY=change-me-in-production
+SECRET_KEY=...                    # signs session cookies + approval tokens
 ADMIN_TOKEN=...                   # bearer for /api/admin/*
 SESSION_PASSWORD=...               # family-shared password for /api/auth/login
 # Email
 EMAIL_BACKEND=console              # 'console' (default) or 'sendgrid'
 SENDGRID_API_KEY=...              # only when EMAIL_BACKEND=sendgrid (R3-C)
 # Lucky / Swiftly
 LUCKY_STORE_ID=757                 # Lucky California — San Pablo
 SWIFTLY_API_BASE=https://prod.swiftlyapi.net
 SWIFTLY_CATEGORIES_URL=https://luckysupermarkets.com/categories
 SWIFTLY_BEARER_TOKEN=...           # Firebase anon JWT, expires hourly
 # Other
 LUCKY_CA_URL=https://luckysupermarkets.com
 AI_IMAGE_ENABLED=false
 LOG_LEVEL=INFO
 ```
 `.env.example` carries a literal expiring bearer token — rotate before any real run. On expiry, the next scrape's `ScrapeLog.error_message` reads `SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)`. Fix: capture a fresh `Authorization: Bearer …` from devtools, update env, restart backend, re-trigger.
 ---
-## Next Steps
+## Verification commands
 ### IMMEDIATE: Verify skeleton with verification matrix
 ```bash
-docker compose config
+# Full local stack
-docker compose build backend
+docker compose --env-file .env.test up -d db backend
-docker compose build frontend
+docker compose --env-file .env.test exec backend alembic upgrade head
-docker compose up -d db
+docker compose --env-file .env.test exec -e TEST_DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner backend pytest -q tests/
-docker compose run --rm backend python -c "from app.main import app; print(app.title)"
+
-docker compose run --rm backend alembic upgrade head
+# Frontend
-docker compose run --rm frontend npm run build
+cd frontend && npm ci && npm run build
 # CI: .github/workflows/ci.yml runs both jobs on push/PR.
 ```
-If any of these fail, fix before proceeding.
+A `.env.test` template lives in the repo root (gitignored) for local stack runs. Pytest uses `TEST_DATABASE_URL`; alembic uses `DATABASE_URL`.
 ### After Verification
 1. Phase 4: Recipe Engine (CRUD, tagging, search)
 2. Phase 5: Meal Planner Engine (generation algorithm, substitutions)
 3. Phase 6: SendGrid Email Integration (meal proposal emails with approval links)
 ---
-## Current Git State
+## Conventions
-```bash
+- Python: Black + isort. TypeScript: Prettier + ESLint.
-git log --oneline -10
+- Conventional commits (`feat:`, `fix:`, `docs:`, `refactor:`, `test:`).
-27bca0e docs: update ORIENTATION.md phase table
+- API paths: no trailing slash, no `/list`/`/planned` suffixes.
-08e196b feat: implement frontend Web UI pages
+- Tests: pytest; `requires_postgres` marker auto-skips locally without `TEST_DATABASE_URL`.
-933a0cc feat: implement Lucky California scraper with Playwright + BeautifulSoup
+- Migrations: Alembic only. Never `Base.metadata.create_all()` at runtime.
-c735d21 feat: implement Phase 2 - Alembic migrations, Pydantic schemas, and real API endpoints
+- Background work: FastAPI `BackgroundTasks` (current). APScheduler with `--workers 1` planned for Phase 11.
 e8706d3 docs: final ORIENTATION update
 624b516 docs: update ORIENTATION.md for Phase 1 complete
 1328ec3 feat: add Phase 1 infrastructure skeleton
 0c5b0aa docs: add complete project documentation
 ```
 ---
-## Important Conventions
+## Where to look
-### Code Style
+- `docs/HANDOFF.md` — comprehensive handoff for fresh agents (start here for non-trivial work).
- Python: Black formatter, isort for imports
+- `docs/SPEC.md` — product spec.
- TypeScript: Prettier, ESLint
+- `docs/ARCHITECTURE.md` — system design.
- No comments unless explaining non-obvious logic
+- `docs/database-schema.md` — full DDL reference.
-
+- `docs/implementation-plan.md` — original phased plan.
-### Git Commits
+- `docs/RUNNING.md` — local dev workflow.
- Conventional commits: `feat:`, `fix:`, `docs:`, `refactor:`, `test:`
+- `.agent/plan.md`, `.agent/context.md`, `.agent/phase-summaries/` — recovery decisions and per-phase summaries from the R1+R2+R3-0 work.
- One logical change per commit where possible
+- `Review/reviewconcensus.md` — the adversarial review that drove the recovery.
 ### API Design
 - RESTful endpoints with proper HTTP methods and status codes
 - Pydantic schemas for request/response validation
 - Email approval links use GET → confirmation page → POST
 - Per-voter tokens, not shared household tokens
 ### Database
 - Always use UUIDs for primary keys
 - Timestamps with timezone (TIMESTAMPTZ)
 - Use ENUMs for status fields (not loose VARCHAR)
 - Use CITEXT or lowercase-on-write for name matching
 ### Testing
 - Write unit tests for services (pytest)
 - Integration tests for API endpoints
 - Frontend: component tests with React Testing Library
 ---
-## Known Issues / Open Questions
+Last updated: 2026-05-05 — after R1+R2 stabilization + R3-0 Swiftly ingestion (commit `8e89f79`).
 1. **Lucky California scraping**: Feasibility not yet spiked. URL is `luckysupermarkets.com`.
 2. **AI image generation**: Config flag to enable/disable. Disabled by default.
 3. **WhatsApp integration**: Planned for future via Twilio. Out of scope for MVP.
 4. **Planned scheduler**: APScheduler with `--workers 1` to avoid duplicate fires.
 ---
 ## Contact / Context
 - **Primary user**: Peter (tech-savvy, hosts the system)
 - **Wife**: Non-technical, will use web UI and email
 - **Children**: 2, eating habits vary (one OK with mushrooms)
 Last updated: 2026-05-04 (all adversarial review fixes committed)