Meal-Planner/docs/HANDOFF.md

# MealPlanner — Agent Handoff

You are taking over a project in mid-flight. Read `docs/ORIENTATION.md` first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.

Date of handoff: 2026-05-06. Last commit before handoff: thin Phase 4 (P4-15).

---

## TL;DR

The project completed a **recovery pass** (R1+R2+R3-0) from 2026-05-04 to 2026-05-05 because the prior agent had marked Phases 2/3/7 "complete" while leaving 12 distinct defects that prevented the app from importing, prevented migrations from upgrading on Postgres, prevented authentication from existing, and prevented any actual grocery items from being persisted.

That is now fixed. The app imports, the schema upgrades cleanly, auth works, scraping persists ~10k rows live in 36 s, and the email approval round-trip works end-to-end. 31/31 pytest tests pass.

The project's reason to exist — the meal-planner generation algorithm — is **still not started**. That is your top priority. See "Suggested next move" at the bottom.

---

## What is real (verified)

### Backend
- `backend/app/main.py` imports cleanly with 22 routes wired.
- Health: `GET /health`, `GET /health/db`.
- Auth: bearer `ADMIN_TOKEN` for admin; signed-cookie session via `app.security.require_session` for mutations on family-facing routes; `/api/auth/login` with shared `SESSION_PASSWORD`.
- Routers (`backend/app/api/`): `profile.py`, `recipes.py`, `meals.py`, `pantry.py`, `shopping_list.py`, `admin.py`, `auth.py`. CRUD shapes are stubbed/partial — they validate request bodies and persist correctly but business logic is thin.
- `POST /api/admin/scrape` enqueues via FastAPI `BackgroundTasks`, returns 202 + `scrape_log_id`. Status polled via `GET /api/admin/logs/{id}`.
- Lucky California ingestion (`backend/app/scraper/lucky_ca_scraper.py`) is a `requests`-based Swiftly JSON API client. **Not Playwright** — that path was deleted. Discovers 17 categories from `https://luckysupermarkets.com/categories`, fetches each from `prod.swiftlyapi.net/search/api/v1/products/categories?cat=…&store=757&limit=10000`. Bearer scoping: token only ever attached to `prod.swiftlyapi.net` requests, never to the public categories page.
- Approval flow (`backend/app/services/approval.py` + meals router): per-voter `URLSafeTimedSerializer` tokens, TTL, single-use enforced in `consume_token`, GET renders an HTMLResponse vote page, POST records the vote and applies the rule (any deny → item denied; all approve → item approved; otherwise pending).
- Email backend (`backend/app/services/email.py`): Protocol + `ConsoleEmailBackend` (writes JSONL to `backend/var/email_outbox.jsonl`) + `SendGridEmailBackend` stub that raises `NotImplementedError`. Selected via `EMAIL_BACKEND` env (default `console`).
- 401 from Swiftly raises `SwiftlyAuthError` carrying the verbatim message `"SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)"`. The bg runner catches it and writes `ScrapeLog.error_message` so it surfaces via the admin logs endpoint.
- Thin Phase 4: ingredient + recipe CRUD endpoints with admin gating; NeverSuggest CRUD (covers both ingredient blocklist and recipe blocklist via the existing schema); ingredient↔grocery_item match layer (rapidfuzz top-3 ranking with confidence threshold 0.75, manual override via /api/admin/ingredients/{id}/matches and /api/admin/ingredient-matches/{id}); 50 canonical ingredients seeded with aliases enriching pre-existing rows from migration 0002; 30 starter recipes spanning chicken/beef/turkey/pork/fish/vegetarian with varied cuisines, all under 45 min for 28/30. Match job runs after each successful scrape; matcher failures don't flip the scrape to FAILED.

### Database
- Postgres 15. Seven migrations: `0001_initial_migration`, `0002_seed_data`, `0003_grocery_item_description`, `0004_family_profile_calorie_target`, `0005_grocery_item_external_id`, `0006_thin_phase4` (ingredient.aliases, recipe.calories_per_serving, ingredient_grocery_match), `0007_seed_canonical_ingredients` (50 ingredients + 30 recipes).
- `0001` downgrade now does a `DO $$ … DROP TABLE … DROP TYPE … END $$;` block that preserves `alembic_version`. Round-trip works.
- `0002` seed is idempotent (`ON CONFLICT (name_lower) DO NOTHING`).
- Every `SQLEnum(...)` column carries `values_callable=lambda obj: [e.value for e in obj]` — without this, name-mode breaks reads against the lowercase Postgres enum values.
- `MealPlan.votes` relationship was removed (it had no FK target). Votes are reachable via `MealPlan.items[*].votes`.
- `grocery_item` upsert key is `(source, external_id)`.

### Frontend
- React 18 + TS + Vite + Tailwind. Dashboard / MealDetail / Pantry / ShoppingList pages exist.
- API client at `frontend/src/api/index.ts` uses `withCredentials: true` for cookie-based session auth and exposes `auth.login(password)` + `auth.logout()`.
- **No login UI yet.** No feedback page. No tests.

### Tests
- 59 tests under `backend/tests/` (31 prior + Phase 4 additions: ingredient/recipe schema + API, matcher, match-hook, never-suggest, resolve-ingredient, thin-phase-4 smoke). All green when `TEST_DATABASE_URL` is set; `requires_postgres` marker auto-skips locally without it.
- Live spike scripts in `scripts/`: `spike_lucky_scrape.py` (R2-A archived path), `send_test_approval.py` (email round-trip prover, supports `--simulate-click {approve|deny}`), `spike_swiftly_ingest.py` (R3-0 live ingestion prover; requires `--confirm-live`).

### CI
- `.github/workflows/ci.yml`: backend job (postgres:15 service, alembic + pytest) + frontend job (npm ci + build). Triggers on push and pull_request.

---

## What is stubbed or missing

### Phase 4 — Recipe Engine (thin slice complete)
- Thin slice landed: ingredient + recipe CRUD, NeverSuggest CRUD, ingredient↔grocery match layer with rapidfuzz + manual override, 50 canonical ingredients + 30 starter recipes seeded. 59/59 tests green.
- Phase 4 ingestion source (Spoonacular / TheMealDB / manual-only) — pros/cons table in `docs/specs/2026-05-05-meal-planner-algorithm-design.md` §6; decision deferred until Phase 9 lands.
- Still thin: full-text recipe search, advanced tag filtering, bulk import endpoints — punted until the engine demonstrates which surfaces it actually needs.

### Phase 5 — Meal-planner orchestration (not started)
- The weekly cycle: scrape Sunday → generate Monday → email Monday-evening → deadline Thursday → finalize Friday.
- All the parts exist (scrape works; email works; vote works; approval rule works) but nothing chains them.

### Phase 6 — SendGrid (stub)
- `SendGridEmailBackend.send` raises `NotImplementedError("Wire SendGrid in R3-C")`. Templates: meal proposal, reminder (T-24h), confirmation, denial.
- `from_email` / `reply_to` config not added to Settings yet.

### Phase 8 — Feedback UI (not started)
- `feedback` table exists with `rating`, `denial_reason`, free-text. No frontend page reads or writes it. No `/api/feedback` router (folded into `meals.py`?).
- The "learn from feedback" loop into Phase 9 is unscoped.

### Phase 9 — **Meal-planner generation algorithm (not started)**
- This is the core of the project.
- Inputs: `family_profile` (size, budget, calorie target), `family_member` preferences (mushroom etc.), `never_suggest` filter, `home_pantry` items, `grocery_item` sales for the week, `recipe` library, prior `feedback` (downweight low-rated, never-suggest-flagged).
- Output: 7 `MealPlanItem` rows (or 21 if 3 meals/day) with chosen `recipe_id`, day, meal_type.
- Constraints from spec: variety (max 2 same protein/sauce per week), respect mushroom rule (3 of 4 don't like → recipes containing mushrooms heavily downweighted unless flagged "kid-only"), prefer pantry items, prefer sale items, hit budget per meal.

### Phase 10 — Images (not started)
- Recipe images: scrape from source sites first, AI fallback (`AI_IMAGE_ENABLED=false` flag exists, no implementation).

### Phase 11 — Polish (not started)
- APScheduler container with `--workers 1` to run weekly cadence.
- Variety analysis dashboard.
- Budget tracking.
- WhatsApp via Twilio (out of MVP scope).

---

## Known caveats and traps

1. **Bootstrap login hatch.** `app/api/auth.py` login: when no `family_profile` row exists, it signs the literal string `"bootstrap"` instead of a UUID. Anyone with `SESSION_PASSWORD` gets a session even with zero data in the DB. Acceptable for self-hosted on a trusted network. Replace with a proper first-run setup gate before exposing the system beyond the LAN/VPN. The decision is documented in `.agent/context.md` under "Decisions".

2. **`SWIFTLY_BEARER_TOKEN` expires hourly.** It's a Firebase anonymous-auth JWT scoped to `swiftly-lu-prod`. The user explicitly chose to surface a request when it expires rather than mint new tokens automatically. On 401, `ScrapeLog.error_message` carries the actionable message. The user captures a fresh token from luckysupermarkets.com devtools and updates the env var. Don't try to automate Firebase auth unless the user asks.

3. **`.env.example` ships a real (expiring) token.** Per user authorization. If it's already expired by the time you read this, that's expected — surface the refresh request to the user. Do not log it.

4. **`ScrapeStatus` enum reuses `STARTED` for the queued state.** R1-C didn't add a `QUEUED` value because that would have churned the Postgres enum type. Cosmetic. If you change it, add a migration.

5. **Pytest's transactional `db` fixture rolls back at teardown.** Background tasks open their own `SessionLocal()` and don't see uncommitted data. `test_swiftly_api.py::test_background_runner_writes_failed_with_token_message` is the example of how to test bg-task behavior — use a separate non-fixture session, commit, run, verify, clean up explicitly.

6. **`alembic downgrade base` in 0001 preserves `alembic_version` table.** Don't change this to `DROP SCHEMA public CASCADE` — that would also drop `alembic_version` and break the alembic state machine on the next upgrade.

7. **Login bootstrap aside, `family_profile` is currently empty in any fresh DB.** Phase 9 must either seed it during the first-run flow or assume the admin manually created the row. Either way, document it.

8. **Routes use `@router.get("")` (no trailing slash).** FastAPI's `redirect_slashes=True` (the default) will 307-redirect `/api/profile/` to `/api/profile`. Tests assert canonical paths (no slash). The frontend client matches.

9. **30 starter recipes seeded.** Migration 0007 loads them; enough to exercise Phase 9 against real data. Bulk ingestion source still deferred.

10. **Frontend doesn't have a login UI.** Until you build one, the family-facing flows can't actually be exercised by a real user — only by tests. The Dashboard/Pantry/etc. pages assume the cookie is already set.

---

## Verification commands

Same as `docs/ORIENTATION.md`:

```bash
# Stack up
docker compose --env-file .env.test up -d db backend
docker compose --env-file .env.test exec backend alembic upgrade head

# Tests
docker compose --env-file .env.test exec \
  -e TEST_DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner \
  backend pytest -q tests/   # → 59 passed

# Frontend
cd frontend && npm ci && npm run build

# Email approval round-trip
docker cp scripts/send_test_approval.py mealplanner-backend-1:/app/send_test_approval.py
docker compose --env-file .env.test exec backend \
  python /app/send_test_approval.py --simulate-click approve

# Live Swiftly ingest (will hit the real API once)
docker cp scripts/spike_swiftly_ingest.py mealplanner-backend-1:/app/spike_swiftly_ingest.py
docker compose --env-file .env.test exec backend \
  python /app/spike_swiftly_ingest.py --confirm-live
```

---

## Suggested next move

**Phase 9 first — meal-planner generation algorithm.**

Why: the schema is provably real now. The grocery feed is live. The approval flow works. Email and recipe ingestion can be developed against fixtures while you build the engine. Without Phase 9 the project has no reason to exist.

Sketch:

1. **Phase 4 (recipe ingestion) just enough to feed Phase 9.** Cheapest path: a CSV / JSON seed of ~30 recipes the family already knows + `POST /api/recipes/import` that takes a JSON body. Punt scraping recipe sites until later.
2. **Phase 9 algorithm.** Inputs listed above. Output: 7 `MealPlanItem` rows. Start dumb — random selection respecting `never_suggest` + mushroom rule. Iterate to add variety, sales bias, pantry bias, budget.
3. **Phase 5 orchestration + Phase 6 SendGrid** — chain everything.
4. **Phase 8 feedback UI** — close the learning loop.

Brainstorm with the user before committing to the algorithm shape. Use the `superpowers:brainstorming` skill.

---

## Open tasks (carried over from recovery)

| ID | Subject | Priority |
|---|---|---|
| #8 | `ScrapeStatus` enum could use a distinct `QUEUED` value | Cosmetic |

Other tasks in the recovery session were closed. See `.agent/phase-summaries/` for the detailed write-ups of each phase (R1A test harness, R1B+D auth+paths, R1C async scrape, R2A live scrape, R2B email approval, R3-0 Swiftly ingestion).

---

## Useful files map

```
.agent/
├── plan.md                              recovery plan (R1, R2, R3 phases)
├── context.md                           locked-in decisions
└── phase-summaries/                     per-phase write-ups
    ├── r1-r2-gate-pass.md
    ├── r3-0-gate-pass.md
    ├── r1a-summary.md
    ├── r1bd-summary.md
    ├── r1c-summary.md
    ├── r2a-summary.md
    ├── r2b-summary.md
    ├── r2b-blockers.md
    └── r3-0-summary.md

backend/app/
├── api/
│   ├── admin.py                         scrape trigger + logs (admin-gated)
│   ├── auth.py                          login/logout (R1-B+D)
│   ├── meals.py                         meal plans + vote routes (per-token)
│   ├── pantry.py
│   ├── profile.py
│   ├── recipes.py
│   └── shopping_list.py
├── scraper/
│   ├── base.py                          rate-limited HTTP base
│   └── lucky_ca_scraper.py              Swiftly JSON API client (R3-0)
├── services/
│   ├── approval.py                      per-voter token issue/verify/consume
│   ├── email.py                         Console + SendGrid stub
│   └── scraper_service.py               enqueue + bg runner
├── security.py                          require_admin / require_session
├── config.py                            pydantic Settings
├── database.py                          engine / SessionLocal / get_db
└── models/__init__.py                   all SQLAlchemy models

backend/alembic/versions/                0001 → 0005
backend/tests/                           31 tests
backend/tests/fixtures/lucky_ca/         categories.html, category_meat_seafood.json, weekly_ad.html (R2-A archive)

scripts/
├── send_test_approval.py                email round-trip prover
├── spike_lucky_scrape.py                R2-A archived
└── spike_swiftly_ingest.py              R3-0 live ingest prover

.github/workflows/ci.yml                 backend (postgres + pytest) + frontend (npm build)
```

---

## Final words

Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself. The recovery happened because the prior agent did the latter without the former.

Last updated: 2026-05-06.