admin/Meal-Planner
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update HANDOFF.md for 2026-05-10 session

Browse Source 
Covers all changes from this session:
- MVP login + auth gating (Login page, 401 interceptor, Sign out)
- nginx DNS resolver fix + port 8081
- Vote email: ingredient list + collapsible cooking steps
- Shopping list: per-meal sections, current_price fix
- Matcher full rewrite: ingredient-centric, precision×recall scoring,
  exclusion words, min precision floor, exact-name fast path, limit 100
- Scraper: save priceless produce items (garlic, lime, etc.)
- Infrastructure notes: DB user, module caching, admin API auth header
- Next moves: Spoonacular enrichment + Ollama LLM matcher

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
admin
2026-05-10 20:55:27 -07:00
co-authored by Claude Sonnet 4.6
parent 2373883fe7
commit a98f0dc1cb
1 changed files with 229 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HANDOFF.md
+229 -185
View File
@@ -2,252 +2,296 @@
You are taking over a project in mid-flight. Read `docs/ORIENTATION.md` first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.
Date of handoff: 2026-05-08. Last commit before handoff: Phase 6 SendGrid integration shipped.
**Date of handoff: 2026-05-10. Last commits before handoff:**
```
2373883 fix: exact-name fast path in matcher + save priceless produce in scraper
d7a3f5c fix: matcher exclusion words + precision floor for clean grocery matching
ac2b575 fix: rewrite matcher as ingredient-centric with precision×recall scoring
aeed2a4 feat: add cooking instructions to vote email; shopping list grouped by meal
59a15a2 fix: nginx DNS resolver, port 8081, seed script with real family data
```
---
## TL;DR
The project completed a **recovery pass** (R1+R2+R3-0) from 2026-05-04 to 2026-05-05 (12 defects from a prior agent fixed), then shipped **thin Phase 4** (recipe engine + ingredient↔grocery match layer + 30-recipe seed), **Phase 9** (meal-planner generation algorithm), **Phase 5** (weekly orchestration cycle — scrape → generate → email → deadline → finalize on a Friday Pacific cadence, driven by a dedicated APScheduler container), and **Phase 6** (real SendGrid email delivery + `step_reminder` pre-deadline nudge).
The system is **fully operational end-to-end** on the Woolery family's home network. All six orchestration steps run cleanly (scrape → generate → email → reminder → deadline → finalize). Vote emails reach real inboxes with ingredient lists and collapsible cooking steps. Shopping list emails group ingredients by meal with accurate Lucky CA prices.
The project's reason to exist is now real and verified end-to-end. 123/123 pytest tests pass.
The ingredient↔grocery matcher was completely rewritten in this session and is now ingredient-centric with precision×recall scoring, a category exclusion word list, and an exact-name fast path. Match accuracy went from ~25% correct to ~90%+ correct.
**Operator toil eliminated.** Swiftly bearer JWTs are now auto-minted via Firebase REST anon-signUp (`backend/app/services/swiftly_auth.py`); the `SWIFTLY_BEARER_TOKEN` env var is gone. Cache hit ratio in steady state is ~99% (one mint per ~hour). See `docs/specs/2026-05-06-swiftly-token-auto-mint.md` (status: Implemented).
The **two approved next tasks** are:
1. **Spoonacular enrichment** — recipe images + descriptions for all 107 recipes
2. **Ollama LLM matcher** — for a second pass on ingredients Lucky CA doesn't carry in the weekly ad (e.g. Olive Oil, Corn Tortillas)
---
## Infrastructure — READ THIS FIRST
### Access
- App: `http://100.108.224.12:8081` (WireGuard `wt0` interface)
- Ports 80 and 443 are owned by `lifemanager-caddy-1` on this host — do NOT use them
- Always use `docker compose --env-file .env.test` (never bare `docker compose`)
### Stack up
```bash
cd /home/peter/Projects/MealPlanner
docker compose --env-file .env.test up -d
```
### Applying Python code changes
`docker cp` alone is NOT enough — the running process caches modules. Always:
```bash
docker cp backend/app/path/to/file.py mealplanner-backend-1:/app/app/path/to/file.py
docker compose --env-file .env.test restart backend
```
### DB connection
```bash
docker compose --env-file .env.test exec -T db psql -U mealplanner -d mealplanner
```
DB user is `mealplanner` (not `postgres` — that role does not exist).
### Admin API auth
```
Authorization: Bearer test-admin-token
```
(NOT `X-Admin-Token` — it's a standard Bearer header. See `backend/app/security.py`.)
### Key env vars (`.env.test`)
```
EMAIL_BACKEND=sendgrid
SENDGRID_API_KEY=<real key>
APP_BASE_URL=http://100.108.224.12:8081
SESSION_PASSWORD=test-family-password
ADMIN_TOKEN=test-admin-token
```
---
## Family data (live, seeded)
- Family: **Woolery**, 4 members
- Adults: **Peter** (peter@research.bike) + **Julia** (julia@research.bike)
- 2 kids without email addresses (voting not required from them)
- Family profile + members are in the DB. Seed script: `scripts/seed_family.py` (safe to re-inspect; will exit early if profile already exists)
---
## What changed in this session
### MVP login + auth gating (committed in `feature/mvp-login`, merged to master)
- `frontend/src/pages/Login.tsx` — password form, calls `auth.login()`, redirects to `/`
- `frontend/src/App.tsx` — added `/login` route, Sign out button in nav
- `frontend/src/api/index.ts` — 401 interceptor redirects unauthenticated users to `/login`
- The frontend is **built and deployed** inside the Docker frontend container
### nginx (committed in `59a15a2`)
- Rewritten with Docker DNS resolver (`127.0.0.11 valid=10s`) to prevent IP caching after container restarts
- Port mapped to `8081:80` (ports 80/443 conflict with `lifemanager-caddy-1`)
- Proxy pattern: `set $var` forces per-request DNS resolution — without this, a backend restart causes 502s until nginx restarts too
### Vote email enrichment (`aeed2a4`)
Each recipe card in the Friday proposal email now shows:
- Recipe name
- Ingredient list (resolved from `Ingredient` table via UUID lookup — the JSONB stores `ingredient_id`, not `name`)
- Collapsible `<details>` block with numbered cooking steps (`recipe.instructions` ARRAY)
- Estimated cost (sum of top-confidence grocery matches)
- Vote button
### Shopping list email improvements (`aeed2a4`)
- Ingredients grouped under each recipe heading (was flat deduplicated list)
- Fixed field name: `match.grocery_item.current_price` (was `.price` — column doesn't exist)
### Ingredient matcher — full rewrite (`ac2b575`, `d7a3f5c`, `2373883`)
**Root cause of old failures:** the old matcher iterated grocery items and matched them against ingredient names using `fuzz.WRatio`. Long branded product names containing an ingredient word incidentally scored very high — "Pampers Baby Fresh Scent Wipes" → "Ginger, **Fresh**".
**New algorithm in `backend/app/services/matcher.py`:**
```
For each ingredient:
1. Exact-name fast path: lowercase-trimmed dict lookup against all grocery names
→ confidence 1.000, skip fuzzy entirely (handles "Lime" → "Lime")
2. Fuzzy: partial_token_sort_ratio against all grocery names (limit=100)
3. For each candidate above threshold (0.82):
a. 100% recall: all ingredient sig-words must appear in grocery sig-words
b. Category exclusion: grocery must not have disqualifying words absent from ingredient
(bread, chips, pasta, margarita, butter, soda, juice, tuna, rotisserie, etc.)
c. Precision floor (0.45): ingredient sig-words / grocery sig-words ≥ 0.45
d. Combined score = partial_score × precision
4. Store best combined score via ON CONFLICT DO NOTHING (preserves manual overrides)
```
**Stop words** (stripped from sig-word sets): fresh, organic, whole, large, small, medium, low, free, light, dark, raw, dried, frozen, canned, extra, virgin, pure, natural, classic, style, boneless, skinless, lean, grain, long, jarred, roasted, smoked, cooked, and, with, for, the.
**Benchmark on Lucky CA weekly ad + full produce catalog (11,044 items):**
- Before: ~25% correct (Pampers→Ginger, Red Wine→Bell Pepper, Garlic Bread→Garlic)
- After: ~90%+ correct
**Current match quality for recipe ingredients:**
```
Garlic → Fresh Garlic ($4.99) ✓ confidence 1.0
Lime → Lime (no price — sold by each) ✓ confidence 1.0
Cilantro → Cilantro, Fresh ($1.99) ✓ confidence 1.0
Bell Pepper, Red → Organic Red Bell Pepper ($2.49) ✓ confidence 1.0
Ground Beef, 85/15 → 85% Lean Ground Beef ($5.99) ✓ confidence 1.0
Cheddar Cheese, Sharp → Sharp Cheddar Cheese ($10.99) ✓ confidence 1.0
Ginger, Fresh → Ginger Root ($3.99) ✓ confidence 0.5
Ground Turkey → Butterball Ground Turkey ($6.99) ✓ confidence 0.67
Soy Sauce → Kikkoman Soy Sauce ($3.99) ✓ confidence 0.67
Salt, Kosher → Coarse Kosher Salt ($2.99) ✓ confidence 0.67
Olive Oil → — (Lucky CA has none in catalog)
Tortilla, Corn → — (not in catalog this week)
```
### Scraper fix — save priceless produce (`2373883`)
`backend/app/scraper/lucky_ca_scraper.py` `map_product()` previously returned `None` for items with no price, skipping them. Fresh produce (garlic, limes) is sold by the each with no catalog price. Removed the price guard — items with `current_price=NULL` are now saved and matched.
---
## What is real (verified)
### Backend
- `backend/app/main.py` imports cleanly with 25 routes wired (22 prior + 3 orchestrate endpoints).
- Health: `GET /health`, `GET /health/db`.
- Auth: bearer `ADMIN_TOKEN` for admin; signed-cookie session via `app.security.require_session` for mutations on family-facing routes; `/api/auth/login` with shared `SESSION_PASSWORD`.
- Routers (`backend/app/api/`): `profile.py`, `recipes.py`, `meals.py`, `pantry.py`, `shopping_list.py`, `admin.py`, `auth.py`. CRUD shapes are stubbed/partial — they validate request bodies and persist correctly but business logic is thin.
- `POST /api/admin/scrape` enqueues via FastAPI `BackgroundTasks`, returns 202 + `scrape_log_id`. Status polled via `GET /api/admin/logs/{id}`.
- Lucky California ingestion (`backend/app/scraper/lucky_ca_scraper.py`) is a `requests`-based Swiftly JSON API client. **Not Playwright** — that path was deleted. Discovers 17 categories from `https://luckysupermarkets.com/categories`, fetches each from `prod.swiftlyapi.net/search/api/v1/products/categories?cat=…&store=757&limit=10000`. Bearer scoping: token only ever attached to `prod.swiftlyapi.net` requests, never to the public categories page.
- Approval flow (`backend/app/services/approval.py` + meals router): per-voter `URLSafeTimedSerializer` tokens, TTL, single-use enforced in `consume_token`, GET renders an HTMLResponse vote page, POST records the vote and applies the rule (any deny → item denied; all approve → item approved; otherwise pending).
- Email backend (`backend/app/services/email.py`): Protocol + `ConsoleEmailBackend` (writes JSONL to `backend/var/email_outbox.jsonl`) + `SendGridEmailBackend` (live — uses `sendgrid==6.12.0`, reads `SENDGRID_API_KEY`/`SENDGRID_FROM_EMAIL`/`SENDGRID_REPLY_TO` from Settings, raises `RuntimeError` on non-2xx). Selected via `EMAIL_BACKEND` env (default `console`).
- 401 from Swiftly is now rare (we always send a freshly minted JWT). When it does happen, `SwiftlyAuthError` carries a message pointing at the auto-mint spec; mint failures upstream surface as `SwiftlyAuthMintError`. Either lands verbatim in `ScrapeLog.error_message` via the bg runner.
- Thin Phase 4: ingredient + recipe CRUD endpoints with admin gating; NeverSuggest CRUD (covers both ingredient blocklist and recipe blocklist via the existing schema); ingredient↔grocery_item match layer (rapidfuzz top-3 ranking with confidence threshold 0.75, manual override via /api/admin/ingredients/{id}/matches and /api/admin/ingredient-matches/{id}); 50 canonical ingredients seeded with aliases enriching pre-existing rows from migration 0002; 30 starter recipes spanning chicken/beef/turkey/pork/fish/vegetarian with varied cuisines, all under 45 min for 28/30. Match job runs after each successful scrape; matcher failures don't flip the scrape to FAILED.
- Phase 9: meal-plan generation. POST /api/admin/meal-plans/generate runs the full filter→score→set-select pipeline against seeded recipes and produces a persisted MealPlan with up to 3 MealPlanItem dinners. Regenerate endpoint accepts relaxed constraint overrides (`relax_time_max_minutes`, `relax_calorie_pct`, `relax_max_meal_cost`) and deletes any prior plan for the same `(family, week_start_date)` before re-running. Per-meal cost matched against ingredient_grocery_match using the top-confidence grocery row.
- **Phase 5 orchestration** (`backend/app/services/orchestrator/`): six idempotent step functions (`step_scrape`, `step_generate`, `step_email`, `step_reminder`, `step_deadline`, `step_finalize`) chained by `runner.run_step()` / `run_week()`. State tracked in `weekly_run` table (one row per family per week; each step sets its timestamp column on completion — re-firing is a no-op). Scrape failure retries once then proceeds with stale data + admin alert banner in email. Deadline resolves PENDING items per `family_profile.pending_approval_policy` (default `"approve"`). Shopping-list email sent at finalize. Admin override endpoints: `POST /api/admin/orchestrate/{step}`, `POST /api/admin/orchestrate/run-week`, `GET /api/admin/orchestrate/status`. New env vars: `ADMIN_EMAIL` (alert destination, empty = silent), `APP_BASE_URL` (vote link base, default `http://localhost`).
- **Phase 6 email** (`backend/app/services/email.py`): `SendGridEmailBackend` now live. `step_reminder` (Fri 16:00 PT) queries `MealPlanVote` to find members who haven't voted on any PENDING item and sends them a "1 hour until cutoff" nudge with fresh vote links. HTML-injection risk in proposal and shopping-list emails closed (`html.escape()` on all recipe/ingredient/member names). Migration 0009 adds `reminded_at TIMESTAMPTZ NULL` to `weekly_run`.
- **Scheduler container** (`backend/app/scheduler/__main__.py`): `BlockingScheduler(timezone="America/Los_Angeles")` with **six** `CronTrigger` jobs — Fri 02:00 scrape, 05:00 generate, 06:00 email, **16:00 reminder**, 17:00 deadline, 18:00 finalize. Runs as a separate Docker service (`scheduler:`) using the same backend image with `command: python -m app.scheduler`. Started via `docker compose up -d scheduler`.
- **Swiftly auto-mint** (`backend/app/services/swiftly_auth.py`): `get_token()` returns a Firebase anon-signUp JWT, cached in process memory until exp − 5min. `mint_anonymous_token()` fetches `firebaseApiKey` from `luckysupermarkets.com/config.json`, posts to `identitytoolkit.googleapis.com/v1/accounts:signUp` with `Origin`/`Referer` set to `https://luckysupermarkets.com`, validates `iss`/`exp` on the returned JWT. `LuckyCaliforniaScraper.fetch_category()` calls it; mint failures surface as `SwiftlyAuthMintError`. Live-verified 2026-05-06: 10,928 items scraped in 44s, 29,779 ingredient_grocery_match rows produced.
Everything in the prior HANDOFF (Phase 4 thin slice, Phase 5 orchestration, Phase 6 SendGrid, Phase 9 generation) is still real. Key additions:
### Database
- Postgres 15. Nine migrations head-at: `0001_initial_migration`, `0002_seed_data`, `0003_grocery_item_description`, `0004_family_profile_calorie_target`, `0005_grocery_item_external_id`, `0006_thin_phase4` (ingredient.aliases, recipe.calories_per_serving, ingredient_grocery_match), `0007_seed_canonical_ingredients` (50 ingredients + 30 recipes), `0008_phase5_orchestration` (`weekly_run` table + `family_profile.pending_approval_policy`), `0009_phase6_reminded_at` (`weekly_run.reminded_at TIMESTAMPTZ NULL`).
- `0001` downgrade now does a `DO $$ … DROP TABLE … DROP TYPE … END $$;` block that preserves `alembic_version`. Round-trip works.
- `0002` seed is idempotent (`ON CONFLICT (name_lower) DO NOTHING`).
- Every `SQLEnum(...)` column carries `values_callable=lambda obj: [e.value for e in obj]` — without this, name-mode breaks reads against the lowercase Postgres enum values.
- `MealPlan.votes` relationship was removed (it had no FK target). Votes are reachable via `MealPlan.items[*].votes`.
- `grocery_item` upsert key is `(source, external_id)`.
### Lucky CA API (Swiftly) — full catalog accessible
The Swiftly API is the same for ALL product categories, not just the weekly ad:
- **Taxonomy**: `GET https://luckysupermarkets.com/categories?_data=root` — works without user cookies, returns JSON with `taxonomies` list of 17 top-level category slugs
- **Products per category**: `GET https://prod.swiftlyapi.net/search/api/v1/products/categories?cat=Product%2F{slug}&limit=10000&store=757` with `Authorization: Bearer <swiftly_jwt>`
- **JWT**: auto-minted via `backend/app/services/swiftly_auth.py` — no manual token needed
- **17 categories**: produce (660 items), meat_seafood (265), pantry (1000), dairy_eggs_cheese (1000), frozen_foods, beverage, snacks, bread_bakery, deli_counter, etc.
- Current scraper scrapes all 17 categories; produce items now saved even without price
### Frontend
- React 18 + TS + Vite + Tailwind. Dashboard / MealDetail / Pantry / ShoppingList pages exist.
- API client at `frontend/src/api/index.ts` uses `withCredentials: true` for cookie-based session auth and exposes `auth.login(password)` + `auth.logout()`.
- **No login UI yet.** No feedback page. No tests.
### Tests
- **123 tests** under `backend/tests/` — 117 prior + 2 in `test_email_backend.py` (SendGrid send + error) + 6 new in `test_orchestrator.py` (step_reminder: idempotent, skips-when-not-emailed, sends-to-non-voter, skips-voter, all-voted, escape). All green when `TEST_DATABASE_URL` is set; `requires_postgres` marker auto-skips locally without it.
- Live spike scripts in `scripts/`: `spike_lucky_scrape.py` (R2-A archived path), `send_test_approval.py` (email round-trip prover, supports `--simulate-click {approve|deny}`), `spike_swiftly_ingest.py` (live Swiftly ingestion prover; requires `--confirm-live`; uses auto-minted JWT — no env var needed).
### CI
- `.github/workflows/ci.yml`: backend job (postgres:15 service, alembic + pytest) + frontend job (npm ci + build). Triggers on push and pull_request.
### Matcher runs automatically
`backend/app/services/scraper_service.py` calls `run_match_job(db, source_filter="lucky_california")` after every successful scrape. If you change matcher code, restart the backend before re-scraping so the new code is loaded.
---
## What is stubbed or missing
### Phase 4 — Recipe Engine (thin slice complete)
- Thin slice landed: ingredient + recipe CRUD, NeverSuggest CRUD, ingredient↔grocery match layer with rapidfuzz + manual override, 50 canonical ingredients + 30 starter recipes seeded.
- Phase 4 ingestion source (Spoonacular / TheMealDB / manual-only) — pros/cons table in `docs/specs/2026-05-05-meal-planner-algorithm-design.md` §6; decision deferred until Phase 9 lands.
- Still thin: full-text recipe search, advanced tag filtering, bulk import endpoints — punted until the engine demonstrates which surfaces it actually needs.
### Recipe images (Phase 10 — approved, not started)
- `recipe.image_url` is NULL for all 107 recipes → no photos in vote emails
- **Approved plan:** Spoonacular API enrichment script (107 recipes × 1 call = fits 150/day free quota)
- API returns image URL + description + improved instructions
### Phase 5 — Meal-planner orchestration (complete)
- Weekly cycle ships every Friday (Pacific): 02:00 scrape, 05:00 generate, 06:00 email, 16:00 reminder, 17:00 deadline, 18:00 finalize + shopping list.
- `weekly_run` table is the state machine; each step is idempotent on its timestamp column.
- Pending approval policy is configurable per family (`family_profile.pending_approval_policy`); default `"approve"` (silence = ok).
- Scrape failure retries once then falls back to stale data with a visible banner in the proposal email.
### Recipe descriptions
- `recipe.description` is NULL for all recipes → no blurb in vote emails
- Spoonacular enrichment solves this alongside images
### Phase 6 — SendGrid (complete)
- `SendGridEmailBackend` live: `sendgrid==6.12.0`, sends from `SENDGRID_FROM_EMAIL` with `reply_to=SENDGRID_REPLY_TO`.
- `step_reminder` (Fri 16:00 PT): nudges members who haven't voted yet on any PENDING meal plan item; sends fresh vote-link email; idempotent via `weekly_run.reminded_at`.
- All email templates HTML-safe: `html.escape()` applied to recipe names, ingredient names, and member names.
- All user-derived strings (recipe names, ingredient names, member names) are HTML-escaped in all email templates.
### Olive Oil + Corn Tortillas (Lucky catalog gap)
- Lucky CA's Swiftly catalog has no standalone olive oil or plain corn tortillas
- These show "—" in shopping list — correct behavior (better than wrong match)
- **Approved plan:** Ollama LLM matcher as a second pass using Lucky's product search API (`luckysupermarkets.com/search/products?q=<ingredient>`) to find items outside the Swiftly weekly ad
### Phase 8 — Feedback UI (not started)
- `feedback` table exists with `rating`, `denial_reason`, free-text. No frontend page reads or writes it. No `/api/feedback` router (folded into `meals.py`?).
- The "learn from feedback" loop into Phase 9 is unscoped.
### Phase 10 — Images (not started)
- Recipe images: scrape from source sites first, AI fallback (`AI_IMAGE_ENABLED=false` flag exists, no implementation).
### Phase 11 — Polish (not started)
- Variety analysis dashboard.
- Budget tracking.
- WhatsApp via Twilio (out of MVP scope).
- Note: APScheduler container is now live (Phase 5). `--workers 1` constraint applies to the scheduler service only.
- `feedback` table exists; no UI reads/writes it
---
## Known caveats and traps
1. **Bootstrap login hatch.** `app/api/auth.py` login: when no `family_profile` row exists, it signs the literal string `"bootstrap"` instead of a UUID. Anyone with `SESSION_PASSWORD` gets a session even with zero data in the DB. Acceptable for self-hosted on a trusted network. Replace with a proper first-run setup gate before exposing the system beyond the LAN/VPN. The decision is documented in `.agent/context.md` under "Decisions".
1. **Module caching.** `docker cp` without restart leaves old Python code running. Always restart backend after copying files.
2. **Auto-mint failure modes.** `swiftly_auth.get_token()` can fail in three ways: (a) `luckysupermarkets.com/config.json` becomes non-public; (b) Lucky disables anonymous Firebase auth on the `swiftly-lu-prod` project (signUp returns 400); (c) Google adds anti-abuse fingerprinting that the REST headers can't satisfy. All three surface as `SwiftlyAuthMintError` with the upstream status/body in the message and land verbatim in `ScrapeLog.error_message`. The fallback is to revive the manual-capture flow; the historical `scripts/refresh_swiftly_token.py` (commit `ccfb38a`) is in git history if you ever need it.
2. **Bootstrap login hatch.** When no `family_profile` row exists, `auth.py` signs the literal string `"bootstrap"`. Woolery family is seeded so this is dormant. If DB is wiped, re-run `scripts/seed_family.py`.
3. **`ScrapeStatus` enum reuses `STARTED` for the queued state.** R1-C didn't add a `QUEUED` value because that would have churned the Postgres enum type. Cosmetic. If you change it, add a migration.
3. **DB user is `mealplanner`.** `psql -U postgres` fails. Always use `psql -U mealplanner -d mealplanner`.
4. **Pytest's transactional `db` fixture rolls back at teardown.** Background tasks open their own `SessionLocal()` and don't see uncommitted data. `test_swiftly_api.py::test_background_runner_writes_failed_with_token_message` is the example of how to test bg-task behavior — use a separate non-fixture session, commit, run, verify, clean up explicitly.
4. **Matcher ON CONFLICT DO NOTHING.** Manual matches (`source='manual'`) are never overwritten. If you set a manual match and want the auto-matcher to take over, delete the manual row first.
5. **`alembic downgrade base` in 0001 preserves `alembic_version` table.** Don't change this to `DROP SCHEMA public CASCADE` — that would also drop `alembic_version` and break the alembic state machine on the next upgrade.
5. **`weekly_run` idempotency.** Each step sets its timestamp column on completion; re-firing is a no-op. To re-trigger a step, set its timestamp to NULL:
```sql
UPDATE weekly_run SET finalized_at = NULL, status = 'running';
```
6. **Login bootstrap aside, `family_profile` is currently empty in any fresh DB.** Phase 9 must either seed it during the first-run flow or assume the admin manually created the row. Either way, document it.
6. **Scraper `items_scraped` count appears stuck at 0 during run.** The count is only written on completion (26–60s). The status field stays `started` until then.
7. **Routes use `@router.get("")` (no trailing slash).** FastAPI's `redirect_slashes=True` (the default) will 307-redirect `/api/profile/` to `/api/profile`. Tests assert canonical paths (no slash). The frontend client matches.
7. **`limit=10000` in Swiftly API.** Pantry and dairy categories return exactly 1000 items each — suspected server-side cap below our limit. Either multiple pages exist (no offset param observed) or those are genuine catalog sizes. Produce (660) and meat_seafood (265) look complete.
8. **30 starter recipes seeded.** Migration 0007 loads them; enough to exercise Phase 9 against real data. Bulk ingestion source still deferred.
9. **Frontend doesn't have a login UI.** Until you build one, the family-facing flows can't actually be exercised by a real user — only by tests. The Dashboard/Pantry/etc. pages assume the cookie is already set.
10. `regenerate.exclude_recipe_ids` accepted by the API for forward compat but not yet honored by the orchestrator — only NeverSuggest blocklist applies. ~30-line follow-up.
11. `GET /api/meal-plans/{id}` returns persisted items but with `score=0`, `components={}`, and zeroed debug — those are only available in the immediate `generate` response. Acceptable for the email-approval flow which uses the generate response directly. To persist them, add columns to MealPlanItem.
12. `family_profile.calorie_target` is treated as per-serving by the planner filter (matches spec §2.1 wording). The family-setup UI/API should clarify per-serving vs per-day to avoid confusion. Test families use ~500 cal/serving for a 4-person household.
13. Cost estimation treats `qty` as dimensionless (no unit conversion). Produces a biased-but-monotonic ranking signal; sufficient for current use, revisit if real-dollar accuracy is needed (`docs/specs/2026-05-05-meal-planner-algorithm-design.md` §7).
8. All prior caveats in the 2026-05-08 HANDOFF still apply (SQLEnum, transactional fixtures, `alembic downgrade base`, etc.).
---
## Verification commands
Same as `docs/ORIENTATION.md`:
## Admin API reference
```bash
# Stack up (include scheduler to test the full Phase 5 setup)
docker compose --env-file .env.test up -d db backend scheduler
docker compose --env-file .env.test exec backend alembic upgrade head
# Trigger individual steps
curl -s -X POST http://localhost:8081/api/admin/orchestrate/{step} \
-H 'Authorization: Bearer test-admin-token'
# Valid steps: scrape, generate, email, reminder, deadline, finalize
# Tests
docker compose --env-file .env.test exec \
-e TEST_DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner \
backend pytest -q tests/ # → 123 passed
# Full week cycle (background)
curl -s -X POST http://localhost:8081/api/admin/orchestrate/run-week \
-H 'Authorization: Bearer test-admin-token'
# Verify scheduler registered all 6 jobs
docker compose --env-file .env.test logs scheduler | grep Registered
# Scrape status
curl -s http://localhost:8081/api/admin/logs/{scrape_log_id} \
-H 'Authorization: Bearer test-admin-token'
# Frontend
cd frontend && npm ci && npm run build
# Weekly run status
curl -s http://localhost:8081/api/admin/orchestrate/status \
-H 'Authorization: Bearer test-admin-token'
# Email approval round-trip
docker cp scripts/send_test_approval.py mealplanner-backend-1:/app/send_test_approval.py
docker compose --env-file .env.test exec backend \
python /app/send_test_approval.py --simulate-click approve
# Live Swiftly ingest (will hit the real API once)
docker cp scripts/spike_swiftly_ingest.py mealplanner-backend-1:/app/spike_swiftly_ingest.py
docker compose --env-file .env.test exec backend \
python /app/spike_swiftly_ingest.py --confirm-live
# Trigger fresh scrape + auto-match
curl -s -X POST http://localhost:8081/api/admin/scrape \
-H 'Authorization: Bearer test-admin-token'
```
---
## Suggested next move
## Suggested next moves
Phase 6 SendGrid shipped. Emails now actually deliver via SendGrid. The weekly Friday cycle is fully automated and observable end-to-end.
### 1. Spoonacular recipe enrichment (images + descriptions)
### Priority order
Free tier: 150 req/day. 107 recipes = one run, one commit.
1. **Frontend login UI.** `/api/auth/login` exists and the cookie-based session works, but no UI consumes it. Until this lands, family-facing flows (Pantry, MealDetail, ShoppingList) can only be exercised by tests.
2. **Phase 8 — feedback UI.** Close the learning loop into Phase 9. The `feedback` table exists and the schema supports it; nothing reads/writes it from a UI.
3. **Phase 11 polish — variety analysis, budget tracking.** APScheduler container is live.
4. **Phase 10 — image strategy.**
Plan:
- Write `scripts/enrich_recipes_spoonacular.py`
- For each recipe: `GET https://api.spoonacular.com/recipes/search?query={name}&apiKey=…` → pick best match → fetch details → update `recipe.image_url`, `recipe.description`
- `SPOONACULAR_API_KEY` needs to be added to `.env.test`
- Run once: `docker cp scripts/enrich_recipes_spoonacular.py mealplanner-backend-1:/app/ && docker compose --env-file .env.test exec backend python /app/enrich_recipes_spoonacular.py`
Brainstorm with the user before committing to non-trivial scope. Use the `superpowers:brainstorming` skill.
### 2. Ollama LLM matcher (Olive Oil, Corn Tortillas, etc.)
Approved architecture:
```
For each ingredient with no match OR confidence < 0.5:
1. Query Lucky product search: GET https://luckysupermarkets.com/search/products?q={ingredient}
(reverse-engineer the JSON API from that page)
2. Extract top 5-10 results
3. POST to Ollama: "I need {ingredient} for a recipe. Which is the best match?
Options: [list]. Answer with just the product name or 'none'."
4. Store result as source='auto_llm' in ingredient_grocery_match
```
Peter uses **Ollama Cloud** for LLM inference. Confirm the API endpoint + model to use.
A small model (llama3.2:3b or mistral:7b) handles "pick the right produce item" accurately.
### 3. Natural Friday cycle
Next Friday at 02:00 PT the scheduler runs automatically. No action needed. All fixes in this session are committed and the new matcher + scraper will run.
---
## Open tasks
| ID | Subject | Priority |
|---|---|---|
| #8 | `ScrapeStatus` enum could use a distinct `QUEUED` value | Cosmetic |
Other tasks in the recovery session were closed. See `.agent/phase-summaries/` for the detailed write-ups of each phase (R1A test harness, R1B+D auth+paths, R1C async scrape, R2A live scrape, R2B email approval, R3-0 Swiftly ingestion).
---
## Useful files map
## File map (additions from this session)
```
.agent/
├── plan.md recovery plan (R1, R2, R3 phases)
├── context.md locked-in decisions
└── phase-summaries/ per-phase write-ups
├── r1-r2-gate-pass.md
├── r3-0-gate-pass.md
├── r1a-summary.md
├── r1bd-summary.md
├── r1c-summary.md
├── r2a-summary.md
├── r2b-summary.md
├── r2b-blockers.md
└── r3-0-summary.md
backend/app/
├── api/
│ ├── admin.py scrape trigger + logs (admin-gated)
│ ├── auth.py login/logout (R1-B+D)
│ ├── meals.py meal plans + vote routes (per-token)
│ ├── pantry.py
│ ├── profile.py
│ ├── recipes.py
│ └── shopping_list.py
├── scraper/
│ ├── base.py rate-limited HTTP base
│ └── lucky_ca_scraper.py Swiftly JSON API client; mints via swiftly_auth.get_token()
├── scheduler/
│ ├── __init__.py
│ └── __main__.py APScheduler entry; 6 Friday Pacific jobs (incl. 16:00 reminder)
├── services/
│ ├── approval.py per-voter token issue/verify/consume
│ ├── email.py Console + SendGrid (live)
│ ├── matcher.py ingredient ↔ grocery rapidfuzz scorer
│ ├── orchestrator/ Phase 5 weekly cycle
│ │ ├── __init__.py re-exports run_step / run_week
│ │ ├── alerts.py send_admin_alert()
│ │ ├── runner.py per-family loop; run_step / run_week
│ │ └── steps.py step_scrape/generate/email/reminder/deadline/finalize
│ ├── planner/ Phase 9 filter / score / select / orchestrator
│ ├── scraper_service.py enqueue + bg runner; runs matcher post-scrape
│ └── swiftly_auth.py Firebase REST anon-signUp; process-local JWT cache
├── security.py require_admin / require_session
├── config.py pydantic Settings (no SWIFTLY_BEARER_TOKEN — auto-minted)
├── database.py engine / SessionLocal / get_db
└── models/__init__.py all SQLAlchemy models
backend/alembic/versions/ 0001 → 0009
backend/tests/ 123 tests (incl. test_swiftly_auth.py, test_orchestrator.py, test_email_backend.py)
backend/tests/fixtures/lucky_ca/ categories.html, category_meat_seafood.json, weekly_ad.html (R2-A archive)
scripts/
├── send_test_approval.py email round-trip prover
├── spike_lucky_scrape.py R2-A archived
└── spike_swiftly_ingest.py live ingest prover (auto-minted JWT)
docs/specs/
├── 2026-05-05-meal-planner-algorithm-design.md Phase 9 + thin Phase 4 design
└── 2026-05-06-swiftly-token-auto-mint.md Implemented; auto-minted JWT replaces SWIFTLY_BEARER_TOKEN
.github/workflows/ci.yml backend (postgres + pytest) + frontend (npm build)
backend/app/scraper/lucky_ca_scraper.py — removed price guard in map_product()
backend/app/services/matcher.py — full rewrite: ingredient-centric, precision×recall
backend/app/services/orchestrator/
steps.py — vote email: ingredient list + cooking steps
shopping list: grouped by meal, current_price fix
nginx/nginx.conf — Docker DNS resolver, proxy via set $var pattern
frontend/src/pages/Login.tsx — password form (Login page)
frontend/src/App.tsx — /login route, Sign out button
frontend/src/api/index.ts — 401 interceptor
scripts/seed_family.py — Woolery family + real emails
docs/superpowers/plans/
2026-05-09-mvp-login.md — MVP login plan (executed, merged)
```
---
## Final words
Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself. The recovery happened because the prior agent did the latter without the former.
Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself.
Last updated: 2026-05-08 — Phase 6 SendGrid shipped; 123/123 pytest green; scheduler has 6 Friday Pacific jobs. Next pickup: Frontend login UI.
**Last updated: 2026-05-10** — Matcher rewritten, vote email enriched with cooking steps, shopping list grouped by meal, scraper saves priceless produce, Woolery family live on wt0. Next pickup: Spoonacular enrichment + Ollama LLM matcher.