Meal-Planner/docs/HANDOFF.md

MealPlanner — Agent Handoff

You are taking over a project in mid-flight. Read docs/ORIENTATION.md first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.

Date of handoff: 2026-05-10. Last commits before handoff:

2373883 fix: exact-name fast path in matcher + save priceless produce in scraper
d7a3f5c fix: matcher exclusion words + precision floor for clean grocery matching
ac2b575 fix: rewrite matcher as ingredient-centric with precision×recall scoring
aeed2a4 feat: add cooking instructions to vote email; shopping list grouped by meal
59a15a2 fix: nginx DNS resolver, port 8081, seed script with real family data

---

TL;DR

The system is fully operational end-to-end on the Woolery family's home network. All six orchestration steps run cleanly (scrape → generate → email → reminder → deadline → finalize). Vote emails reach real inboxes with ingredient lists and collapsible cooking steps. Shopping list emails group ingredients by meal with accurate Lucky CA prices.

The ingredient↔grocery matcher was completely rewritten in this session and is now ingredient-centric with precision×recall scoring, a category exclusion word list, and an exact-name fast path. Match accuracy went from ~25% correct to ~90%+ correct.

The two approved next tasks are:

1. Spoonacular enrichment — recipe images + descriptions for all 107 recipes
2. Ollama LLM matcher — for a second pass on ingredients Lucky CA doesn't carry in the weekly ad (e.g. Olive Oil, Corn Tortillas)

---

Infrastructure — READ THIS FIRST

Access

• App: http://100.108.224.12:8081 (WireGuard wt0 interface)
• Ports 80 and 443 are owned by lifemanager-caddy-1 on this host — do NOT use them
• Always use docker compose --env-file .env.test (never bare docker compose)

Stack up

cd /home/peter/Projects/MealPlanner
docker compose --env-file .env.test up -d

Applying Python code changes

docker cp alone is NOT enough — the running process caches modules. Always:

docker cp backend/app/path/to/file.py mealplanner-backend-1:/app/app/path/to/file.py
docker compose --env-file .env.test restart backend

DB connection

docker compose --env-file .env.test exec -T db psql -U mealplanner -d mealplanner

DB user is mealplanner (not postgres — that role does not exist).

Admin API auth

Authorization: Bearer test-admin-token

(NOT X-Admin-Token — it's a standard Bearer header. See backend/app/security.py.)

Key env vars (.env.test)

EMAIL_BACKEND=sendgrid
SENDGRID_API_KEY=<real key>
APP_BASE_URL=http://100.108.224.12:8081
SESSION_PASSWORD=test-family-password
ADMIN_TOKEN=test-admin-token

---

Family data (live, seeded)

• Family: Woolery, 4 members
• Adults: Peter (peter@research.bike) + Julia (julia@research.bike)
• 2 kids without email addresses (voting not required from them)
• Family profile + members are in the DB. Seed script: scripts/seed_family.py (safe to re-inspect; will exit early if profile already exists)

---

What changed in this session

MVP login + auth gating (committed in feature/mvp-login, merged to master)

• frontend/src/pages/Login.tsx — password form, calls auth.login(), redirects to /
• frontend/src/App.tsx — added /login route, Sign out button in nav
• frontend/src/api/index.ts — 401 interceptor redirects unauthenticated users to /login
• The frontend is built and deployed inside the Docker frontend container

nginx (committed in 59a15a2)

• Rewritten with Docker DNS resolver (127.0.0.11 valid=10s) to prevent IP caching after container restarts
• Port mapped to 8081:80 (ports 80/443 conflict with lifemanager-caddy-1)
• Proxy pattern: set $var forces per-request DNS resolution — without this, a backend restart causes 502s until nginx restarts too

Vote email enrichment (aeed2a4)

Each recipe card in the Friday proposal email now shows:

• Recipe name
• Ingredient list (resolved from Ingredient table via UUID lookup — the JSONB stores ingredient_id, not name)
• Collapsible <details> block with numbered cooking steps (recipe.instructions ARRAY)
• Estimated cost (sum of top-confidence grocery matches)
• Vote button

Shopping list email improvements (aeed2a4)

• Ingredients grouped under each recipe heading (was flat deduplicated list)
• Fixed field name: match.grocery_item.current_price (was .price — column doesn't exist)

Ingredient matcher — full rewrite (ac2b575, d7a3f5c, 2373883)

Root cause of old failures: the old matcher iterated grocery items and matched them against ingredient names using fuzz.WRatio. Long branded product names containing an ingredient word incidentally scored very high — "Pampers Baby Fresh Scent Wipes" → "Ginger, Fresh".

New algorithm in backend/app/services/matcher.py:

For each ingredient:
  1. Exact-name fast path: lowercase-trimmed dict lookup against all grocery names
     → confidence 1.000, skip fuzzy entirely (handles "Lime" → "Lime")
  2. Fuzzy: partial_token_sort_ratio against all grocery names (limit=100)
  3. For each candidate above threshold (0.82):
     a. 100% recall: all ingredient sig-words must appear in grocery sig-words
     b. Category exclusion: grocery must not have disqualifying words absent from ingredient
        (bread, chips, pasta, margarita, butter, soda, juice, tuna, rotisserie, etc.)
     c. Precision floor (0.45): ingredient sig-words / grocery sig-words ≥ 0.45
     d. Combined score = partial_score × precision
  4. Store best combined score via ON CONFLICT DO NOTHING (preserves manual overrides)

Stop words (stripped from sig-word sets): fresh, organic, whole, large, small, medium, low, free, light, dark, raw, dried, frozen, canned, extra, virgin, pure, natural, classic, style, boneless, skinless, lean, grain, long, jarred, roasted, smoked, cooked, and, with, for, the.

Benchmark on Lucky CA weekly ad + full produce catalog (11,044 items):

• Before: ~25% correct (Pampers→Ginger, Red Wine→Bell Pepper, Garlic Bread→Garlic)
• After: ~90%+ correct

Current match quality for recipe ingredients:

Garlic                → Fresh Garlic ($4.99)           ✓ confidence 1.0
Lime                  → Lime (no price — sold by each) ✓ confidence 1.0
Cilantro              → Cilantro, Fresh ($1.99)         ✓ confidence 1.0
Bell Pepper, Red      → Organic Red Bell Pepper ($2.49) ✓ confidence 1.0
Ground Beef, 85/15    → 85% Lean Ground Beef ($5.99)   ✓ confidence 1.0
Cheddar Cheese, Sharp → Sharp Cheddar Cheese ($10.99)  ✓ confidence 1.0
Ginger, Fresh         → Ginger Root ($3.99)             ✓ confidence 0.5
Ground Turkey         → Butterball Ground Turkey ($6.99) ✓ confidence 0.67
Soy Sauce             → Kikkoman Soy Sauce ($3.99)      ✓ confidence 0.67
Salt, Kosher          → Coarse Kosher Salt ($2.99)      ✓ confidence 0.67
Olive Oil             → — (Lucky CA has none in catalog)
Tortilla, Corn        → — (not in catalog this week)

Scraper fix — save priceless produce (2373883)

backend/app/scraper/lucky_ca_scraper.py map_product() previously returned None for items with no price, skipping them. Fresh produce (garlic, limes) is sold by the each with no catalog price. Removed the price guard — items with current_price=NULL are now saved and matched.

---

What is real (verified)

Everything in the prior HANDOFF (Phase 4 thin slice, Phase 5 orchestration, Phase 6 SendGrid, Phase 9 generation) is still real. Key additions:

Lucky CA API (Swiftly) — full catalog accessible

The Swiftly API is the same for ALL product categories, not just the weekly ad:

• Taxonomy: GET https://luckysupermarkets.com/categories?_data=root — works without user cookies, returns JSON with taxonomies list of 17 top-level category slugs
• Products per category: GET https://prod.swiftlyapi.net/search/api/v1/products/categories?cat=Product%2F{slug}&limit=10000&store=757 with Authorization: Bearer <swiftly_jwt>
• JWT: auto-minted via backend/app/services/swiftly_auth.py — no manual token needed
• 17 categories: produce (660 items), meat_seafood (265), pantry (1000), dairy_eggs_cheese (1000), frozen_foods, beverage, snacks, bread_bakery, deli_counter, etc.
• Current scraper scrapes all 17 categories; produce items now saved even without price

Matcher runs automatically

backend/app/services/scraper_service.py calls run_match_job(db, source_filter="lucky_california") after every successful scrape. If you change matcher code, restart the backend before re-scraping so the new code is loaded.

---

What is stubbed or missing

Recipe images (Phase 10 — approved, not started)

• recipe.image_url is NULL for all 107 recipes → no photos in vote emails
• Approved plan: Spoonacular API enrichment script (107 recipes × 1 call = fits 150/day free quota)
• API returns image URL + description + improved instructions

Recipe descriptions

• recipe.description is NULL for all recipes → no blurb in vote emails
• Spoonacular enrichment solves this alongside images

Olive Oil + Corn Tortillas (Lucky catalog gap)

• Lucky CA's Swiftly catalog has no standalone olive oil or plain corn tortillas
• These show "—" in shopping list — correct behavior (better than wrong match)
• Approved plan: Ollama LLM matcher as a second pass using Lucky's product search API (luckysupermarkets.com/search/products?q=<ingredient>) to find items outside the Swiftly weekly ad

Phase 8 — Feedback UI (not started)

• feedback table exists; no UI reads/writes it

---

Known caveats and traps

1. Module caching. docker cp without restart leaves old Python code running. Always restart backend after copying files.

2. Bootstrap login hatch. When no family_profile row exists, auth.py signs the literal string "bootstrap". Woolery family is seeded so this is dormant. If DB is wiped, re-run scripts/seed_family.py.

3. DB user is mealplanner. psql -U postgres fails. Always use psql -U mealplanner -d mealplanner.

4. Matcher ON CONFLICT DO NOTHING. Manual matches (source='manual') are never overwritten. If you set a manual match and want the auto-matcher to take over, delete the manual row first.

5. weekly_run idempotency. Each step sets its timestamp column on completion; re-firing is a no-op. To re-trigger a step, set its timestamp to NULL:

   UPDATE weekly_run SET finalized_at = NULL, status = 'running';
   

6. Scraper items_scraped count appears stuck at 0 during run. The count is only written on completion (26–60s). The status field stays started until then.

7. limit=10000 in Swiftly API. Pantry and dairy categories return exactly 1000 items each — suspected server-side cap below our limit. Either multiple pages exist (no offset param observed) or those are genuine catalog sizes. Produce (660) and meat_seafood (265) look complete.

8. All prior caveats in the 2026-05-08 HANDOFF still apply (SQLEnum, transactional fixtures, alembic downgrade base, etc.).

---

Admin API reference

# Trigger individual steps
curl -s -X POST http://localhost:8081/api/admin/orchestrate/{step} \
  -H 'Authorization: Bearer test-admin-token'
# Valid steps: scrape, generate, email, reminder, deadline, finalize

# Full week cycle (background)
curl -s -X POST http://localhost:8081/api/admin/orchestrate/run-week \
  -H 'Authorization: Bearer test-admin-token'

# Scrape status
curl -s http://localhost:8081/api/admin/logs/{scrape_log_id} \
  -H 'Authorization: Bearer test-admin-token'

# Weekly run status
curl -s http://localhost:8081/api/admin/orchestrate/status \
  -H 'Authorization: Bearer test-admin-token'

# Trigger fresh scrape + auto-match
curl -s -X POST http://localhost:8081/api/admin/scrape \
  -H 'Authorization: Bearer test-admin-token'

---

Suggested next moves

1. Spoonacular recipe enrichment (images + descriptions)

Free tier: 150 req/day. 107 recipes = one run, one commit.

Plan:

• Write scripts/enrich_recipes_spoonacular.py
• For each recipe: GET https://api.spoonacular.com/recipes/search?query={name}&apiKey=… → pick best match → fetch details → update recipe.image_url, recipe.description
• SPOONACULAR_API_KEY needs to be added to .env.test
• Run once: docker cp scripts/enrich_recipes_spoonacular.py mealplanner-backend-1:/app/ && docker compose --env-file .env.test exec backend python /app/enrich_recipes_spoonacular.py

2. Ollama LLM matcher (Olive Oil, Corn Tortillas, etc.)

Approved architecture:

For each ingredient with no match OR confidence < 0.5:
  1. Query Lucky product search: GET https://luckysupermarkets.com/search/products?q={ingredient}
     (reverse-engineer the JSON API from that page)
  2. Extract top 5-10 results
  3. POST to Ollama: "I need {ingredient} for a recipe. Which is the best match?
     Options: [list]. Answer with just the product name or 'none'."
  4. Store result as source='auto_llm' in ingredient_grocery_match

Peter uses Ollama Cloud for LLM inference. Confirm the API endpoint + model to use. A small model (llama3.2:3b or mistral:7b) handles "pick the right produce item" accurately.

3. Natural Friday cycle

Next Friday at 02:00 PT the scheduler runs automatically. No action needed. All fixes in this session are committed and the new matcher + scraper will run.

---

File map (additions from this session)

backend/app/scraper/lucky_ca_scraper.py  — removed price guard in map_product()
backend/app/services/matcher.py          — full rewrite: ingredient-centric, precision×recall
backend/app/services/orchestrator/
  steps.py                               — vote email: ingredient list + cooking steps
                                           shopping list: grouped by meal, current_price fix
nginx/nginx.conf                         — Docker DNS resolver, proxy via set $var pattern
frontend/src/pages/Login.tsx             — password form (Login page)
frontend/src/App.tsx                     — /login route, Sign out button
frontend/src/api/index.ts                — 401 interceptor
scripts/seed_family.py                   — Woolery family + real emails
docs/superpowers/plans/
  2026-05-09-mvp-login.md               — MVP login plan (executed, merged)

---

Final words

Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself.

Last updated: 2026-05-10 — Matcher rewritten, vote email enriched with cooking steps, shopping list grouped by meal, scraper saves priceless produce, Woolery family live on wt0. Next pickup: Spoonacular enrichment + Ollama LLM matcher.