# MealPlanner — Agent Handoff

You are taking over a project in mid-flight. Read `docs/ORIENTATION.md` first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.

**Date of handoff: 2026-06-03. Last commits before handoff:**
```
427d8ac docs(review): add handoff document for UI audit work
e90a9d6 feat(ui): close 3 P2 audit findings + a11y sweep (Sprint 3)
f5fb755 fix(migration): simplify aisle migration + add persistent backup script
ccc70aa feat(ui): close 6 P1 audit findings + 1 bonus mobile fix (Sprint 2)
36038bb docs(review): mark Sprint 1 P0 fixes addressed in commit f3e4a44
f3e4a44 fix(ui): close 5 P0 audit findings (ingredients, cost, routing, mobile slots)
b522760 fix: cast qty/unit to str before html.escape in vote email shopping preview
```

**Focused UI/UX audit handoff** (Sprints 1, 2, 3 — 14 findings closed across 3 commits):
see `Review/handoff-ui-audit.md`. That doc is the right starting point for
anyone continuing the UI/UX work; the present file remains the project-wide
overview (backend, infra, family data, admin API, prior phases).

---

## TL;DR

The system is **fully operational end-to-end** on the Woolery family's home network. Vote emails now show recipe images, descriptions, ingredient lists, cooking steps, estimated costs, and a shopping list preview. Peter confirmed the email looks polished; Julia's feedback pending.

**Match accuracy:** 10,140 AUTO + 3 AUTO_LLM matches. 22 ingredients remain unmatched (genuine Lucky CA catalog gaps: olive oil, dried spices, chickpeas, etc.).

**Next tasks:**
1. **Spoonacular enrichment (5 remaining)** — run `scripts/enrich_recipes_spoonacular.py` again; 5 recipes still need images (daily quota was hit on 2026-05-11)
2. **Phase 8 Feedback UI** — `feedback` table exists; no UI reads/writes it yet

---

## Infrastructure — READ THIS FIRST

### Access
- App: `http://100.108.224.12:8081` (WireGuard `wt0` interface)
- Ports 80 and 443 are owned by `lifemanager-caddy-1` on this host — do NOT use them
- Always use `docker compose --env-file .env.test` (never bare `docker compose`)

### Stack up
```bash
cd /home/peter/Projects/MealPlanner
docker compose --env-file .env.test up -d
```

### Applying Python code changes
`docker cp` alone is NOT enough — the running process caches modules. Always:
```bash
docker cp backend/app/path/to/file.py mealplanner-backend-1:/app/app/path/to/file.py
docker compose --env-file .env.test restart backend
```

### DB connection
```bash
docker compose --env-file .env.test exec -T db psql -U mealplanner -d mealplanner
```
DB user is `mealplanner` (not `postgres` — that role does not exist).

### Admin API auth
```
Authorization: Bearer test-admin-token
```
(NOT `X-Admin-Token` — it's a standard Bearer header. See `backend/app/security.py`.)

### Key env vars (`.env.test`)
```
EMAIL_BACKEND=sendgrid
SENDGRID_API_KEY=<real key>
APP_BASE_URL=http://100.108.224.12:8081
SESSION_PASSWORD=test-family-password
ADMIN_TOKEN=test-admin-token
```

---

## Family data (live, seeded)

- Family: **Woolery**, 4 members
- Adults: **Peter** (peter@research.bike) + **Julia** (julia@research.bike)
- 2 kids without email addresses (voting not required from them)
- Family profile + members are in the DB. Seed script: `scripts/seed_family.py` (safe to re-inspect; will exit early if profile already exists)

---

## What changed in this session

### MVP login + auth gating (committed in `feature/mvp-login`, merged to master)
- `frontend/src/pages/Login.tsx` — password form, calls `auth.login()`, redirects to `/`
- `frontend/src/App.tsx` — added `/login` route, Sign out button in nav
- `frontend/src/api/index.ts` — 401 interceptor redirects unauthenticated users to `/login`
- The frontend is **built and deployed** inside the Docker frontend container

### nginx (committed in `59a15a2`)
- Rewritten with Docker DNS resolver (`127.0.0.11 valid=10s`) to prevent IP caching after container restarts
- Port mapped to `8081:80` (ports 80/443 conflict with `lifemanager-caddy-1`)
- Proxy pattern: `set $var` forces per-request DNS resolution — without this, a backend restart causes 502s until nginx restarts too

### Vote email enrichment (`aeed2a4`)
Each recipe card in the Friday proposal email now shows:
- Recipe name
- Ingredient list (resolved from `Ingredient` table via UUID lookup — the JSONB stores `ingredient_id`, not `name`)
- Collapsible `<details>` block with numbered cooking steps (`recipe.instructions` ARRAY)
- Estimated cost (sum of top-confidence grocery matches)
- Vote button

### Shopping list email improvements (`aeed2a4`)
- Ingredients grouped under each recipe heading (was flat deduplicated list)
- Fixed field name: `match.grocery_item.current_price` (was `.price` — column doesn't exist)

### Ingredient matcher — full rewrite (`ac2b575`, `d7a3f5c`, `2373883`)

**Root cause of old failures:** the old matcher iterated grocery items and matched them against ingredient names using `fuzz.WRatio`. Long branded product names containing an ingredient word incidentally scored very high — "Pampers Baby Fresh Scent Wipes" → "Ginger, **Fresh**".

**New algorithm in `backend/app/services/matcher.py`:**

```
For each ingredient:
  1. Exact-name fast path: lowercase-trimmed dict lookup against all grocery names
     → confidence 1.000, skip fuzzy entirely (handles "Lime" → "Lime")
  2. Fuzzy: partial_token_sort_ratio against all grocery names (limit=100)
  3. For each candidate above threshold (0.82):
     a. 100% recall: all ingredient sig-words must appear in grocery sig-words
     b. Category exclusion: grocery must not have disqualifying words absent from ingredient
        (bread, chips, pasta, margarita, butter, soda, juice, tuna, rotisserie, etc.)
     c. Precision floor (0.45): ingredient sig-words / grocery sig-words ≥ 0.45
     d. Combined score = partial_score × precision
  4. Store best combined score via ON CONFLICT DO NOTHING (preserves manual overrides)
```

**Stop words** (stripped from sig-word sets): fresh, organic, whole, large, small, medium, low, free, light, dark, raw, dried, frozen, canned, extra, virgin, pure, natural, classic, style, boneless, skinless, lean, grain, long, jarred, roasted, smoked, cooked, and, with, for, the.

**Benchmark on Lucky CA weekly ad + full produce catalog (11,044 items):**
- Before: ~25% correct (Pampers→Ginger, Red Wine→Bell Pepper, Garlic Bread→Garlic)
- After: ~90%+ correct

**Current match quality for recipe ingredients:**
```
Garlic                → Fresh Garlic ($4.99)           ✓ confidence 1.0
Lime                  → Lime (no price — sold by each) ✓ confidence 1.0
Cilantro              → Cilantro, Fresh ($1.99)         ✓ confidence 1.0
Bell Pepper, Red      → Organic Red Bell Pepper ($2.49) ✓ confidence 1.0
Ground Beef, 85/15    → 85% Lean Ground Beef ($5.99)   ✓ confidence 1.0
Cheddar Cheese, Sharp → Sharp Cheddar Cheese ($10.99)  ✓ confidence 1.0
Ginger, Fresh         → Ginger Root ($3.99)             ✓ confidence 0.5
Ground Turkey         → Butterball Ground Turkey ($6.99) ✓ confidence 0.67
Soy Sauce             → Kikkoman Soy Sauce ($3.99)      ✓ confidence 0.67
Salt, Kosher          → Coarse Kosher Salt ($2.99)      ✓ confidence 0.67
Olive Oil             → — (Lucky CA has none in catalog)
Tortilla, Corn        → — (not in catalog this week)
```

### Scraper fix — save priceless produce (`2373883`)
`backend/app/scraper/lucky_ca_scraper.py` `map_product()` previously returned `None` for items with no price, skipping them. Fresh produce (garlic, limes) is sold by the each with no catalog price. Removed the price guard — items with `current_price=NULL` are now saved and matched.

---

## What is real (verified)

Everything in the prior HANDOFF (Phase 4 thin slice, Phase 5 orchestration, Phase 6 SendGrid, Phase 9 generation) is still real. Key additions:

### Lucky CA API (Swiftly) — full catalog accessible
The Swiftly API is the same for ALL product categories, not just the weekly ad:
- **Taxonomy**: `GET https://luckysupermarkets.com/categories?_data=root` — works without user cookies, returns JSON with `taxonomies` list of 17 top-level category slugs
- **Products per category**: `GET https://prod.swiftlyapi.net/search/api/v1/products/categories?cat=Product%2F{slug}&limit=10000&store=757` with `Authorization: Bearer <swiftly_jwt>`
- **JWT**: auto-minted via `backend/app/services/swiftly_auth.py` — no manual token needed
- **17 categories**: produce (660 items), meat_seafood (265), pantry (1000), dairy_eggs_cheese (1000), frozen_foods, beverage, snacks, bread_bakery, deli_counter, etc.
- Current scraper scrapes all 17 categories; produce items now saved even without price

### Matcher runs automatically
`backend/app/services/scraper_service.py` calls `run_match_job(db, source_filter="lucky_california")` after every successful scrape. If you change matcher code, restart the backend before re-scraping so the new code is loaded.

---

## What is stubbed or missing

### Recipe images (Phase 10 — approved, not started)
- `recipe.image_url` is NULL for all 107 recipes → no photos in vote emails
- **Approved plan:** Spoonacular API enrichment script (107 recipes × 1 call = fits 150/day free quota)
- API returns image URL + description + improved instructions

### Recipe descriptions
- `recipe.description` is NULL for all recipes → no blurb in vote emails
- Spoonacular enrichment solves this alongside images

### Olive Oil + Corn Tortillas (Lucky catalog gap)
- Lucky CA's Swiftly catalog has no standalone olive oil or plain corn tortillas
- These show "—" in shopping list — correct behavior (better than wrong match)
- **Approved plan:** Ollama LLM matcher as a second pass using Lucky's product search API (`luckysupermarkets.com/search/products?q=<ingredient>`) to find items outside the Swiftly weekly ad

### Phase 8 — Feedback UI (done)
- `feedback` table now read/written via REST API
- Meal detail page shows star rating, never-suggest, reason dropdown, free-text comments

---

## Known caveats and traps

1. **Module caching.** `docker cp` without restart leaves old Python code running. Always restart backend after copying files.

2. **Bootstrap login hatch.** When no `family_profile` row exists, `auth.py` signs the literal string `"bootstrap"`. Woolery family is seeded so this is dormant. If DB is wiped, re-run `scripts/seed_family.py`.

3. **DB user is `mealplanner`.** `psql -U postgres` fails. Always use `psql -U mealplanner -d mealplanner`.

4. **Matcher ON CONFLICT DO NOTHING.** Manual matches (`source='manual'`) are never overwritten. If you set a manual match and want the auto-matcher to take over, delete the manual row first.

5. **`weekly_run` idempotency.** Each step sets its timestamp column on completion; re-firing is a no-op. To re-trigger a step, set its timestamp to NULL:
   ```sql
   UPDATE weekly_run SET finalized_at = NULL, status = 'running';
   ```

6. **Scraper `items_scraped` count appears stuck at 0 during run.** The count is only written on completion (26–60s). The status field stays `started` until then.

7. **`limit=10000` in Swiftly API.** Pantry and dairy categories return exactly 1000 items each — suspected server-side cap below our limit. Either multiple pages exist (no offset param observed) or those are genuine catalog sizes. Produce (660) and meat_seafood (265) look complete.

8. All prior caveats in the 2026-05-08 HANDOFF still apply (SQLEnum, transactional fixtures, `alembic downgrade base`, etc.).

---

## Admin API reference

```bash
# Trigger individual steps
curl -s -X POST http://localhost:8081/api/admin/orchestrate/{step} \
  -H 'Authorization: Bearer test-admin-token'
# Valid steps: scrape, generate, email, reminder, deadline, finalize

# Full week cycle (background)
curl -s -X POST http://localhost:8081/api/admin/orchestrate/run-week \
  -H 'Authorization: Bearer test-admin-token'

# Scrape status
curl -s http://localhost:8081/api/admin/logs/{scrape_log_id} \
  -H 'Authorization: Bearer test-admin-token'

# Weekly run status
curl -s http://localhost:8081/api/admin/orchestrate/status \
  -H 'Authorization: Bearer test-admin-token'

# Trigger fresh scrape + auto-match
curl -s -X POST http://localhost:8081/api/admin/scrape \
  -H 'Authorization: Bearer test-admin-token'
```

---

## Suggested next moves

### 1. Spoonacular recipe enrichment (images + descriptions)

Free tier: 150 req/day. 107 recipes = one run, one commit.

Plan:
- Write `scripts/enrich_recipes_spoonacular.py`
- For each recipe: `GET https://api.spoonacular.com/recipes/search?query={name}&apiKey=…` → pick best match → fetch details → update `recipe.image_url`, `recipe.description`
- `SPOONACULAR_API_KEY` needs to be added to `.env.test`
- Run once: `docker cp scripts/enrich_recipes_spoonacular.py mealplanner-backend-1:/app/ && docker compose --env-file .env.test exec backend python /app/enrich_recipes_spoonacular.py`

### 2. Ollama LLM matcher (Olive Oil, Corn Tortillas, etc.)

Approved architecture:
```
For each ingredient with no match OR confidence < 0.5:
  1. Query Lucky product search: GET https://luckysupermarkets.com/search/products?q={ingredient}
     (reverse-engineer the JSON API from that page)
  2. Extract top 5-10 results
  3. POST to Ollama: "I need {ingredient} for a recipe. Which is the best match?
     Options: [list]. Answer with just the product name or 'none'."
  4. Store result as source='auto_llm' in ingredient_grocery_match
```

Peter uses **Ollama Cloud** for LLM inference. Confirm the API endpoint + model to use.
A small model (llama3.2:3b or mistral:7b) handles "pick the right produce item" accurately.

### 3. Natural Friday cycle

Next Friday at 02:00 PT the scheduler runs automatically. No action needed. All fixes in this session are committed and the new matcher + scraper will run.

---

## File map (additions from this session)

```
backend/app/api/feedback.py                — new: GET/POST feedback endpoints
frontend/src/pages/MealDetail.tsx          — added Feedback section (rating, never-suggest, reasons)
frontend/src/api/index.ts                  — added feedback API methods
frontend/src/types/index.ts                — added Feedback interface
backend/app/schemas/__init__.py           — RecipeIngredient model_validator qty→quantity
```

---

## Final words

Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself.

**Current open proposals:**
- `docs/proposals/2026-05-23-feedback-driven-recipe-discovery.md` — pending user approval. No code yet (per the 2026-05-23 section below).

**Last updated: 2026-06-03** — UI/UX audit & fix cycle (Sprints 1, 2, 3) complete. 14 findings closed, code committed (`f3e4a44`, `ccc70aa`, `f5fb755`, `e90a9d6`), build green. Sprint 1 deployed; Sprints 2 and 3 awaiting deploy on the user's home server. Backend migration `0015_normalize_pantry_aisles.py` not yet run on prod (dry-run shows 21k rows would change on dev DB). Full UI-audit handoff at `Review/handoff-ui-audit.md`.

---

## New session: 2026-06-03

### UI/UX audit & fix — 3 sprints, 14 findings closed

A full Nielsen-10-heuristics audit of the live deployment at `http://100.108.208.56:8082/` was performed using Playwright (NixOS-compatible Chromium at `/run/current-system/sw/bin/chromium --no-sandbox`; original screenshots in `/tmp/opencode/mp-review/screenshots/`). 14 findings (5 P0, 6 P1, 3 P2) plus 3 a11y items were addressed in three sprints, each ending in `npm run build` green.

**Audit & plan documents (all kept in sync, all in `Review/`):**
- `Review/ui-nielsen-audit.md` — the audit itself, with status blocks per sprint at the top
- `fix-ui-audit.md` — the implementation plan, with per-task implementation notes
- `Review/sprint2-verification.md` — Sprint 2 deploy + smoke-check checklist (includes the **backend migration** step)
- `Review/sprint3-verification.md` — Sprint 3 deploy + smoke-check checklist (frontend only)
- `Review/handoff-ui-audit.md` — focused handoff for a fresh agent continuing UI-audit work

**Commits on `main` (ahead of `origin/main` by 9 prior WIP commits plus these 6):**

| Commit | Sprint | What |
|---|---|---|
| `f3e4a44` | 1 | 5 P0 blockers: recipe/meal ingredient field names, `$N/A` cost, `/recommended` 404, mobile empty slots |
| `36038bb` | 1 (docs) | Mark Sprint 1 P0 fixes in audit doc |
| `ccc70aa` | 2 | 6 P1s + S3.3: meal-card title clamp, MealDetail hero + SEO strip, pantry aisle select, shopping-list aisle map, mobile pantry scroll hint, recipes filters w/ Apply/Reset/active-count, mobile shopping-list 3-col grid |
| `f5fb755` | 2 (fix) | Migration 0015 simplification + persistent backup script (`persist_aisle_backup.sql`) + corrected container-based deploy commands |
| `e90a9d6` | 3 | 3 P2s + a11y: undo-toast (Dashboard refills slot; Pantry fully reversible), mobile nav nowrap, aria-current, `<main id="main-content">`, Badge `aria-label`/`icon` props |
| `427d8ac` | (docs) | Review/handoff-ui-audit.md |

**Critical Sprint 2 deploy note:** the user must run on the deployment host *after* `git pull`:

```bash
# 1. Persistent backup BEFORE the migration (recommended)
docker compose exec -T db psql -U mealplanner -d mealplanner \
  -f /dev/stdin < backend/scripts/persist_aisle_backup.sql

# 2. Dry-run preview (no writes)
docker compose exec -T db psql -U mealplanner -d mealplanner \
  -f /dev/stdin < backend/scripts/dry_run_aisle_migration.sql

# 3. Apply the migration
docker compose exec backend alembic upgrade head

# 4. Frontend rebuild + restart
docker compose -f docker-compose.yml up -d --build frontend
```

The dev DB dry-run on this host shows 10,657 ingredient rows + 10,539 grocery_item rows = **21,196 rows** would change. The deployment-host DB will differ — operator judgment required. The `persist_aisle_backup.sql` creates two permanent `public.*_aisle_backup_0015` tables the operator can `DROP` after confidence is established.

**Sprint 3 deploy is frontend-only:**
```bash
git pull
docker compose -f docker-compose.yml up -d --build frontend
```

### Deployment-host vs dev-host (Tailscale gotcha)

This repo lives on a dev host (Tailscale `100.108.146.47`). The user's home server (Tailscale `100.108.224.12`) serves the live app at `100.108.208.56:8082`. The user's workflow is **commit locally, `git pull` on the deployment host, rebuild there**. Don't `docker compose up` on the local dev host expecting it to update the live site — it won't.

### Repo quirk: `.gitignore` blocks `frontend/src/lib/`

Pre-existing bug: `.gitignore` line 17 is `lib/` (the Python ignore), and it catches `frontend/src/lib/`. New files there need `git add -f` (the `toast.tsx` rename in Sprint 3 was force-added). The `lib/` ignore should arguably be `^lib/$` or `/lib/`, but that's a separate cleanup.

### Files added by this session

```
Review/handoff-ui-audit.md                       # Focused UI-audit handoff
Review/sprint2-verification.md                   # Deploy + smoke-check for Sprint 2
Review/sprint3-verification.md                   # Deploy + smoke-check for Sprint 3
Review/ui-nielsen-audit.md                       # (rewritten) Audit with status blocks per sprint
fix-ui-audit.md                                  # The plan, with per-task implementation notes
frontend/src/pages/NotFound.tsx                  # 404 catch-all (B4)
backend/alembic/versions/0015_normalize_pantry_aisles.py  # Sprint 2 migration
backend/scripts/dry_run_aisle_migration.sql      # Read-only preview
backend/scripts/persist_aisle_backup.sql         # Persistent backup
```

### Files modified by this session

```
frontend/src/App.tsx                             # Routes, nav, aria
frontend/src/api/index.ts                        # (pre-existing WIP, not mine)
frontend/src/components/ui/Badge.tsx             # icon + aria-label props
frontend/src/components/ui/EmptyState.tsx        # optional to prop
frontend/src/lib/toast.ts → toast.tsx            # renamed for JSX; showToast.undo()
frontend/src/lib/utils.ts                        # cleanDescription() (B7)
frontend/src/pages/Dashboard.tsx                 # B5, B6, B12
frontend/src/pages/MealDetail.tsx                # B2, B3, B7
frontend/src/pages/Pantry.tsx                    # B8, B10, B12
frontend/src/pages/RecipeDetail.tsx              # B1
frontend/src/pages/Recipes.tsx                   # B11
frontend/src/pages/ShoppingList.tsx              # B9, S3.3
frontend/src/types/index.ts                       # PANTRY_AISLES, RecipeIngredient extensions
```

---

## New session: 2026-05-24

### Unit conversion implementation
Completed implementation of recipe-to-grocery unit conversion to make cost estimates accurate.

**Files added:**
- `backend/app/utils/units.py` — `UnitConverter` class
  - Normalization: maps synonyms to canonical units (e.g. "TBS" → "tbsp", "pounds" → "lb")
  - Within-family linear conversion: lb↔oz↔g, cup↔tbsp↔tsp, dozen↔ea
  - Cross-family via density tables for ~30 canonical ingredients (e.g. rice cup→lb via 185g/cup / 453.592g/lb)
  - Fallback to dimensionless qty when conversion is impossible (preserves monotonic ranking signal)

**Files modified:**
- `backend/app/services/planner/cost.py` — multiplies `current_price` by `convert_qty(qty, recipe_unit, grocery_unit, ingredient_name)`
- `backend/app/services/planner/generate.py` — `_load_match_index` now joins `Ingredient` table and returns `ingredient_name` + `grocery_unit` for each match
- `backend/app/services/orchestrator/steps.py` — both email cost block and shopping-list total now use unit conversion
- `backend/tests/test_planner_cost.py` — updated fixture to include new fields
- `backend/tests/test_units.py` — 19 tests covering normalization, within-family, density, and fallback

**Test results:** `test_units.py` 19/19 pass; planner cost/score/select 36 passed.

---

## New session: 2026-05-23

### Context
User observed that the system is constrained to 30 seed recipes and asked whether feedback triggers new recipe discovery. Investigation confirmed:
- **No feedback analysis service exists.** `feedback_text`, `rating`, `denial_reason` are persisted but never read downstream.
- **No recipe discovery pipeline exists.** External recipe APIs (Spoonacular, TheMealDB) are only used for image/description enrichment (`scripts/enrich_recipes_spoonacular.py`), not for discovering new recipes based on preferences.
- **Planner only reads blocklist + recency.** No signal from free-form feedback reaches `score.py` or `generate.py`.

### Proposal written
A comprehensive proposal for **Feedback-Driven Recipe Discovery** has been authored at `docs/proposals/2026-05-23-feedback-driven-recipe-discovery.md` with:
- Feedback Analyzer service (reads feedback → positive/negative signals + discovery queries)
- Recipe Discovery Service (queries Spoonacular/TheMealDB)
- Recipe Ingestion Pipeline (normalizes external recipes → our schema)
- Review Queue table (admin approval gate before recipes enter planner)
- Full architecture diagram, API changes, schema changes, cost analysis, risk matrix

### Files written
- `docs/proposals/2026-05-23-feedback-driven-recipe-discovery.md`

### Files NOT yet modified (blocked on approval)
- No code changes. No schema migrations. No API endpoints added.
- `backend/app/services/feedback_analyzer.py` — planned
- `backend/app/services/recipe_discovery.py` — planned
- `backend/alembic/versions/0010_feedback_analysis_and_review_queue.py` — planned

### Next step
Await user approval on the proposal. If approved, create `.agent/plan.md` and begin Phase A (Feedback Analyzer).