admin/Meal-Planner
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh HANDOFF + ORIENTATION for Phase 6 completion

Browse Source
This commit is contained in:
admin
2026-05-08 21:46:04 -07:00
parent aea47d8365
commit 0d70fb118d
2 changed files with 43 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HANDOFF.md
+28 -30
View File
@@ -2,15 +2,15 @@
You are taking over a project in mid-flight. Read `docs/ORIENTATION.md` first for the high-level. This file is the deep dive: what's real, what's stubbed, where the bodies are buried, and what to do next.
Date of handoff: 2026-05-07. Last commit before handoff: Phase 5 weekly orchestration shipped.
Date of handoff: 2026-05-08. Last commit before handoff: Phase 6 SendGrid integration shipped.
---
## TL;DR
The project completed a **recovery pass** (R1+R2+R3-0) from 2026-05-04 to 2026-05-05 (12 defects from a prior agent fixed), then shipped **thin Phase 4** (recipe engine + ingredient↔grocery match layer + 30-recipe seed), **Phase 9** (meal-planner generation algorithm), and **Phase 5** (weekly orchestration cycle — scrape → generate → email → deadline → finalize on a Friday Pacific cadence, driven by a dedicated APScheduler container).
The project completed a **recovery pass** (R1+R2+R3-0) from 2026-05-04 to 2026-05-05 (12 defects from a prior agent fixed), then shipped **thin Phase 4** (recipe engine + ingredient↔grocery match layer + 30-recipe seed), **Phase 9** (meal-planner generation algorithm), **Phase 5** (weekly orchestration cycle — scrape → generate → email → deadline → finalize on a Friday Pacific cadence, driven by a dedicated APScheduler container), and **Phase 6** (real SendGrid email delivery + `step_reminder` pre-deadline nudge).
The project's reason to exist is now real and verified end-to-end. 115/115 pytest tests pass.
The project's reason to exist is now real and verified end-to-end. 123/123 pytest tests pass.
**Operator toil eliminated.** Swiftly bearer JWTs are now auto-minted via Firebase REST anon-signUp (`backend/app/services/swiftly_auth.py`); the `SWIFTLY_BEARER_TOKEN` env var is gone. Cache hit ratio in steady state is ~99% (one mint per ~hour). See `docs/specs/2026-05-06-swiftly-token-auto-mint.md` (status: Implemented).
@@ -26,16 +26,17 @@ The project's reason to exist is now real and verified end-to-end. 115/115 pytes
- `POST /api/admin/scrape` enqueues via FastAPI `BackgroundTasks`, returns 202 + `scrape_log_id`. Status polled via `GET /api/admin/logs/{id}`.
- Lucky California ingestion (`backend/app/scraper/lucky_ca_scraper.py`) is a `requests`-based Swiftly JSON API client. **Not Playwright** — that path was deleted. Discovers 17 categories from `https://luckysupermarkets.com/categories`, fetches each from `prod.swiftlyapi.net/search/api/v1/products/categories?cat=…&store=757&limit=10000`. Bearer scoping: token only ever attached to `prod.swiftlyapi.net` requests, never to the public categories page.
- Approval flow (`backend/app/services/approval.py` + meals router): per-voter `URLSafeTimedSerializer` tokens, TTL, single-use enforced in `consume_token`, GET renders an HTMLResponse vote page, POST records the vote and applies the rule (any deny → item denied; all approve → item approved; otherwise pending).
- Email backend (`backend/app/services/email.py`): Protocol + `ConsoleEmailBackend` (writes JSONL to `backend/var/email_outbox.jsonl`) + `SendGridEmailBackend` stub that raises `NotImplementedError`. Selected via `EMAIL_BACKEND` env (default `console`).
- Email backend (`backend/app/services/email.py`): Protocol + `ConsoleEmailBackend` (writes JSONL to `backend/var/email_outbox.jsonl`) + `SendGridEmailBackend` (live — uses `sendgrid==6.12.0`, reads `SENDGRID_API_KEY`/`SENDGRID_FROM_EMAIL`/`SENDGRID_REPLY_TO` from Settings, raises `RuntimeError` on non-2xx). Selected via `EMAIL_BACKEND` env (default `console`).
- 401 from Swiftly is now rare (we always send a freshly minted JWT). When it does happen, `SwiftlyAuthError` carries a message pointing at the auto-mint spec; mint failures upstream surface as `SwiftlyAuthMintError`. Either lands verbatim in `ScrapeLog.error_message` via the bg runner.
- Thin Phase 4: ingredient + recipe CRUD endpoints with admin gating; NeverSuggest CRUD (covers both ingredient blocklist and recipe blocklist via the existing schema); ingredient↔grocery_item match layer (rapidfuzz top-3 ranking with confidence threshold 0.75, manual override via /api/admin/ingredients/{id}/matches and /api/admin/ingredient-matches/{id}); 50 canonical ingredients seeded with aliases enriching pre-existing rows from migration 0002; 30 starter recipes spanning chicken/beef/turkey/pork/fish/vegetarian with varied cuisines, all under 45 min for 28/30. Match job runs after each successful scrape; matcher failures don't flip the scrape to FAILED.
- Phase 9: meal-plan generation. POST /api/admin/meal-plans/generate runs the full filter→score→set-select pipeline against seeded recipes and produces a persisted MealPlan with up to 3 MealPlanItem dinners. Regenerate endpoint accepts relaxed constraint overrides (`relax_time_max_minutes`, `relax_calorie_pct`, `relax_max_meal_cost`) and deletes any prior plan for the same `(family, week_start_date)` before re-running. Per-meal cost matched against ingredient_grocery_match using the top-confidence grocery row.
- **Phase 5 orchestration** (`backend/app/services/orchestrator/`): five idempotent step functions (`step_scrape`, `step_generate`, `step_email`, `step_deadline`, `step_finalize`) chained by `runner.run_step()` / `run_week()`. State tracked in `weekly_run` table (one row per family per week; each step sets its timestamp column on completion — re-firing is a no-op). Scrape failure retries once then proceeds with stale data + admin alert banner in email. Deadline resolves PENDING items per `family_profile.pending_approval_policy` (default `"approve"`). Shopping-list email sent at finalize. Admin override endpoints: `POST /api/admin/orchestrate/{step}`, `POST /api/admin/orchestrate/run-week`, `GET /api/admin/orchestrate/status`. New env vars: `ADMIN_EMAIL` (alert destination, empty = silent), `APP_BASE_URL` (vote link base, default `http://localhost`).
- **Scheduler container** (`backend/app/scheduler/__main__.py`): `BlockingScheduler(timezone="America/Los_Angeles")` with five `CronTrigger` jobs — Fri 02:00 scrape, 05:00 generate, 06:00 email, 17:00 deadline, 18:00 finalize. Runs as a separate Docker service (`scheduler:`) using the same backend image with `command: python -m app.scheduler`. Started via `docker compose up -d scheduler`.
- **Phase 5 orchestration** (`backend/app/services/orchestrator/`): six idempotent step functions (`step_scrape`, `step_generate`, `step_email`, `step_reminder`, `step_deadline`, `step_finalize`) chained by `runner.run_step()` / `run_week()`. State tracked in `weekly_run` table (one row per family per week; each step sets its timestamp column on completion — re-firing is a no-op). Scrape failure retries once then proceeds with stale data + admin alert banner in email. Deadline resolves PENDING items per `family_profile.pending_approval_policy` (default `"approve"`). Shopping-list email sent at finalize. Admin override endpoints: `POST /api/admin/orchestrate/{step}`, `POST /api/admin/orchestrate/run-week`, `GET /api/admin/orchestrate/status`. New env vars: `ADMIN_EMAIL` (alert destination, empty = silent), `APP_BASE_URL` (vote link base, default `http://localhost`).
- **Phase 6 email** (`backend/app/services/email.py`): `SendGridEmailBackend` now live. `step_reminder` (Fri 16:00 PT) queries `MealPlanVote` to find members who haven't voted on any PENDING item and sends them a "1 hour until cutoff" nudge with fresh vote links. HTML-injection risk in proposal and shopping-list emails closed (`html.escape()` on all recipe/ingredient/member names). Migration 0009 adds `reminded_at TIMESTAMPTZ NULL` to `weekly_run`.
- **Scheduler container** (`backend/app/scheduler/__main__.py`): `BlockingScheduler(timezone="America/Los_Angeles")` with **six** `CronTrigger` jobs — Fri 02:00 scrape, 05:00 generate, 06:00 email, **16:00 reminder**, 17:00 deadline, 18:00 finalize. Runs as a separate Docker service (`scheduler:`) using the same backend image with `command: python -m app.scheduler`. Started via `docker compose up -d scheduler`.
- **Swiftly auto-mint** (`backend/app/services/swiftly_auth.py`): `get_token()` returns a Firebase anon-signUp JWT, cached in process memory until exp − 5min. `mint_anonymous_token()` fetches `firebaseApiKey` from `luckysupermarkets.com/config.json`, posts to `identitytoolkit.googleapis.com/v1/accounts:signUp` with `Origin`/`Referer` set to `https://luckysupermarkets.com`, validates `iss`/`exp` on the returned JWT. `LuckyCaliforniaScraper.fetch_category()` calls it; mint failures surface as `SwiftlyAuthMintError`. Live-verified 2026-05-06: 10,928 items scraped in 44s, 29,779 ingredient_grocery_match rows produced.
### Database
- Postgres 15. Eight migrations head-at: `0001_initial_migration`, `0002_seed_data`, `0003_grocery_item_description`, `0004_family_profile_calorie_target`, `0005_grocery_item_external_id`, `0006_thin_phase4` (ingredient.aliases, recipe.calories_per_serving, ingredient_grocery_match), `0007_seed_canonical_ingredients` (50 ingredients + 30 recipes), `0008_phase5_orchestration` (`weekly_run` table + `family_profile.pending_approval_policy`).
- Postgres 15. Nine migrations head-at: `0001_initial_migration`, `0002_seed_data`, `0003_grocery_item_description`, `0004_family_profile_calorie_target`, `0005_grocery_item_external_id`, `0006_thin_phase4` (ingredient.aliases, recipe.calories_per_serving, ingredient_grocery_match), `0007_seed_canonical_ingredients` (50 ingredients + 30 recipes), `0008_phase5_orchestration` (`weekly_run` table + `family_profile.pending_approval_policy`), `0009_phase6_reminded_at` (`weekly_run.reminded_at TIMESTAMPTZ NULL`).
- `0001` downgrade now does a `DO $$ … DROP TABLE … DROP TYPE … END $$;` block that preserves `alembic_version`. Round-trip works.
- `0002` seed is idempotent (`ON CONFLICT (name_lower) DO NOTHING`).
- Every `SQLEnum(...)` column carries `values_callable=lambda obj: [e.value for e in obj]` — without this, name-mode breaks reads against the lowercase Postgres enum values.
@@ -48,7 +49,7 @@ The project's reason to exist is now real and verified end-to-end. 115/115 pytes
- **No login UI yet.** No feedback page. No tests.
### Tests
- **115 tests** under `backend/tests/` — 92 prior + 23 new in `test_orchestrator.py` (all 5 step functions, runner, admin endpoints, fixtures). All green when `TEST_DATABASE_URL` is set; `requires_postgres` marker auto-skips locally without it.
- **123 tests** under `backend/tests/` — 117 prior + 2 in `test_email_backend.py` (SendGrid send + error) + 6 new in `test_orchestrator.py` (step_reminder: idempotent, skips-when-not-emailed, sends-to-non-voter, skips-voter, all-voted, escape). All green when `TEST_DATABASE_URL` is set; `requires_postgres` marker auto-skips locally without it.
- Live spike scripts in `scripts/`: `spike_lucky_scrape.py` (R2-A archived path), `send_test_approval.py` (email round-trip prover, supports `--simulate-click {approve|deny}`), `spike_swiftly_ingest.py` (live Swiftly ingestion prover; requires `--confirm-live`; uses auto-minted JWT — no env var needed).
### CI
@@ -64,15 +65,16 @@ The project's reason to exist is now real and verified end-to-end. 115/115 pytes
- Still thin: full-text recipe search, advanced tag filtering, bulk import endpoints — punted until the engine demonstrates which surfaces it actually needs.
### Phase 5 — Meal-planner orchestration (complete)
- Weekly cycle ships every Friday (Pacific): 02:00 scrape, 05:00 generate, 06:00 email, 17:00 deadline, 18:00 finalize + shopping list.
- Weekly cycle ships every Friday (Pacific): 02:00 scrape, 05:00 generate, 06:00 email, 16:00 reminder, 17:00 deadline, 18:00 finalize + shopping list.
- `weekly_run` table is the state machine; each step is idempotent on its timestamp column.
- Pending approval policy is configurable per family (`family_profile.pending_approval_policy`); default `"approve"` (silence = ok).
- Scrape failure retries once then falls back to stale data with a visible banner in the proposal email.
- Minor follow-ups: (a) HTML-escape recipe/ingredient names in email templates; (b) simplify dead `getattr` default in `step_deadline`.
### Phase 6 — SendGrid (stub)
- `SendGridEmailBackend.send` raises `NotImplementedError("Wire SendGrid in R3-C")`. Templates: meal proposal, reminder (T-24h), confirmation, denial.
- `from_email` / `reply_to` config not added to Settings yet.
### Phase 6 — SendGrid (complete)
- `SendGridEmailBackend` live: `sendgrid==6.12.0`, sends from `SENDGRID_FROM_EMAIL` with `reply_to=SENDGRID_REPLY_TO`.
- `step_reminder` (Fri 16:00 PT): nudges members who haven't voted yet on any PENDING meal plan item; sends fresh vote-link email; idempotent via `weekly_run.reminded_at`.
- All email templates HTML-safe: `html.escape()` applied to recipe names, ingredient names, and member names.
- All user-derived strings (recipe names, ingredient names, member names) are HTML-escaped in all email templates.
### Phase 8 — Feedback UI (not started)
- `feedback` table exists with `rating`, `denial_reason`, free-text. No frontend page reads or writes it. No `/api/feedback` router (folded into `meals.py`?).
@@ -131,9 +133,9 @@ docker compose --env-file .env.test exec backend alembic upgrade head
# Tests
docker compose --env-file .env.test exec \
-e TEST_DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner \
backend pytest -q tests/ # → 115 passed
backend pytest -q tests/ # → 123 passed
# Verify scheduler registered all 5 jobs
# Verify scheduler registered all 6 jobs
docker compose --env-file .env.test logs scheduler | grep Registered
# Frontend
@@ -154,15 +156,14 @@ docker compose --env-file .env.test exec backend \
## Suggested next move
Phase 5 weekly orchestration shipped. The system now runs fully automated: scrape → match → generate → email → vote → finalize every Friday without operator intervention.
Phase 6 SendGrid shipped. Emails now actually deliver via SendGrid. The weekly Friday cycle is fully automated and observable end-to-end.
### Priority order
1. **Phase 6 — SendGrid.** Replace the `ConsoleEmailBackend` JSONL stub with real SendGrid. The proposal email and shopping-list email are now being sent by Phase 5 — they currently land in `backend/var/email_outbox.jsonl`. Templates needed: meal proposal (already rendered in `step_email`), shopping list (already rendered in `step_finalize`), T-24h reminder, denial notification. `from_email`/`reply_to` config still needs adding to Settings.
2. **Frontend login UI.** `/api/auth/login` exists and the cookie-based session works, but no UI consumes it. Until this lands, family-facing flows (Pantry, MealDetail, ShoppingList) can only be exercised by tests.
3. **Phase 8 — feedback UI.** Close the learning loop into Phase 9. The `feedback` table exists and the schema supports it; nothing reads/writes it from a UI.
4. **Phase 11 polish — variety analysis, budget tracking.** APScheduler container is now live.
5. **Phase 10 — image strategy.**
1. **Frontend login UI.** `/api/auth/login` exists and the cookie-based session works, but no UI consumes it. Until this lands, family-facing flows (Pantry, MealDetail, ShoppingList) can only be exercised by tests.
2. **Phase 8 — feedback UI.** Close the learning loop into Phase 9. The `feedback` table exists and the schema supports it; nothing reads/writes it from a UI.
3. **Phase 11 polish — variety analysis, budget tracking.** APScheduler container is live.
4. **Phase 10 — image strategy.**
Brainstorm with the user before committing to non-trivial scope. Use the `superpowers:brainstorming` skill.
@@ -172,9 +173,6 @@ Brainstorm with the user before committing to non-trivial scope. Use the `superp
| ID | Subject | Priority |
|---|---|---|
| Phase 6 | Replace `ConsoleEmailBackend` stub with real SendGrid | High |
| #P5-a | HTML-escape recipe/ingredient names in step_email / step_finalize email templates | Minor |
| #P5-b | Simplify dead `getattr` default in `step_deadline` (line ~179 of steps.py) | Minor |
| #8 | `ScrapeStatus` enum could use a distinct `QUEUED` value | Cosmetic |
Other tasks in the recovery session were closed. See `.agent/phase-summaries/` for the detailed write-ups of each phase (R1A test harness, R1B+D auth+paths, R1C async scrape, R2A live scrape, R2B email approval, R3-0 Swiftly ingestion).
@@ -212,16 +210,16 @@ backend/app/
│ └── lucky_ca_scraper.py Swiftly JSON API client; mints via swiftly_auth.get_token()
├── scheduler/
│ ├── __init__.py
│ └── __main__.py APScheduler entry; 5 Friday Pacific jobs
│ └── __main__.py APScheduler entry; 6 Friday Pacific jobs (incl. 16:00 reminder)
├── services/
│ ├── approval.py per-voter token issue/verify/consume
│ ├── email.py Console + SendGrid stub
│ ├── email.py Console + SendGrid (live)
│ ├── matcher.py ingredient ↔ grocery rapidfuzz scorer
│ ├── orchestrator/ Phase 5 weekly cycle
│ │ ├── __init__.py re-exports run_step / run_week
│ │ ├── alerts.py send_admin_alert()
│ │ ├── runner.py per-family loop; run_step / run_week
│ │ └── steps.py step_scrape/generate/email/deadline/finalize
│ │ └── steps.py step_scrape/generate/email/reminder/deadline/finalize
│ ├── planner/ Phase 9 filter / score / select / orchestrator
│ ├── scraper_service.py enqueue + bg runner; runs matcher post-scrape
│ └── swiftly_auth.py Firebase REST anon-signUp; process-local JWT cache
@@ -230,8 +228,8 @@ backend/app/
├── database.py engine / SessionLocal / get_db
└── models/__init__.py all SQLAlchemy models
backend/alembic/versions/ 0001 → 0008
backend/tests/ 115 tests (incl. test_swiftly_auth.py, test_orchestrator.py)
backend/alembic/versions/ 0001 → 0009
backend/tests/ 123 tests (incl. test_swiftly_auth.py, test_orchestrator.py, test_email_backend.py)
backend/tests/fixtures/lucky_ca/ categories.html, category_meat_seafood.json, weekly_ad.html (R2-A archive)
scripts/
@@ -252,4 +250,4 @@ docs/specs/
Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself. The recovery happened because the prior agent did the latter without the former.
Last updated: 2026-05-07 — Phase 5 weekly orchestration shipped; 115/115 pytest green. Next pickup: Phase 6 SendGrid.
Last updated: 2026-05-08 — Phase 6 SendGrid shipped; 123/123 pytest green; scheduler has 6 Friday Pacific jobs. Next pickup: Frontend login UI.