feat: feedback-driven recipe discovery (auto-ingest via Spoonacular)

2026-05-24 13:17:39 -07:00
parent 35f736a052
commit 3885d7d0dc
20 changed files with 1962 additions and 9 deletions
@@ -286,4 +286,37 @@ backend/app/schemas/__init__.py           — RecipeIngredient model_validator q

 Trust the tests. Trust the live runs. Don't trust prose claims that something is "complete" without running the verification gate yourself.

-**Last updated: 2026-05-14** — Spoonacular enrichment complete (102/107 matched, 5 unmatched). Phase 8 Feedback UI built: REST API + frontend. Pydantic validation fixed for recipe ingredients (qty↔quantity schema drift). .env restored after accidental overwrite.
+**Current open proposals:**
+- None — feedback-driven recipe discovery implemented and verified.
+
+**Last updated: 2026-05-24** — Feedback-driven recipe discovery (Phases A–D) complete.
+
+---
+
+## New session: 2026-05-23
+
+### Context
+User observed that the system is constrained to 30 seed recipes and asked whether feedback triggers new recipe discovery. Investigation confirmed:
+- **No feedback analysis service exists.** `feedback_text`, `rating`, `denial_reason` are persisted but never read downstream.
+- **No recipe discovery pipeline exists.** External recipe APIs (Spoonacular, TheMealDB) are only used for image/description enrichment (`scripts/enrich_recipes_spoonacular.py`), not for discovering new recipes based on preferences.
+- **Planner only reads blocklist + recency.** No signal from free-form feedback reaches `score.py` or `generate.py`.
+
+### Proposal written
+A comprehensive proposal for **Feedback-Driven Recipe Discovery** has been authored at `docs/proposals/2026-05-23-feedback-driven-recipe-discovery.md` with:
+- Feedback Analyzer service (reads feedback → positive/negative signals + discovery queries)
+- Recipe Discovery Service (queries Spoonacular/TheMealDB)
+- Recipe Ingestion Pipeline (normalizes external recipes → our schema)
+- Review Queue table (admin approval gate before recipes enter planner)
+- Full architecture diagram, API changes, schema changes, cost analysis, risk matrix
+
+### Files written
+- `docs/proposals/2026-05-23-feedback-driven-recipe-discovery.md`
+
+### Files NOT yet modified (blocked on approval)
+- No code changes. No schema migrations. No API endpoints added.
+- `backend/app/services/feedback_analyzer.py` — planned
+- `backend/app/services/recipe_discovery.py` — planned
+- `backend/alembic/versions/0010_feedback_analysis_and_review_queue.py` — planned
+
+### Next step
+Await user approval on the proposal. If approved, create `.agent/plan.md` and begin Phase A (Feedback Analyzer).
@@ -0,0 +1,344 @@
+# Proposal: Feedback-Driven Recipe Discovery
+
+Date: 2026-05-23
+Status: Draft — awaiting approval
+Author: Agent handoff
+
+---
+
+## 1. Problem Statement
+
+The system currently has **30 hardcoded seed recipes** and no automated mechanism to grow the recipe pool based on family feedback. Feedback (`rating`, `feedback_text`, `denial_reason`) is collected via the Phase 8 UI but **never read** by any downstream service. As the family uses the system, preferences evolve ("too spicy", "loved the Thai one", "boring chicken again"), but the planner cannot discover new recipes that better match these signals.
+
+**Current state:**
+- 30 recipes → planner selects 3/week → cycle repeats every 10 weeks
+- Feedback is write-only: stored but not analyzed
+- Manual recipe creation is the only growth path
+
+**Desired state:**
+- Feedback text/ratings automatically analyzed
+- Low-rated recipes or categories trigger external discovery
+- New recipes fetched, normalized, and added to the database
+- Family sees increasing variety aligned with their tastes
+
+---
+
+## 2. Design Principles
+
+1. **Privacy-first:** External API calls only when necessary; no bulk uploads of family data
+2. **Idempotent:** Re-running discovery with the same feedback produces no duplicate recipes
+3. **Admin-gated:** New recipes enter as `needs_review` status; admin approves before the planner sees them
+4. **Budget-aware:** Free tiers prioritized; paid quotas tracked and logged
+5. **Graceful degradation:** If external APIs fail, the system continues with existing recipes
+
+---
+
+## 3. Proposed Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  FEEDBACK-DRIVEN RECIPE DISCOVERY                           │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐  │
+│  │ Feedback     │    │ Feedback     │    │ Recipe       │  │
+│  │ Collector    │───▶│ Analyzer     │───▶│ Discovery    │  │
+│  │ (exists)     │    │ (new)        │    │ Service      │  │
+│  └──────────────┘    └──────────────┘    └──────────────┘  │
+│                              │                              │
+│                              ▼                              │
+│                       ┌──────────────┐                     │
+│                       │ Recipe       │                     │
+│                       │ Ingestion    │                     │
+│                       │ Pipeline     │                     │
+│                       └──────────────┘                     │
+│                              │                              │
+│                              ▼                              │
+│                       ┌──────────────┐                     │
+│                       │ Review Queue │                     │
+│                       │ (new table)  │                     │
+│                       └──────────────┘                     │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### 3.1 Feedback Analyzer
+
+Periodic job (weekly, post-finalize) that reads all feedback from the past N weeks and produces a structured analysis.
+
+**Inputs:**
+- `feedback` table rows for the lookback window
+- `recipe` metadata (tags, ingredients, cuisine)
+- `meal_plan_item` history
+
+**Outputs** (`feedback_analysis` JSONB on `weekly_run`):
+```json
+{
+  "negative_signals": [
+    {"type": "avoid_tag': "dietary:vegetarian", "count": 3, "source": "member_A"},
+    {"type": "avoid_ingredient": "mushroom", "count": 2, "source": "member_B"},
+    {"type": "too_spicy", "count": 1, "source": "feedback_text"}
+  ],
+  "positive_signals": [
+    {"type": "prefer_cuisine": "mexican", "avg_rating": 5.0, "count": 4},
+    {"type": "prefer_protein": "shrimp", "avg_rating": 4.5, "count": 3}
+  ],
+  "discovery_queries": [
+    "mexican shrimp",
+    "mild thai chicken",
+    "vegetarian pasta"
+  ],
+  "confidence": 0.82
+}
+```
+
+### 3.2 Recipe Discovery Service
+
+Queries external recipe APIs using the discovery queries from the Analyzer.
+
+**Primary source:** Spoonacular (`/recipes/complexSearch`)
+- Quota: 150 req/day free tier (1 query per recipe page of 10 results = 15 calls/day)
+- Rate limit: built-in via API key
+
+**Fallback source:** TheMealDB
+- Free, no auth required for basic usage
+- ~300 recipes total (limited variety)
+- Good for backup or initial seeding
+
+**Query construction from feedback signals:**
+```python
+def build_queries(analysis: dict) -> list[str]:
+    """Convert positive signals into API search queries."""
+    queries = []
+    
+    # Combine preferred cuisines + proteins
+    for cuisine in analysis.positive_signals["cuisines"]:
+        for protein in analysis.positive_signals["proteins"]:
+            queries.append(f"{cuisine} {protein}")
+    
+    # Weight by rating: high-rated specific recipes → "similar to X"
+    for recipe in analysis.top_rated_recipes:
+        queries.append(recipe.name)  # Spoonacular search by name
+    
+    # Apply negative filters: exclude avoided ingredients/tags
+    for avoid in analysis.negative_signals:
+        if avoid.type == "avoid_ingredient":
+            queries.append(f"-{avoid.value}")  # Spoonacular supports excludeIngredients
+    
+    return queries[:5]  # Cap at 5 queries per run to stay within free quota
+```
+
+### 3.3 Recipe Ingestion Pipeline
+
+Transforms external API responses into our `recipe` schema.
+
+**Spoonacular → Normalized Recipe:**
+```python
+@dataclass
+class ExternalRecipe:
+    name: str
+    source_url: str
+    image_url: str
+    description: str
+    prep_time_minutes: int
+    cook_time_minutes: int
+    servings: int
+    cuisine_tags: list[str]
+    dietary_tags: list[str]
+    protein_type: str  # inferred from ingredients
+    calories_per_serving: int
+    ingredients: list[dict]  # {ingredient_id?, name, qty, unit}
+    instructions: list[str]
+    external_source: str  # "spoonacular"
+    external_id: str  # spoonacular recipe ID
+```
+
+**Challenges & Mitigations:**
+
+| Challenge | Mitigation |
+|---|---|
+| External ingredients use different names than our canonical `ingredient` table | LLM-assisted mapping (reuse `llm_matcher.py` pattern) or fuzzy match + admin review |
+| Units differ (cups vs grams) | Phase 9 spec §7 already flags unit conversion as follow-up; defer ingestion-side conversion |
+| Conflicting cuisine/protein tags | Normalize via small mapping table ("mexican"↔"mexico", "chicken breast"→"chicken") |
+| Duplicate external recipes | ON CONFLICT on `external_source` + `external_id` |
+| Image licensing | Store as URLs only (per ARCHITECTURE.md §6.1), hotlink with attribution |
+
+### 3.4 Review Queue
+
+New table to hold externally discovered recipes pending admin approval:
+
+```sql
+CREATE TABLE recipe_review_queue (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    recipe JSONB NOT NULL,  -- full normalized recipe payload
+    external_source TEXT NOT NULL,
+    external_id TEXT,
+    discovery_reason TEXT,  -- e.g., "positive_signal: mexican shrimp"
+    status TEXT NOT NULL CHECK (status IN ('pending', 'approved', 'rejected')),
+    similarity_score FLOAT,  -- cosine/embedding similarity to existing recipes
+    created_at TIMESTAMPTZ DEFAULT now(),
+    reviewed_at TIMESTAMPTZ,
+    reviewed_by UUID REFERENCES family_member(id)  -- who approved/rejected
+);
+
+CREATE INDEX idx_recipe_review_status ON recipe_review_queue(status);
+```
+
+**Admin API:**
+- `GET /api/admin/recipes/review-queue` — list pending
+- `POST /api/admin/recipes/review-queue/{id}/approve` — move to `recipe` table
+- `POST /api/admin/recipes/review-queue/{id}/reject` — mark rejected (never re-discover)
+
+**Similarity gate:** Before adding to queue, compute embedding similarity against existing recipes. Reject if >0.90 similar (exact or near-duplicate). This prevents re-adding recipes the family already has.
+
+---
+
+## 4. Data Flow
+
+```
+1. Weekly finalize step completes
+   → triggers `analyze_feedback()`
+
+2. Analyzer reads feedback from past 8 weeks
+   → produces positive/negative signals + discovery_queries
+
+3. If confidence > 0.5 and queries exist:
+   → call Spoonacular complexSearch for each query
+   → normalize responses
+   → dedupe by (external_source, external_id)
+   → similarity check vs existing recipes
+   → insert into recipe_review_queue with status='pending'
+
+4. Admin (Peter) reviews queue via UI
+   → approves → recipe moved to `recipe` table, `is_manually_added=false`
+   → rejects → stays in queue, marked rejected
+
+5. Next week's planner generation
+   → `db.query(Recipe).all()` now includes new approved recipes
+```
+
+---
+
+## 5. API Changes
+
+### New Endpoints
+
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| POST | `/api/admin/recipes/discover` | admin | Trigger manual discovery run |
+| GET | `/api/admin/recipes/review-queue` | admin | List pending recipes |
+| POST | `/api/admin/recipes/review-queue/{id}/approve` | admin | Approve a queued recipe |
+| POST | `/api/admin/recipes/review-queue/{id}/reject` | admin | Reject a queued recipe |
+| GET | `/api/analytics/feedback` | admin | Feedback analysis summary |
+
+### Schema Changes
+
+1. **`recipe` table additions:**
+   - `external_source TEXT` — e.g., "spoonacular", "themealdb"
+   - `external_id TEXT` — ID in the external system
+   - `discovery_reason TEXT` — why this recipe was found
+
+2. **`weekly_run` table additions:**
+   - `feedback_analysis JSONB` — output of the analyzer
+
+3. **`recipe_review_queue` table:** (new)
+   - As defined in §3.4
+
+---
+
+## 6. Implementation Phases
+
+### Phase A — Feedback Analyzer (1–2 days)
+- [ ] Implement `FeedbackAnalyzer` service
+- [ ] Add `feedback_analysis` JSONB column to `weekly_run`
+- [ ] Unit tests for analysis logic (mocked feedback data)
+- [ ] Hook into `step_finalize` (orchestrator)
+
+### Phase B — Recipe Discovery + Ingestion (2–3 days)
+- [ ] Implement `RecipeDiscoveryService` (Spoonacular client)
+- [ ] Implement ingestion normalization pipeline
+- [ ] Add `recipe_review_queue` table + CRUD API
+- [ ] Add `external_source`/`external_id`/`discovery_reason` to `recipe`
+- [ ] Rate-limiting and quota tracking
+- [ ] Unit + integration tests (mocked Spoonacular responses)
+
+### Phase C — Review Queue UI (1–2 days)
+- [ ] Admin page: list pending recipes with cards (image, title, ingredients)
+- [ ] Approve/Reject actions
+- [ ] Filter by status, discovery reason
+
+### Phase D — Orchestrator Integration (0.5 day)
+- [ ] Weekly finalize step calls analyzer → triggers discovery if conditions met
+- [ ] Configuration: enable/disable auto-discovery, quota limits
+
+### Phase E — Verification & Hardening (1 day)
+- [ ] End-to-end test: submit feedback → run discovery → approve recipe → verify in planner
+- [ ] Quota exhaustion handling
+- [ ] Duplicate detection accuracy
+
+---
+
+## 7. Cost & Quota Analysis
+
+| Source | Free Tier | Proposed Usage | Cost |
+|--------|-----------|----------------|------|
+| Spoonacular | 150 req/day | 5 queries × 1 call = 5/day | Free |
+| TheMealDB | 100% free | Backup only | Free |
+| Ollama (ingredient mapping) | Self-hosted | ~20 LLM calls per batch of recipes | Free |
+
+**Spoonacular points math:**
+- `complexSearch` = 1 point + 0.01 per result
+- 5 queries × 10 results = 5 + 0.5 = 5.5 points/day
+- Weekly total: ~38.5 points / 150 free = well within limits
+- If we need more, paid tiers start at $29/mo (5,000 points/day)
+
+---
+
+## 8. Risks & Mitigations
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| Spoonacular API changes/breaks schema | Medium | TheMealDB fallback; version-pinned client; mocked fixtures in CI |
+| Ingredient mapping is inaccurate | Medium | LLM-assisted + admin review queue gate; never auto-add without review |
+| Recipes not matched to our grocery catalog | Medium | Same as above — review queue lets admin see if ingredients are shoppable |
+| Family data leaked to external APIs | Low | Only recipe search queries (keywords) leave the system; no family data |
+| Duplicate recipes slip through | Low | Embedding similarity check + human review |
+| Free quota exhausted | Low | 5.5/day usage; explicit quota tracker; admin alert on 80% |
+
+---
+
+## 9. Files to Create / Modify
+
+### New Files
+```
+backend/app/services/feedback_analyzer.py
+backend/app/services/recipe_discovery.py
+backend/app/api/recipe_discovery.py  # or merge into recipes.py
+backend/alembic/versions/0010_feedback_analysis_and_review_queue.py
+docs/specs/2026-05-23-recipe-discovery-pipeline.md
+```
+
+### Modified Files
+```
+backend/app/services/orchestrator/steps.py       # hook analyzer into finalize
+backend/app/models/__init__.py                   # new columns + RecipeReviewQueue
+backend/app/schemas/__init__.py                  # response schemas
+backend/app/api/admin.py                         # new admin endpoints
+backend/app/api/recipes.py                       # external_source fields
+backend/app/models/...                           # Recipe additions
+```
+
+---
+
+## 10. Open Questions
+
+1. **Admin review cadence:** Should new recipes email Peter, or is a dashboard badge enough?
+2. **Ollama endpoint:** What's the exact model + base URL Peter uses? (Confirmed in `.agent/context.md`: Ollama Cloud)
+3. **Embedding model:** Use Ollama embeddings or a lightweight local model (sentence-transformers)?
+4. **Feedback lookback:** 8 weeks feels right for a 4-person family; confirm.
+5. **Auto-approval threshold:** Should very high-confidence discoveries (spoonacular score > 90, all ingredients mapped) skip the review queue? (Recommended: no — always review.)
+
+---
+
+## 11. Next Step
+
+Await approval on this proposal. Once approved, agent will create the implementation plan (`.agent/plan.md`) and execute Phase A.