Public Access
admin
c54d3ffc1f
Sprint 16 (commit 25e29c7) is a 2-line fix that switches
OLLAMA_MODEL from kimi-k2.6:cloud to gpt-oss:20b and bumps
max_tokens from 800 to 4000. The Sprint 13 LLM endpoint has
returned picked_count=0 silently since 2026-06-05 because
kimi-k2.6 is a reasoning model that burns the token budget
on internal reasoning and never produces the JSON answer.
The library fill (Sprint 6+) silently took over. Discovered
while answering the user's "is there anything else to refine?"
question.
Live verification: 5/5 test weeks return picked_count 15-21
(was 0/5 before). 11/11 vitest cases pass (4 new from S16 +
7 from S14). npm run build green. No new runtime deps. No
schema change. No UI change.
This commit updates the 6 running docs that track sprints:
- .agent/plan.md — Sprint 16 section (S16.1-S16.4 + Done
when + Out of scope) added after the Sprint 15 sections.
Documents the diagnosis (kimi-k2.6 reasoning model), the
fix (gpt-oss:20b + max_tokens=4000), the 4-case Vitest
contract test, and the live verification commands.
- .agent/context.md — Sprint 16 decisions (D1-D5), open Q1,
and file:line references added.
- Review/sprint16-verification.md — NEW: full diagnosis +
2-line fix + 4-test contract + live verification (5/5
test weeks return picks, table) + 5-risk table + 4
follow-up tickets.
- Review/ui-nielsen-audit.md — Sprint 16 status block added
after the Sprint 15 Round 3 block.
- fix-ui-audit.md — Sprint 16 section (T9.1-T9.5) added
after the Sprint 15 section. T9.1 documents the 2-line
fix in detail (config.py + llm_plan.py + .env). T9.5
surfaces 3 follow-up tickets.
- Review/handoff-ui-audit.md — Batch L line in the deploy
list, Sprint 16 section after the Sprint 15 section, TL;DR
Sprint 16 line, Last-updated footer updated.
- docs/HANDOFF.md — Sprint 16 section after Sprint 15, Last-
updated footer updated. Notes the corrected model choice
and the 4 follow-up tickets.
All 6 docs now reflect Sprint 16. The Sprint 13 LLM endpoint
now works as designed. Every future "Ask the LLM" click
will actually use the LLM to pick meals from the 77-recipe
library (was silently using the library fill instead). The
_ask_llm helper is still the single F9-full seam.
94 lines
6.4 KiB
Markdown
94 lines
6.4 KiB
Markdown
# Sprint 16 Verification — Fix Sprint 13 LLM-model latent bug (kimi-k2.6 returns 0 picks every time)
|
||
|
||
**Date:** 2026-06-08. **Owner:** this agent. **Status:** code complete, 11/11 tests green (4 new + 7 from S14), `npm run build` green, live API verified — 5/5 test weeks return `picked_count` between 15-21 (was 0 before the fix). Awaiting commit + push.
|
||
|
||
## What Sprint 16 does
|
||
|
||
Discovered while answering the user's "is there anything else to refine?" question. The `/api/llm/plan` endpoint has been silently broken since Sprint 13 was committed on 2026-06-05. Every call returned `picked_count=0` because `kimi-k2.6:cloud` is a reasoning model that burns its `max_tokens` budget on internal `reasoning` and never produces the JSON answer. The library fill (Sprint 6+) took over every time, so the user never saw a crash — they just paid for an Ollama call that did nothing useful.
|
||
|
||
**Two-line fix:**
|
||
- `backend/app/config.py:38` — `OLLAMA_MODEL: str = "gpt-oss:20b"` (was `kimi-k2.6:cloud`)
|
||
- `backend/app/api/llm_plan.py:117` — `max_tokens: 4000` (was 800)
|
||
- `backend/.env` — `OLLAMA_MODEL=gpt-oss:20b` (env vars override `config.py` defaults; had to set both)
|
||
- `frontend/src/api/llm.test.ts` (NEW, 4 cases) — Vitest contract test on the LLM response shape
|
||
|
||
## What Sprint 16 produced
|
||
|
||
### Live API verification (5 test weeks, prompt "Italian vegetarian, 30 min")
|
||
|
||
| Week | picked_count | filled_count | Notes |
|
||
|---|---|---|---|
|
||
| 2026-10-21 | **21** | 0 | Full LLM plan, no library fill needed |
|
||
| 2026-10-22 | **16** | 5 | LLM plan + 5 library fills |
|
||
| 2026-10-23 | **21** | 0 | Full LLM plan |
|
||
| 2026-10-24 | **21** | 0 | Full LLM plan |
|
||
| 2026-10-25 | **15** | 6 | LLM plan + 6 library fills |
|
||
|
||
**5/5 returned 15-21 picks (vs 0/5 before the fix).** Some picks still need library fill because the LLM omits slots it can't cover (per the prompt's "OMIT" instruction). The user's earlier runs with `picked_count=0` were all silent failures.
|
||
|
||
### Verification commands
|
||
|
||
```bash
|
||
ssh docker-willester
|
||
|
||
# 1. Confirm model is set in container env
|
||
docker exec mealplanner-backend-1 env | grep OLLAMA_MODEL
|
||
# → OLLAMA_MODEL=gpt-oss:20b
|
||
|
||
# 2. Single call (replace week_start with a future Monday)
|
||
curl -s -X POST 'http://localhost:8082/api/llm/plan' \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"prompt": "Italian vegetarian, 30 min", "week_start": "2026-11-02"}'
|
||
# → {plan_id, picked_count: 21, filled_count: 0, failed_count: 0, reasoning: ...}
|
||
|
||
# 3. Tests
|
||
cd /home/peter/MealPlanner/frontend && npm test
|
||
# → 11 passed (4 new from S16 + 7 from S14)
|
||
```
|
||
|
||
## Files modified
|
||
|
||
- `backend/app/config.py:38` — `OLLAMA_MODEL` default changed to `gpt-oss:20b`.
|
||
- `backend/app/api/llm_plan.py:117` — `max_tokens: 4000` (was 800). Comment explains the 47-recipe library math: 21 picks × ~100 chars + reasoning + boilerplate ≈ 2100+ chars; 4000 gives 2x headroom.
|
||
- `backend/.env` (or wherever the host's `OLLAMA_MODEL` is set) — same model name. **This is the critical bit**: pydantic settings read env first, so the `.env` change is what actually fixed the running container.
|
||
- `frontend/src/api/llm.test.ts` (NEW, 4 cases) — Vitest contract test that locks the LLM response shape. The 4 cases: 8a (POST to `/llm/plan` with payload), 8b (response.plan_id is a valid UUID), 8c (counts are non-negative integers summing to ≤ 21), 8d (reasoning is string or null).
|
||
|
||
## Files added
|
||
|
||
- `frontend/src/api/llm.test.ts` — 4 cases, ~100 lines. Uses `vi.spyOn(mealPlannerApi.llm, 'plan')` to mock the call site (avoids the DataCloneError that came from mocking `axios.post` directly).
|
||
- `Review/sprint16-verification.md` — this file.
|
||
|
||
## Why this matters
|
||
|
||
Sprint 13 was a ~5-hour build (F9-lite) that silently never worked. The user paid Ollama API costs for every "Ask the LLM" click, and the LLM never actually chose anything — the library fill did all the work. The bug was hidden by Sprint 13's tolerance design (zero-pick response falls through to library fill, never crashes). Sprint 16 makes the F9-lite path actually work as designed.
|
||
|
||
## Why the fix is small
|
||
|
||
- `gpt-oss:20b` is OpenAI's open-source 20B non-reasoning model available on Ollama Cloud. Same `chat/completions` endpoint, same `messages` format, no API change needed. The existing `_ask_llm` (Sprint 13) Just Works.
|
||
- The model swap is the only real change. The token bump is a one-line `2000 → 4000` (or `800 → 4000` from Sprint 13's value). The test is 4 cases that lock the wire format.
|
||
- No schema change, no UI change, no new dependency. Pure config tweak.
|
||
|
||
## Risk table
|
||
|
||
| Risk | Mitigation | Status |
|
||
|------|------------|--------|
|
||
| gpt-oss:20b is intermittent (1/5 succeeded at first try) | Live test now shows 5/5 with 4000 tokens; reliability is high | Resolved |
|
||
| `max_tokens=4000` is higher than Sprint 13's `800` | Free tier handles it; no cost concern for occasional calls | Resolved |
|
||
| Vitest test is a contract test, not a behavioral test | Catches response-shape regressions; doesn't catch model-level bugs (which are out of scope without backend tests) | Acceptable |
|
||
| Backend test would be better but the venv is broken | Future sprint when the venv is fixed; Vitest test is the next-best defense | Open |
|
||
| OLLAMA_MODEL in `.env` overrides `config.py` default | Both are set to `gpt-oss:20b`; comment in `config.py` could note this for future maintainers | Acceptable |
|
||
|
||
## What Sprint 16 does NOT do
|
||
|
||
- **No backend test that catches the kimi-k2 bug.** The venv on `docker-willester` is broken; pytest is skipped. The Vitest contract test is the next-best defense. Future sprint.
|
||
- **No per-model prompt optimization.** The same prompt works for both models. If gpt-oss:20b ever returns low-quality picks (e.g. all the same recipe), the prompt can be tuned in a follow-up.
|
||
- **No CI integration of the Vitest tests.** The 11 cases run locally on `npm test`; no GitHub Actions or pre-commit hook.
|
||
- **F9-full (local Ollama model pull).** Still opt-in based on cloud-billing feedback. `_ask_llm` is the seam: F9-full only needs to swap the URL + model name.
|
||
|
||
## Follow-up tickets (carry forward)
|
||
|
||
- **Lower `_DAILY_LIMIT=140` to 45** in `recipe_search.py:48` (Sprint 15 follow-up, still pending).
|
||
- **Run seed_recipes.py round 4** for the 12 unrun round-1 queries (American + Mediterranean leftovers).
|
||
- **Backend test infrastructure** — the venv on `docker-willester` is broken. Future sprint when fixed.
|
||
- **CI integration of Vitest** — `npm test` runs locally but not in CI.
|