Public Access
feat: phase r1+r2 recovery + r3-0 swiftly api ingestion
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -0,0 +1,60 @@
|
||||
# Context — Recovery Takeover
|
||||
|
||||
## Why this plan exists
|
||||
|
||||
Prior agent marked Phases 1, 2, 3, 7 complete and consensus blockers "addressed" in docs, but verification of the repo shows:
|
||||
|
||||
1. Auth blocker (review §1.2) closed in docs only — no auth dependency on any router; `/api/admin/scrape` is open.
|
||||
2. No tests, no CI; verification matrix from `Review/reviewconcensus.md §6` was never run.
|
||||
3. Review §2.4 explicitly warned: spike scrape + email-approval BEFORE schema/UI commits. Prior agent did the opposite — schema, full API surface, and UI shell first; scrape unverified, email-approval not started.
|
||||
4. `/api/admin/scrape` runs Playwright synchronously inside the request handler; will time out in production.
|
||||
5. Phase 7 UI ships above engines (4/5/9) that don't exist — Dashboard renders meal plans the system can't generate.
|
||||
|
||||
## Decisions (locked in for this recovery branch)
|
||||
|
||||
- **Auth model:** bearer-token admin (single shared `ADMIN_TOKEN` env var) + signed-cookie session for family web UI. Matches what was claimed in ORIENTATION.md "Adversarial Review" section. No public-internet exposure assumed; nginx is sole entrypoint, already correct in `docker-compose.yml`.
|
||||
- **Path canonicalization (R1-B+D):** dropped `/list` and `/planned` suffixes; routers use `@router.get("")` (no trailing slash) so the canonical paths are `/api/profile`, `/api/recipes`, `/api/recipes/ingredients`, `/api/meals`, `/api/pantry`, `/api/shopping-list`. Frontend `frontend/src/api/index.ts` and smoke tests updated to enforce.
|
||||
- **Login bootstrap:** `/api/auth/login` signs the family-profile id; if no profile row exists yet, signs literal "bootstrap" so first-run isn't blocked. Cookie validates regardless; downstream code that needs a real id should re-issue after profile creation.
|
||||
- **Recipe-ingredient:** stay JSONB-only (already chosen). Do not reopen.
|
||||
- **Household model:** keep `family_member` table (already chosen). Do not reopen.
|
||||
- **Day-of-week:** ISO (1=Mon). Already chosen.
|
||||
- **Migrations:** Alembic only. Never `Base.metadata.create_all()` at runtime.
|
||||
- **Background work:** FastAPI `BackgroundTasks` for the scrape now; APScheduler container with `--workers 1` later (R3-E).
|
||||
|
||||
## Open questions to surface to the user, not to assume
|
||||
|
||||
- Is `ADMIN_TOKEN` acceptable, or does the user want OIDC/Tailscale-style auth? Default for now: bearer token, easy to swap.
|
||||
- Email backend for the spike: real SendGrid (needs key) or a console/file backend? Default for spike: console backend, swap to SendGrid in R3-C.
|
||||
|
||||
## Verification gate (Phase R1 must pass all)
|
||||
|
||||
- `cd backend && pytest` → green
|
||||
- `docker compose run --rm backend alembic upgrade head` → no error, schema matches models
|
||||
- `docker compose run --rm backend python -c "from app.main import app; print(app.title)"` → "MealPlanner"
|
||||
- `docker compose run --rm frontend npm run build` → no error
|
||||
- `curl -X POST http://localhost/api/admin/scrape` (no token) → 401
|
||||
- `curl http://localhost/api/profile` (no session) → 200 (read), POST/PUT → 401
|
||||
- CI workflow runs all of the above on push.
|
||||
|
||||
## Phase ordering rule (do not violate)
|
||||
|
||||
R1 and R2 are independent and run in parallel. R3 cannot start until BOTH R1 verification and R2 spikes pass. If R2 reveals schema impact, schema changes happen on this branch BEFORE R3-A.
|
||||
|
||||
## Swiftly API (R3-0, replaces Playwright path)
|
||||
|
||||
- Discovery: `GET https://luckysupermarkets.com/categories` (HTML, no auth). Selector: `<a class="swiftlyCouponCategory" href="/categories/<urlencoded slug>">`. Slug regex: `/categories/(.+)$` then `urllib.parse.unquote`. Fixture (2026-05-05) yielded 17 distinct slugs (e.g. `Product/meat_seafood`, `Product/produce`, ...).
|
||||
- Products: `GET https://prod.swiftlyapi.net/search/api/v1/products/categories?cat=<slug>&store=757&limit=10000` with `Authorization: Bearer <SWIFTLY_BEARER_TOKEN>`. Response shape: `{"products": {"info": {"count": N}, "items": [...], "facets": [...]}}`. `meat_seafood` returned 256 items.
|
||||
- Field mapping (item dict → grocery_item):
|
||||
- `id` (string) → new `external_id` column (migration 0005)
|
||||
- `name` → `name`
|
||||
- `description` → `description`
|
||||
- `brand` → `brand`
|
||||
- `primaryImage.url` → `image_url`
|
||||
- `price.ok.regPriceText` (e.g. `"$3.49 /lb"`) → parsed `regular_price` (Decimal) + `unit` (e.g. `"lb"`, may be NULL when no `/unit` suffix)
|
||||
- `price.ok.promoArea.promoText` (e.g. `"$2.49 /lb"`) → parsed `sale_price` (Decimal); when present `is_on_sale=True`, else `is_on_sale=False`
|
||||
- `price.ok.promoArea.validityText` (e.g. `"Valid 04/29/26 - 05/05/26"`) → ignored for v1 (no migration to add date columns; existing `sale_start_date` / `sale_end_date` left null)
|
||||
- aisle: extracted from the queried category slug (`Product/meat_seafood` → `meat_seafood`)
|
||||
- `product_url` → NULL (site has no public product page; per R2-A note kept nullable)
|
||||
- Auth scoping: bearer header is attached ONLY to `prod.swiftlyapi.net` requests, NOT to the public `luckysupermarkets.com` HTML page. Two `requests.Session` objects (one with default UA, one with the bearer header).
|
||||
- 401 detection: cannot use `BaseScraper._get` because it swallows HTTPError into a `None` return. The new client calls `session.get(...)` directly and checks `resp.status_code == 401` BEFORE `raise_for_status` to raise `SwiftlyAuthError`. Token in `.env.example` expires hourly per spec; on 401 the scraper aborts with a fixed error_message instructing the admin to refresh the token.
|
||||
- Idempotency key: `(source, external_id)` upserts. Migration 0005 adds `grocery_item.external_id` (nullable text, indexed; not unique because legacy R2-A rows lack one).
|
||||
@@ -0,0 +1,63 @@
|
||||
# Phase R1 + R2 Verification Gate — PASSED
|
||||
|
||||
Date: 2026-05-04
|
||||
|
||||
## Test results
|
||||
|
||||
```
|
||||
docker compose run pytest tests/ -q
|
||||
26 passed in 2.54s
|
||||
```
|
||||
|
||||
Suite: `test_smoke`, `test_alembic` (forward + roundtrip), `test_config`,
|
||||
`test_auth`, `test_scrape_endpoint`, `test_approval`, `test_lucky_ca_scraper`.
|
||||
|
||||
## Verification matrix
|
||||
|
||||
| Item | Result |
|
||||
|------|--------|
|
||||
| `from app.main import app; app.title` | "MealPlanner", 22 routes |
|
||||
| `alembic upgrade head` (fresh DB) | 0001 → 0002 → 0003 → 0004 clean |
|
||||
| `alembic upgrade → downgrade base → upgrade head` | clean (in-test) |
|
||||
| `pytest -q` | 26 passed, 0 failed |
|
||||
| `npm run build` (frontend) | clean, 264 KB |
|
||||
| `POST /api/admin/scrape` no token | 401 |
|
||||
| `POST /api/admin/scrape` bad bearer | 401 |
|
||||
| `POST /api/admin/scrape` good bearer | 202 + scrape_log_id |
|
||||
| `GET /api/profile` no session (read) | 200 |
|
||||
| `PUT /api/profile` no session (mutation) | 401 |
|
||||
| `POST /api/auth/login` good password | 204 + Set-Cookie |
|
||||
| `POST /api/auth/login` bad password | 401 |
|
||||
| Email approval round-trip (R2-B) | OK: send → GET 200 → POST 200 → replay 409, item_status=approved |
|
||||
| Live Lucky CA scrape (R2-A) | 11 coupons parsed, schema survives |
|
||||
|
||||
## Pre-existing defects discovered and fixed during the recovery pass
|
||||
|
||||
1. `backend/app/api/shopping_list.py:59` — `Ingredient.id.in_ all_ingredient_ids` syntax error blocking app import. One-char fix.
|
||||
2. `backend/app/models/__init__.py:201` — `MealPlan.votes` relationship had no FK target. Removed (votes are reachable via `MealPlan.items[*].votes`).
|
||||
3. `backend/alembic/versions/0001_initial_migration.py:101` — `JSONB(astext=True)` invalid kwarg. Dropped.
|
||||
4. `backend/alembic/versions/0001_initial_migration.py:288` — `downgrade()` was `pass`. Replaced with a DO block dropping all non-alembic tables + all public enum types.
|
||||
5. `backend/alembic/versions/0002_seed_data.py` — duplicate Chickpeas + Black Beans seed rows; INSERT not idempotent. Removed dupes, added `ON CONFLICT (name_lower) DO NOTHING`.
|
||||
6. `backend/requirements.txt` — scraper imports `requests` but it wasn't pinned. Added `requests==2.31.0`.
|
||||
7. `backend/app/models/__init__.py` — every `SQLEnum(...)` used the default name-based mapping, but the Postgres enum types use lowercase values. All occurrences now use `values_callable=lambda obj: [e.value for e in obj]`.
|
||||
8. Schema drift: `FamilyProfile.calorie_target` existed in the model but not in the migration. New migration `0004_family_profile_calorie_target.py` adds it.
|
||||
9. `docker-compose.yml` — backend service didn't pass `ADMIN_TOKEN` / `SESSION_PASSWORD` / `EMAIL_BACKEND` env to the container. Added.
|
||||
10. `backend/tests/test_config.py` — `importlib.reload` fired the `RuntimeError` outside `pytest.raises`. Refactored to construct `Settings(_env_file=None, DATABASE_URL="")` inside the assertion.
|
||||
11. `backend/app/api/admin.py` `POST /scrape` ran Playwright synchronously inside the request handler. Now enqueues via `BackgroundTasks` and returns 202.
|
||||
12. `grocery_item.description` column added (migration 0003) — the live scraper produces description but the model/schema didn't have a column.
|
||||
|
||||
## Decisions captured in `.agent/context.md`
|
||||
|
||||
- Auth model: bearer `ADMIN_TOKEN` for admin, signed-cookie session (itsdangerous, key=SECRET_KEY) for family UI mutations. Reads stay open inside the trusted network.
|
||||
- Login bootstrap: signs literal `"bootstrap"` if no FamilyProfile exists yet — first-run hatch. **Worth flagging to the user explicitly.**
|
||||
- Path canonicalization: dropped `/list` and `/planned` suffixes; routers use `@router.get("")`. Frontend updated.
|
||||
- Email backend: `ConsoleEmailBackend` for dev; SendGrid stub raises NotImplementedError until R3-C.
|
||||
|
||||
## Open issues tracked but not blocking the gate
|
||||
|
||||
- Task #8: `ScrapeStatus` enum lacks distinct queued/started states (R1-C reused STARTED).
|
||||
- Task #10: `scraper_service._run_scrape_in_background` line 105 has tz-aware/naive datetime subtraction TypeError. Affects bg task happy-path post-202; admin gate verified independently.
|
||||
|
||||
## Phase R1 + R2 — DONE
|
||||
|
||||
Per `.agent/plan.md`, R3 (Phase 4 recipe engine, Phase 5 planner algo, Phase 6 SendGrid, Phase 8 feedback UI, Phase 9 generation, Phase 10 images) is now unblocked. Schema has survived contact with the deferred-risk spikes the adversarial review demanded.
|
||||
@@ -0,0 +1,29 @@
|
||||
# R1-A — Verification Harness
|
||||
|
||||
## Files created (NEW only — no edits to existing files)
|
||||
|
||||
- `backend/pytest.ini` — testpaths=tests, asyncio auto, deprecation filter.
|
||||
- `backend/tests/__init__.py` — empty package marker.
|
||||
- `backend/tests/conftest.py` — sets `DATABASE_URL` from `TEST_DATABASE_URL` before app import; `requires_postgres` marker auto-skips when PG unreachable; session-scoped `_schema` runs `alembic upgrade head`; per-test `db` fixture uses connection+transaction rollback for isolation; `client` fixture overrides `get_db`.
|
||||
- `backend/tests/test_smoke.py` — `test_app_imports`, `test_health`, `test_health_db`, parametrized `test_router_list_endpoints` over the 8 paths in the spec. Asserts `< 500` and accepts {200, 307, 401, 404, 422}.
|
||||
- `backend/tests/test_alembic.py` — downgrade-base then upgrade-head round-trip.
|
||||
- `backend/requirements-dev.txt` — pytest, pytest-cov, pytest-asyncio, httpx (note: pytest + httpx + pytest-asyncio also pinned in `requirements.txt`; dev file holds the canonical dev set).
|
||||
- `.github/workflows/ci.yml` — `backend` job (postgres:15 service, py3.11, alembic upgrade, pytest -q) + `frontend` job (node20, npm ci, npm run build). Triggers on push + pull_request.
|
||||
|
||||
## Test outcomes (could not execute locally — Bash python denied)
|
||||
|
||||
By design:
|
||||
- `test_app_imports` — passes if `app.main` imports cleanly (DATABASE_URL set in conftest pre-import).
|
||||
- `test_health`, `test_health_db`, `test_router_list_endpoints[*]`, `test_alembic_upgrade_head_roundtrip` — all gated on `requires_postgres`. Pass when CI Postgres service is up; auto-skipped locally without `TEST_DATABASE_URL`.
|
||||
- No xfails added.
|
||||
|
||||
## Blockers / risks found
|
||||
|
||||
- `app/config.py::Settings` has no default for `DATABASE_URL` — any import path without env var crashes. Conftest works around it but production code is fragile. Flag for R1-B.
|
||||
- Alembic migrations use `postgresql.UUID/JSONB/ARRAY` — SQLite fallback impossible. CI requires PG service (already wired).
|
||||
- Spec listed `/api/recipes/ingredients` and `/api/meals` but actual routes are `/api/recipes/ingredients/list` and `/api/meals/planned`. Smoke test still validates router wiring (404 acceptable, no 5xx).
|
||||
- `/health` declares an unused `db` dependency — works but odd.
|
||||
|
||||
## CI yaml one-liner
|
||||
|
||||
Two jobs (`backend` w/ postgres:15 service runs alembic+pytest, `frontend` runs `npm ci && npm run build`) on push/PR.
|
||||
@@ -0,0 +1,39 @@
|
||||
# R1-B+D — Auth + path canonicalization + DATABASE_URL fail-fast
|
||||
|
||||
## Files added
|
||||
- `backend/app/security.py` — `require_admin` (bearer token vs `settings.ADMIN_TOKEN`), `require_session` (signed-cookie via itsdangerous), `issue_session`, `SESSION_COOKIE`, `SESSION_MAX_AGE`.
|
||||
- `backend/app/api/auth.py` — `POST /api/auth/login` (shared `SESSION_PASSWORD`, sets httponly+secure+samesite=lax cookie), `POST /api/auth/logout` (clears cookie). Both return 204.
|
||||
- `backend/tests/test_auth.py` — admin bearer required, session cookie required for mutation, login/logout round-trip.
|
||||
- `backend/tests/test_config.py` — `Settings(_env_file=None, DATABASE_URL="")` raises `RuntimeError("DATABASE_URL is required")`.
|
||||
|
||||
## Files modified
|
||||
- `backend/app/config.py` — added `ADMIN_TOKEN`, `SESSION_PASSWORD`; `DATABASE_URL` no longer typed as required (default `""`) but a `model_validator(mode="after")` raises `RuntimeError` if blank — gives a clear error instead of pydantic's confusing ValidationError.
|
||||
- `backend/app/main.py` — wired `auth.router` at `/api/auth`.
|
||||
- `backend/app/api/admin.py` — `APIRouter(dependencies=[Depends(require_admin)])` so EVERY admin route is bearer-gated.
|
||||
- `backend/app/api/profile.py` — session-gated: `PUT /`, `POST /members`, `DELETE /members/{id}`. GETs open. Trailing slashes dropped.
|
||||
- `backend/app/api/recipes.py` — rewritten so `/ingredients` GET/POST come before `/{recipe_id}`. Session-gated: `POST /`, `DELETE /{id}`, `POST /ingredients`. GETs open. `GET /ingredients/list` renamed → `GET /ingredients`. Trailing slashes dropped.
|
||||
- `backend/app/api/meals.py` — `GET /planned` renamed → `GET ""`. Session-gated: `POST /`, `POST /{id}/lock`, `POST /items/{id}/swap`. Per-voter approval token routes (`GET/POST /items/{id}/vote/{token}`) UNCHANGED.
|
||||
- `backend/app/api/pantry.py` — session-gated: `POST /`, `PUT /{id}`, `DELETE /{id}`. GETs open. Trailing slashes dropped.
|
||||
- `backend/app/api/shopping_list.py` — trailing slash dropped on root GET. No mutations exist.
|
||||
- `backend/tests/test_smoke.py` — canonical paths enforced; 307 (slash redirect) is now a failure.
|
||||
- `backend/requirements.txt` — added `itsdangerous==2.1.2`.
|
||||
- `frontend/src/api/index.ts` — `withCredentials: true`, added `auth.login/logout`, `recipes.listIngredients` → `/recipes/ingredients`, `meals.getPlanned` → `/meals`.
|
||||
- `.env.example` — added `ADMIN_TOKEN`, `SESSION_PASSWORD`.
|
||||
|
||||
## Routes admin-gated (bearer token)
|
||||
All `/api/admin/*`: `POST /scrape`, `GET /logs`, `GET /logs/{id}`, `GET /email-logs`, `GET /meal-plans`, `POST /test-email`, `GET /stats`.
|
||||
|
||||
## Routes session-gated (cookie)
|
||||
- `PUT /api/profile`, `POST /api/profile/members`, `DELETE /api/profile/members/{id}`
|
||||
- `POST /api/recipes`, `DELETE /api/recipes/{id}`, `POST /api/recipes/ingredients`
|
||||
- `POST /api/meals`, `POST /api/meals/{id}/lock`, `POST /api/meals/items/{id}/swap`
|
||||
- `POST /api/pantry`, `PUT /api/pantry/{id}`, `DELETE /api/pantry/{id}`
|
||||
|
||||
## Paths renamed
|
||||
- `GET /api/recipes/ingredients/list` → `GET /api/recipes/ingredients`
|
||||
- `GET /api/meals/planned` → `GET /api/meals`
|
||||
- All routers: `@router.get("/")` → `@router.get("")` (no trailing slash on resource roots)
|
||||
|
||||
## Blockers
|
||||
- Could not execute `pytest` or `python -c "from app.main import app"` locally — Bash python execution denied (same as R1-A). Verification deferred to CI. Logical review of imports/wiring done.
|
||||
- TestClient runs http; the `secure=True` cookie won't auto-roundtrip. `test_session_login_logout` manually re-sets the cookie to validate the signing path. Production (https via nginx) is unaffected.
|
||||
@@ -0,0 +1,64 @@
|
||||
# R1-C — Async scrape + grocery_item.description
|
||||
|
||||
## What changed
|
||||
|
||||
- `backend/app/api/admin.py` — `POST /api/admin/scrape` now returns **202**
|
||||
with body `{"status": "queued", "scrape_log_id": "<uuid>"}`. Endpoint
|
||||
delegates to `enqueue_scrape(db, source, scrape_type, background_tasks)`.
|
||||
- `backend/app/services/scraper_service.py` — split:
|
||||
- `enqueue_scrape()` inserts a `ScrapeLog` row (status=`STARTED`),
|
||||
`db.commit()`, then `background_tasks.add_task(_run_scrape_in_background, log.id, ...)`.
|
||||
Returns the refreshed log row.
|
||||
- `_run_scrape_in_background(log_id, source, scrape_type)` opens its OWN
|
||||
`SessionLocal()` (request session is gone by then), runs the scraper,
|
||||
sets terminal status (`SUCCESS`/`FAILED` + `error_message` + `duration_seconds`),
|
||||
`db.close()` in `finally`. Rolls back before writing FAILED to avoid
|
||||
detached instances.
|
||||
- `ScraperService.run_scrape` retained for direct/test callers; both paths
|
||||
share `_do_scrape()` and `_save_grocery_item()`.
|
||||
- `_save_grocery_item` now maps `item_data["description"]` → `GroceryItem.description`
|
||||
on both insert and update.
|
||||
- `backend/app/models/__init__.py` — added `description = Column(Text, nullable=True)` to `GroceryItem`.
|
||||
- `backend/alembic/versions/0003_grocery_item_description.py` — `revision="0003"`,
|
||||
`down_revision="0002"`. `op.add_column("grocery_item", sa.Column("description", sa.Text(), nullable=True))` / `op.drop_column` on downgrade.
|
||||
- `backend/tests/test_scrape_endpoint.py` — monkeypatches
|
||||
`_run_scrape_in_background` to a no-op, POSTs with valid bearer, asserts
|
||||
202 + `scrape_log_id`, reads back the row through the test session, asserts
|
||||
`status == ScrapeStatus.STARTED` and `completed_at is None`.
|
||||
|
||||
## Endpoint contract
|
||||
|
||||
`POST /api/admin/scrape` (admin bearer required):
|
||||
- Request: optional query `source` (default `lucky_california`), `scrape_type` (default `weekly_ad`).
|
||||
- Response: **202** + `{"status": "queued", "scrape_log_id": "<uuid>"}`.
|
||||
- Poll `GET /api/admin/logs/{id}` for terminal state.
|
||||
|
||||
## Status enum note
|
||||
|
||||
Existing `ScrapeStatus` enum is `started|success|failed` only — no
|
||||
`pending`/`running`/`completed` members. Added migration would have to alter
|
||||
the PG enum type (and seed data assumes 3-value). Reused `STARTED` as the
|
||||
queued/in-flight state; spec semantics map cleanly. If you want explicit
|
||||
`PENDING`/`RUNNING`/`COMPLETED`, that's a separate enum migration.
|
||||
|
||||
## DB session in the bg task
|
||||
|
||||
Imports `SessionLocal` from `app.database` at function call time and opens a
|
||||
fresh session per task; the request-scoped session is closed when the 202 is
|
||||
sent.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pytest tests/test_lucky_ca_scraper.py -q` → 1 passed (R2-A still green).
|
||||
- `tests/test_scrape_endpoint.py` parses and is wired correctly; runs only
|
||||
with `TEST_DATABASE_URL` set (other tests follow the same convention).
|
||||
- `python -c "from app.services.scraper_service import enqueue_scrape, _run_scrape_in_background, ScraperService; from app.models import GroceryItem, ScrapeStatus"` → ok.
|
||||
|
||||
## Blocker (pre-existing, OUT OF R1-C SCOPE)
|
||||
|
||||
`backend/app/api/shopping_list.py:59` has a syntax error from commit c735d21:
|
||||
`Ingredient.id.in_ all_ingredient_ids` — missing parens, should be
|
||||
`Ingredient.id.in_(all_ingredient_ids)`. This blocks `from app.main import app`
|
||||
and therefore `tests/test_smoke.py::test_app_imports`. Untouched by R1-A,
|
||||
R1-B+D, R2-A, or R1-C — they all couldn't actually exercise the import path.
|
||||
Recommend a one-character fix in the next round (it's outside my owned files).
|
||||
@@ -0,0 +1,35 @@
|
||||
# R2-A — Lucky California live-scrape spike (DONE)
|
||||
|
||||
## Outcome
|
||||
|
||||
- ONE live fetch performed: `https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad`, HTTP 200, 201 KB rendered HTML.
|
||||
- Fixture saved: `backend/tests/fixtures/lucky_ca/weekly_ad.html` + `weekly_ad.png` + `META.md`.
|
||||
- Parser now yields **11 items** from the captured page, each with non-empty `name`, `current_price`, and `image_url`.
|
||||
- Test `backend/tests/test_lucky_ca_scraper.py::test_parse_fixture` passes (`pytest` exit 0, 0.09 s, no network, no Playwright).
|
||||
|
||||
## Selectors (verified 2026-05-04)
|
||||
|
||||
- Card root: `div.coupon-card-wrapper` (Swiftly-rendered tiles).
|
||||
- Price + short name: `.coupon-card-value-text` ("$13.97 Pepsi 24 packs").
|
||||
- Long description: `.coupon-card-short-description`.
|
||||
- Image: nested `<img>` from `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`.
|
||||
- No per-card hyperlink in DOM; offers are non-navigable tiles.
|
||||
|
||||
## Scraper diff (minimal, additive)
|
||||
|
||||
`backend/app/scraper/lucky_ca_scraper.py`: added `parse_featured_coupons_html(html)` and `_parse_coupon_card(card)`; rewrote `scrape_featured_coupons` to delegate to the parser. Old `_parse_coupon_item` retained as fallback. The original `h2/h3/a` regex selector was stale — it produced 1 item against the live DOM.
|
||||
|
||||
## Captcha / blocker check
|
||||
|
||||
The HTML contains a `<span id="recaptcha-element">` placeholder element (no challenge served). Title is `"Featured in Ad | Luckys Supermarket"`, all 13 `coupon-card-wrapper` cards rendered. **Not a wall.** The spike script's heuristic flagged the keyword; META.md documents the false positive.
|
||||
|
||||
## Schema impact (grocery_item)
|
||||
|
||||
- Survives contact with reality. Populated columns: `name`, `current_price`, `image_url`, `is_on_sale`, `scraped_at`, `scraped_url`.
|
||||
- `product_url` is **always NULL** on this site — keep nullable, don't index as required.
|
||||
- Parser also produces `description` (long offer text). Recommend either adding `description TEXT NULL` to `grocery_item` or dropping the field — currently no model column exists for it.
|
||||
- 2 of 13 cards filter out (no parseable price), expected (header/footer rows, e.g. "$35 minimum order"). Not a regression.
|
||||
|
||||
## Blockers
|
||||
|
||||
None. Phase R2-A clears the gate. Recommend a tiny migration adding `grocery_item.description TEXT NULL` before R3-A, otherwise the description is dropped silently when persisting.
|
||||
@@ -0,0 +1,82 @@
|
||||
# R2-B blockers — pre-existing defects discovered by the spike
|
||||
|
||||
The R2-B email + per-voter approval round-trip cannot complete the
|
||||
end-to-end DB-backed proof until both of the following pre-existing bugs
|
||||
are fixed. Both are OUT OF SCOPE for R2-B per the task brief.
|
||||
|
||||
## Blocker 1: model relationship bug — `MealPlan.votes`
|
||||
|
||||
**File:** `backend/app/models/__init__.py:201`
|
||||
|
||||
```python
|
||||
class MealPlan(Base):
|
||||
...
|
||||
votes = relationship("MealPlanVote", back_populates="meal_plan", cascade="all, delete-orphan")
|
||||
```
|
||||
|
||||
But `MealPlanVote` has no `meal_plan` relationship and no FK to
|
||||
`meal_plan.id` — only to `meal_plan_item.id` (line 235).
|
||||
|
||||
**Symptom:** `sqlalchemy.exc.NoForeignKeysError: Could not determine join
|
||||
condition between parent/child tables on relationship MealPlan.votes`
|
||||
fires the moment ANY mapper is configured (i.e. on the first ORM use of
|
||||
any model). Blocks every flow, not just the vote flow.
|
||||
|
||||
**Repro (with schema in place):**
|
||||
|
||||
```bash
|
||||
DATABASE_URL='postgresql://...' python -c "
|
||||
from app import models
|
||||
from sqlalchemy.orm import configure_mappers
|
||||
configure_mappers() # raises NoForeignKeysError
|
||||
"
|
||||
```
|
||||
|
||||
**Fix options (for whichever agent owns models):**
|
||||
1. Remove `MealPlan.votes` (votes are reachable via `MealPlan.items[*].votes`).
|
||||
2. Add `meal_plan_id` FK to `MealPlanVote` and a back-ref. Requires migration.
|
||||
3. Specify `primaryjoin="MealPlan.id == foreign(remote(MealPlanVote.meal_plan_item_id))"` via `MealPlanItem` — viewonly only.
|
||||
|
||||
Option 1 is least invasive and matches existing usage in `meals.py` (no
|
||||
code reads `MealPlan.votes`).
|
||||
|
||||
## Blocker 2: migration uses invalid kwarg — `JSONB(astext=True)`
|
||||
|
||||
**File:** `backend/alembic/versions/0001_initial_migration.py:101`
|
||||
|
||||
```python
|
||||
sa.Column('ingredients', postgresql.JSONB(astext=True), nullable=False),
|
||||
```
|
||||
|
||||
`astext` is not a valid `JSONB.__init__` kwarg in SQLAlchemy 2.x.
|
||||
|
||||
**Symptom:** `alembic upgrade head` fails with:
|
||||
`TypeError: JSON.__init__() got an unexpected keyword argument 'astext'`.
|
||||
|
||||
**Impact:** `backend/tests/test_alembic.py::test_alembic_upgrade_head_roundtrip`
|
||||
fails. `backend/tests/conftest.py::_schema` fixture errors, so any test
|
||||
marked `requires_postgres` errors at session bootstrap.
|
||||
|
||||
**Fix:** drop the kwarg. `astext` is a *runtime* attribute on a `JSONB`
|
||||
column expression for casting to text, not a column definition arg.
|
||||
|
||||
```python
|
||||
sa.Column('ingredients', postgresql.JSONB(), nullable=False),
|
||||
```
|
||||
|
||||
## Effect on R2-B
|
||||
|
||||
- Parts 1 (services), 2 (routes), and the unit-test portion of Part 4
|
||||
are complete and verified (3 unit tests pass, 3 DB-backed tests skip
|
||||
cleanly on no-PG environments).
|
||||
- Part 3 `--simulate-click` and the 3 DB-backed tests in Part 4 require
|
||||
a working schema. They will run as soon as Blocker 1 is fixed (the
|
||||
spike script can bootstrap its own schema via SQLAlchemy
|
||||
`Base.metadata.create_all` once the relationship resolves).
|
||||
|
||||
## Sandbox cleanup note
|
||||
|
||||
While diagnosing Blocker 1, this agent ran `Base.metadata.create_all`
|
||||
against `mealplanner-db-1` (192.168.144.2:5432, db `mealplanner`) and
|
||||
then dropped all created tables/types after the model error surfaced.
|
||||
Final state: only `alembic_version` (matches pre-spike state).
|
||||
@@ -0,0 +1,82 @@
|
||||
# R2-B — Email + per-voter approval round-trip spike
|
||||
|
||||
## Schema gap (top, per spec)
|
||||
|
||||
The spike surfaced a pre-existing **model relationship bug** that blocks
|
||||
the round-trip even after a clean schema bootstrap:
|
||||
`backend/app/models/__init__.py:201` declares
|
||||
`MealPlan.votes = relationship("MealPlanVote", back_populates="meal_plan", ...)`
|
||||
but `MealPlanVote` has no `meal_plan` FK or relationship. SQLAlchemy
|
||||
fails at first mapper configure with `NoForeignKeysError`. Details +
|
||||
two pre-existing migration bug in `r2b-blockers.md`. Per scope rule
|
||||
("If the spike reveals the schema can't support the flow, STOP, do not
|
||||
workaround"), I did NOT touch models or migrations.
|
||||
|
||||
## What landed
|
||||
|
||||
- `backend/app/services/email.py` — `EmailBackend` Protocol,
|
||||
`ConsoleEmailBackend` (stdout + JSONL outbox at
|
||||
`backend/var/email_outbox.jsonl`), `SendGridEmailBackend` stub
|
||||
raising `NotImplementedError("Wire SendGrid in R3-C")`,
|
||||
`get_email_backend()` switching on `settings.EMAIL_BACKEND`.
|
||||
- `backend/app/services/approval.py` — itsdangerous
|
||||
`URLSafeTimedSerializer`, salt `meal-approval-v1`, key
|
||||
`settings.SECRET_KEY`. `issue_token`, `verify_token` (raises 401),
|
||||
`consume_token` (verifies, matches URL `item_id`, looks up voter,
|
||||
enforces single-use via existing `MealPlanVote` row → 409). Single-
|
||||
use enforcement lives ONLY in `consume_token`.
|
||||
- `backend/app/api/meals.py` — replaced two vote routes:
|
||||
- `GET /api/meals/vote/{item_id}?token=...` → minimal accessible
|
||||
HTML page (lang attr, contrast, ARIA labels, JS-enhanced JSON
|
||||
POST with form fallback). Token only inside form `action`,
|
||||
never visible body.
|
||||
- `POST /api/meals/vote/{item_id}?token=...` body `{"vote": "approve"|"deny"}`.
|
||||
Records `MealPlanVote` (Boolean: True/False), then applies rule:
|
||||
any deny → DENIED; full electorate approved → APPROVED; else PENDING.
|
||||
Returns `{"status": "recorded", "item_status": "..."}`.
|
||||
- `backend/app/config.py` — added `EMAIL_BACKEND: str = "console"`.
|
||||
- `backend/requirements-dev.txt` — added `freezegun>=1.4`.
|
||||
- `.gitignore` — appended `backend/var/`.
|
||||
- `scripts/send_test_approval.py` — bootstraps profile/voter/recipe/
|
||||
plan/item, issues a token, sends via `ConsoleEmailBackend`, prints
|
||||
the URL. `--simulate-click {approve|deny}` drives a `TestClient`
|
||||
GET (asserts 200 + html) + POST (asserts 200, prints
|
||||
`item_status=...`) + a second POST (asserts 409 single-use).
|
||||
- `backend/tests/test_approval.py` — 6 tests as spec'd; 3 unit tests
|
||||
green; 3 `requires_postgres` tests written and skip cleanly on
|
||||
no-PG. They will pass once Blocker 1 is fixed.
|
||||
|
||||
## URL contract
|
||||
|
||||
`/api/meals/vote/{item_id}?token=<itsdangerous-signed JSON>`.
|
||||
Token payload: `{"item": str(uuid), "voter": str(uuid)}`. TTL 7 days.
|
||||
Single use enforced by `UniqueConstraint(meal_plan_item_id, family_member_id)`
|
||||
on `meal_plan_vote` (already in the schema).
|
||||
|
||||
## What the round-trip proved (and didn't)
|
||||
|
||||
- **Token layer** proved end-to-end (issue → verify → tamper → expire).
|
||||
- **Schema fit (paper)**: `family_member`, `meal_plan_item`,
|
||||
`meal_plan_vote` columns map cleanly to the flow.
|
||||
- **Schema fit (runtime)**: BLOCKED by `MealPlan.votes` defect.
|
||||
Cannot run the `--simulate-click` proof or the 3 DB tests until
|
||||
another agent fixes the model.
|
||||
|
||||
## Verification (what I ran)
|
||||
|
||||
- `pytest backend/tests/test_approval.py -q` → **3 passed, 3 skipped**.
|
||||
- `python -c "from app.services.email import get_email_backend; print(...)"`
|
||||
→ `ConsoleEmailBackend`.
|
||||
- `python scripts/send_test_approval.py --simulate-click approve` →
|
||||
fails at ORM init with the documented Blocker 1.
|
||||
|
||||
## Files
|
||||
|
||||
- created: `backend/app/services/__init__.py`,
|
||||
`backend/app/services/email.py`, `backend/app/services/approval.py`,
|
||||
`scripts/send_test_approval.py`,
|
||||
`backend/tests/test_approval.py`,
|
||||
`.agent/phase-summaries/r2b-blockers.md`,
|
||||
`.agent/phase-summaries/r2b-summary.md`.
|
||||
- modified: `backend/app/api/meals.py`, `backend/app/config.py`,
|
||||
`backend/requirements-dev.txt`, `.gitignore`.
|
||||
@@ -0,0 +1,56 @@
|
||||
# R3-0 (Swiftly API ingestion) — GATE PASSED
|
||||
|
||||
Date: 2026-05-05
|
||||
|
||||
## Live end-to-end scrape
|
||||
|
||||
```
|
||||
POST /api/admin/scrape (Bearer test-admin-token) → 202 + scrape_log_id
|
||||
[bg] Swiftly: discovered 17 categories
|
||||
[bg] Background scrape complete: 10908/10908 items saved
|
||||
final status: success
|
||||
items_scraped: 10908
|
||||
duration_seconds: 36
|
||||
grocery_item count (after upsert): 9960
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```
|
||||
pytest -q tests/ → 31 passed in 2.41s
|
||||
```
|
||||
|
||||
Includes:
|
||||
- 26 tests from R1+R2 gate
|
||||
- 5 new Swiftly tests: parse categories fixture, parse category JSON, mapping invariants, 401 raises `SwiftlyAuthError`, background runner writes FAILED with actionable message.
|
||||
|
||||
## Schema
|
||||
|
||||
Migration `0005_grocery_item_external_id.py` adds:
|
||||
- `grocery_item.external_id TEXT NULLABLE` (Swiftly product id)
|
||||
- `grocery_item.source TEXT NULLABLE`
|
||||
- composite index `ix_grocery_item_source_external_id`
|
||||
|
||||
Idempotency key for upserts: `(source, external_id)` with fallback `(name, scraped_url)` for legacy R2-A rows.
|
||||
|
||||
## Resolved
|
||||
|
||||
- Task #11 (LuckyCaliforniaScraper `_browser` attr bug): obsoleted — the new client uses `requests`, no Playwright lifecycle.
|
||||
- The "Phase 3 complete on paper, broken in practice" condition flagged in the original synopsis is no longer true.
|
||||
|
||||
## Behavior to know
|
||||
|
||||
- `SWIFTLY_BEARER_TOKEN` is read from env at scraper init. The token in `.env.example` and `.env.test` is the user-supplied one and will expire (~hourly). On 401, scrape fails with `error_message` instructing the admin to capture a fresh token from luckysupermarkets.com's devtools network tab and update the env var.
|
||||
- Bearer header is scoped to `prod.swiftlyapi.net` only; never sent to the public categories page.
|
||||
- `LUCKY_STORE_ID=757` (Lucky California in San Pablo). Switch via env to target a different store.
|
||||
- Playwright is still pinned in `requirements.txt` (other future scrapers may want it). Removable later if no other use case lands.
|
||||
|
||||
## Open items
|
||||
|
||||
- Task #8: ScrapeStatus lacks distinct queued/started states. Cosmetic.
|
||||
- `.env.example` carries the live bearer token; user authorized. Refresh whenever the token expires.
|
||||
|
||||
## Phase 3 status
|
||||
|
||||
Was: Playwright scraper, 11 coupons, broken `_browser` attribute, 0 rows persisted via the endpoint.
|
||||
Now: JSON API client, 17 categories × N products, 9,960 rows persisted live, 31 tests green, 401-handling proven.
|
||||
@@ -0,0 +1,64 @@
|
||||
# R3-0 — Swiftly product API client (replaces Playwright path)
|
||||
|
||||
## Schema migration
|
||||
|
||||
Added `0005_grocery_item_external_id.py` (`down_revision='0004'`):
|
||||
|
||||
- `grocery_item.external_id` (`String(100)`, nullable, indexed)
|
||||
- `grocery_item.source` (`String(50)`, nullable)
|
||||
- composite index `ix_grocery_item_source_external_id`
|
||||
|
||||
Idempotency key for upserts is `(source, external_id)`. Both nullable so legacy R2-A rows (which lacked an external id) keep validating; the upsert path falls back to `(name, scraped_url)` when external_id is absent.
|
||||
|
||||
## Discovery + sample counts
|
||||
|
||||
- Categories page: 17 distinct slugs (e.g. `Product/meat_seafood`, `Product/produce`, `Product/dairy_eggs_cheese`, ...). Selector: `<a class="swiftlyCouponCategory" href="/categories/<urlencoded slug>">`, regex with double-lookahead so attribute order doesn't matter.
|
||||
- `Product/meat_seafood` JSON returned 256 items, all 256 mapped successfully — 82 on sale (`promoArea` present), 174 regular-only.
|
||||
|
||||
## Field mapping
|
||||
|
||||
| Swiftly JSON | grocery_item column |
|
||||
| --- | --- |
|
||||
| `id` | `external_id` (new) |
|
||||
| `name` | `name` |
|
||||
| `description` | `description` |
|
||||
| `brand` | `brand` |
|
||||
| `primaryImage.url` | `image_url` |
|
||||
| `price.ok.regPriceText` (e.g. `"$3.49 /lb"`) | `regular_price`, `unit` |
|
||||
| `price.ok.promoArea.promoText` | `sale_price`, `is_on_sale=True` |
|
||||
| (queried slug → tail) | `aisle` (e.g. `meat_seafood`) |
|
||||
| (constant) | `source = "lucky_california"` |
|
||||
| (none) | `product_url = NULL` (Swiftly exposes none) |
|
||||
| `validityText` | not parsed — `sale_start_date`/`sale_end_date` left null |
|
||||
|
||||
`current_price` = `sale_price` when on sale, else `regular_price`. Prices stored as `Decimal`.
|
||||
|
||||
## 401 handling
|
||||
|
||||
`SwiftlyAuthError` is a custom exception raised:
|
||||
|
||||
1. Up-front when `SWIFTLY_BEARER_TOKEN` is empty (no HTTP call).
|
||||
2. On `response.status_code == 401` BEFORE `raise_for_status` (which would have masked the 401 as a generic `HTTPError`). The new client uses `requests.Session.get` directly — `BaseScraper._get`'s retry-and-swallow path was bypassed deliberately, advisor flagged this as load-bearing.
|
||||
|
||||
`_run_scrape_in_background` catches all exceptions (existing behavior), writes `status=FAILED` + `error_message="SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)"`. Admin sees it via `GET /api/admin/logs/<id>`.
|
||||
|
||||
## Auth scoping
|
||||
|
||||
Two `requests.Session` objects: `public_session` (no auth, hits `luckysupermarkets.com`) and `api_session` (Authorization header attached per-request, hits `prod.swiftlyapi.net`). Bearer is never sent to the public host.
|
||||
|
||||
## Verification
|
||||
|
||||
- `pytest -q tests/test_swiftly_api.py` → 5 passed, 1 skipped (Postgres-only).
|
||||
- `pytest -q tests/` → 10 passed, 21 skipped (all skipped because no live Postgres in dev shell — same as R1+R2 gate-pass baseline).
|
||||
- `python scripts/spike_swiftly_ingest.py --confirm-live` → 17 categories discovered, 256 items in meat_seafood, sale + reg-only samples both render correctly with Decimal prices and unit="lb".
|
||||
- Docker stack POST verification deferred: no live Postgres in this shell. The test `test_background_runner_writes_failed_with_token_message` covers that path under `requires_postgres` and will run in CI / `docker compose` env.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Token in `.env.example` is a real (expiring) credential**, per task spec. It expired 2026-05-05 ~07:13 PT (`exp:1777991217`); I used it during the spike and it still worked. The user explicitly authorized this. Recommend gitignoring `.env.example` or rotating to a placeholder in a future cleanup task.
|
||||
- Old test `tests/test_lucky_ca_scraper.py` was deleted (it asserted on `parse_featured_coupons_html`, which no longer exists). R2-A's fixtures `weekly_ad.html`, `weekly_ad.png`, `META.md` retained per task spec.
|
||||
- `BaseScraper` and `SeleniumScraper` classes in `app/scraper/base.py` are no longer subclassed but kept untouched — they are still imported via `app.scraper.__init__` and may be useful for a future second store. No Playwright code path is exercised by `LuckyCaliforniaScraper` anymore, so the `_browser` attribute bug cannot recur.
|
||||
- Rate limit set to 0.75s/req in the new client (between the spec's 1–2 req/sec). Sequential walk of 17 categories @ ~250 items/category should run in ~15s server-side.
|
||||
- `_save_grocery_item` now `flush()`es instead of `commit()`ting per row; the outer commit happens in `_run_scrape_in_background` / `ScraperService.run_scrape`. Trade-off: a single bad row aborts the whole scrape's transaction. The API data is well-typed so this is acceptable; a future hardening could wrap each row in a savepoint.
|
||||
- Legacy fallback `(name, scraped_url)` upsert path is intentionally non-colliding with new rows: the new scraper writes `scraped_url="LuckyCaliforniaScraper:Product/<slug>"` whereas R2-A wrote `scraped_url=base_url`, so the two epochs of rows coexist without false matches.
|
||||
- Verification #4 (POST `/api/admin/scrape` against the live docker stack) was NOT run from this subagent shell — no live Postgres reachable. The unit test `test_background_runner_writes_failed_with_token_message` (Postgres-required, skips cleanly without it) covers the failure-path persistence; the success-path will run when the parent agent runs the suite inside `docker compose` per the R1+R2 gate-pass precedent.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Recovery Plan — MealPlanner
|
||||
|
||||
Goal: bring implementation back into alignment with `Review/reviewconcensus.md`. Stop building forward features until the deferred-risk spikes and the verification matrix pass.
|
||||
|
||||
## Phase R1 — Stabilize (parallel-safe)
|
||||
|
||||
- [ ] R1-A: Verification harness. Add `backend/tests/` with pytest config, a `conftest.py` with a transactional DB fixture, and smoke tests covering: app import, `/health`, `/health/db`, every router's GET list endpoint, Alembic `upgrade head` round-trip on a throwaway DB. Add `.github/workflows/ci.yml` running lint + pytest + frontend `npm run build`.
|
||||
- [ ] R1-B: Auth dependencies on existing routers. Implement an `app.security` module with: (1) `require_admin` dep — bearer token compared to `settings.ADMIN_TOKEN`, applied to ALL `/api/admin/*` routes; (2) `require_session` dep — signed-cookie session (itsdangerous, key = `SECRET_KEY`) for profile/pantry/recipes/meals/shopping-list mutations; reads stay open inside the trusted network. Per-voter approval token flow stays as-is. Update `.env.example` with `ADMIN_TOKEN`. Document the model in `docs/SECURITY.md`.
|
||||
- [ ] R1-C: Make `/api/admin/scrape` async. Convert the endpoint to enqueue a background job (FastAPI `BackgroundTasks` for now; APScheduler later). Endpoint returns 202 + `scrape_log_id`; status polled via `/api/admin/logs/{id}`. ScraperService must open its own DB session inside the task (the request-scoped `db` is gone by then).
|
||||
|
||||
## Phase R2 — De-risk deferred work (parallel-safe, must run BEFORE further feature work per review §2.4)
|
||||
|
||||
- [ ] R2-A: Live-scrape spike. Run `LuckyCaliforniaScraper` against `https://luckysupermarkets.com` once, capture the raw HTML/PNG to `backend/tests/fixtures/lucky_ca/`, write a unit test that parses the captured fixture (no live network in CI). Document selector decisions in `.agent/context.md`. If the page can't be parsed, file the schema impact before going further.
|
||||
- [ ] R2-B: Email + approval round-trip spike. Implement minimal SendGrid sender (`app/services/email.py`), an `app/services/approval.py` that issues per-voter signed tokens (TTL, single-use), the GET confirmation page + POST submit handler (the routes already exist as stubs in `meals.py`), and a CLI script `scripts/send_test_approval.py` that creates a fake meal plan, emails one voter, and verifies the click→POST→DB write path end to end against a sandboxed inbox or `MAIL_BACKEND=console`. Goal: prove the schema (family_member, approval_token tables) survives one full round trip BEFORE building Phase 4/5/9.
|
||||
|
||||
## Phase R3 — Resume feature work (sequential, only after R1+R2 green)
|
||||
|
||||
- [ ] R3-A: Phase 4 Recipe Engine — search, tagging, never-suggest filter.
|
||||
- [ ] R3-B: Phase 9 Meal Planner generation algorithm.
|
||||
- [ ] R3-C: Phase 6 SendGrid templated emails (proposal, reminder, confirmation).
|
||||
- [ ] R3-D: Phase 8 Feedback UI.
|
||||
- [ ] R3-E: APScheduler with `--workers 1` for weekly scrape + plan generation + email send.
|
||||
- [ ] R3-F: Phase 10 image strategy.
|
||||
|
||||
## Halt conditions
|
||||
|
||||
- R2 spikes fail → stop, propose schema/spec change, await approval.
|
||||
- Verification matrix in `Review/reviewconcensus.md §6` not green → no R3 work begins.
|
||||
@@ -13,6 +13,15 @@ RECIPES_EMAIL=you@example.com
|
||||
# Lucky California
|
||||
LUCKY_CA_URL=https://luckysupermarkets.com
|
||||
|
||||
# Swiftly product API (Lucky CA backend). Token expires hourly; capture a
|
||||
# fresh one from the luckysupermarkets.com network panel on any
|
||||
# /search/api/v1 request and paste it here. The scraper aborts with a
|
||||
# FAILED ScrapeLog row whose error_message asks for a refresh on 401.
|
||||
LUCKY_STORE_ID=757
|
||||
SWIFTLY_API_BASE=https://prod.swiftlyapi.net
|
||||
SWIFTLY_CATEGORIES_URL=https://luckysupermarkets.com/categories
|
||||
SWIFTLY_BEARER_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6IjJiMzZhYjQxYTczOTJlMTRlNjM1ZmRlM2M2YWYwOWZlYmFhM2YyZDYiLCJ0eXAiOiJKV1QifQ.eyJzd2lmdGx5U2hvcHBlcklkIjoiYWQ2NjMyYjItOGY1Ny00NzgyLTgwZDAtOTJjMDZjNDI4YjY0IiwiYmFubmVySWQiOiJhM2IxMTcxNy1mNGNhLTQxOTYtYjY3MC1lNTE0MmMyMDVkZWUiLCJzZWxlY3RlZFN0b3JlIjoiNzU3IiwicHJvdmlkZXJfaWQiOiJhbm9ueW1vdXMiLCJpc3MiOiJodHRwczovL3NlY3VyZXRva2VuLmdvb2dsZS5jb20vc3dpZnRseS1sdS1wcm9kIiwiYXVkIjoic3dpZnRseS1sdS1wcm9kIiwiYXV0aF90aW1lIjoxNzc3OTg3NTcwLCJ1c2VyX2lkIjoicHJWUkh0Ujd5OFBKNEpyQTJtaGw4RW04VkpzMSIsInN1YiI6InByVlJIdFI3eThQSjRKckEybWhsOEVtOFZKczEiLCJpYXQiOjE3Nzc5ODc2MTcsImV4cCI6MTc3Nzk5MTIxNywiZmlyZWJhc2UiOnsiaWRlbnRpdGllcyI6e30sInNpZ25faW5fcHJvdmlkZXIiOiJhbm9ueW1vdXMifX0.VjlmFQVKrHy9RAqqqXpA8anaOhtYrgR9Odi77qlDcArnxb_4k_6imwNm48yuoH4umCMRKJSb3qcbWyBWSvcSjwtAKFhZLfH_QGX5VYMuyrJXtov8Hyy26jA12zLWxGxmOwmhow30ZTiElwqS0FSrA2GNPNF3l7ZLCIPnED8ijQo4XGb7NnZJ7egFdcvHwbpiZ6a4pFgY-74VBM411c9f4NQBamc3S5Xwez5T7bSA3KJ2ST6MpJ3G5_fIG-53ah1XqfsDBllqM4KxgQ0c1Nbux37kW4U3kVzNx9rCx8JZHBp8gHe_mDsbNO0bSXAzicQJuLIKGRPQtNLcZdtckMAetA
|
||||
|
||||
# AI Image Generation (optional)
|
||||
AI_IMAGE_ENABLED=false
|
||||
AI_IMAGE_PROVIDER=openai
|
||||
@@ -21,3 +30,7 @@ AI_IMAGE_API_KEY=sk-your-api-key
|
||||
# Application
|
||||
LOG_LEVEL=INFO
|
||||
SECRET_KEY=change-me-to-a-random-secret-key
|
||||
|
||||
# Auth
|
||||
ADMIN_TOKEN=change-me-to-a-random-admin-token
|
||||
SESSION_PASSWORD=change-me-to-the-family-shared-password
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["**"]
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
backend:
|
||||
name: backend (pytest + alembic)
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_USER: mealplanner
|
||||
POSTGRES_PASSWORD: password
|
||||
POSTGRES_DB: mealplanner_test
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 5s
|
||||
--health-timeout 5s
|
||||
--health-retries 10
|
||||
env:
|
||||
TEST_DATABASE_URL: postgresql://mealplanner:password@localhost:5432/mealplanner_test
|
||||
DATABASE_URL: postgresql://mealplanner:password@localhost:5432/mealplanner_test
|
||||
SECRET_KEY: ci-secret-key-not-for-prod
|
||||
ADMIN_TOKEN: ci-admin-token
|
||||
SESSION_PASSWORD: test-family-password
|
||||
EMAIL_BACKEND: console
|
||||
defaults:
|
||||
run:
|
||||
working-directory: backend
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.11
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
cache: pip
|
||||
cache-dependency-path: |
|
||||
backend/requirements.txt
|
||||
backend/requirements-dev.txt
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
pip install -r requirements-dev.txt
|
||||
|
||||
- name: Wait for Postgres
|
||||
run: |
|
||||
for i in $(seq 1 30); do
|
||||
pg_isready -h localhost -p 5432 -U mealplanner && exit 0
|
||||
sleep 1
|
||||
done
|
||||
echo "Postgres never became ready" >&2
|
||||
exit 1
|
||||
|
||||
- name: Alembic upgrade head
|
||||
run: alembic upgrade head
|
||||
|
||||
- name: Run pytest
|
||||
run: pytest -q
|
||||
|
||||
frontend:
|
||||
name: frontend (build)
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: frontend
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node 20
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "20"
|
||||
cache: npm
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build
|
||||
run: npm run build
|
||||
@@ -146,3 +146,9 @@ nginx/ssl/*.pem
|
||||
|
||||
# Node
|
||||
node_modules/
|
||||
|
||||
# R2-B email console outbox (local-only spike artifact)
|
||||
backend/var/
|
||||
|
||||
# Local verification helper — has weak test credentials, not for the repo
|
||||
.env.test
|
||||
|
||||
@@ -98,7 +98,7 @@ def upgrade() -> None:
|
||||
sa.Column('dietary_tags', postgresql.ARRAY(sa.String(length=50)), nullable=True),
|
||||
sa.Column('protein_type', sa.String(length=50), nullable=True),
|
||||
sa.Column('spice_level', sa.Integer(), nullable=True),
|
||||
sa.Column('ingredients', postgresql.JSONB(astext=True), nullable=False),
|
||||
sa.Column('ingredients', postgresql.JSONB(), nullable=False),
|
||||
sa.Column('instructions', postgresql.ARRAY(sa.Text()), nullable=False),
|
||||
sa.Column('source_url', sa.Text(), nullable=True),
|
||||
sa.Column('scraped_at', sa.DateTime(timezone=True), nullable=True),
|
||||
@@ -286,4 +286,17 @@ def upgrade() -> None:
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
pass
|
||||
op.execute(
|
||||
"""
|
||||
DO $$ DECLARE
|
||||
r RECORD;
|
||||
BEGIN
|
||||
FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename != 'alembic_version') LOOP
|
||||
EXECUTE 'DROP TABLE IF EXISTS public.' || quote_ident(r.tablename) || ' CASCADE';
|
||||
END LOOP;
|
||||
FOR r IN (SELECT t.typname FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace WHERE t.typtype = 'e' AND n.nspname = 'public') LOOP
|
||||
EXECUTE 'DROP TYPE IF EXISTS public.' || quote_ident(r.typname) || ' CASCADE';
|
||||
END LOOP;
|
||||
END $$;
|
||||
"""
|
||||
)
|
||||
@@ -95,8 +95,6 @@ def upgrade() -> None:
|
||||
('Sugar', 'sugar', 'lb', 'Pantry', 2.49),
|
||||
('Brown Rice', 'brown rice', 'lb', 'Grains', 3.49),
|
||||
('Oats', 'oats', 'lb', 'Grains', 2.99),
|
||||
('Chickpeas', 'chickpeas', 'can', 'Canned Goods', 1.49),
|
||||
('Black Beans', 'black beans', 'can', 'Canned Goods', 1.29),
|
||||
('Kidney Beans', 'kidney beans', 'can', 'Canned Goods', 1.29),
|
||||
('Corn', 'corn', 'can', 'Canned Goods', 1.49),
|
||||
('Green Beans', 'green beans', 'can', 'Canned Goods', 1.49),
|
||||
@@ -107,6 +105,7 @@ def upgrade() -> None:
|
||||
op.execute(f"""
|
||||
INSERT INTO ingredient (id, name, name_lower, unit, aisle, typical_price)
|
||||
VALUES (uuid_generate_v4(), '{name}', '{name_lower}', '{unit}', '{aisle}', {price})
|
||||
ON CONFLICT (name_lower) DO NOTHING
|
||||
""")
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Add nullable description column to grocery_item.
|
||||
|
||||
The Lucky California parser produces a long-form description per coupon
|
||||
(class ``coupon-card-short-description``); without this column it is dropped
|
||||
silently when persisting. See R2-A spike notes in
|
||||
``.agent/phase-summaries/r2a-summary.md``.
|
||||
|
||||
Revision ID: 0003
|
||||
Revises: 0002
|
||||
Create Date: 2026-05-04
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = '0003'
|
||||
down_revision: Union[str, None] = '0002'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"grocery_item",
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("grocery_item", "description")
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Add calorie_target to family_profile
|
||||
|
||||
Revision ID: 0004
|
||||
Revises: 0003
|
||||
Create Date: 2026-05-04
|
||||
|
||||
The model has carried `calorie_target` on FamilyProfile since Phase 2,
|
||||
but the initial migration omitted it. SELECT * from family_profile fails
|
||||
without this column. Adversarial review §1.6 flagged the model/schema
|
||||
drift.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "0004"
|
||||
down_revision = "0003"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"family_profile",
|
||||
sa.Column("calorie_target", sa.Integer(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("family_profile", "calorie_target")
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Add nullable indexed external_id and source columns to grocery_item.
|
||||
|
||||
The Swiftly API exposes a stable ``id`` per product (e.g. ``"46556"``).
|
||||
Persisting it as ``grocery_item.external_id`` lets the scraper UPSERT by
|
||||
``(source, external_id)`` rather than create duplicates on every run.
|
||||
``source`` distinguishes overlapping IDs across future banners (e.g. a
|
||||
second Save Mart store using the same Swiftly tenant). See R3-0 notes in
|
||||
``.agent/phase-summaries/r3-0-summary.md``.
|
||||
|
||||
Revision ID: 0005
|
||||
Revises: 0004
|
||||
Create Date: 2026-05-05
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = '0005'
|
||||
down_revision: Union[str, None] = '0004'
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"grocery_item",
|
||||
sa.Column("external_id", sa.String(length=100), nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"grocery_item",
|
||||
sa.Column(
|
||||
"source", sa.String(length=50), nullable=True,
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_grocery_item_source_external_id",
|
||||
"grocery_item",
|
||||
["source", "external_id"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("ix_grocery_item_source_external_id", table_name="grocery_item")
|
||||
op.drop_column("grocery_item", "source")
|
||||
op.drop_column("grocery_item", "external_id")
|
||||
@@ -1,20 +1,29 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
from app.database import get_db
|
||||
from app.models import ScrapeLog, EmailLog, MealPlan
|
||||
from app.services.scraper_service import ScraperService
|
||||
from app.security import require_admin
|
||||
from app.services.scraper_service import ScraperService, enqueue_scrape
|
||||
from typing import List, Optional
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
router = APIRouter()
|
||||
router = APIRouter(dependencies=[Depends(require_admin)])
|
||||
|
||||
|
||||
@router.post("/scrape")
|
||||
def trigger_scrape(source: str = "lucky_california", scrape_type: str = "weekly_ad", db: Session = Depends(get_db)):
|
||||
scraper_service = ScraperService(db)
|
||||
result = scraper_service.run_scrape(source=source, scrape_type=scrape_type)
|
||||
|
||||
return result
|
||||
@router.post("/scrape", status_code=202)
|
||||
def trigger_scrape(
|
||||
background_tasks: BackgroundTasks,
|
||||
source: str = "lucky_california",
|
||||
scrape_type: str = "weekly_ad",
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
log = enqueue_scrape(
|
||||
db,
|
||||
source=source,
|
||||
scrape_type=scrape_type,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
return {"status": "queued", "scrape_log_id": str(log.id)}
|
||||
|
||||
|
||||
@router.get("/logs")
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
"""
|
||||
Family-shared session login.
|
||||
|
||||
POST /api/auth/login — body ``{"password": "..."}`` — must match
|
||||
``settings.SESSION_PASSWORD``. On success: signs the first FamilyProfile.id
|
||||
and writes it as the ``mp_session`` cookie, returns 204.
|
||||
|
||||
POST /api/auth/logout — clears the cookie, returns 204.
|
||||
|
||||
The session is intentionally simple: a single shared family password gates
|
||||
mutations behind nginx on the trusted network. No per-user auth.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response, status
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import settings
|
||||
from app.database import get_db
|
||||
from app.models import FamilyProfile
|
||||
from app.security import SESSION_COOKIE, SESSION_MAX_AGE, issue_session
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class LoginRequest(BaseModel):
|
||||
password: str
|
||||
|
||||
|
||||
@router.post("/login")
|
||||
def login(payload: LoginRequest, db: Session = Depends(get_db)):
|
||||
expected = settings.SESSION_PASSWORD
|
||||
if not expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="Session auth not configured",
|
||||
)
|
||||
if payload.password != expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid password"
|
||||
)
|
||||
|
||||
profile = db.query(FamilyProfile).first()
|
||||
# If no profile exists yet, sign a placeholder so the cookie still
|
||||
# validates; the family-id will be re-issued the first time a profile
|
||||
# is created. This avoids login being blocked on first-run.
|
||||
family_id = str(profile.id) if profile else "bootstrap"
|
||||
cookie_value = issue_session(family_id)
|
||||
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.set_cookie(
|
||||
key=SESSION_COOKIE,
|
||||
value=cookie_value,
|
||||
max_age=SESSION_MAX_AGE,
|
||||
httponly=True,
|
||||
secure=True,
|
||||
samesite="lax",
|
||||
path="/",
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
@router.post("/logout")
|
||||
def logout():
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.delete_cookie(key=SESSION_COOKIE, path="/")
|
||||
return response
|
||||
+158
-60
@@ -1,4 +1,7 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse
|
||||
from html import escape as _html_escape
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.orm import Session, joinedload
|
||||
from app.database import get_db
|
||||
from app.models import (
|
||||
@@ -10,6 +13,8 @@ from app.schemas import (
|
||||
MealPlanResponse, MealPlanCreate,
|
||||
MealPlanItemResponse, VoteRequest, VoteResponse
|
||||
)
|
||||
from app.security import require_session
|
||||
from app.services import approval as approval_service
|
||||
from uuid import UUID
|
||||
from typing import List, Optional
|
||||
from datetime import datetime, timedelta
|
||||
@@ -17,7 +22,7 @@ from datetime import datetime, timedelta
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/planned", response_model=Optional[MealPlanResponse])
|
||||
@router.get("", response_model=Optional[MealPlanResponse])
|
||||
def get_planned_meals(db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -33,7 +38,7 @@ def get_planned_meals(db: Session = Depends(get_db)):
|
||||
return meal_plan
|
||||
|
||||
|
||||
@router.post("/", response_model=MealPlanResponse)
|
||||
@router.post("", response_model=MealPlanResponse, dependencies=[Depends(require_session)])
|
||||
def create_meal_plan(meal_plan_data: MealPlanCreate, db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -80,7 +85,7 @@ def get_meal_plan(meal_plan_id: UUID, db: Session = Depends(get_db)):
|
||||
return meal_plan
|
||||
|
||||
|
||||
@router.post("/{meal_plan_id}/lock")
|
||||
@router.post("/{meal_plan_id}/lock", dependencies=[Depends(require_session)])
|
||||
def lock_meal_plan(meal_plan_id: UUID, db: Session = Depends(get_db)):
|
||||
meal_plan = db.query(MealPlan).filter(MealPlan.id == meal_plan_id).first()
|
||||
if not meal_plan:
|
||||
@@ -91,76 +96,169 @@ def lock_meal_plan(meal_plan_id: UUID, db: Session = Depends(get_db)):
|
||||
return {"message": "Meal plan locked", "status": meal_plan.status.value}
|
||||
|
||||
|
||||
@router.get("/items/{item_id}/vote/{token}")
|
||||
def get_vote_page(item_id: UUID, token: str, db: Session = Depends(get_db)):
|
||||
approval_token = db.query(ApprovalToken).filter(ApprovalToken.token == token).first()
|
||||
if not approval_token:
|
||||
raise HTTPException(status_code=404, detail="Invalid token")
|
||||
_DAY_NAMES = {
|
||||
1: "Monday", 2: "Tuesday", 3: "Wednesday", 4: "Thursday",
|
||||
5: "Friday", 6: "Saturday", 7: "Sunday",
|
||||
}
|
||||
|
||||
if approval_token.meal_plan_item_id != item_id:
|
||||
|
||||
class VoteSubmission(BaseModel):
|
||||
"""Body for POST /vote/{item_id}?token=...
|
||||
|
||||
Spec body: {"vote": "approve" | "deny"}.
|
||||
"""
|
||||
|
||||
vote: str = Field(..., pattern="^(approve|deny)$")
|
||||
|
||||
|
||||
@router.get("/vote/{item_id}", response_class=HTMLResponse)
|
||||
def get_vote_page(
|
||||
item_id: UUID,
|
||||
token: str = Query(..., description="Per-voter signed token"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Render the per-voter approval confirmation page.
|
||||
|
||||
Verifies the signed token (no consume) and returns minimal accessible
|
||||
HTML with Approve / Deny buttons that POST to the same URL.
|
||||
"""
|
||||
payload = approval_service.verify_token(token)
|
||||
if str(payload.get("item")) != str(item_id):
|
||||
raise HTTPException(status_code=400, detail="Token not valid for this meal")
|
||||
|
||||
if approval_token.status != ApprovalTokenStatus.ACTIVE:
|
||||
raise HTTPException(status_code=400, detail="Token has already been used or expired")
|
||||
voter = (
|
||||
db.query(FamilyMember)
|
||||
.filter(FamilyMember.id == UUID(str(payload["voter"])))
|
||||
.first()
|
||||
)
|
||||
if not voter:
|
||||
raise HTTPException(status_code=404, detail="Voter not found")
|
||||
|
||||
if approval_token.expires_at < datetime.now():
|
||||
raise HTTPException(status_code=400, detail="Token has expired")
|
||||
item = (
|
||||
db.query(MealPlanItem)
|
||||
.options(joinedload(MealPlanItem.recipe))
|
||||
.filter(MealPlanItem.id == item_id)
|
||||
.first()
|
||||
)
|
||||
if not item:
|
||||
raise HTTPException(status_code=404, detail="Meal plan item not found")
|
||||
|
||||
recipe_name = item.recipe.name if item.recipe else "Unnamed meal"
|
||||
day_name = _DAY_NAMES.get(int(item.day_of_week), str(item.day_of_week))
|
||||
meal_type = item.meal_type.value if item.meal_type else ""
|
||||
|
||||
# Token is included only inside the form action (href), never in the
|
||||
# visible body. POST is performed by JS so we can submit JSON without
|
||||
# leaving the page; non-JS users still get a usable form fallback.
|
||||
safe_voter = _html_escape(voter.name)
|
||||
safe_recipe = _html_escape(recipe_name)
|
||||
safe_day = _html_escape(day_name)
|
||||
safe_meal = _html_escape(meal_type)
|
||||
action_url = f"/api/meals/vote/{item_id}?token={_html_escape(token, quote=True)}"
|
||||
|
||||
html = f"""<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>Approve meal</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<style>
|
||||
body {{ font-family: system-ui, sans-serif; max-width: 36rem; margin: 2rem auto;
|
||||
padding: 0 1rem; color: #111; background: #fff; line-height: 1.5; }}
|
||||
h1 {{ font-size: 1.4rem; }}
|
||||
.meal {{ padding: 1rem; border: 1px solid #444; border-radius: 6px; margin: 1rem 0; }}
|
||||
button {{ font-size: 1rem; padding: .6rem 1.2rem; margin-right: .5rem;
|
||||
border: 2px solid #111; border-radius: 4px; cursor: pointer; }}
|
||||
.approve {{ background: #0a6b2b; color: #fff; }}
|
||||
.deny {{ background: #b00020; color: #fff; }}
|
||||
#status {{ margin-top: 1rem; font-weight: bold; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Hi {safe_voter}, please vote on this meal</h1>
|
||||
<div class="meal">
|
||||
<div><strong>{safe_recipe}</strong></div>
|
||||
<div>{safe_day} · {safe_meal}</div>
|
||||
</div>
|
||||
<form id="voteForm" method="post" action="{action_url}">
|
||||
<button type="submit" name="vote" value="approve" class="approve" aria-label="Approve this meal">Approve</button>
|
||||
<button type="submit" name="vote" value="deny" class="deny" aria-label="Deny this meal">Deny</button>
|
||||
</form>
|
||||
<div id="status" role="status" aria-live="polite"></div>
|
||||
<script>
|
||||
document.getElementById('voteForm').addEventListener('submit', async function(e) {{
|
||||
e.preventDefault();
|
||||
var btn = e.submitter || document.activeElement;
|
||||
var vote = btn && btn.value ? btn.value : 'approve';
|
||||
var resp = await fetch(this.action, {{
|
||||
method: 'POST',
|
||||
headers: {{ 'Content-Type': 'application/json' }},
|
||||
body: JSON.stringify({{ vote: vote }})
|
||||
}});
|
||||
var data = {{}};
|
||||
try {{ data = await resp.json(); }} catch (_) {{}}
|
||||
var s = document.getElementById('status');
|
||||
if (resp.ok) {{
|
||||
s.textContent = 'Recorded: ' + (data.item_status || vote);
|
||||
}} else {{
|
||||
s.textContent = 'Error: ' + (data.detail || resp.status);
|
||||
}}
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
return HTMLResponse(content=html, status_code=200)
|
||||
|
||||
|
||||
@router.post("/vote/{item_id}")
|
||||
def submit_vote(
|
||||
item_id: UUID,
|
||||
submission: VoteSubmission,
|
||||
token: str = Query(..., description="Per-voter signed token"),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Record a per-voter vote and apply the approval rule.
|
||||
|
||||
- Single-use enforcement lives in `approval_service.consume_token`.
|
||||
- Approval rule: any deny -> item.denied; all-approve -> item.approved;
|
||||
otherwise pending (waiting on remaining voters).
|
||||
"""
|
||||
voter = approval_service.consume_token(db, token, item_id)
|
||||
|
||||
item = db.query(MealPlanItem).filter(MealPlanItem.id == item_id).first()
|
||||
if not item:
|
||||
raise HTTPException(status_code=404, detail="Meal plan item not found")
|
||||
|
||||
return {
|
||||
"item_id": str(item_id),
|
||||
"family_member_id": str(approval_token.family_member_id),
|
||||
"meal_plan_item": item
|
||||
}
|
||||
|
||||
|
||||
@router.post("/items/{item_id}/vote/{token}", response_model=VoteResponse)
|
||||
def submit_vote(item_id: UUID, token: str, vote_data: VoteRequest, db: Session = Depends(get_db)):
|
||||
approval_token = db.query(ApprovalToken).filter(ApprovalToken.token == token).first()
|
||||
if not approval_token:
|
||||
raise HTTPException(status_code=404, detail="Invalid token")
|
||||
|
||||
if approval_token.meal_plan_item_id != item_id:
|
||||
raise HTTPException(status_code=400, detail="Token not valid for this meal")
|
||||
|
||||
if approval_token.status != ApprovalTokenStatus.ACTIVE:
|
||||
raise HTTPException(status_code=400, detail="Token has already been used or expired")
|
||||
|
||||
if approval_token.expires_at < datetime.now():
|
||||
approval_token.status = ApprovalTokenStatus.EXPIRED
|
||||
db.commit()
|
||||
raise HTTPException(status_code=400, detail="Token has expired")
|
||||
|
||||
existing_vote = db.query(MealPlanVote).filter(
|
||||
MealPlanVote.meal_plan_item_id == item_id,
|
||||
MealPlanVote.family_member_id == approval_token.family_member_id
|
||||
).first()
|
||||
|
||||
if existing_vote:
|
||||
raise HTTPException(status_code=400, detail="You have already voted on this meal")
|
||||
|
||||
vote = MealPlanVote(
|
||||
vote_bool = submission.vote == "approve"
|
||||
db.add(MealPlanVote(
|
||||
meal_plan_item_id=item_id,
|
||||
family_member_id=approval_token.family_member_id,
|
||||
vote=vote_data.vote
|
||||
)
|
||||
db.add(vote)
|
||||
family_member_id=voter.id,
|
||||
vote=vote_bool,
|
||||
))
|
||||
db.flush()
|
||||
|
||||
approval_token.status = ApprovalTokenStatus.USED
|
||||
approval_token.used_at = datetime.now()
|
||||
# Approval rule: count electorate (all family members on this profile)
|
||||
# vs votes recorded so far.
|
||||
profile_id = item.meal_plan.family_profile_id
|
||||
electorate_ids = {
|
||||
m.id for m in db.query(FamilyMember)
|
||||
.filter(FamilyMember.family_profile_id == profile_id)
|
||||
.all()
|
||||
}
|
||||
votes = db.query(MealPlanVote).filter(
|
||||
MealPlanVote.meal_plan_item_id == item_id,
|
||||
).all()
|
||||
|
||||
item = db.query(MealPlanItem).filter(MealPlanItem.id == item_id).first()
|
||||
if not vote_data.vote and vote_data.denial_reason:
|
||||
if any(v.vote is False for v in votes):
|
||||
item.approval_status = MealPlanItemStatus.DENIED
|
||||
item.denial_reason = vote_data.denial_reason
|
||||
item.denial_details = vote_data.denial_details
|
||||
elif electorate_ids and {v.family_member_id for v in votes} >= electorate_ids:
|
||||
item.approval_status = MealPlanItemStatus.APPROVED
|
||||
else:
|
||||
item.approval_status = MealPlanItemStatus.PENDING
|
||||
|
||||
db.commit()
|
||||
db.refresh(vote)
|
||||
return vote
|
||||
return {"status": "recorded", "item_status": item.approval_status.value}
|
||||
|
||||
|
||||
@router.get("/items/{item_id}", response_model=MealPlanItemResponse)
|
||||
@@ -173,7 +271,7 @@ def get_meal_item(item_id: UUID, db: Session = Depends(get_db)):
|
||||
return item
|
||||
|
||||
|
||||
@router.post("/items/{item_id}/swap")
|
||||
@router.post("/items/{item_id}/swap", dependencies=[Depends(require_session)])
|
||||
def swap_meal_item(item_id: UUID, new_recipe_id: UUID, db: Session = Depends(get_db)):
|
||||
item = db.query(MealPlanItem).filter(MealPlanItem.id == item_id).first()
|
||||
if not item:
|
||||
|
||||
@@ -3,13 +3,14 @@ from sqlalchemy.orm import Session
|
||||
from app.database import get_db
|
||||
from app.models import HomePantry, Ingredient, FamilyProfile
|
||||
from app.schemas import HomePantryResponse, HomePantryCreate
|
||||
from app.security import require_session
|
||||
from uuid import UUID
|
||||
from typing import List
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/", response_model=List[HomePantryResponse])
|
||||
@router.get("", response_model=List[HomePantryResponse])
|
||||
def get_pantry_items(db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -21,7 +22,7 @@ def get_pantry_items(db: Session = Depends(get_db)):
|
||||
return items
|
||||
|
||||
|
||||
@router.post("/", response_model=HomePantryResponse)
|
||||
@router.post("", response_model=HomePantryResponse, dependencies=[Depends(require_session)])
|
||||
def add_pantry_item(item: HomePantryCreate, db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -54,7 +55,7 @@ def add_pantry_item(item: HomePantryCreate, db: Session = Depends(get_db)):
|
||||
return db_item
|
||||
|
||||
|
||||
@router.delete("/{item_id}")
|
||||
@router.delete("/{item_id}", dependencies=[Depends(require_session)])
|
||||
def remove_pantry_item(item_id: UUID, db: Session = Depends(get_db)):
|
||||
item = db.query(HomePantry).filter(HomePantry.id == item_id).first()
|
||||
if not item:
|
||||
@@ -65,7 +66,7 @@ def remove_pantry_item(item_id: UUID, db: Session = Depends(get_db)):
|
||||
return {"message": "Pantry item removed"}
|
||||
|
||||
|
||||
@router.put("/{item_id}", response_model=HomePantryResponse)
|
||||
@router.put("/{item_id}", response_model=HomePantryResponse, dependencies=[Depends(require_session)])
|
||||
def update_pantry_item(item_id: UUID, update: HomePantryCreate, db: Session = Depends(get_db)):
|
||||
item = db.query(HomePantry).filter(HomePantry.id == item_id).first()
|
||||
if not item:
|
||||
|
||||
@@ -6,13 +6,14 @@ from app.schemas import (
|
||||
FamilyProfileResponse, FamilyProfileUpdate, FamilyProfileCreate,
|
||||
FamilyMemberResponse, FamilyMemberCreate
|
||||
)
|
||||
from app.security import require_session
|
||||
from uuid import UUID
|
||||
from typing import List
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/", response_model=FamilyProfileResponse)
|
||||
@router.get("", response_model=FamilyProfileResponse)
|
||||
def get_profile(db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -20,7 +21,7 @@ def get_profile(db: Session = Depends(get_db)):
|
||||
return profile
|
||||
|
||||
|
||||
@router.put("/", response_model=FamilyProfileResponse)
|
||||
@router.put("", response_model=FamilyProfileResponse, dependencies=[Depends(require_session)])
|
||||
def update_profile(update: FamilyProfileUpdate, db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -43,7 +44,7 @@ def get_members(db: Session = Depends(get_db)):
|
||||
return profile.members
|
||||
|
||||
|
||||
@router.post("/members", response_model=FamilyMemberResponse)
|
||||
@router.post("/members", response_model=FamilyMemberResponse, dependencies=[Depends(require_session)])
|
||||
def add_member(member: FamilyMemberCreate, db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -69,7 +70,7 @@ def add_member(member: FamilyMemberCreate, db: Session = Depends(get_db)):
|
||||
return db_member
|
||||
|
||||
|
||||
@router.delete("/members/{member_id}")
|
||||
@router.delete("/members/{member_id}", dependencies=[Depends(require_session)])
|
||||
def delete_member(member_id: UUID, db: Session = Depends(get_db)):
|
||||
member = db.query(FamilyMember).filter(FamilyMember.id == member_id).first()
|
||||
if not member:
|
||||
|
||||
+35
-34
@@ -3,13 +3,14 @@ from sqlalchemy.orm import Session, joinedload
|
||||
from app.database import get_db
|
||||
from app.models import Recipe, FamilyProfile
|
||||
from app.schemas import RecipeResponse, RecipeCreate, IngredientCreate, IngredientResponse
|
||||
from app.security import require_session
|
||||
from uuid import UUID
|
||||
from typing import List, Optional
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/", response_model=List[RecipeResponse])
|
||||
@router.get("", response_model=List[RecipeResponse])
|
||||
def get_recipes(
|
||||
skip: int = 0,
|
||||
limit: int = 50,
|
||||
@@ -31,6 +32,37 @@ def get_recipes(
|
||||
return recipes
|
||||
|
||||
|
||||
@router.get("/ingredients", response_model=List[IngredientResponse])
|
||||
def list_ingredients(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)):
|
||||
from app.models import Ingredient
|
||||
ingredients = db.query(Ingredient).offset(skip).limit(limit).all()
|
||||
return ingredients
|
||||
|
||||
|
||||
@router.post("/ingredients", response_model=IngredientResponse, dependencies=[Depends(require_session)])
|
||||
def create_ingredient(ingredient: IngredientCreate, db: Session = Depends(get_db)):
|
||||
from app.models import Ingredient
|
||||
|
||||
existing = db.query(Ingredient).filter(Ingredient.name_lower == ingredient.name_lower).first()
|
||||
if existing:
|
||||
raise HTTPException(status_code=400, detail="Ingredient with this name already exists")
|
||||
|
||||
db_ingredient = Ingredient(
|
||||
name=ingredient.name,
|
||||
name_lower=ingredient.name_lower,
|
||||
plural_name=ingredient.plural_name,
|
||||
aisle=ingredient.aisle,
|
||||
typical_price=ingredient.typical_price,
|
||||
unit=ingredient.unit,
|
||||
season_months=ingredient.season_months
|
||||
)
|
||||
|
||||
db.add(db_ingredient)
|
||||
db.commit()
|
||||
db.refresh(db_ingredient)
|
||||
return db_ingredient
|
||||
|
||||
|
||||
@router.get("/{recipe_id}", response_model=RecipeResponse)
|
||||
def get_recipe(recipe_id: UUID, db: Session = Depends(get_db)):
|
||||
recipe = db.query(Recipe).filter(Recipe.id == recipe_id).first()
|
||||
@@ -39,7 +71,7 @@ def get_recipe(recipe_id: UUID, db: Session = Depends(get_db)):
|
||||
return recipe
|
||||
|
||||
|
||||
@router.post("/", response_model=RecipeResponse)
|
||||
@router.post("", response_model=RecipeResponse, dependencies=[Depends(require_session)])
|
||||
def create_recipe(recipe: RecipeCreate, db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
|
||||
@@ -69,7 +101,7 @@ def create_recipe(recipe: RecipeCreate, db: Session = Depends(get_db)):
|
||||
return db_recipe
|
||||
|
||||
|
||||
@router.delete("/{recipe_id}")
|
||||
@router.delete("/{recipe_id}", dependencies=[Depends(require_session)])
|
||||
def delete_recipe(recipe_id: UUID, db: Session = Depends(get_db)):
|
||||
recipe = db.query(Recipe).filter(Recipe.id == recipe_id).first()
|
||||
if not recipe:
|
||||
@@ -78,34 +110,3 @@ def delete_recipe(recipe_id: UUID, db: Session = Depends(get_db)):
|
||||
db.delete(recipe)
|
||||
db.commit()
|
||||
return {"message": "Recipe deleted"}
|
||||
|
||||
|
||||
@router.get("/ingredients/list", response_model=List[IngredientResponse])
|
||||
def list_ingredients(skip: int = 0, limit: int = 100, db: Session = Depends(get_db)):
|
||||
from app.models import Ingredient
|
||||
ingredients = db.query(Ingredient).offset(skip).limit(limit).all()
|
||||
return ingredients
|
||||
|
||||
|
||||
@router.post("/ingredients", response_model=IngredientResponse)
|
||||
def create_ingredient(ingredient: IngredientCreate, db: Session = Depends(get_db)):
|
||||
from app.models import Ingredient
|
||||
|
||||
existing = db.query(Ingredient).filter(Ingredient.name_lower == ingredient.name_lower).first()
|
||||
if existing:
|
||||
raise HTTPException(status_code=400, detail="Ingredient with this name already exists")
|
||||
|
||||
db_ingredient = Ingredient(
|
||||
name=ingredient.name,
|
||||
name_lower=ingredient.name_lower,
|
||||
plural_name=ingredient.plural_name,
|
||||
aisle=ingredient.aisle,
|
||||
typical_price=ingredient.typical_price,
|
||||
unit=ingredient.unit,
|
||||
season_months=ingredient.season_months
|
||||
)
|
||||
|
||||
db.add(db_ingredient)
|
||||
db.commit()
|
||||
db.refresh(db_ingredient)
|
||||
return db_ingredient
|
||||
@@ -14,7 +14,7 @@ from collections import defaultdict
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/", response_model=ShoppingListResponse)
|
||||
@router.get("", response_model=ShoppingListResponse)
|
||||
def get_shopping_list(db: Session = Depends(get_db)):
|
||||
profile = db.query(FamilyProfile).first()
|
||||
if not profile:
|
||||
@@ -56,7 +56,7 @@ def get_shopping_list(db: Session = Depends(get_db)):
|
||||
|
||||
if all_ingredient_ids:
|
||||
ingredients = db.query(Ingredient).filter(
|
||||
Ingredient.id.in_ all_ingredient_ids
|
||||
Ingredient.id.in_(all_ingredient_ids)
|
||||
).all()
|
||||
ingredient_map = {i.id: i for i in ingredients}
|
||||
|
||||
|
||||
+20
-1
@@ -1,17 +1,30 @@
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import model_validator
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
DATABASE_URL: str
|
||||
# Required — fail fast at import time if unset.
|
||||
DATABASE_URL: str = ""
|
||||
|
||||
SENDGRID_API_KEY: Optional[str] = None
|
||||
EMAIL_BACKEND: str = "console"
|
||||
LUCKY_CA_URL: str = "https://www.luckyncal.com"
|
||||
# R3-0: Swiftly product API (replaces Playwright path).
|
||||
SWIFTLY_BEARER_TOKEN: str = ""
|
||||
LUCKY_STORE_ID: str = "757"
|
||||
SWIFTLY_API_BASE: str = "https://prod.swiftlyapi.net"
|
||||
SWIFTLY_CATEGORIES_URL: str = "https://luckysupermarkets.com/categories"
|
||||
AI_IMAGE_ENABLED: bool = False
|
||||
AI_IMAGE_PROVIDER: Optional[str] = None
|
||||
AI_IMAGE_API_KEY: Optional[str] = None
|
||||
LOG_LEVEL: str = "INFO"
|
||||
SECRET_KEY: str = "dev-secret-key"
|
||||
|
||||
# Auth (R1-B+D)
|
||||
ADMIN_TOKEN: str = ""
|
||||
SESSION_PASSWORD: str = ""
|
||||
|
||||
FAMILY_EMAIL_1: Optional[str] = None
|
||||
FAMILY_EMAIL_2: Optional[str] = None
|
||||
RECIPES_EMAIL: Optional[str] = None
|
||||
@@ -19,5 +32,11 @@ class Settings(BaseSettings):
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_database_url(self) -> "Settings":
|
||||
if not self.DATABASE_URL:
|
||||
raise RuntimeError("DATABASE_URL is required")
|
||||
return self
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
||||
+2
-1
@@ -29,8 +29,9 @@ def health_check_db(db: Session = Depends(get_db)):
|
||||
return {"status": "error", "database": "disconnected", "error": str(e)}
|
||||
|
||||
|
||||
from app.api import profile, recipes, meals, shopping_list, pantry, admin
|
||||
from app.api import profile, recipes, meals, shopping_list, pantry, admin, auth
|
||||
|
||||
app.include_router(auth.router, prefix="/api/auth", tags=["auth"])
|
||||
app.include_router(profile.router, prefix="/api/profile", tags=["profile"])
|
||||
app.include_router(recipes.router, prefix="/api/recipes", tags=["recipes"])
|
||||
app.include_router(meals.router, prefix="/api/meals", tags=["meals"])
|
||||
|
||||
@@ -114,7 +114,7 @@ class FamilyMember(Base):
|
||||
family_profile_id = Column(UUID(as_uuid=True), ForeignKey("family_profile.id", ondelete="CASCADE"))
|
||||
name = Column(String(100), nullable=False)
|
||||
email = Column(String(300))
|
||||
role = Column(SQLEnum(FamilyMemberRole, name="family_member_role_enum", create_type=False), nullable=False)
|
||||
role = Column(SQLEnum(FamilyMemberRole, name="family_member_role_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), nullable=False)
|
||||
likes_mushrooms = Column(Boolean, default=False)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
@@ -185,7 +185,7 @@ class MealPlan(Base):
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
family_profile_id = Column(UUID(as_uuid=True), ForeignKey("family_profile.id"))
|
||||
week_start_date = Column(Date, nullable=False)
|
||||
status = Column(SQLEnum(MealPlanStatus, name="meal_plan_status_enum", create_type=False), nullable=False, default=MealPlanStatus.DRAFT)
|
||||
status = Column(SQLEnum(MealPlanStatus, name="meal_plan_status_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), nullable=False, default=MealPlanStatus.DRAFT)
|
||||
approval_deadline = Column(DateTime(timezone=True))
|
||||
total_estimated_cost = Column(Numeric(10, 2))
|
||||
notes = Column(Text)
|
||||
@@ -198,7 +198,6 @@ class MealPlan(Base):
|
||||
|
||||
family_profile = relationship("FamilyProfile", back_populates="meal_plans")
|
||||
items = relationship("MealPlanItem", back_populates="meal_plan", cascade="all, delete-orphan")
|
||||
votes = relationship("MealPlanVote", back_populates="meal_plan", cascade="all, delete-orphan")
|
||||
|
||||
|
||||
class MealPlanItem(Base):
|
||||
@@ -208,9 +207,9 @@ class MealPlanItem(Base):
|
||||
meal_plan_id = Column(UUID(as_uuid=True), ForeignKey("meal_plan.id", ondelete="CASCADE"))
|
||||
recipe_id = Column(UUID(as_uuid=True), ForeignKey("recipe.id"))
|
||||
day_of_week = Column(Integer, nullable=False)
|
||||
meal_type = Column(SQLEnum(MealType, name="meal_type_enum", create_type=False), nullable=False)
|
||||
approval_status = Column(SQLEnum(MealPlanItemStatus, name="meal_plan_item_status_enum", create_type=False), default=MealPlanItemStatus.PENDING)
|
||||
denial_reason = Column(SQLEnum(DenialReason, name="denial_reason_enum", create_type=False))
|
||||
meal_type = Column(SQLEnum(MealType, name="meal_type_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), nullable=False)
|
||||
approval_status = Column(SQLEnum(MealPlanItemStatus, name="meal_plan_item_status_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), default=MealPlanItemStatus.PENDING)
|
||||
denial_reason = Column(SQLEnum(DenialReason, name="denial_reason_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]))
|
||||
denial_details = Column(Text)
|
||||
estimated_cost = Column(Numeric(10, 2))
|
||||
used_pantry_items = Column(ARRAY(UUID(as_uuid=True)))
|
||||
@@ -252,7 +251,7 @@ class ApprovalToken(Base):
|
||||
meal_plan_item_id = Column(UUID(as_uuid=True), ForeignKey("meal_plan_item.id", ondelete="CASCADE"))
|
||||
family_member_id = Column(UUID(as_uuid=True), ForeignKey("family_member.id", ondelete="CASCADE"))
|
||||
token = Column(String(64), nullable=False, unique=True)
|
||||
status = Column(SQLEnum(ApprovalTokenStatus, name="approval_token_status_enum", create_type=False), default=ApprovalTokenStatus.ACTIVE)
|
||||
status = Column(SQLEnum(ApprovalTokenStatus, name="approval_token_status_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), default=ApprovalTokenStatus.ACTIVE)
|
||||
expires_at = Column(DateTime(timezone=True), nullable=False)
|
||||
used_at = Column(DateTime(timezone=True))
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -294,7 +293,7 @@ class Feedback(Base):
|
||||
meal_plan_item_id = Column(UUID(as_uuid=True), ForeignKey("meal_plan_item.id", ondelete="CASCADE"))
|
||||
rating = Column(Integer)
|
||||
never_suggest = Column(Boolean, default=False)
|
||||
denial_reason = Column(SQLEnum(DenialReason, name="denial_reason_enum", create_type=False))
|
||||
denial_reason = Column(SQLEnum(DenialReason, name="denial_reason_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]))
|
||||
feedback_text = Column(Text)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
@@ -314,7 +313,7 @@ class NeverSuggest(Base):
|
||||
family_profile_id = Column(UUID(as_uuid=True), ForeignKey("family_profile.id", ondelete="CASCADE"))
|
||||
ingredient_id = Column(UUID(as_uuid=True), ForeignKey("ingredient.id", ondelete="CASCADE"))
|
||||
recipe_id = Column(UUID(as_uuid=True), ForeignKey("recipe.id", ondelete="CASCADE"))
|
||||
reason = Column(SQLEnum(NeverSuggestReason, name="never_suggest_reason_enum", create_type=False))
|
||||
reason = Column(SQLEnum(NeverSuggestReason, name="never_suggest_reason_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]))
|
||||
notes = Column(Text)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
@@ -335,12 +334,16 @@ class GroceryItem(Base):
|
||||
aisle = Column(String(100))
|
||||
image_url = Column(Text)
|
||||
product_url = Column(Text)
|
||||
description = Column(Text, nullable=True)
|
||||
is_on_sale = Column(Boolean, default=False)
|
||||
sale_start_date = Column(Date)
|
||||
sale_end_date = Column(Date)
|
||||
in_season = Column(Boolean, default=False)
|
||||
scraped_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
scraped_url = Column(Text)
|
||||
# R3-0: Swiftly API idempotency key. (source, external_id) → upsert.
|
||||
external_id = Column(String(100), nullable=True, index=True)
|
||||
source = Column(String(50), nullable=True)
|
||||
|
||||
ingredient = relationship("Ingredient", back_populates="grocery_item_links")
|
||||
|
||||
@@ -351,7 +354,7 @@ class ScrapeLog(Base):
|
||||
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
source = Column(String(50), nullable=False)
|
||||
scrape_type = Column(String(50), nullable=False)
|
||||
status = Column(SQLEnum(ScrapeStatus, name="scrape_status_enum", create_type=False), nullable=False)
|
||||
status = Column(SQLEnum(ScrapeStatus, name="scrape_status_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), nullable=False)
|
||||
items_scraped = Column(Integer, default=0)
|
||||
error_message = Column(Text)
|
||||
started_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
@@ -369,7 +372,7 @@ class EmailLog(Base):
|
||||
meal_plan_id = Column(UUID(as_uuid=True), ForeignKey("meal_plan.id"))
|
||||
meal_plan_item_id = Column(UUID(as_uuid=True), ForeignKey("meal_plan_item.id"))
|
||||
sendgrid_message_id = Column(String(100))
|
||||
status = Column(SQLEnum(EmailStatus, name="email_status_enum", create_type=False), nullable=False)
|
||||
status = Column(SQLEnum(EmailStatus, name="email_status_enum", create_type=False, values_callable=lambda obj: [e.value for e in obj]), nullable=False)
|
||||
error_message = Column(Text)
|
||||
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
||||
delivered_at = Column(DateTime(timezone=True))
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from .base import BaseScraper
|
||||
from .lucky_ca_scraper import LuckyCaliforniaScraper
|
||||
from .lucky_ca_scraper import LuckyCaliforniaScraper, SwiftlyAuthError
|
||||
|
||||
__all__ = ["BaseScraper", "LuckyCaliforniaScraper"]
|
||||
__all__ = ["BaseScraper", "LuckyCaliforniaScraper", "SwiftlyAuthError"]
|
||||
@@ -1,180 +1,325 @@
|
||||
"""Lucky California / Swiftly product API client.
|
||||
|
||||
Replaces the prior Playwright + HTML coupon-card path (R2-A) with the
|
||||
underlying JSON API the website itself calls. The API exposes the full
|
||||
inventory per category — no rendering, no Chromium, and far more items
|
||||
than the visible coupon strip (256 in `Product/meat_seafood` vs the 11
|
||||
the HTML parser saw).
|
||||
|
||||
Two endpoints (no public docs; reverse-engineered from the network panel
|
||||
on luckysupermarkets.com — see ``.agent/context.md`` "Swiftly API"):
|
||||
|
||||
GET https://luckysupermarkets.com/categories
|
||||
→ HTML page; anchors with ``class="swiftlyCouponCategory"`` carry
|
||||
``href="/categories/Product%2F<slug>"``.
|
||||
|
||||
GET https://prod.swiftlyapi.net/search/api/v1/products/categories
|
||||
?cat=<slug>&store=<store_id>&limit=10000
|
||||
Authorization: Bearer <SWIFTLY_BEARER_TOKEN>
|
||||
→ ``{"products": {"info": {...}, "items": [...], "facets": [...]}}``
|
||||
|
||||
The bearer token expires roughly hourly. On 401 we raise
|
||||
``SwiftlyAuthError`` so the background runner records the error_message
|
||||
that asks the admin to refresh ``SWIFTLY_BEARER_TOKEN`` and retry.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Dict, Any, List, Optional
|
||||
import time
|
||||
import urllib.parse
|
||||
from datetime import datetime
|
||||
from urllib.parse import urljoin
|
||||
from decimal import Decimal, InvalidOperation
|
||||
from typing import Any, Dict, Iterator, List, Optional, Tuple
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from .base import SeleniumScraper
|
||||
|
||||
from app.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LuckyCaliforniaScraper(SeleniumScraper):
|
||||
def __init__(self, base_url: str = "https://luckysupermarkets.com"):
|
||||
super().__init__(base_url=base_url, rate_limit_seconds=3.0)
|
||||
self.ingredients_cache = {}
|
||||
class SwiftlyAuthError(Exception):
|
||||
"""Raised when the Swiftly API returns 401.
|
||||
|
||||
The exception message is surfaced verbatim to the ScrapeLog row by
|
||||
``_run_scrape_in_background``; keep it actionable.
|
||||
"""
|
||||
|
||||
|
||||
_USER_AGENT = (
|
||||
"MealPlannerBot/1.0 (+https://mealplanner.local; contact peter@research.bike)"
|
||||
)
|
||||
|
||||
_AUTH_ERROR_MESSAGE = (
|
||||
"SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user "
|
||||
"(capture from luckysupermarkets.com network tab on a /search/api/v1 request)"
|
||||
)
|
||||
|
||||
|
||||
class LuckyCaliforniaScraper:
|
||||
"""Swiftly product-API client for the Lucky California banner.
|
||||
|
||||
The class name is preserved (``LuckyCaliforniaScraper``) so existing
|
||||
import sites in ``app.scraper.__init__`` and
|
||||
``ScraperService._do_scrape`` keep working. Internally it is no
|
||||
longer a ``BaseScraper`` subclass: that class wires Playwright +
|
||||
``_get`` retry-with-swallow, neither of which we want here. The
|
||||
client uses two ``requests.Session`` objects so the bearer header
|
||||
is scoped strictly to the API host (the public categories page
|
||||
is unauthenticated).
|
||||
"""
|
||||
|
||||
SOURCE = "lucky_california"
|
||||
SLUG_HREF_RE = re.compile(
|
||||
r'<a\b(?=[^>]*\bclass="swiftlyCouponCategory")(?=[^>]*\bhref="([^"]+)")[^>]*>',
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
bearer_token: Optional[str] = None,
|
||||
store_id: Optional[str] = None,
|
||||
api_base: Optional[str] = None,
|
||||
categories_url: Optional[str] = None,
|
||||
rate_limit_seconds: float = 0.75,
|
||||
timeout: int = 60,
|
||||
) -> None:
|
||||
self.bearer_token = bearer_token or settings.SWIFTLY_BEARER_TOKEN
|
||||
self.store_id = store_id or settings.LUCKY_STORE_ID
|
||||
self.api_base = (api_base or settings.SWIFTLY_API_BASE).rstrip("/")
|
||||
self.categories_url = categories_url or settings.SWIFTLY_CATEGORIES_URL
|
||||
self.rate_limit_seconds = rate_limit_seconds
|
||||
self.timeout = timeout
|
||||
self._last_request = 0.0
|
||||
|
||||
self.public_session = requests.Session()
|
||||
self.public_session.headers.update(
|
||||
{
|
||||
"User-Agent": _USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.5",
|
||||
}
|
||||
)
|
||||
|
||||
self.api_session = requests.Session()
|
||||
self.api_session.headers.update(
|
||||
{
|
||||
"User-Agent": _USER_AGENT,
|
||||
"Accept": "application/json",
|
||||
}
|
||||
)
|
||||
# base_url retained for code paths that still introspect it.
|
||||
self.base_url = "https://luckysupermarkets.com"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle (no-op; preserves cleanup() contract from BaseScraper).
|
||||
# ------------------------------------------------------------------
|
||||
def cleanup(self) -> None:
|
||||
try:
|
||||
self.public_session.close()
|
||||
finally:
|
||||
self.api_session.close()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public scrape entrypoints
|
||||
# ------------------------------------------------------------------
|
||||
def scrape(self) -> Dict[str, Any]:
|
||||
logger.info("Starting Lucky California scrape")
|
||||
result = {
|
||||
"source": "lucky_california",
|
||||
"""Synchronous top-level scrape.
|
||||
|
||||
Kept for the legacy ``ScraperService.run_scrape`` path used by
|
||||
tests. Walks every category and returns ``{"items": [...], ...}``
|
||||
with mapped product dicts ready for ``_save_grocery_item``.
|
||||
"""
|
||||
started = datetime.now().isoformat()
|
||||
items: List[Dict[str, Any]] = list(self.fetch_all())
|
||||
return {
|
||||
"source": self.SOURCE,
|
||||
"scrape_type": "weekly_ad",
|
||||
"started_at": datetime.now().isoformat(),
|
||||
"items_scraped": 0,
|
||||
"items": []
|
||||
"started_at": started,
|
||||
"completed_at": datetime.now().isoformat(),
|
||||
"items_scraped": len(items),
|
||||
"items": items,
|
||||
"status": "success",
|
||||
}
|
||||
|
||||
def fetch_all(self) -> Iterator[Dict[str, Any]]:
|
||||
"""Yield mapped product dicts across every discovered category."""
|
||||
slugs = self.discover_categories()
|
||||
logger.info("Swiftly: discovered %d categories", len(slugs))
|
||||
for slug in slugs:
|
||||
try:
|
||||
featured_items = self.scrape_featured_coupons()
|
||||
result["items"].extend(featured_items)
|
||||
result["items_scraped"] = len(featured_items)
|
||||
|
||||
result["completed_at"] = datetime.now().isoformat()
|
||||
result["status"] = "success"
|
||||
logger.info(f"Lucky California scrape complete: {result['items_scraped']} items")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Lucky California scrape failed: {e}")
|
||||
result["status"] = "failed"
|
||||
result["error_message"] = str(e)
|
||||
result["completed_at"] = datetime.now().isoformat()
|
||||
|
||||
return result
|
||||
|
||||
def scrape_featured_coupons(self) -> List[Dict[str, Any]]:
|
||||
url = f"{self.base_url}/coupons/Coupon%2Flu-featured-in-ad"
|
||||
logger.info(f"Scraping featured coupons from {url}")
|
||||
|
||||
page = self.get_browser_page(url)
|
||||
content = page.content()
|
||||
page.close()
|
||||
|
||||
soup = BeautifulSoup(content, "html.parser")
|
||||
items = []
|
||||
|
||||
coupon_items = soup.find_all("div", class_=re.compile(r"coupon|item|product", re.I))
|
||||
if not coupon_items:
|
||||
headline = soup.find("h1")
|
||||
if headline:
|
||||
logger.info(f"Page loaded, headline: {headline.get_text().strip()}")
|
||||
|
||||
titles = soup.find_all(["h2", "h3", "a"], string=re.compile(r"\$[\d\.]+"))
|
||||
for title_elem in titles[:20]:
|
||||
try:
|
||||
item = self._parse_coupon_item(title_elem)
|
||||
if item:
|
||||
items.append(item)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to parse item: {e}")
|
||||
products = self.fetch_category(slug)
|
||||
except SwiftlyAuthError:
|
||||
# Hard fail — token must be refreshed before any further work.
|
||||
raise
|
||||
except requests.RequestException as exc:
|
||||
logger.warning("Swiftly: skipping %s after error: %s", slug, exc)
|
||||
continue
|
||||
aisle = self._aisle_from_slug(slug)
|
||||
for product in products:
|
||||
mapped = self.map_product(product, aisle=aisle, source_slug=slug)
|
||||
if mapped is not None:
|
||||
yield mapped
|
||||
|
||||
logger.info(f"Found {len(items)} coupon items")
|
||||
# ------------------------------------------------------------------
|
||||
# Category discovery
|
||||
# ------------------------------------------------------------------
|
||||
def discover_categories(self) -> List[str]:
|
||||
"""Fetch the categories page and return the list of API slugs.
|
||||
|
||||
The slugs look like ``Product/meat_seafood``. Order is preserved
|
||||
from the HTML (which is the order shown to the user).
|
||||
"""
|
||||
self._rate_limit()
|
||||
resp = self.public_session.get(self.categories_url, timeout=self.timeout)
|
||||
resp.raise_for_status()
|
||||
return self.parse_categories_html(resp.text)
|
||||
|
||||
@classmethod
|
||||
def parse_categories_html(cls, html: str) -> List[str]:
|
||||
"""Pure parser used by tests against a saved fixture."""
|
||||
slugs: List[str] = []
|
||||
seen: set[str] = set()
|
||||
for href in cls.SLUG_HREF_RE.findall(html):
|
||||
m = re.match(r"^/categories/(.+)$", href)
|
||||
if not m:
|
||||
continue
|
||||
slug = urllib.parse.unquote(m.group(1))
|
||||
if slug not in seen:
|
||||
seen.add(slug)
|
||||
slugs.append(slug)
|
||||
return slugs
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Per-category fetch
|
||||
# ------------------------------------------------------------------
|
||||
def fetch_category(self, slug: str) -> List[Dict[str, Any]]:
|
||||
"""Fetch every product in a category. Raises ``SwiftlyAuthError`` on 401."""
|
||||
if not self.bearer_token:
|
||||
raise SwiftlyAuthError(_AUTH_ERROR_MESSAGE)
|
||||
|
||||
self._rate_limit()
|
||||
url = f"{self.api_base}/search/api/v1/products/categories"
|
||||
params = {"cat": slug, "store": self.store_id, "limit": 10000}
|
||||
headers = {"Authorization": f"Bearer {self.bearer_token}"}
|
||||
resp = self.api_session.get(
|
||||
url, params=params, headers=headers, timeout=self.timeout
|
||||
)
|
||||
if resp.status_code == 401:
|
||||
raise SwiftlyAuthError(_AUTH_ERROR_MESSAGE)
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
return self.parse_category_response(payload)
|
||||
|
||||
@staticmethod
|
||||
def parse_category_response(payload: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Pure parser used by tests against a saved fixture."""
|
||||
if not isinstance(payload, dict):
|
||||
return []
|
||||
products = payload.get("products") or {}
|
||||
items = products.get("items")
|
||||
if not isinstance(items, list):
|
||||
return []
|
||||
return items
|
||||
|
||||
def _parse_coupon_item(self, element) -> Optional[Dict[str, Any]]:
|
||||
text = element.get_text().strip()
|
||||
# ------------------------------------------------------------------
|
||||
# Field mapping
|
||||
# ------------------------------------------------------------------
|
||||
@classmethod
|
||||
def map_product(
|
||||
cls,
|
||||
product: Dict[str, Any],
|
||||
*,
|
||||
aisle: Optional[str] = None,
|
||||
source_slug: Optional[str] = None,
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""Convert one Swiftly product dict to a grocery_item-ready dict.
|
||||
|
||||
price_match = re.search(r'\$[\d,]+\.?\d*', text)
|
||||
if not price_match:
|
||||
Returns ``None`` for products with no parseable price or no name —
|
||||
those are usually placeholder/unavailable rows.
|
||||
"""
|
||||
external_id = product.get("id")
|
||||
name = (product.get("name") or "").strip()
|
||||
if not name:
|
||||
return None
|
||||
|
||||
price_str = price_match.group().replace("$", "").replace(",", "")
|
||||
try:
|
||||
price = float(price_str)
|
||||
except ValueError:
|
||||
price_block = (product.get("price") or {}).get("ok") or {}
|
||||
reg_price, reg_unit = cls._parse_price(price_block.get("regPriceText"))
|
||||
promo = price_block.get("promoArea") or {}
|
||||
sale_price, sale_unit = cls._parse_price(promo.get("promoText"))
|
||||
|
||||
if reg_price is None and sale_price is None:
|
||||
# No usable price; skip rather than persist garbage.
|
||||
return None
|
||||
|
||||
name_elem = element.find_parent("a") or element.find_parent("div")
|
||||
name = text.split("$")[0].strip() if "$" in text else text
|
||||
name = re.sub(r'\s+', " ", name).strip()[:200]
|
||||
unit = sale_unit or reg_unit
|
||||
is_on_sale = sale_price is not None and reg_price is not None and sale_price < reg_price
|
||||
# current_price = "what the customer pays today" → sale_price when on sale.
|
||||
current_price = sale_price if is_on_sale else reg_price
|
||||
|
||||
if not name or len(name) < 3:
|
||||
return None
|
||||
image_url: Optional[str] = None
|
||||
primary_image = product.get("primaryImage")
|
||||
if isinstance(primary_image, dict):
|
||||
image_url = primary_image.get("url")
|
||||
|
||||
image_url = None
|
||||
img_elem = element.find_parent().find("img") if element.find_parent() else None
|
||||
if img_elem and img_elem.get("src"):
|
||||
image_url = img_elem["src"]
|
||||
brand = product.get("brand")
|
||||
description = product.get("description")
|
||||
|
||||
product_url = None
|
||||
link_elem = element.find_parent("a") if element.find_parent() else element.find("a")
|
||||
if link_elem and link_elem.get("href"):
|
||||
product_url = urljoin(self.base_url, link_elem["href"])
|
||||
|
||||
item = {
|
||||
"name": name,
|
||||
"current_price": price,
|
||||
return {
|
||||
"external_id": str(external_id) if external_id is not None else None,
|
||||
"source": cls.SOURCE,
|
||||
"name": name[:300],
|
||||
"brand": (brand or None),
|
||||
"description": description,
|
||||
"current_price": current_price,
|
||||
"regular_price": reg_price,
|
||||
"sale_price": sale_price,
|
||||
"is_on_sale": bool(is_on_sale),
|
||||
"unit": unit,
|
||||
"aisle": aisle,
|
||||
"image_url": image_url,
|
||||
"product_url": product_url,
|
||||
"is_on_sale": True,
|
||||
"product_url": None, # Swiftly does not expose a public product URL
|
||||
"scraped_at": datetime.now().isoformat(),
|
||||
"scraped_url": self.base_url
|
||||
"scraped_url": f"{cls.__name__}:{source_slug}" if source_slug else None,
|
||||
}
|
||||
|
||||
return item
|
||||
# ------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ------------------------------------------------------------------
|
||||
@staticmethod
|
||||
def _parse_price(text: Optional[str]) -> Tuple[Optional[Decimal], Optional[str]]:
|
||||
"""Extract ``(price, unit)`` from strings like ``"$3.49 /lb"``.
|
||||
|
||||
def scrape_produce(self) -> List[Dict[str, Any]]:
|
||||
url = f"{self.base_url}/coupons/Coupon%2Flu-produce"
|
||||
logger.info(f"Scraping produce from {url}")
|
||||
|
||||
page = self.get_browser_page(url)
|
||||
content = page.content()
|
||||
page.close()
|
||||
|
||||
soup = BeautifulSoup(content, "html.parser")
|
||||
items = []
|
||||
|
||||
for item_elem in soup.find_all("div", class_=re.compile(r"product|item|coupon")):
|
||||
Returns ``(None, None)`` when ``text`` is empty or unparseable.
|
||||
"""
|
||||
if not text:
|
||||
return None, None
|
||||
m = re.search(r"\$\s*([\d,]+(?:\.\d+)?)", text)
|
||||
if not m:
|
||||
return None, None
|
||||
raw = m.group(1).replace(",", "")
|
||||
try:
|
||||
item = self._parse_coupon_item(item_elem)
|
||||
if item:
|
||||
item["aisle"] = "Produce"
|
||||
items.append(item)
|
||||
except Exception:
|
||||
continue
|
||||
price = Decimal(raw)
|
||||
except (InvalidOperation, ValueError):
|
||||
return None, None
|
||||
unit_match = re.search(r"/\s*([A-Za-z]+)", text)
|
||||
unit = unit_match.group(1).lower() if unit_match else None
|
||||
return price, unit
|
||||
|
||||
return items
|
||||
@staticmethod
|
||||
def _aisle_from_slug(slug: str) -> Optional[str]:
|
||||
"""``Product/meat_seafood`` → ``meat_seafood``."""
|
||||
if not slug:
|
||||
return None
|
||||
if "/" in slug:
|
||||
return slug.rsplit("/", 1)[1]
|
||||
return slug
|
||||
|
||||
def scrape_meat_seafood(self) -> List[Dict[str, Any]]:
|
||||
url = f"{self.base_url}/coupons/Coupon%2Flu-meat-seafood"
|
||||
logger.info(f"Scraping meat & seafood from {url}")
|
||||
|
||||
page = self.get_browser_page(url)
|
||||
content = page.content()
|
||||
page.close()
|
||||
|
||||
soup = BeautifulSoup(content, "html.parser")
|
||||
items = []
|
||||
|
||||
for item_elem in soup.find_all("div", class_=re.compile(r"product|item|coupon")):
|
||||
try:
|
||||
item = self._parse_coupon_item(item_elem)
|
||||
if item:
|
||||
item["aisle"] = "Meat"
|
||||
items.append(item)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return items
|
||||
|
||||
def scrape_dairy_eggs(self) -> List[Dict[str, Any]]:
|
||||
url = f"{self.base_url}/coupons/Coupon%2Flu-dairy-eggs"
|
||||
logger.info(f"Scraping dairy & eggs from {url}")
|
||||
|
||||
page = self.get_browser_page(url)
|
||||
content = page.content()
|
||||
page.close()
|
||||
|
||||
soup = BeautifulSoup(content, "html.parser")
|
||||
items = []
|
||||
|
||||
for item_elem in soup.find_all("div", class_=re.compile(r"product|item|coupon")):
|
||||
try:
|
||||
item = self._parse_coupon_item(item_elem)
|
||||
if item:
|
||||
item["aisle"] = "Dairy"
|
||||
items.append(item)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return items
|
||||
def _rate_limit(self) -> None:
|
||||
elapsed = time.time() - self._last_request
|
||||
if elapsed < self.rate_limit_seconds:
|
||||
time.sleep(self.rate_limit_seconds - elapsed)
|
||||
self._last_request = time.time()
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""
|
||||
Auth dependencies for the MealPlanner backend.
|
||||
|
||||
Two flavors:
|
||||
- ``require_admin`` — bearer token for ``/api/admin/*`` routes; token compared
|
||||
to ``settings.ADMIN_TOKEN`` (must be set in env).
|
||||
- ``require_session`` — signed-cookie session (``itsdangerous``) gating
|
||||
mutations on the family-facing routers; reads stay open inside the
|
||||
trusted network.
|
||||
|
||||
The per-voter approval-token flow on meal items is intentionally NOT covered
|
||||
here — it has its own short-lived single-use tokens elsewhere.
|
||||
"""
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
from itsdangerous import BadSignature, SignatureExpired, TimestampSigner
|
||||
|
||||
from app.config import settings
|
||||
|
||||
bearer_header = "Authorization"
|
||||
|
||||
SESSION_COOKIE = "mp_session"
|
||||
SESSION_MAX_AGE = 60 * 60 * 24 * 30 # 30 days
|
||||
|
||||
|
||||
def require_admin(request: Request) -> None:
|
||||
"""Enforce a shared bearer token. 401 on bad/missing token, 503 if unset."""
|
||||
expected = settings.ADMIN_TOKEN
|
||||
if not expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="Admin auth not configured",
|
||||
)
|
||||
auth = request.headers.get(bearer_header, "")
|
||||
if not auth.startswith("Bearer ") or auth[7:] != expected:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="Invalid admin token",
|
||||
)
|
||||
|
||||
|
||||
def _signer() -> TimestampSigner:
|
||||
return TimestampSigner(settings.SECRET_KEY)
|
||||
|
||||
|
||||
def issue_session(family_profile_id: str) -> str:
|
||||
"""Sign the family_profile_id and return the cookie value."""
|
||||
return _signer().sign(family_profile_id.encode()).decode()
|
||||
|
||||
|
||||
def require_session(request: Request) -> str:
|
||||
"""Return the family_profile_id stored in the signed session cookie."""
|
||||
raw = request.cookies.get(SESSION_COOKIE)
|
||||
if not raw:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Session required"
|
||||
)
|
||||
try:
|
||||
family_id = (
|
||||
_signer().unsign(raw.encode(), max_age=SESSION_MAX_AGE).decode()
|
||||
)
|
||||
except SignatureExpired:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Session expired"
|
||||
)
|
||||
except BadSignature:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid session"
|
||||
)
|
||||
return family_id
|
||||
@@ -0,0 +1,89 @@
|
||||
"""
|
||||
Per-voter approval tokens.
|
||||
|
||||
Stateless signed tokens (itsdangerous) keyed on settings.SECRET_KEY with a
|
||||
versioned salt. Single-use is enforced by the presence of a MealPlanVote
|
||||
row for (item, voter) — the table already has UniqueConstraint on that
|
||||
pair, so the DB is the source of truth, not a token-status column.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import HTTPException
|
||||
from itsdangerous import (
|
||||
BadSignature,
|
||||
SignatureExpired,
|
||||
URLSafeTimedSerializer,
|
||||
)
|
||||
|
||||
from app.config import settings
|
||||
from app.models import FamilyMember, MealPlanVote
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
|
||||
SALT = "meal-approval-v1"
|
||||
DEFAULT_MAX_AGE_SECONDS = 7 * 24 * 3600
|
||||
|
||||
|
||||
def _serializer() -> URLSafeTimedSerializer:
|
||||
return URLSafeTimedSerializer(secret_key=settings.SECRET_KEY, salt=SALT)
|
||||
|
||||
|
||||
def issue_token(meal_plan_item_id: UUID, family_member_id: UUID) -> str:
|
||||
payload = {
|
||||
"item": str(meal_plan_item_id),
|
||||
"voter": str(family_member_id),
|
||||
}
|
||||
return _serializer().dumps(payload)
|
||||
|
||||
|
||||
def verify_token(token: str, max_age_seconds: int = DEFAULT_MAX_AGE_SECONDS) -> dict:
|
||||
try:
|
||||
payload = _serializer().loads(token, max_age=max_age_seconds)
|
||||
except SignatureExpired:
|
||||
raise HTTPException(status_code=401, detail="Token expired")
|
||||
except BadSignature:
|
||||
raise HTTPException(status_code=401, detail="Invalid token")
|
||||
if not isinstance(payload, dict) or "item" not in payload or "voter" not in payload:
|
||||
raise HTTPException(status_code=401, detail="Invalid token payload")
|
||||
return payload
|
||||
|
||||
|
||||
def consume_token(
|
||||
db: "Session",
|
||||
token: str,
|
||||
meal_plan_item_id: UUID,
|
||||
max_age_seconds: int = DEFAULT_MAX_AGE_SECONDS,
|
||||
) -> FamilyMember:
|
||||
"""Verify + match URL + enforce single-use. Returns the voter on success.
|
||||
|
||||
Single-use is checked by looking for an existing MealPlanVote row for
|
||||
(item, voter). If one exists, raise 409.
|
||||
"""
|
||||
payload = verify_token(token, max_age_seconds=max_age_seconds)
|
||||
|
||||
if str(payload["item"]) != str(meal_plan_item_id):
|
||||
raise HTTPException(status_code=400, detail="Token not valid for this meal")
|
||||
|
||||
voter_id = UUID(str(payload["voter"]))
|
||||
voter = db.query(FamilyMember).filter(FamilyMember.id == voter_id).first()
|
||||
if not voter:
|
||||
raise HTTPException(status_code=404, detail="Voter not found")
|
||||
|
||||
existing = (
|
||||
db.query(MealPlanVote)
|
||||
.filter(
|
||||
MealPlanVote.meal_plan_item_id == meal_plan_item_id,
|
||||
MealPlanVote.family_member_id == voter_id,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if existing:
|
||||
raise HTTPException(status_code=409, detail="Already voted")
|
||||
|
||||
return voter
|
||||
@@ -0,0 +1,89 @@
|
||||
"""
|
||||
Email backends.
|
||||
|
||||
R2-B spike: only ConsoleEmailBackend is functional. SendGrid is a stub
|
||||
intentionally left to be wired in R3-C.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional, Protocol
|
||||
|
||||
from app.config import settings
|
||||
|
||||
|
||||
# Repository convention: backend/var/email_outbox.jsonl
|
||||
# This module lives at backend/app/services/email.py — go up two levels to
|
||||
# reach backend/, then var/.
|
||||
_BACKEND_ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
OUTBOX_PATH = _BACKEND_ROOT / "var" / "email_outbox.jsonl"
|
||||
|
||||
|
||||
class EmailBackend(Protocol):
|
||||
def send(
|
||||
self,
|
||||
to: str,
|
||||
subject: str,
|
||||
html: str,
|
||||
text: Optional[str] = None,
|
||||
) -> None:
|
||||
...
|
||||
|
||||
|
||||
class ConsoleEmailBackend:
|
||||
"""Dev/spike backend. Prints to stdout AND appends a JSON line to
|
||||
backend/var/email_outbox.jsonl so the round-trip can be inspected
|
||||
after the fact.
|
||||
"""
|
||||
|
||||
def send(
|
||||
self,
|
||||
to: str,
|
||||
subject: str,
|
||||
html: str,
|
||||
text: Optional[str] = None,
|
||||
) -> None:
|
||||
record = {
|
||||
"ts": datetime.now(timezone.utc).isoformat(),
|
||||
"to": to,
|
||||
"subject": subject,
|
||||
"html": html,
|
||||
"text": text,
|
||||
}
|
||||
print(
|
||||
f"[ConsoleEmailBackend] -> {to} | {subject}",
|
||||
file=sys.stdout,
|
||||
flush=True,
|
||||
)
|
||||
OUTBOX_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
with OUTBOX_PATH.open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record) + "\n")
|
||||
|
||||
|
||||
class SendGridEmailBackend:
|
||||
"""Stub. Wire in R3-C (real SendGrid client + sandbox mode + retries).
|
||||
|
||||
Deliberately raises so a misconfigured prod env fails loudly instead of
|
||||
silently dropping mail.
|
||||
"""
|
||||
|
||||
def send(
|
||||
self,
|
||||
to: str,
|
||||
subject: str,
|
||||
html: str,
|
||||
text: Optional[str] = None,
|
||||
) -> None: # pragma: no cover - stub
|
||||
raise NotImplementedError("Wire SendGrid in R3-C")
|
||||
|
||||
|
||||
def get_email_backend() -> EmailBackend:
|
||||
backend = (settings.EMAIL_BACKEND or "console").lower()
|
||||
if backend == "sendgrid":
|
||||
return SendGridEmailBackend()
|
||||
return ConsoleEmailBackend()
|
||||
@@ -1,87 +1,204 @@
|
||||
"""Scraper service.
|
||||
|
||||
Two entry points:
|
||||
|
||||
- ``enqueue_scrape(db, source, scrape_type, background_tasks)``: synchronously
|
||||
inserts a ``ScrapeLog`` row in status ``STARTED`` (the existing enum has no
|
||||
``pending`` member; ``STARTED`` is reused for the queued state) and registers
|
||||
``_run_scrape_in_background`` to fire after the response is sent.
|
||||
- ``_run_scrape_in_background(log_id, source, scrape_type)``: runs in a
|
||||
FastAPI background task with its OWN ``SessionLocal()`` (the request-scoped
|
||||
``db`` is closed by the time this fires). Writes terminal status
|
||||
(``SUCCESS``/``FAILED``) and ``error_message``.
|
||||
|
||||
``ScraperService.run_scrape`` is preserved for direct/test invocation; the
|
||||
``/api/admin/scrape`` endpoint now goes through ``enqueue_scrape``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
from uuid import uuid4
|
||||
from typing import Any, Dict, Optional
|
||||
from datetime import datetime, timezone
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from fastapi import BackgroundTasks
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.database import SessionLocal
|
||||
from app.models import GroceryItem, Ingredient, ScrapeLog, ScrapeStatus
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API: enqueue + background runner
|
||||
# ---------------------------------------------------------------------------
|
||||
def enqueue_scrape(
|
||||
db: Session,
|
||||
*,
|
||||
source: str,
|
||||
scrape_type: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> ScrapeLog:
|
||||
"""Create the ScrapeLog row, commit, and schedule the background scrape.
|
||||
|
||||
Returns the persisted ``ScrapeLog`` instance (refreshed). The actual
|
||||
scraping work runs after FastAPI sends the 202 response.
|
||||
"""
|
||||
log = ScrapeLog(
|
||||
id=uuid4(),
|
||||
source=source,
|
||||
scrape_type=scrape_type,
|
||||
status=ScrapeStatus.STARTED,
|
||||
started_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db.add(log)
|
||||
db.commit()
|
||||
db.refresh(log)
|
||||
|
||||
background_tasks.add_task(
|
||||
_run_scrape_in_background, log.id, source, scrape_type
|
||||
)
|
||||
return log
|
||||
|
||||
|
||||
def _run_scrape_in_background(
|
||||
log_id: UUID, source: str, scrape_type: str
|
||||
) -> None:
|
||||
"""Background entry point. Opens a fresh DB session — the request-scoped
|
||||
session is gone by the time this runs.
|
||||
"""
|
||||
db: Session = SessionLocal()
|
||||
try:
|
||||
log = db.query(ScrapeLog).filter(ScrapeLog.id == log_id).first()
|
||||
if log is None:
|
||||
logger.error("ScrapeLog %s vanished before background run", log_id)
|
||||
return
|
||||
|
||||
# No "running" state in the enum; STARTED already covers in-flight.
|
||||
# Mark started_at fresh in case there was lag between enqueue and run.
|
||||
try:
|
||||
service = ScraperService(db)
|
||||
saved_count, items_found = service._do_scrape(source=source)
|
||||
log.status = ScrapeStatus.SUCCESS
|
||||
log.items_scraped = saved_count
|
||||
log.completed_at = datetime.now(timezone.utc)
|
||||
log.duration_seconds = int(
|
||||
(log.completed_at - log.started_at).total_seconds()
|
||||
)
|
||||
db.commit()
|
||||
logger.info(
|
||||
"Background scrape %s complete: %s/%s items saved",
|
||||
log_id, saved_count, items_found,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 — must catch all to mark failed
|
||||
logger.exception("Background scrape %s failed", log_id)
|
||||
db.rollback()
|
||||
# Re-fetch in case the rollback detached the instance.
|
||||
log = db.query(ScrapeLog).filter(ScrapeLog.id == log_id).first()
|
||||
if log is not None:
|
||||
log.status = ScrapeStatus.FAILED
|
||||
log.error_message = str(exc)
|
||||
log.completed_at = datetime.now(timezone.utc)
|
||||
if log.started_at:
|
||||
log.duration_seconds = int(
|
||||
(log.completed_at - log.started_at).total_seconds()
|
||||
)
|
||||
db.commit()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Service class (kept for direct/test use)
|
||||
# ---------------------------------------------------------------------------
|
||||
class ScraperService:
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def run_scrape(self, source: str = "lucky_california", scrape_type: str = "weekly_ad") -> Dict[str, Any]:
|
||||
from app.scraper import LuckyCaliforniaScraper
|
||||
from app.models import ScrapeLog, GroceryItem, Ingredient
|
||||
|
||||
scrape_log = ScrapeLog(
|
||||
def run_scrape(
|
||||
self, source: str = "lucky_california", scrape_type: str = "weekly_ad"
|
||||
) -> Dict[str, Any]:
|
||||
"""Synchronous end-to-end scrape (legacy path). Creates the log row,
|
||||
runs the scrape, commits terminal status. Used by direct callers and
|
||||
tests; the API endpoint goes through ``enqueue_scrape``.
|
||||
"""
|
||||
log = ScrapeLog(
|
||||
id=uuid4(),
|
||||
source=source,
|
||||
scrape_type=scrape_type,
|
||||
status="started",
|
||||
started_at=datetime.now()
|
||||
status=ScrapeStatus.STARTED,
|
||||
started_at=datetime.now(timezone.utc),
|
||||
)
|
||||
self.db.add(scrape_log)
|
||||
self.db.add(log)
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Starting {source} {scrape_type} scrape")
|
||||
logger.info("Starting %s %s scrape", source, scrape_type)
|
||||
|
||||
try:
|
||||
scraper = LuckyCaliforniaScraper()
|
||||
result = scraper.scrape()
|
||||
scraper.cleanup()
|
||||
|
||||
items = result.get("items", [])
|
||||
saved_count = 0
|
||||
|
||||
for item_data in items:
|
||||
saved_item = self._save_grocery_item(item_data)
|
||||
if saved_item:
|
||||
saved_count += 1
|
||||
|
||||
scrape_log.status = "success"
|
||||
scrape_log.items_scraped = saved_count
|
||||
scrape_log.completed_at = datetime.now()
|
||||
scrape_log.duration_seconds = int(
|
||||
(scrape_log.completed_at - scrape_log.started_at).total_seconds()
|
||||
saved_count, items_found = self._do_scrape(source=source)
|
||||
log.status = ScrapeStatus.SUCCESS
|
||||
log.items_scraped = saved_count
|
||||
log.completed_at = datetime.now(timezone.utc)
|
||||
log.duration_seconds = int(
|
||||
(log.completed_at - log.started_at).total_seconds()
|
||||
)
|
||||
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Scrape complete: {saved_count} items saved")
|
||||
|
||||
logger.info("Scrape complete: %s items saved", saved_count)
|
||||
return {
|
||||
"scrape_id": str(scrape_log.id),
|
||||
"scrape_id": str(log.id),
|
||||
"status": "success",
|
||||
"items_scraped": saved_count,
|
||||
"items_found": len(items)
|
||||
"items_found": items_found,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Scrape failed: {e}")
|
||||
scrape_log.status = "failed"
|
||||
scrape_log.error_message = str(e)
|
||||
scrape_log.completed_at = datetime.now()
|
||||
scrape_log.duration_seconds = int(
|
||||
(scrape_log.completed_at - scrape_log.started_at).total_seconds()
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.error("Scrape failed: %s", e)
|
||||
log.status = ScrapeStatus.FAILED
|
||||
log.error_message = str(e)
|
||||
log.completed_at = datetime.now(timezone.utc)
|
||||
log.duration_seconds = int(
|
||||
(log.completed_at - log.started_at).total_seconds()
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
return {
|
||||
"scrape_id": str(scrape_log.id),
|
||||
"scrape_id": str(log.id),
|
||||
"status": "failed",
|
||||
"error": str(e)
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
def _save_grocery_item(self, item_data: Dict[str, Any]) -> Optional[GroceryItem]:
|
||||
from app.models import GroceryItem, Ingredient
|
||||
# ------------------------------------------------------------------
|
||||
# Internals
|
||||
# ------------------------------------------------------------------
|
||||
def _do_scrape(self, *, source: str) -> tuple[int, int]:
|
||||
"""Run the scraper and persist items. Returns (saved, found).
|
||||
|
||||
name = item_data.get("name", "").strip()
|
||||
Raises any scraper exception (including ``SwiftlyAuthError``) to
|
||||
the caller for status mapping.
|
||||
"""
|
||||
from app.scraper import LuckyCaliforniaScraper
|
||||
|
||||
scraper = LuckyCaliforniaScraper()
|
||||
saved_count = 0
|
||||
found_count = 0
|
||||
try:
|
||||
for item_data in scraper.fetch_all():
|
||||
found_count += 1
|
||||
if self._save_grocery_item(item_data) is not None:
|
||||
saved_count += 1
|
||||
finally:
|
||||
scraper.cleanup()
|
||||
return saved_count, found_count
|
||||
|
||||
def _save_grocery_item(
|
||||
self, item_data: Dict[str, Any]
|
||||
) -> Optional[GroceryItem]:
|
||||
name = (item_data.get("name") or "").strip()
|
||||
if not name:
|
||||
return None
|
||||
|
||||
name_lower = name.lower()
|
||||
source = item_data.get("source") or "lucky_california"
|
||||
external_id = item_data.get("external_id")
|
||||
|
||||
ingredient = self.db.query(Ingredient).filter(
|
||||
Ingredient.name_lower == name_lower
|
||||
@@ -93,22 +210,41 @@ class ScraperService:
|
||||
name=name,
|
||||
name_lower=name_lower,
|
||||
aisle=item_data.get("aisle"),
|
||||
typical_price=item_data.get("current_price")
|
||||
typical_price=item_data.get("current_price"),
|
||||
)
|
||||
self.db.add(ingredient)
|
||||
self.db.flush()
|
||||
|
||||
# Idempotency:
|
||||
# 1) (source, external_id) when both present (Swiftly path);
|
||||
# 2) fall back to (name, scraped_url) for legacy rows from R2-A.
|
||||
existing: Optional[GroceryItem] = None
|
||||
if external_id:
|
||||
existing = self.db.query(GroceryItem).filter(
|
||||
GroceryItem.source == source,
|
||||
GroceryItem.external_id == external_id,
|
||||
).first()
|
||||
if existing is None:
|
||||
existing = self.db.query(GroceryItem).filter(
|
||||
GroceryItem.name == name,
|
||||
GroceryItem.scraped_url == item_data.get("scraped_url")
|
||||
GroceryItem.scraped_url == item_data.get("scraped_url"),
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
existing.ingredient_id = ingredient.id
|
||||
existing.brand = item_data.get("brand")
|
||||
existing.current_price = item_data.get("current_price")
|
||||
existing.is_on_sale = item_data.get("is_on_sale", True)
|
||||
existing.regular_price = item_data.get("regular_price")
|
||||
existing.unit = item_data.get("unit")
|
||||
existing.aisle = item_data.get("aisle")
|
||||
existing.image_url = item_data.get("image_url")
|
||||
existing.product_url = item_data.get("product_url")
|
||||
existing.scraped_at = datetime.now()
|
||||
existing.description = item_data.get("description")
|
||||
existing.is_on_sale = bool(item_data.get("is_on_sale", False))
|
||||
existing.scraped_at = datetime.now(timezone.utc)
|
||||
existing.scraped_url = item_data.get("scraped_url")
|
||||
existing.external_id = external_id
|
||||
existing.source = source
|
||||
self.db.flush()
|
||||
return existing
|
||||
|
||||
@@ -116,31 +252,31 @@ class ScraperService:
|
||||
id=uuid4(),
|
||||
ingredient_id=ingredient.id,
|
||||
name=name,
|
||||
brand=item_data.get("brand"),
|
||||
current_price=item_data.get("current_price"),
|
||||
regular_price=item_data.get("regular_price"),
|
||||
unit=item_data.get("unit"),
|
||||
aisle=item_data.get("aisle"),
|
||||
image_url=item_data.get("image_url"),
|
||||
product_url=item_data.get("product_url"),
|
||||
is_on_sale=item_data.get("is_on_sale", True),
|
||||
description=item_data.get("description"),
|
||||
is_on_sale=bool(item_data.get("is_on_sale", False)),
|
||||
sale_start_date=item_data.get("sale_start_date"),
|
||||
sale_end_date=item_data.get("sale_end_date"),
|
||||
in_season=item_data.get("in_season", False),
|
||||
scraped_at=datetime.now(),
|
||||
scraped_url=item_data.get("scraped_url")
|
||||
scraped_at=datetime.now(timezone.utc),
|
||||
scraped_url=item_data.get("scraped_url"),
|
||||
external_id=external_id,
|
||||
source=source,
|
||||
)
|
||||
|
||||
self.db.add(grocery_item)
|
||||
self.db.commit()
|
||||
self.db.refresh(grocery_item)
|
||||
|
||||
self.db.flush()
|
||||
return grocery_item
|
||||
|
||||
def get_sale_items(self, limit: int = 50) -> list:
|
||||
from app.models import GroceryItem
|
||||
|
||||
items = self.db.query(GroceryItem).filter(
|
||||
GroceryItem.is_on_sale == True
|
||||
GroceryItem.is_on_sale == True # noqa: E712 — SQLAlchemy idiom
|
||||
).order_by(GroceryItem.scraped_at.desc()).limit(limit).all()
|
||||
|
||||
return [
|
||||
@@ -152,7 +288,8 @@ class ScraperService:
|
||||
"aisle": item.aisle,
|
||||
"image_url": item.image_url,
|
||||
"product_url": item.product_url,
|
||||
"scraped_at": item.scraped_at.isoformat() if item.scraped_at else None
|
||||
"description": item.description,
|
||||
"scraped_at": item.scraped_at.isoformat() if item.scraped_at else None,
|
||||
}
|
||||
for item in items
|
||||
]
|
||||
@@ -0,0 +1,6 @@
|
||||
[pytest]
|
||||
testpaths = tests
|
||||
asyncio_mode = auto
|
||||
addopts = -ra
|
||||
filterwarnings =
|
||||
ignore::DeprecationWarning
|
||||
@@ -0,0 +1,6 @@
|
||||
# Dev / test-only dependencies. Production deps live in requirements.txt.
|
||||
pytest>=7.4
|
||||
pytest-cov>=4.1
|
||||
pytest-asyncio>=0.23
|
||||
httpx>=0.25
|
||||
freezegun>=1.4
|
||||
@@ -11,6 +11,8 @@ beautifulsoup4==4.12.3
|
||||
lxml==5.1.0
|
||||
apscheduler==3.10.4
|
||||
python-dotenv==1.0.0
|
||||
itsdangerous==2.1.2
|
||||
httpx==0.26.0
|
||||
requests==2.31.0
|
||||
pytest==7.4.4
|
||||
pytest-asyncio==0.23.3
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
"""
|
||||
Pytest fixtures for MealPlanner backend.
|
||||
|
||||
DB strategy
|
||||
-----------
|
||||
- If env ``TEST_DATABASE_URL`` is set, use it (must be a Postgres URL — the
|
||||
current Alembic migrations use ``postgresql.UUID``/``JSONB``/``ARRAY`` types
|
||||
which are not portable to SQLite).
|
||||
- Else fall back to ``DATABASE_URL`` if it points at Postgres.
|
||||
- Else skip Postgres-only tests (marker: ``requires_postgres``).
|
||||
|
||||
Each test using the ``db`` fixture runs inside a SAVEPOINT-style nested
|
||||
transaction that rolls back on teardown so tests do not leak state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import pathlib
|
||||
import subprocess
|
||||
from typing import Iterator
|
||||
|
||||
import pytest
|
||||
|
||||
# Ensure DATABASE_URL is set BEFORE importing app.config (Settings requires it).
|
||||
# We default to the TEST_DATABASE_URL or a sentinel that lets imports succeed;
|
||||
# tests that actually need the DB rely on the marker / fixture skip path below.
|
||||
_DEFAULT_DSN = "postgresql://mealplanner:password@localhost:5432/mealplanner_test"
|
||||
os.environ.setdefault(
|
||||
"DATABASE_URL",
|
||||
os.environ.get("TEST_DATABASE_URL", _DEFAULT_DSN),
|
||||
)
|
||||
|
||||
# Make backend/ importable when pytest is invoked from repo root.
|
||||
BACKEND_ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
if str(BACKEND_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(BACKEND_ROOT))
|
||||
|
||||
from sqlalchemy import create_engine, text # noqa: E402
|
||||
from sqlalchemy.orm import sessionmaker # noqa: E402
|
||||
from sqlalchemy.exc import OperationalError # noqa: E402
|
||||
|
||||
|
||||
def _resolve_test_dsn() -> str | None:
|
||||
dsn = os.environ.get("TEST_DATABASE_URL") or os.environ.get("DATABASE_URL")
|
||||
if not dsn:
|
||||
return None
|
||||
if not dsn.startswith(("postgresql://", "postgresql+psycopg2://")):
|
||||
return None
|
||||
return dsn
|
||||
|
||||
|
||||
def _postgres_reachable(dsn: str) -> bool:
|
||||
try:
|
||||
eng = create_engine(
|
||||
dsn.replace("postgresql://", "postgresql+psycopg2://"),
|
||||
pool_pre_ping=True,
|
||||
)
|
||||
with eng.connect() as conn:
|
||||
conn.execute(text("SELECT 1"))
|
||||
eng.dispose()
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
_DSN = _resolve_test_dsn()
|
||||
_PG_AVAILABLE = bool(_DSN) and _postgres_reachable(_DSN)
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config, items):
|
||||
"""Skip postgres-only tests when no live Postgres is available."""
|
||||
if _PG_AVAILABLE:
|
||||
return
|
||||
skip_pg = pytest.mark.skip(
|
||||
reason="Postgres not reachable; set TEST_DATABASE_URL to enable."
|
||||
)
|
||||
for item in items:
|
||||
if "requires_postgres" in item.keywords:
|
||||
item.add_marker(skip_pg)
|
||||
|
||||
|
||||
def pytest_configure(config):
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"requires_postgres: test needs a live Postgres reachable via TEST_DATABASE_URL",
|
||||
)
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"scraper_offline: parser-only test; uses saved HTML fixture; no "
|
||||
"network or Playwright/Chromium required (CI-safe).",
|
||||
)
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"scraper_live: hits the live grocery website; requires Playwright "
|
||||
"and Chromium; skipped by default in CI.",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Schema bootstrap (session-scoped): run alembic upgrade head once per session.
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.fixture(scope="session")
|
||||
def _schema() -> Iterator[None]:
|
||||
if not _PG_AVAILABLE:
|
||||
yield
|
||||
return
|
||||
env = os.environ.copy()
|
||||
env["DATABASE_URL"] = _DSN # Alembic env.py reads from settings.DATABASE_URL
|
||||
# alembic.ini lives in backend/, run from there.
|
||||
subprocess.run(
|
||||
["alembic", "upgrade", "head"],
|
||||
cwd=str(BACKEND_ROOT),
|
||||
env=env,
|
||||
check=True,
|
||||
)
|
||||
yield
|
||||
# Best-effort cleanup so re-running the suite locally is idempotent.
|
||||
subprocess.run(
|
||||
["alembic", "downgrade", "base"],
|
||||
cwd=str(BACKEND_ROOT),
|
||||
env=env,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def _engine(_schema):
|
||||
if not _PG_AVAILABLE:
|
||||
yield None
|
||||
return
|
||||
eng = create_engine(
|
||||
_DSN.replace("postgresql://", "postgresql+psycopg2://"),
|
||||
pool_pre_ping=True,
|
||||
)
|
||||
yield eng
|
||||
eng.dispose()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db(_engine):
|
||||
"""Per-test transactional session that rolls back at teardown."""
|
||||
if _engine is None:
|
||||
pytest.skip("Postgres not reachable")
|
||||
connection = _engine.connect()
|
||||
trans = connection.begin()
|
||||
Session = sessionmaker(bind=connection, autocommit=False, autoflush=False)
|
||||
session = Session()
|
||||
try:
|
||||
yield session
|
||||
finally:
|
||||
session.close()
|
||||
trans.rollback()
|
||||
connection.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(db):
|
||||
"""TestClient with get_db overridden to yield the test session."""
|
||||
from fastapi.testclient import TestClient
|
||||
from app.main import app
|
||||
from app.database import get_db
|
||||
|
||||
def _override():
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
pass
|
||||
|
||||
app.dependency_overrides[get_db] = _override
|
||||
try:
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
finally:
|
||||
app.dependency_overrides.pop(get_db, None)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client_no_db():
|
||||
"""TestClient that does NOT require a live DB — for pure import/wiring smoke."""
|
||||
from fastapi.testclient import TestClient
|
||||
from app.main import app
|
||||
from app.database import get_db
|
||||
|
||||
class _StubSession:
|
||||
def execute(self, *a, **kw):
|
||||
from sqlalchemy.engine import Result # noqa: F401
|
||||
raise RuntimeError("DB not available in this fixture")
|
||||
|
||||
def query(self, *a, **kw):
|
||||
raise RuntimeError("DB not available in this fixture")
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
def _override():
|
||||
yield _StubSession()
|
||||
|
||||
app.dependency_overrides[get_db] = _override
|
||||
try:
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
finally:
|
||||
app.dependency_overrides.pop(get_db, None)
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
# Lucky California weekly-ad spike (R2-A)
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
|
||||
| Final URL | https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad |
|
||||
| Fetched (UTC) | 2026-05-04 (single live fetch via scripts/spike_lucky_scrape.py) |
|
||||
| HTTP status | 200 |
|
||||
| HTML bytes | 201224 |
|
||||
| Items parsed (current parser) | 11 |
|
||||
| Captcha/block signal | benign — page contains an empty `<span id="recaptcha-element">` placeholder, no actual challenge served. Title: "Featured in Ad \| Luckys Supermarket". 13 real `coupon-card-wrapper` cards rendered. |
|
||||
| Error | none |
|
||||
| User-Agent | `Mozilla/5.0 (compatible; MealPlannerSpike/0.1; +https://github.com/MealPlanner; spike=R2-A)` |
|
||||
|
||||
## Selectors used (verified against this fixture)
|
||||
|
||||
- Card root: `div.coupon-card-wrapper` (fallback: `div.coupon-card-container`)
|
||||
- Price + short name: `.coupon-card-value-text` (e.g. `"$13.97 Pepsi 24 packs"`)
|
||||
- Long description: `.coupon-card-short-description`
|
||||
- Image: `img` inside `.coupon-card-img-container` (CDN URL: `cdn.luckysupermarkets.com/loyalty/offer/<id>.jpg`)
|
||||
- No per-card link is present in the rendered DOM (offers are non-navigable tiles).
|
||||
|
||||
## First parsed item (sample)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Pepsi 24 packs",
|
||||
"description": "$13.97 Pepsi Products 24 pack, Poppi 8 pack, Gatorade 18 pack, Rockstar 10 pack, or Pure Leaf 12 pack, select varieties +CRV in CA. While supplies last.",
|
||||
"current_price": 13.97,
|
||||
"image_url": "https://cdn.luckysupermarkets.com/loyalty/offer/205213.jpg",
|
||||
"product_url": null,
|
||||
"is_on_sale": true,
|
||||
"scraped_at": "2026-05-04T21:42:56.952936",
|
||||
"scraped_url": "https://luckysupermarkets.com"
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- ONE live fetch performed by `scripts/spike_lucky_scrape.py`. Do not rerun without reason.
|
||||
- HTML and PNG saved alongside this file (`weekly_ad.html`, `weekly_ad.png`).
|
||||
- Initial parser (regex-based on `h2/h3/a` text) produced only 1 item; selectors were
|
||||
stale. A minimal additive fix landed in `backend/app/scraper/lucky_ca_scraper.py`:
|
||||
new `_parse_coupon_card` + `parse_featured_coupons_html` methods that target
|
||||
Swiftly-style `.coupon-card-wrapper` cards. Old `_parse_coupon_item` retained
|
||||
as a fallback path.
|
||||
- Schema impact: `grocery_item` columns (name, current_price, image_url, is_on_sale,
|
||||
scraped_at, scraped_url) are all populated. `product_url` is None for every
|
||||
card (no per-offer link in DOM) — keep nullable. New optional `description`
|
||||
field is produced; either add a `description TEXT` column or drop it.
|
||||
+80
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+89
File diff suppressed because one or more lines are too long
BIN
Binary file not shown.
|
After Width: | Height: | Size: 582 KiB |
@@ -0,0 +1,47 @@
|
||||
"""
|
||||
Alembic round-trip test.
|
||||
|
||||
Verifies that ``alembic upgrade head`` followed by ``alembic downgrade base``
|
||||
runs without error against a Postgres throwaway database. Skipped when no
|
||||
Postgres is reachable (the migrations rely on PG-specific types and cannot
|
||||
target SQLite).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import pytest
|
||||
|
||||
BACKEND_ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_alembic_upgrade_head_roundtrip():
|
||||
dsn = os.environ.get("TEST_DATABASE_URL") or os.environ.get("DATABASE_URL")
|
||||
assert dsn, "TEST_DATABASE_URL or DATABASE_URL must be set"
|
||||
|
||||
env = os.environ.copy()
|
||||
env["DATABASE_URL"] = dsn
|
||||
|
||||
# The session fixture has already upgraded; downgrade then re-upgrade to
|
||||
# exercise both paths inside this test without polluting the rest of the
|
||||
# session schema state.
|
||||
down = subprocess.run(
|
||||
["alembic", "downgrade", "base"],
|
||||
cwd=str(BACKEND_ROOT),
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert down.returncode == 0, f"downgrade failed: {down.stderr}"
|
||||
|
||||
up = subprocess.run(
|
||||
["alembic", "upgrade", "head"],
|
||||
cwd=str(BACKEND_ROOT),
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert up.returncode == 0, f"upgrade failed: {up.stderr}"
|
||||
@@ -0,0 +1,237 @@
|
||||
"""
|
||||
R2-B: approval-token + per-voter vote round-trip tests.
|
||||
|
||||
- Pure unit tests for `app.services.approval` (no DB).
|
||||
- DB-backed tests for `consume_token` single-use + the two vote routes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
import uuid
|
||||
from datetime import date, timedelta
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
from app.services import approval as approval_service
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure-unit token tests.
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_issue_and_verify_token_roundtrip():
|
||||
item_id = uuid.uuid4()
|
||||
voter_id = uuid.uuid4()
|
||||
token = approval_service.issue_token(item_id, voter_id)
|
||||
payload = approval_service.verify_token(token)
|
||||
assert payload["item"] == str(item_id)
|
||||
assert payload["voter"] == str(voter_id)
|
||||
|
||||
|
||||
def test_verify_rejects_tampered_token():
|
||||
item_id = uuid.uuid4()
|
||||
voter_id = uuid.uuid4()
|
||||
token = approval_service.issue_token(item_id, voter_id)
|
||||
tampered = token[:-2] + ("AA" if not token.endswith("AA") else "BB")
|
||||
with pytest.raises(HTTPException) as excinfo:
|
||||
approval_service.verify_token(tampered)
|
||||
assert excinfo.value.status_code == 401
|
||||
|
||||
|
||||
def test_verify_rejects_expired_token(monkeypatch):
|
||||
"""itsdangerous reads `time.time()` — patch the time module's `time`
|
||||
attribute on `itsdangerous.timed` to simulate clock advance.
|
||||
"""
|
||||
import itsdangerous.timed as _timed_mod
|
||||
|
||||
item_id = uuid.uuid4()
|
||||
voter_id = uuid.uuid4()
|
||||
|
||||
# Issue at "now".
|
||||
token = approval_service.issue_token(item_id, voter_id)
|
||||
|
||||
# Advance the clock 8 days past issue time (default TTL is 7 days).
|
||||
real_now = time.time()
|
||||
fake_now = real_now + (8 * 24 * 3600)
|
||||
|
||||
class _FakeTime:
|
||||
@staticmethod
|
||||
def time():
|
||||
return fake_now
|
||||
|
||||
monkeypatch.setattr(_timed_mod, "time", _FakeTime)
|
||||
|
||||
with pytest.raises(HTTPException) as excinfo:
|
||||
approval_service.verify_token(token)
|
||||
assert excinfo.value.status_code == 401
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB-backed fixtures: build a minimal scenario reused by several tests.
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.fixture()
|
||||
def scenario(db):
|
||||
"""Create profile + 2 voters + recipe + plan + item, all in one shot.
|
||||
|
||||
Uses the per-test transactional session, so changes roll back on
|
||||
teardown (no cross-test pollution).
|
||||
"""
|
||||
from app.models import (
|
||||
FamilyMember,
|
||||
FamilyMemberRole,
|
||||
FamilyProfile,
|
||||
MealPlan,
|
||||
MealPlanItem,
|
||||
MealPlanStatus,
|
||||
MealType,
|
||||
Recipe,
|
||||
)
|
||||
|
||||
suffix = uuid.uuid4().hex[:8]
|
||||
|
||||
profile = FamilyProfile(
|
||||
name=f"Test Family {suffix}",
|
||||
household_size=2,
|
||||
adult_count=2,
|
||||
child_count=0,
|
||||
)
|
||||
db.add(profile)
|
||||
db.flush()
|
||||
|
||||
voter_a = FamilyMember(
|
||||
family_profile_id=profile.id,
|
||||
name="Alice",
|
||||
email=f"alice+{suffix}@example.com",
|
||||
role=FamilyMemberRole.ADULT,
|
||||
)
|
||||
voter_b = FamilyMember(
|
||||
family_profile_id=profile.id,
|
||||
name="Bob",
|
||||
email=f"bob+{suffix}@example.com",
|
||||
role=FamilyMemberRole.ADULT,
|
||||
)
|
||||
db.add_all([voter_a, voter_b])
|
||||
db.flush()
|
||||
|
||||
recipe = Recipe(
|
||||
family_profile_id=profile.id,
|
||||
name=f"Test Pasta {suffix}",
|
||||
servings=2,
|
||||
ingredients=[{"name": "pasta", "qty": "200g"}],
|
||||
instructions=["Boil", "Drain"],
|
||||
is_manually_added=True,
|
||||
)
|
||||
db.add(recipe)
|
||||
db.flush()
|
||||
|
||||
plan = MealPlan(
|
||||
family_profile_id=profile.id,
|
||||
week_start_date=date.today() + timedelta(days=14),
|
||||
status=MealPlanStatus.DRAFT,
|
||||
)
|
||||
db.add(plan)
|
||||
db.flush()
|
||||
|
||||
item = MealPlanItem(
|
||||
meal_plan_id=plan.id,
|
||||
recipe_id=recipe.id,
|
||||
day_of_week=1,
|
||||
meal_type=MealType.DINNER,
|
||||
)
|
||||
db.add(item)
|
||||
db.flush()
|
||||
|
||||
return {
|
||||
"profile": profile,
|
||||
"voter_a": voter_a,
|
||||
"voter_b": voter_b,
|
||||
"recipe": recipe,
|
||||
"plan": plan,
|
||||
"item": item,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB-backed tests.
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.requires_postgres
|
||||
def test_consume_token_single_use(db, scenario):
|
||||
from app.models import MealPlanVote
|
||||
|
||||
item = scenario["item"]
|
||||
voter = scenario["voter_a"]
|
||||
token = approval_service.issue_token(item.id, voter.id)
|
||||
|
||||
# First call: succeeds, returns the voter.
|
||||
out = approval_service.consume_token(db, token, item.id)
|
||||
assert out.id == voter.id
|
||||
|
||||
# Simulate the route writing the vote row (consume_token itself does NOT
|
||||
# write — single-use is enforced by the presence of the vote row).
|
||||
db.add(MealPlanVote(
|
||||
meal_plan_item_id=item.id,
|
||||
family_member_id=voter.id,
|
||||
vote=True,
|
||||
))
|
||||
db.flush()
|
||||
|
||||
# Second call: raises 409.
|
||||
with pytest.raises(HTTPException) as excinfo:
|
||||
approval_service.consume_token(db, token, item.id)
|
||||
assert excinfo.value.status_code == 409
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_vote_get_renders_html(client, db, scenario):
|
||||
item = scenario["item"]
|
||||
voter = scenario["voter_a"]
|
||||
token = approval_service.issue_token(item.id, voter.id)
|
||||
|
||||
r = client.get(f"/api/meals/vote/{item.id}", params={"token": token})
|
||||
assert r.status_code == 200, r.text
|
||||
assert "text/html" in r.headers.get("content-type", "")
|
||||
# Voter name appears in the visible body.
|
||||
assert "Alice" in r.text
|
||||
# Token is allowed inside the form `action` attribute (the link's href)
|
||||
# but must not appear in the visible meal-card text.
|
||||
body = r.text
|
||||
card_start = body.find('<div class="meal">')
|
||||
card_end = body.find("</div>", card_start)
|
||||
assert card_start != -1 and card_end != -1
|
||||
visible_card = body[card_start:card_end]
|
||||
assert token not in visible_card
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_vote_post_records_and_decides(client, db, scenario):
|
||||
"""First voter approves -> still pending (Bob hasn't voted).
|
||||
|
||||
Then Bob approves -> approved.
|
||||
"""
|
||||
item = scenario["item"]
|
||||
voter_a = scenario["voter_a"]
|
||||
voter_b = scenario["voter_b"]
|
||||
|
||||
token_a = approval_service.issue_token(item.id, voter_a.id)
|
||||
r1 = client.post(
|
||||
f"/api/meals/vote/{item.id}",
|
||||
params={"token": token_a},
|
||||
json={"vote": "approve"},
|
||||
)
|
||||
assert r1.status_code == 200, r1.text
|
||||
body1 = r1.json()
|
||||
assert body1["status"] == "recorded"
|
||||
assert body1["item_status"] in ("pending", "approved")
|
||||
# With 2 voters, after 1 approve we should still be pending.
|
||||
assert body1["item_status"] == "pending"
|
||||
|
||||
token_b = approval_service.issue_token(item.id, voter_b.id)
|
||||
r2 = client.post(
|
||||
f"/api/meals/vote/{item.id}",
|
||||
params={"token": token_b},
|
||||
json={"vote": "approve"},
|
||||
)
|
||||
assert r2.status_code == 200, r2.text
|
||||
body2 = r2.json()
|
||||
assert body2["item_status"] == "approved"
|
||||
@@ -0,0 +1,114 @@
|
||||
"""
|
||||
Auth gate tests for R1-B+D.
|
||||
|
||||
Covers:
|
||||
- /api/admin/* requires bearer token (401 without, not 401 with valid).
|
||||
- Mutating routes on family routers require a session cookie.
|
||||
- /api/auth/login + /api/auth/logout round-trip with the shared password.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
# Configure auth secrets BEFORE app import. conftest.py runs first and sets
|
||||
# DATABASE_URL; we layer auth env on top here.
|
||||
os.environ.setdefault("ADMIN_TOKEN", "test-admin-token")
|
||||
os.environ.setdefault("SESSION_PASSWORD", "test-family-password")
|
||||
os.environ.setdefault("SECRET_KEY", "test-secret-key-do-not-use-in-prod")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reload_settings(monkeypatch):
|
||||
"""Force ``settings`` to re-read env (test isolation)."""
|
||||
monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token")
|
||||
monkeypatch.setenv("SESSION_PASSWORD", "test-family-password")
|
||||
monkeypatch.setenv("SECRET_KEY", "test-secret-key-do-not-use-in-prod")
|
||||
# Re-instantiate the singleton so dependents pick up env.
|
||||
from app import config as app_config
|
||||
|
||||
app_config.settings = app_config.Settings()
|
||||
yield
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Admin bearer token
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.requires_postgres
|
||||
def test_admin_requires_token(client):
|
||||
"""No bearer → 401. Valid bearer → not 401 (handler runs)."""
|
||||
r = client.post("/api/admin/scrape")
|
||||
assert r.status_code == 401, r.text
|
||||
|
||||
r = client.post(
|
||||
"/api/admin/scrape",
|
||||
headers={"Authorization": "Bearer test-admin-token"},
|
||||
)
|
||||
# Handler may 200/202/500 (Playwright not installed in CI), but NOT 401.
|
||||
assert r.status_code != 401, r.text
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_admin_logs_requires_token(client):
|
||||
r = client.get("/api/admin/logs")
|
||||
assert r.status_code == 401
|
||||
|
||||
r = client.get(
|
||||
"/api/admin/logs", headers={"Authorization": "Bearer test-admin-token"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Session-gated mutations
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.requires_postgres
|
||||
def test_session_required_for_mutation(client):
|
||||
"""POST /api/profile/members without cookie → 401."""
|
||||
r = client.post(
|
||||
"/api/profile/members",
|
||||
json={"name": "x", "email": "x@y.z", "role": "voter"},
|
||||
)
|
||||
assert r.status_code == 401, r.text
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_session_open_for_reads(client):
|
||||
"""GET /api/profile is NOT auth-gated (reads stay open)."""
|
||||
r = client.get("/api/profile")
|
||||
# Either 200 (profile exists) or 404 (no profile yet) — never 401.
|
||||
assert r.status_code in (200, 404), r.text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Login / logout
|
||||
# ---------------------------------------------------------------------------
|
||||
@pytest.mark.requires_postgres
|
||||
def test_session_login_logout(client):
|
||||
# Wrong password
|
||||
r = client.post("/api/auth/login", json={"password": "nope"})
|
||||
assert r.status_code == 401
|
||||
|
||||
# Right password sets the cookie
|
||||
r = client.post(
|
||||
"/api/auth/login", json={"password": "test-family-password"}
|
||||
)
|
||||
assert r.status_code == 204
|
||||
assert "mp_session" in r.cookies, r.headers
|
||||
|
||||
# With cookie, mutation succeeds (or fails for non-auth reasons)
|
||||
cookie_value = r.cookies.get("mp_session")
|
||||
client.cookies.set("mp_session", cookie_value)
|
||||
r2 = client.post(
|
||||
"/api/profile/members",
|
||||
json={"name": "x", "email": "test@example.com", "role": "voter"},
|
||||
)
|
||||
# 401 means the cookie was rejected — that's the bug we're guarding.
|
||||
assert r2.status_code != 401, r2.text
|
||||
|
||||
# Logout clears the cookie
|
||||
r3 = client.post("/api/auth/logout")
|
||||
assert r3.status_code == 204
|
||||
@@ -0,0 +1,19 @@
|
||||
"""
|
||||
Config fail-fast: importing app.config with DATABASE_URL unset must raise.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def test_database_url_required(monkeypatch):
|
||||
"""Settings() with empty DATABASE_URL → RuntimeError at instantiation."""
|
||||
from app import config as app_config
|
||||
|
||||
monkeypatch.setenv("DATABASE_URL", "")
|
||||
# Pass _env_file=None so a stray .env on disk can't satisfy the field.
|
||||
with pytest.raises(RuntimeError, match="DATABASE_URL is required"):
|
||||
app_config.Settings(_env_file=None, DATABASE_URL="")
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Async scrape endpoint contract.
|
||||
|
||||
Verifies that ``POST /api/admin/scrape``:
|
||||
- returns 202 + ``scrape_log_id`` synchronously,
|
||||
- persists a ``ScrapeLog`` row in status STARTED before the background task
|
||||
runs (the task is monkey-patched out so it never reaches Playwright and
|
||||
never opens a session outside the test transaction).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
|
||||
os.environ.setdefault("ADMIN_TOKEN", "test-admin-token")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _admin_token(monkeypatch):
|
||||
monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token")
|
||||
yield
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_scrape_returns_202_and_log_id(client, db, monkeypatch):
|
||||
"""Endpoint enqueues the scrape and returns 202 + scrape_log_id.
|
||||
|
||||
We replace the background runner with a no-op so the test does NOT spin up
|
||||
Playwright and does NOT open a session outside the rolled-back test
|
||||
transaction.
|
||||
"""
|
||||
calls: list[tuple] = []
|
||||
|
||||
def _fake_run(log_id, source, scrape_type):
|
||||
calls.append((log_id, source, scrape_type))
|
||||
|
||||
# Patch in BOTH the service module (definition site) and the api module
|
||||
# (import site) so whichever symbol the route resolved to is replaced.
|
||||
monkeypatch.setattr(
|
||||
"app.services.scraper_service._run_scrape_in_background", _fake_run
|
||||
)
|
||||
|
||||
r = client.post(
|
||||
"/api/admin/scrape",
|
||||
headers={"Authorization": "Bearer test-admin-token"},
|
||||
)
|
||||
assert r.status_code == 202, r.text
|
||||
body = r.json()
|
||||
assert body["status"] == "queued"
|
||||
assert "scrape_log_id" in body
|
||||
|
||||
log_id = uuid.UUID(body["scrape_log_id"])
|
||||
|
||||
# Row was committed inside enqueue_scrape — visible on the test session.
|
||||
from app.models import ScrapeLog, ScrapeStatus
|
||||
|
||||
log = db.query(ScrapeLog).filter(ScrapeLog.id == log_id).first()
|
||||
assert log is not None, "ScrapeLog row should exist after enqueue"
|
||||
assert log.status == ScrapeStatus.STARTED
|
||||
assert log.source == "lucky_california"
|
||||
assert log.scrape_type == "weekly_ad"
|
||||
assert log.completed_at is None
|
||||
|
||||
# TestClient runs background tasks before returning from the context
|
||||
# manager exit — by the time we get here, the fake runner ran exactly once.
|
||||
assert len(calls) == 1
|
||||
assert calls[0][0] == log_id
|
||||
@@ -0,0 +1,63 @@
|
||||
"""
|
||||
Smoke tests: app boots, routers wire up, no import-time crashes.
|
||||
|
||||
The router-list test asserts each endpoint returns 200 or 401 (auth gate not
|
||||
yet implemented in R1-A's scope) but explicitly NOT 5xx — the goal is to catch
|
||||
import errors and crashing handlers, not to validate business logic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import pytest
|
||||
|
||||
|
||||
def test_app_imports():
|
||||
from app.main import app
|
||||
assert app.title == "MealPlanner"
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_health(client):
|
||||
r = client.get("/health")
|
||||
assert r.status_code == 200
|
||||
assert r.json() == {"status": "ok"}
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_health_db(client):
|
||||
r = client.get("/health/db")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body.get("database") == "connected"
|
||||
|
||||
|
||||
ROUTER_GET_PATHS = [
|
||||
"/api/profile",
|
||||
"/api/profile/members",
|
||||
"/api/recipes",
|
||||
"/api/recipes/ingredients",
|
||||
"/api/meals",
|
||||
"/api/pantry",
|
||||
"/api/shopping-list",
|
||||
"/api/admin/logs",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
@pytest.mark.parametrize("path", ROUTER_GET_PATHS)
|
||||
def test_router_list_endpoints(client, path):
|
||||
"""
|
||||
Each canonical GET must respond. 200 (handled), 401 (admin-gated),
|
||||
404 (handler ran but no row found) — all OK. 5xx means import/runtime
|
||||
crash; 307 (trailing-slash redirect) means a handler is still mounted
|
||||
on the wrong path.
|
||||
"""
|
||||
r = client.get(path, follow_redirects=False)
|
||||
assert r.status_code < 500, (
|
||||
f"{path} returned {r.status_code}: {r.text[:300]}"
|
||||
)
|
||||
# 307 is a regression — canonical paths must be the route definitions.
|
||||
assert r.status_code in (200, 401, 404, 422), (
|
||||
f"{path} returned unexpected {r.status_code}"
|
||||
)
|
||||
@@ -0,0 +1,223 @@
|
||||
"""Offline tests for the Swiftly product-API client (R3-0).
|
||||
|
||||
All tests run against saved fixtures and mocked HTTP — no live network,
|
||||
no Playwright/Chromium. Captured 2026-05-05 from a single live spike;
|
||||
see ``.agent/context.md`` "Swiftly API" for the field-mapping rationale.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import uuid
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
# Make `app.*` importable when pytest is invoked from the repo root.
|
||||
BACKEND_DIR = Path(__file__).resolve().parent.parent
|
||||
if str(BACKEND_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(BACKEND_DIR))
|
||||
|
||||
from app.scraper.lucky_ca_scraper import ( # noqa: E402
|
||||
LuckyCaliforniaScraper,
|
||||
SwiftlyAuthError,
|
||||
)
|
||||
|
||||
FIXTURE_DIR = Path(__file__).resolve().parent / "fixtures" / "lucky_ca"
|
||||
CATEGORIES_HTML = FIXTURE_DIR / "categories.html"
|
||||
CATEGORY_JSON = FIXTURE_DIR / "category_meat_seafood.json"
|
||||
|
||||
|
||||
pytestmark = pytest.mark.scraper_offline
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pure-parser tests against captured fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
def test_parse_categories_fixture() -> None:
|
||||
"""Parser returns >=10 distinct API slugs from the captured page."""
|
||||
if not CATEGORIES_HTML.exists():
|
||||
pytest.skip(f"fixture missing: {CATEGORIES_HTML}")
|
||||
html = CATEGORIES_HTML.read_text(encoding="utf-8")
|
||||
|
||||
slugs = LuckyCaliforniaScraper.parse_categories_html(html)
|
||||
|
||||
assert len(slugs) >= 10, f"expected >=10 categories, got {len(slugs)}"
|
||||
assert len(set(slugs)) == len(slugs), "slugs must be deduplicated"
|
||||
# Every slug should look like `Product/<name>` per the API contract.
|
||||
for s in slugs:
|
||||
assert s.startswith("Product/"), f"unexpected slug shape: {s!r}"
|
||||
# Spot-check the one we know is in the captured snapshot.
|
||||
assert "Product/meat_seafood" in slugs
|
||||
|
||||
|
||||
def test_parse_category_response_fixture() -> None:
|
||||
"""Parser returns >=10 product dicts each with the mapped fields populated."""
|
||||
if not CATEGORY_JSON.exists():
|
||||
pytest.skip(f"fixture missing: {CATEGORY_JSON}")
|
||||
payload = json.loads(CATEGORY_JSON.read_text(encoding="utf-8"))
|
||||
|
||||
raw_items = LuckyCaliforniaScraper.parse_category_response(payload)
|
||||
assert len(raw_items) >= 10, f"expected >=10 raw items, got {len(raw_items)}"
|
||||
|
||||
mapped: list[dict] = []
|
||||
for raw in raw_items:
|
||||
m = LuckyCaliforniaScraper.map_product(
|
||||
raw, aisle="meat_seafood", source_slug="Product/meat_seafood"
|
||||
)
|
||||
if m is not None:
|
||||
mapped.append(m)
|
||||
|
||||
assert len(mapped) >= 10, (
|
||||
f"expected >=10 mapped products, got {len(mapped)} "
|
||||
f"(from {len(raw_items)} raw)"
|
||||
)
|
||||
|
||||
sample = mapped[0]
|
||||
# Required fields per the field-mapping contract.
|
||||
for key in (
|
||||
"external_id",
|
||||
"source",
|
||||
"name",
|
||||
"current_price",
|
||||
"regular_price",
|
||||
"is_on_sale",
|
||||
"image_url",
|
||||
"aisle",
|
||||
):
|
||||
assert key in sample, f"missing key {key!r} in mapped product: {sample!r}"
|
||||
|
||||
assert sample["source"] == "lucky_california"
|
||||
assert sample["aisle"] == "meat_seafood"
|
||||
assert isinstance(sample["external_id"], str) and sample["external_id"]
|
||||
assert isinstance(sample["name"], str) and sample["name"].strip()
|
||||
assert isinstance(sample["regular_price"], Decimal)
|
||||
assert sample["regular_price"] > 0
|
||||
assert isinstance(sample["is_on_sale"], bool)
|
||||
|
||||
# Across the whole category at least SOME items should be on sale and
|
||||
# at least some should have a regular-only price (sanity for the parser).
|
||||
assert any(m["is_on_sale"] for m in mapped), "expected at least one sale item"
|
||||
assert any(not m["is_on_sale"] for m in mapped), "expected at least one reg-only item"
|
||||
|
||||
|
||||
def test_map_product_returns_none_for_unparseable() -> None:
|
||||
"""Products with no name AND no parseable price are dropped."""
|
||||
assert LuckyCaliforniaScraper.map_product({"name": ""}) is None
|
||||
assert (
|
||||
LuckyCaliforniaScraper.map_product(
|
||||
{"id": "x", "name": "Foo", "price": {"ok": {}}}
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 401 → SwiftlyAuthError → FAILED ScrapeLog
|
||||
# ---------------------------------------------------------------------------
|
||||
def _mock_response(status_code: int, payload=None) -> MagicMock:
|
||||
resp = MagicMock(spec=requests.Response)
|
||||
resp.status_code = status_code
|
||||
if payload is not None:
|
||||
resp.json.return_value = payload
|
||||
if status_code >= 400:
|
||||
resp.raise_for_status.side_effect = requests.HTTPError(
|
||||
f"{status_code} error", response=resp
|
||||
)
|
||||
else:
|
||||
resp.raise_for_status.return_value = None
|
||||
return resp
|
||||
|
||||
|
||||
def test_swiftly_auth_error_on_401_from_api() -> None:
|
||||
"""A 401 from the API host raises SwiftlyAuthError before raise_for_status."""
|
||||
scraper = LuckyCaliforniaScraper(bearer_token="stale-token")
|
||||
|
||||
with patch.object(scraper.api_session, "get", return_value=_mock_response(401)):
|
||||
with pytest.raises(SwiftlyAuthError) as excinfo:
|
||||
scraper.fetch_category("Product/meat_seafood")
|
||||
|
||||
assert "SWIFTLY_BEARER_TOKEN expired" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_swiftly_auth_error_when_token_missing() -> None:
|
||||
"""An empty token short-circuits to SwiftlyAuthError without any HTTP call.
|
||||
|
||||
Force the token empty AFTER construction so the test is independent of
|
||||
whatever ``SWIFTLY_BEARER_TOKEN`` happens to be set in the environment
|
||||
(it WILL be set when pytest runs inside ``docker compose``).
|
||||
"""
|
||||
scraper = LuckyCaliforniaScraper()
|
||||
scraper.bearer_token = ""
|
||||
with patch.object(scraper.api_session, "get") as mock_get:
|
||||
with pytest.raises(SwiftlyAuthError):
|
||||
scraper.fetch_category("Product/meat_seafood")
|
||||
mock_get.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.requires_postgres
|
||||
def test_background_runner_writes_failed_with_token_message(monkeypatch):
|
||||
"""A 401 during the background scrape lands in ScrapeLog as FAILED + message.
|
||||
|
||||
Uses a real (non-fixture) session so the bg runner's rollback+re-query
|
||||
path mirrors production. The bg runner commits the FAILED row; we clean
|
||||
up explicitly at the end.
|
||||
"""
|
||||
from app.models import ScrapeLog, ScrapeStatus
|
||||
from app.services import scraper_service
|
||||
from app.scraper.lucky_ca_scraper import SwiftlyAuthError, LuckyCaliforniaScraper
|
||||
from app.database import SessionLocal
|
||||
from datetime import datetime, timezone
|
||||
|
||||
log_id = uuid.uuid4()
|
||||
setup_session = SessionLocal()
|
||||
try:
|
||||
setup_session.add(
|
||||
ScrapeLog(
|
||||
id=log_id,
|
||||
source="lucky_california",
|
||||
scrape_type="weekly_ad",
|
||||
status=ScrapeStatus.STARTED,
|
||||
started_at=datetime.now(timezone.utc),
|
||||
)
|
||||
)
|
||||
setup_session.commit()
|
||||
finally:
|
||||
setup_session.close()
|
||||
|
||||
def _explode(self):
|
||||
raise SwiftlyAuthError(
|
||||
"SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user "
|
||||
"(capture from luckysupermarkets.com network tab on a /search/api/v1 request)"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(LuckyCaliforniaScraper, "fetch_all", _explode)
|
||||
|
||||
try:
|
||||
scraper_service._run_scrape_in_background(
|
||||
log_id, "lucky_california", "weekly_ad"
|
||||
)
|
||||
|
||||
verify_session = SessionLocal()
|
||||
try:
|
||||
refreshed = (
|
||||
verify_session.query(ScrapeLog)
|
||||
.filter(ScrapeLog.id == log_id)
|
||||
.first()
|
||||
)
|
||||
assert refreshed is not None
|
||||
assert refreshed.status == ScrapeStatus.FAILED
|
||||
assert "SWIFTLY_BEARER_TOKEN expired" in (refreshed.error_message or "")
|
||||
assert refreshed.completed_at is not None
|
||||
finally:
|
||||
verify_session.close()
|
||||
finally:
|
||||
cleanup_session = SessionLocal()
|
||||
try:
|
||||
cleanup_session.query(ScrapeLog).filter(ScrapeLog.id == log_id).delete()
|
||||
cleanup_session.commit()
|
||||
finally:
|
||||
cleanup_session.close()
|
||||
@@ -11,9 +11,16 @@ services:
|
||||
- DATABASE_URL=postgresql://mealplanner:${POSTGRES_PASSWORD}@db:5432/mealplanner
|
||||
- SENDGRID_API_KEY=${SENDGRID_API_KEY}
|
||||
- LUCKY_CA_URL=${LUCKY_CA_URL:-https://luckysupermarkets.com}
|
||||
- LUCKY_STORE_ID=${LUCKY_STORE_ID:-757}
|
||||
- SWIFTLY_API_BASE=${SWIFTLY_API_BASE:-https://prod.swiftlyapi.net}
|
||||
- SWIFTLY_CATEGORIES_URL=${SWIFTLY_CATEGORIES_URL:-https://luckysupermarkets.com/categories}
|
||||
- SWIFTLY_BEARER_TOKEN=${SWIFTLY_BEARER_TOKEN}
|
||||
- AI_IMAGE_ENABLED=${AI_IMAGE_ENABLED:-false}
|
||||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||||
- SECRET_KEY=${SECRET_KEY}
|
||||
- ADMIN_TOKEN=${ADMIN_TOKEN}
|
||||
- SESSION_PASSWORD=${SESSION_PASSWORD}
|
||||
- EMAIL_BACKEND=${EMAIL_BACKEND:-console}
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
|
||||
@@ -7,9 +7,15 @@ const api = axios.create({
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
withCredentials: true,
|
||||
})
|
||||
|
||||
export const mealPlannerApi = {
|
||||
auth: {
|
||||
login: (password: string) => api.post('/auth/login', { password }),
|
||||
logout: () => api.post('/auth/logout'),
|
||||
},
|
||||
|
||||
profile: {
|
||||
get: () => api.get('/profile'),
|
||||
update: (data: any) => api.put('/profile', data),
|
||||
@@ -23,12 +29,12 @@ export const mealPlannerApi = {
|
||||
get: (id: string) => api.get(`/recipes/${id}`),
|
||||
create: (data: any) => api.post('/recipes', data),
|
||||
delete: (id: string) => api.delete(`/recipes/${id}`),
|
||||
listIngredients: () => api.get('/recipes/ingredients/list'),
|
||||
listIngredients: () => api.get('/recipes/ingredients'),
|
||||
createIngredient: (data: any) => api.post('/recipes/ingredients', data),
|
||||
},
|
||||
|
||||
meals: {
|
||||
getPlanned: () => api.get('/meals/planned'),
|
||||
getPlanned: () => api.get('/meals'),
|
||||
get: (id: string) => api.get(`/meals/${id}`),
|
||||
getItem: (id: string) => api.get(`/meals/items/${id}`),
|
||||
create: (data: any) => api.post('/meals', data),
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
R2-B end-to-end approval round-trip spike.
|
||||
|
||||
Usage:
|
||||
# Print URL only (you can click it in a browser running the server):
|
||||
python scripts/send_test_approval.py
|
||||
|
||||
# Bypass the email and exercise the full POST against a TestClient:
|
||||
python scripts/send_test_approval.py --simulate-click approve
|
||||
python scripts/send_test_approval.py --simulate-click deny
|
||||
|
||||
This is the actual proof that the email + per-voter approval click
|
||||
round-trip works against the real schema (`family_member`,
|
||||
`meal_plan_item`, `meal_plan_vote`). If the script exits 0 with
|
||||
`item_status=approved` (or `denied`) the spike is green.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
from datetime import date, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
# Make backend/ importable when invoked from repo root.
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
_BACKEND_ROOT = _REPO_ROOT / "backend"
|
||||
if str(_BACKEND_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_BACKEND_ROOT))
|
||||
|
||||
# DATABASE_URL must be set before importing app.config.
|
||||
os.environ.setdefault(
|
||||
"DATABASE_URL",
|
||||
os.environ.get(
|
||||
"TEST_DATABASE_URL",
|
||||
"postgresql://mealplanner:password@localhost:5432/mealplanner_test",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--simulate-click",
|
||||
choices=["approve", "deny"],
|
||||
help="Bypass email; POST the vote via TestClient and assert success.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--base-url",
|
||||
default="http://localhost:8000",
|
||||
help="Base URL for the printed link (informational only).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from app.config import settings
|
||||
from app.models import (
|
||||
FamilyMember,
|
||||
FamilyMemberRole,
|
||||
FamilyProfile,
|
||||
MealPlan,
|
||||
MealPlanItem,
|
||||
MealPlanStatus,
|
||||
MealType,
|
||||
Recipe,
|
||||
)
|
||||
from app.services import approval as approval_service
|
||||
from app.services.email import get_email_backend
|
||||
|
||||
engine = create_engine(
|
||||
settings.DATABASE_URL.replace("postgresql://", "postgresql+psycopg2://")
|
||||
)
|
||||
Session = sessionmaker(bind=engine, autoflush=False, autocommit=False)
|
||||
db = Session()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Bootstrap a minimal scenario. Suffix everything with a UUID4 so the
|
||||
# script is idempotent and can run repeatedly without unique-constraint
|
||||
# collisions.
|
||||
# ------------------------------------------------------------------
|
||||
suffix = uuid.uuid4().hex[:8]
|
||||
|
||||
profile = FamilyProfile(
|
||||
name=f"Spike Family {suffix}",
|
||||
household_size=1,
|
||||
adult_count=1,
|
||||
child_count=0,
|
||||
)
|
||||
db.add(profile)
|
||||
db.flush()
|
||||
|
||||
voter = FamilyMember(
|
||||
family_profile_id=profile.id,
|
||||
name="Spike Voter",
|
||||
email=f"spike+{suffix}@example.com",
|
||||
role=FamilyMemberRole.ADULT,
|
||||
)
|
||||
db.add(voter)
|
||||
db.flush()
|
||||
|
||||
recipe = Recipe(
|
||||
family_profile_id=profile.id,
|
||||
name=f"Spike Pasta {suffix}",
|
||||
servings=2,
|
||||
ingredients=[{"name": "pasta", "qty": "200g"}],
|
||||
instructions=["Boil water", "Cook pasta"],
|
||||
is_manually_added=True,
|
||||
)
|
||||
db.add(recipe)
|
||||
db.flush()
|
||||
|
||||
plan = MealPlan(
|
||||
family_profile_id=profile.id,
|
||||
week_start_date=date.today() + timedelta(days=(7 - date.today().weekday())),
|
||||
status=MealPlanStatus.DRAFT,
|
||||
)
|
||||
db.add(plan)
|
||||
db.flush()
|
||||
|
||||
item = MealPlanItem(
|
||||
meal_plan_id=plan.id,
|
||||
recipe_id=recipe.id,
|
||||
day_of_week=1,
|
||||
meal_type=MealType.DINNER,
|
||||
)
|
||||
db.add(item)
|
||||
db.commit()
|
||||
|
||||
token = approval_service.issue_token(item.id, voter.id)
|
||||
url = f"{args.base_url}/api/meals/vote/{item.id}?token={token}"
|
||||
|
||||
# Send via configured email backend (Console writes to stdout + outbox).
|
||||
backend = get_email_backend()
|
||||
subject = "Action required: please vote on this week's meal"
|
||||
html = (
|
||||
f"<p>Hi {voter.name}, please review this meal:</p>"
|
||||
f"<p><strong>{recipe.name}</strong></p>"
|
||||
f'<p><a href="{url}">Open the approval page</a></p>'
|
||||
)
|
||||
text = f"Hi {voter.name}, please review {recipe.name}: {url}"
|
||||
backend.send(to=voter.email, subject=subject, html=html, text=text)
|
||||
|
||||
print(f"item_id={item.id}")
|
||||
print(f"voter_id={voter.id}")
|
||||
print(f"approval_url={url}")
|
||||
|
||||
if args.simulate_click is None:
|
||||
return 0
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# End-to-end proof: drive the POST through a TestClient.
|
||||
# ------------------------------------------------------------------
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.main import app
|
||||
|
||||
with TestClient(app) as client:
|
||||
# Render the GET page first to mirror a real click.
|
||||
r_get = client.get(f"/api/meals/vote/{item.id}", params={"token": token})
|
||||
assert r_get.status_code == 200, (r_get.status_code, r_get.text)
|
||||
assert "text/html" in r_get.headers.get("content-type", "")
|
||||
|
||||
r_post = client.post(
|
||||
f"/api/meals/vote/{item.id}",
|
||||
params={"token": token},
|
||||
json={"vote": args.simulate_click},
|
||||
)
|
||||
assert r_post.status_code == 200, (r_post.status_code, r_post.text)
|
||||
body = r_post.json()
|
||||
print(f"item_status={body['item_status']}")
|
||||
assert body["status"] == "recorded"
|
||||
|
||||
# Single-use: a second POST must 409.
|
||||
r_post2 = client.post(
|
||||
f"/api/meals/vote/{item.id}",
|
||||
params={"token": token},
|
||||
json={"vote": args.simulate_click},
|
||||
)
|
||||
assert r_post2.status_code == 409, (r_post2.status_code, r_post2.text)
|
||||
print("single_use=enforced")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,169 @@
|
||||
"""
|
||||
R2-A live-scrape spike for Lucky California weekly ad.
|
||||
|
||||
Performs ONE live fetch of https://luckysupermarkets.com weekly-ad page,
|
||||
saves the rendered HTML, a full-page screenshot, and a META.md describing
|
||||
the fetch result. This is the deferred-risk spike demanded by review §2.4.
|
||||
|
||||
Run once. Do not loop. Polite citizen: identifying user-agent, single
|
||||
request, no auth bypass attempts.
|
||||
|
||||
Usage:
|
||||
python scripts/spike_lucky_scrape.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
# Path to backend/app on import path so we can reuse the parser.
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO_ROOT / "backend"))
|
||||
|
||||
from app.scraper.lucky_ca_scraper import LuckyCaliforniaScraper # noqa: E402
|
||||
from bs4 import BeautifulSoup # noqa: E402
|
||||
|
||||
FIXTURE_DIR = REPO_ROOT / "backend" / "tests" / "fixtures" / "lucky_ca"
|
||||
TARGET_URL = "https://luckysupermarkets.com/coupons/Coupon%2Flu-featured-in-ad"
|
||||
USER_AGENT = (
|
||||
"Mozilla/5.0 (compatible; MealPlannerSpike/0.1; "
|
||||
"+https://github.com/MealPlanner; spike=R2-A)"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
html_path = FIXTURE_DIR / "weekly_ad.html"
|
||||
png_path = FIXTURE_DIR / "weekly_ad.png"
|
||||
meta_path = FIXTURE_DIR / "META.md"
|
||||
|
||||
started = datetime.now(timezone.utc)
|
||||
print(f"[spike] Fetching {TARGET_URL}")
|
||||
|
||||
status_code: int | None = None
|
||||
final_url: str = TARGET_URL
|
||||
error: str | None = None
|
||||
captcha_or_block = False
|
||||
captcha_signal = ""
|
||||
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
try:
|
||||
ctx = browser.new_context(user_agent=USER_AGENT)
|
||||
page = ctx.new_page()
|
||||
|
||||
response = None
|
||||
|
||||
def _capture(resp):
|
||||
nonlocal response
|
||||
# Capture only the main document response.
|
||||
if response is None and resp.url.rstrip("/") == TARGET_URL.rstrip("/"):
|
||||
response = resp
|
||||
|
||||
page.on("response", _capture)
|
||||
|
||||
try:
|
||||
nav_resp = page.goto(
|
||||
TARGET_URL,
|
||||
wait_until="networkidle",
|
||||
timeout=45_000,
|
||||
)
|
||||
if nav_resp is not None:
|
||||
status_code = nav_resp.status
|
||||
final_url = nav_resp.url
|
||||
elif response is not None:
|
||||
status_code = response.status
|
||||
final_url = response.url
|
||||
except Exception as exc: # noqa: BLE001 -- spike, capture and report
|
||||
error = f"{type(exc).__name__}: {exc}"
|
||||
|
||||
try:
|
||||
html = page.content()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
html = ""
|
||||
error = (error or "") + f" content_err={exc}"
|
||||
|
||||
html_path.write_text(html, encoding="utf-8")
|
||||
|
||||
try:
|
||||
page.screenshot(path=str(png_path), full_page=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
error = (error or "") + f" screenshot_err={exc}"
|
||||
|
||||
# Heuristic captcha / block detection.
|
||||
lower = html.lower()
|
||||
for needle in (
|
||||
"captcha",
|
||||
"are you a robot",
|
||||
"access denied",
|
||||
"akamai",
|
||||
"cloudflare",
|
||||
"px-captcha",
|
||||
"perimeterx",
|
||||
"incapsula",
|
||||
):
|
||||
if needle in lower:
|
||||
captcha_or_block = True
|
||||
captcha_signal = needle
|
||||
break
|
||||
|
||||
finally:
|
||||
browser.close()
|
||||
|
||||
# Run parser portion against the captured HTML (no network).
|
||||
scraper = LuckyCaliforniaScraper()
|
||||
items: list[dict] = scraper.parse_featured_coupons_html(html)
|
||||
_ = BeautifulSoup # keep import for type-stability if reused later
|
||||
|
||||
completed = datetime.now(timezone.utc)
|
||||
|
||||
meta = f"""# Lucky California weekly-ad spike (R2-A)
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| URL | {TARGET_URL} |
|
||||
| Final URL | {final_url} |
|
||||
| Started (UTC) | {started.isoformat()} |
|
||||
| Completed (UTC) | {completed.isoformat()} |
|
||||
| HTTP status | {status_code} |
|
||||
| HTML bytes | {len(html)} |
|
||||
| Items parsed | {len(items)} |
|
||||
| Captcha/block signal | {"YES (" + captcha_signal + ")" if captcha_or_block else "no"} |
|
||||
| Error | {error or "none"} |
|
||||
| User-Agent | `{USER_AGENT}` |
|
||||
|
||||
## First parsed item (sample)
|
||||
|
||||
```json
|
||||
{__import__("json").dumps(items[0], indent=2) if items else "null"}
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Single live fetch performed. Do not rerun without reason.
|
||||
- HTML and PNG saved alongside this file.
|
||||
- Parser used: `LuckyCaliforniaScraper._parse_coupon_item` against
|
||||
elements matching `h2/h3/a` with `$N.NN` text.
|
||||
"""
|
||||
meta_path.write_text(meta, encoding="utf-8")
|
||||
|
||||
print(f"[spike] status={status_code} html_bytes={len(html)} items={len(items)} "
|
||||
f"block={captcha_or_block}")
|
||||
print(f"[spike] wrote: {html_path}")
|
||||
print(f"[spike] wrote: {png_path}")
|
||||
print(f"[spike] wrote: {meta_path}")
|
||||
|
||||
if captcha_or_block:
|
||||
print("[spike] WARNING: captcha/anti-bot signal detected; review META.md")
|
||||
if not items:
|
||||
print("[spike] WARNING: zero items parsed; selectors may be stale")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,101 @@
|
||||
"""Live spike for the Swiftly product API (R3-0, ad-hoc admin tool).
|
||||
|
||||
Usage:
|
||||
python scripts/spike_swiftly_ingest.py --confirm-live
|
||||
|
||||
Fetches the categories page + ONE category (default ``Product/meat_seafood``)
|
||||
against the live Swiftly API, prints the item count, and pretty-prints
|
||||
two sample mapped products. Does NOT persist anything to Postgres — the
|
||||
purpose is to verify the token + parser round-trip on demand without
|
||||
running the full scrape.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Make `app.*` importable when invoked from the repo root.
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
BACKEND_DIR = REPO_ROOT / "backend"
|
||||
if str(BACKEND_DIR) not in sys.path:
|
||||
sys.path.insert(0, str(BACKEND_DIR))
|
||||
|
||||
# Allow running outside docker without a real Postgres URL just to exercise
|
||||
# the scraper. Settings still requires DATABASE_URL to be non-empty.
|
||||
os.environ.setdefault("DATABASE_URL", "postgresql://placeholder@localhost:5432/placeholder")
|
||||
|
||||
from app.scraper.lucky_ca_scraper import LuckyCaliforniaScraper, SwiftlyAuthError # noqa: E402
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--confirm-live",
|
||||
action="store_true",
|
||||
help="Required: hit the live Swiftly API. Without this flag the script no-ops.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--category",
|
||||
default="Product/meat_seafood",
|
||||
help="API slug to fetch (default: Product/meat_seafood).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=2,
|
||||
help="How many sample mapped products to print (default 2).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.confirm_live:
|
||||
print(
|
||||
"Refusing to make live HTTP calls without --confirm-live.\n"
|
||||
"Pass --confirm-live to opt in.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
scraper = LuckyCaliforniaScraper()
|
||||
try:
|
||||
slugs = scraper.discover_categories()
|
||||
print(f"discovered {len(slugs)} categories")
|
||||
for s in slugs[:5]:
|
||||
print(f" - {s}")
|
||||
if len(slugs) > 5:
|
||||
print(f" ... ({len(slugs) - 5} more)")
|
||||
|
||||
try:
|
||||
raw_items = scraper.fetch_category(args.category)
|
||||
except SwiftlyAuthError as exc:
|
||||
print(f"\nAUTH FAILURE: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"\ncategory={args.category!r}: {len(raw_items)} raw items")
|
||||
|
||||
aisle = LuckyCaliforniaScraper._aisle_from_slug(args.category)
|
||||
mapped = []
|
||||
for raw in raw_items:
|
||||
m = LuckyCaliforniaScraper.map_product(
|
||||
raw, aisle=aisle, source_slug=args.category
|
||||
)
|
||||
if m is not None:
|
||||
mapped.append(m)
|
||||
print(f"{len(mapped)} mapped products (after dropping unparseable rows)")
|
||||
|
||||
print("\n=== sample mapped products ===")
|
||||
for m in mapped[: args.limit]:
|
||||
# Decimal isn't JSON-serializable; coerce for display.
|
||||
display = {k: (str(v) if k in {"current_price", "regular_price", "sale_price"} else v) for k, v in m.items()}
|
||||
print(json.dumps(display, indent=2, default=str))
|
||||
print()
|
||||
|
||||
return 0
|
||||
finally:
|
||||
scraper.cleanup()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user