feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-05 14:08:19 -07:00
co-authored by Claude Opus 4.7
parent b9434967ed
commit 8e89f793d5
58 changed files with 3594 additions and 348 deletions
+68
View File
@@ -0,0 +1,68 @@
"""Async scrape endpoint contract.
Verifies that ``POST /api/admin/scrape``:
- returns 202 + ``scrape_log_id`` synchronously,
- persists a ``ScrapeLog`` row in status STARTED before the background task
runs (the task is monkey-patched out so it never reaches Playwright and
never opens a session outside the test transaction).
"""
from __future__ import annotations
import os
import uuid
import pytest
os.environ.setdefault("ADMIN_TOKEN", "test-admin-token")
@pytest.fixture(autouse=True)
def _admin_token(monkeypatch):
monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token")
yield
@pytest.mark.requires_postgres
def test_scrape_returns_202_and_log_id(client, db, monkeypatch):
"""Endpoint enqueues the scrape and returns 202 + scrape_log_id.
We replace the background runner with a no-op so the test does NOT spin up
Playwright and does NOT open a session outside the rolled-back test
transaction.
"""
calls: list[tuple] = []
def _fake_run(log_id, source, scrape_type):
calls.append((log_id, source, scrape_type))
# Patch in BOTH the service module (definition site) and the api module
# (import site) so whichever symbol the route resolved to is replaced.
monkeypatch.setattr(
"app.services.scraper_service._run_scrape_in_background", _fake_run
)
r = client.post(
"/api/admin/scrape",
headers={"Authorization": "Bearer test-admin-token"},
)
assert r.status_code == 202, r.text
body = r.json()
assert body["status"] == "queued"
assert "scrape_log_id" in body
log_id = uuid.UUID(body["scrape_log_id"])
# Row was committed inside enqueue_scrape — visible on the test session.
from app.models import ScrapeLog, ScrapeStatus
log = db.query(ScrapeLog).filter(ScrapeLog.id == log_id).first()
assert log is not None, "ScrapeLog row should exist after enqueue"
assert log.status == ScrapeStatus.STARTED
assert log.source == "lucky_california"
assert log.scrape_type == "weekly_ad"
assert log.completed_at is None
# TestClient runs background tasks before returning from the context
# manager exit — by the time we get here, the fake runner ran exactly once.
assert len(calls) == 1
assert calls[0][0] == log_id