Public Access

Files

T

adminandClaude Opus 4.7 8e89f793d5 feat: phase r1+r2 recovery + r3-0 swiftly api ingestion

R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-05 14:08:19 -07:00

3.5 KiB

Raw Permalink Blame History

R1-C — Async scrape + grocery_item.description

What changed

backend/app/api/admin.py — POST /api/admin/scrape now returns 202 with body {"status": "queued", "scrape_log_id": "<uuid>"}. Endpoint delegates to enqueue_scrape(db, source, scrape_type, background_tasks).
backend/app/services/scraper_service.py — split:
- enqueue_scrape() inserts a ScrapeLog row (status=STARTED), db.commit(), then background_tasks.add_task(_run_scrape_in_background, log.id, ...). Returns the refreshed log row.
- _run_scrape_in_background(log_id, source, scrape_type) opens its OWN SessionLocal() (request session is gone by then), runs the scraper, sets terminal status (SUCCESS/FAILED + error_message + duration_seconds), db.close() in finally. Rolls back before writing FAILED to avoid detached instances.
- ScraperService.run_scrape retained for direct/test callers; both paths share _do_scrape() and _save_grocery_item().
- _save_grocery_item now maps item_data["description"] → GroceryItem.description on both insert and update.
backend/app/models/__init__.py — added description = Column(Text, nullable=True) to GroceryItem.
backend/alembic/versions/0003_grocery_item_description.py — revision="0003", down_revision="0002". op.add_column("grocery_item", sa.Column("description", sa.Text(), nullable=True)) / op.drop_column on downgrade.
backend/tests/test_scrape_endpoint.py — monkeypatches _run_scrape_in_background to a no-op, POSTs with valid bearer, asserts 202 + scrape_log_id, reads back the row through the test session, asserts status == ScrapeStatus.STARTED and completed_at is None.

Endpoint contract

POST /api/admin/scrape (admin bearer required):

Request: optional query source (default lucky_california), scrape_type (default weekly_ad).
Response: 202 + {"status": "queued", "scrape_log_id": "<uuid>"}.
Poll GET /api/admin/logs/{id} for terminal state.

Status enum note

Existing ScrapeStatus enum is started|success|failed only — no pending/running/completed members. Added migration would have to alter the PG enum type (and seed data assumes 3-value). Reused STARTED as the queued/in-flight state; spec semantics map cleanly. If you want explicit PENDING/RUNNING/COMPLETED, that's a separate enum migration.

DB session in the bg task

Imports SessionLocal from app.database at function call time and opens a fresh session per task; the request-scoped session is closed when the 202 is sent.

Verification

pytest tests/test_lucky_ca_scraper.py -q → 1 passed (R2-A still green).
tests/test_scrape_endpoint.py parses and is wired correctly; runs only with TEST_DATABASE_URL set (other tests follow the same convention).
python -c "from app.services.scraper_service import enqueue_scrape, _run_scrape_in_background, ScraperService; from app.models import GroceryItem, ScrapeStatus" → ok.

Blocker (pre-existing, OUT OF R1-C SCOPE)

backend/app/api/shopping_list.py:59 has a syntax error from commit c735d21: Ingredient.id.in_ all_ingredient_ids — missing parens, should be Ingredient.id.in_(all_ingredient_ids). This blocks from app.main import app and therefore tests/test_smoke.py::test_app_imports. Untouched by R1-A, R1-B+D, R2-A, or R1-C — they all couldn't actually exercise the import path. Recommend a one-character fix in the next round (it's outside my owned files).

3.5 KiB Raw Permalink Blame History