Public Access
adminandClaude Opus 4.7
8e89f793d5
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
65 lines
3.5 KiB
Markdown
65 lines
3.5 KiB
Markdown
# R1-C — Async scrape + grocery_item.description
|
|
|
|
## What changed
|
|
|
|
- `backend/app/api/admin.py` — `POST /api/admin/scrape` now returns **202**
|
|
with body `{"status": "queued", "scrape_log_id": "<uuid>"}`. Endpoint
|
|
delegates to `enqueue_scrape(db, source, scrape_type, background_tasks)`.
|
|
- `backend/app/services/scraper_service.py` — split:
|
|
- `enqueue_scrape()` inserts a `ScrapeLog` row (status=`STARTED`),
|
|
`db.commit()`, then `background_tasks.add_task(_run_scrape_in_background, log.id, ...)`.
|
|
Returns the refreshed log row.
|
|
- `_run_scrape_in_background(log_id, source, scrape_type)` opens its OWN
|
|
`SessionLocal()` (request session is gone by then), runs the scraper,
|
|
sets terminal status (`SUCCESS`/`FAILED` + `error_message` + `duration_seconds`),
|
|
`db.close()` in `finally`. Rolls back before writing FAILED to avoid
|
|
detached instances.
|
|
- `ScraperService.run_scrape` retained for direct/test callers; both paths
|
|
share `_do_scrape()` and `_save_grocery_item()`.
|
|
- `_save_grocery_item` now maps `item_data["description"]` → `GroceryItem.description`
|
|
on both insert and update.
|
|
- `backend/app/models/__init__.py` — added `description = Column(Text, nullable=True)` to `GroceryItem`.
|
|
- `backend/alembic/versions/0003_grocery_item_description.py` — `revision="0003"`,
|
|
`down_revision="0002"`. `op.add_column("grocery_item", sa.Column("description", sa.Text(), nullable=True))` / `op.drop_column` on downgrade.
|
|
- `backend/tests/test_scrape_endpoint.py` — monkeypatches
|
|
`_run_scrape_in_background` to a no-op, POSTs with valid bearer, asserts
|
|
202 + `scrape_log_id`, reads back the row through the test session, asserts
|
|
`status == ScrapeStatus.STARTED` and `completed_at is None`.
|
|
|
|
## Endpoint contract
|
|
|
|
`POST /api/admin/scrape` (admin bearer required):
|
|
- Request: optional query `source` (default `lucky_california`), `scrape_type` (default `weekly_ad`).
|
|
- Response: **202** + `{"status": "queued", "scrape_log_id": "<uuid>"}`.
|
|
- Poll `GET /api/admin/logs/{id}` for terminal state.
|
|
|
|
## Status enum note
|
|
|
|
Existing `ScrapeStatus` enum is `started|success|failed` only — no
|
|
`pending`/`running`/`completed` members. Added migration would have to alter
|
|
the PG enum type (and seed data assumes 3-value). Reused `STARTED` as the
|
|
queued/in-flight state; spec semantics map cleanly. If you want explicit
|
|
`PENDING`/`RUNNING`/`COMPLETED`, that's a separate enum migration.
|
|
|
|
## DB session in the bg task
|
|
|
|
Imports `SessionLocal` from `app.database` at function call time and opens a
|
|
fresh session per task; the request-scoped session is closed when the 202 is
|
|
sent.
|
|
|
|
## Verification
|
|
|
|
- `pytest tests/test_lucky_ca_scraper.py -q` → 1 passed (R2-A still green).
|
|
- `tests/test_scrape_endpoint.py` parses and is wired correctly; runs only
|
|
with `TEST_DATABASE_URL` set (other tests follow the same convention).
|
|
- `python -c "from app.services.scraper_service import enqueue_scrape, _run_scrape_in_background, ScraperService; from app.models import GroceryItem, ScrapeStatus"` → ok.
|
|
|
|
## Blocker (pre-existing, OUT OF R1-C SCOPE)
|
|
|
|
`backend/app/api/shopping_list.py:59` has a syntax error from commit c735d21:
|
|
`Ingredient.id.in_ all_ingredient_ids` — missing parens, should be
|
|
`Ingredient.id.in_(all_ingredient_ids)`. This blocks `from app.main import app`
|
|
and therefore `tests/test_smoke.py::test_app_imports`. Untouched by R1-A,
|
|
R1-B+D, R2-A, or R1-C — they all couldn't actually exercise the import path.
|
|
Recommend a one-character fix in the next round (it's outside my owned files).
|