# R3-0 — Swiftly product API client (replaces Playwright path)

## Schema migration

Added `0005_grocery_item_external_id.py` (`down_revision='0004'`):

- `grocery_item.external_id` (`String(100)`, nullable, indexed)
- `grocery_item.source` (`String(50)`, nullable)
- composite index `ix_grocery_item_source_external_id`

Idempotency key for upserts is `(source, external_id)`. Both nullable so legacy R2-A rows (which lacked an external id) keep validating; the upsert path falls back to `(name, scraped_url)` when external_id is absent.

## Discovery + sample counts

- Categories page: 17 distinct slugs (e.g. `Product/meat_seafood`, `Product/produce`, `Product/dairy_eggs_cheese`, ...). Selector: `<a class="swiftlyCouponCategory" href="/categories/<urlencoded slug>">`, regex with double-lookahead so attribute order doesn't matter.
- `Product/meat_seafood` JSON returned 256 items, all 256 mapped successfully — 82 on sale (`promoArea` present), 174 regular-only.

## Field mapping

| Swiftly JSON | grocery_item column |
| --- | --- |
| `id` | `external_id` (new) |
| `name` | `name` |
| `description` | `description` |
| `brand` | `brand` |
| `primaryImage.url` | `image_url` |
| `price.ok.regPriceText` (e.g. `"$3.49 /lb"`) | `regular_price`, `unit` |
| `price.ok.promoArea.promoText` | `sale_price`, `is_on_sale=True` |
| (queried slug → tail) | `aisle` (e.g. `meat_seafood`) |
| (constant) | `source = "lucky_california"` |
| (none) | `product_url = NULL` (Swiftly exposes none) |
| `validityText` | not parsed — `sale_start_date`/`sale_end_date` left null |

`current_price` = `sale_price` when on sale, else `regular_price`. Prices stored as `Decimal`.

## 401 handling

`SwiftlyAuthError` is a custom exception raised:

1. Up-front when `SWIFTLY_BEARER_TOKEN` is empty (no HTTP call).
2. On `response.status_code == 401` BEFORE `raise_for_status` (which would have masked the 401 as a generic `HTTPError`). The new client uses `requests.Session.get` directly — `BaseScraper._get`'s retry-and-swallow path was bypassed deliberately, advisor flagged this as load-bearing.

`_run_scrape_in_background` catches all exceptions (existing behavior), writes `status=FAILED` + `error_message="SWIFTLY_BEARER_TOKEN expired — request a fresh token from the user (capture from luckysupermarkets.com network tab on a /search/api/v1 request)"`. Admin sees it via `GET /api/admin/logs/<id>`.

## Auth scoping

Two `requests.Session` objects: `public_session` (no auth, hits `luckysupermarkets.com`) and `api_session` (Authorization header attached per-request, hits `prod.swiftlyapi.net`). Bearer is never sent to the public host.

## Verification

- `pytest -q tests/test_swiftly_api.py` → 5 passed, 1 skipped (Postgres-only).
- `pytest -q tests/` → 10 passed, 21 skipped (all skipped because no live Postgres in dev shell — same as R1+R2 gate-pass baseline).
- `python scripts/spike_swiftly_ingest.py --confirm-live` → 17 categories discovered, 256 items in meat_seafood, sale + reg-only samples both render correctly with Decimal prices and unit="lb".
- Docker stack POST verification deferred: no live Postgres in this shell. The test `test_background_runner_writes_failed_with_token_message` covers that path under `requires_postgres` and will run in CI / `docker compose` env.

## Caveats

- **Token in `.env.example` is a real (expiring) credential**, per task spec. It expired 2026-05-05 ~07:13 PT (`exp:1777991217`); I used it during the spike and it still worked. The user explicitly authorized this. Recommend gitignoring `.env.example` or rotating to a placeholder in a future cleanup task.
- Old test `tests/test_lucky_ca_scraper.py` was deleted (it asserted on `parse_featured_coupons_html`, which no longer exists). R2-A's fixtures `weekly_ad.html`, `weekly_ad.png`, `META.md` retained per task spec.
- `BaseScraper` and `SeleniumScraper` classes in `app/scraper/base.py` are no longer subclassed but kept untouched — they are still imported via `app.scraper.__init__` and may be useful for a future second store. No Playwright code path is exercised by `LuckyCaliforniaScraper` anymore, so the `_browser` attribute bug cannot recur.
- Rate limit set to 0.75s/req in the new client (between the spec's 1–2 req/sec). Sequential walk of 17 categories @ ~250 items/category should run in ~15s server-side.
- `_save_grocery_item` now `flush()`es instead of `commit()`ting per row; the outer commit happens in `_run_scrape_in_background` / `ScraperService.run_scrape`. Trade-off: a single bad row aborts the whole scrape's transaction. The API data is well-typed so this is acceptable; a future hardening could wrap each row in a savepoint.
- Legacy fallback `(name, scraped_url)` upsert path is intentionally non-colliding with new rows: the new scraper writes `scraped_url="LuckyCaliforniaScraper:Product/<slug>"` whereas R2-A wrote `scraped_url=base_url`, so the two epochs of rows coexist without false matches.
- Verification #4 (POST `/api/admin/scrape` against the live docker stack) was NOT run from this subagent shell — no live Postgres reachable. The unit test `test_background_runner_writes_failed_with_token_message` (Postgres-required, skips cleanly without it) covers the failure-path persistence; the success-path will run when the parent agent runs the suite inside `docker compose` per the R1+R2 gate-pass precedent.