Commit Graph
6 Commits
Author SHA1 Message Date
admin e92cc3a074 scrapers: drop Swiftly products with unparseable price
Products missing a parseable sale or regular price would previously yield
a GroceryItem with current_price=None. That broke the downstream matcher
(ingredient typical_price is non-null) and cluttered the table.

Added a guard in map_product() to return None when both reg_price and
sale_price are None. Fixes test_map_product_returns_none_for_unparseable.
2026-05-18 17:22:29 -07:00
adminandClaude Sonnet 4.6 2373883fe7 fix: exact-name fast path in matcher + save priceless produce in scraper
Scraper: remove price guard in map_product so produce items without a
catalog price (e.g. Fresh Garlic, Lime sold by weight) are saved to
grocery_item with current_price=NULL rather than skipped.

Matcher:
- Add exact-name fast path: build a lowercase-trimmed name→index map
  and skip fuzzy search entirely when the ingredient name matches a
  grocery item exactly. Lime → Lime (confidence 1.0), Garlic → Fresh
  Garlic from fuzzy (confidence 1.0).
- Add exclusion words: juice, gelatin to prevent beverage/dessert
  products from matching cooking ingredients.
- Increase fuzzy candidate limit 20→100 so exact-name items buried in
  large tie groups are not missed.
- Add 'juice' to exclusion: prevents '100% Lime Juice' from winning
  over plain 'Lime'.

Result: all recipe ingredients now match correct Lucky CA products or
show '—' (no match); zero category cross-contamination remaining.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-10 20:17:03 -07:00
adminandClaude Opus 4.7 95b8e0c1b5 feat: AM-3..AM-6 strip SWIFTLY_BEARER_TOKEN env var, delete superseded script, refresh docs
AM-3: SWIFTLY_BEARER_TOKEN removed from .env.example, .env.test (local),
docker-compose.yml service env, and Settings (backend/app/config.py).
The scraper docstring is updated to reflect the auto-mint path.

AM-4: scripts/refresh_swiftly_token.py (commit ccfb38a, seleniumbase
click-through capture) deleted; superseded by swiftly_auth.py.

AM-5: docs refreshed.
- spec status header → "Implemented 2026-05-06" with live-verification
  evidence
- HANDOFF.md TL;DR + caveats #2/#3 collapsed; replaced with the
  auto-mint failure-modes caveat; "Suggested next move" rewritten
  pointing to Phase 5 orchestration; file-map and last-updated touched
- ORIENTATION.md env-var section updated (no bearer var) + footer

AM-6 verification gate (run 2026-05-06):
- pytest -q tests/ → 92/92 green (88 prior + 4 new swiftly_auth)
- POST /api/admin/scrape → status=success, items_scraped=10928 in 44s
- grocery_item rows: 9980 (after dedup-by external_id)
- ingredient_grocery_match rows: 29779 (matcher post-hook populated)
- Container env confirmed clean of SWIFTLY_BEARER_TOKEN

The system now scrapes, matches, and generates plans without any
operator-managed credential. Live JWT lifecycle: Firebase REST anon
signUp → cache for ~55min → re-mint as needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:47:15 -07:00
adminandClaude Opus 4.7 dfd79a9d08 feat: AM-2 wire swiftly_auth.get_token() into LuckyCaliforniaScraper
fetch_category() now calls swiftly_auth.get_token() to mint a fresh
Firebase JWT on demand when no explicit bearer_token override is
pinned by tests. The cache short-circuit means the per-call mint
overhead is ~zero in the steady state.

- Removed the empty-token short-circuit; auto-mint makes it moot
- Updated _AUTH_ERROR_MESSAGE: 401-after-mint now points at the spec
  (Lucky tightening anon-auth) rather than asking for manual capture
- Replaced test_swiftly_auth_error_when_token_missing with a positive
  test that verifies fetch_category mints when bearer_token is None
- bearer_token constructor arg preserved for the 401-path test

Full suite: 92/92 green. Live verification via
scripts/spike_swiftly_ingest.py --confirm-live deferred to next step
per HANDOFF AM-2 halt boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 15:27:37 -07:00
adminandClaude Opus 4.7 8e89f793d5 feat: phase r1+r2 recovery + r3-0 swiftly api ingestion
R1 stabilization: pytest harness with transactional db fixture, smoke
+ alembic + auth + scrape + approval + swiftly tests, github actions
ci yaml. Bearer-token admin auth + signed-cookie session for family
ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns
202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL
fail-fast on empty.

R2 deferred-risk spikes: live lucky california fetch (R2-A), full
email+per-voter approval click round trip with single-use enforcement
(R2-B, console email backend, sendgrid stub).

R3-0 phase 3 redesign: replaced playwright html scraper with requests
based swiftly json api client. 17 categories, ~10k products per scrape,
upsert by (source, external_id). 401 surfaces actionable token-refresh
message via ScrapeLog.error_message.

Pre-existing defects fixed: shopping_list.py syntax error blocking app
import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid
kwarg, missing requests dep, calorie_target schema drift, every SQLEnum
needed values_callable, 0001 had empty downgrade(), seed had duplicate
ingredient rows.

Migrations added: 0003 grocery_item.description, 0004 family_profile.
calorie_target, 0005 grocery_item.external_id + source + composite index.

Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean,
frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 14:08:19 -07:00
admin 933a0cc9db feat: implement Lucky California scraper with Playwright + BeautifulSoup
- Add BaseScraper with rate limiting, retries, session management
- Add LuckyCaliforniaScraper with Playwright for dynamic content
- Add ScraperService to save scraped items to grocery_item table
- Connect /api/admin/scrape to ScraperService
- Update ORIENTATION.md phase table
2026-05-04 20:50:27 -07:00