Public Access
feat: phase r1+r2 recovery + r3-0 swiftly api ingestion
R1 stabilization: pytest harness with transactional db fixture, smoke + alembic + auth + scrape + approval + swiftly tests, github actions ci yaml. Bearer-token admin auth + signed-cookie session for family ui mutations. Async POST /api/admin/scrape (BackgroundTasks, returns 202). Path canonicalization (no /list, /planned suffixes). DATABASE_URL fail-fast on empty. R2 deferred-risk spikes: live lucky california fetch (R2-A), full email+per-voter approval click round trip with single-use enforcement (R2-B, console email backend, sendgrid stub). R3-0 phase 3 redesign: replaced playwright html scraper with requests based swiftly json api client. 17 categories, ~10k products per scrape, upsert by (source, external_id). 401 surfaces actionable token-refresh message via ScrapeLog.error_message. Pre-existing defects fixed: shopping_list.py syntax error blocking app import, MealPlan.votes orphan relationship, JSONB(astext=True) invalid kwarg, missing requests dep, calorie_target schema drift, every SQLEnum needed values_callable, 0001 had empty downgrade(), seed had duplicate ingredient rows. Migrations added: 0003 grocery_item.description, 0004 family_profile. calorie_target, 0005 grocery_item.external_id + source + composite index. Verified: 31/31 pytest green, alembic upgrade->downgrade->upgrade clean, frontend npm run build clean, live scrape 9,960 grocery_item rows in 36s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,20 +1,29 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
from app.database import get_db
|
||||
from app.models import ScrapeLog, EmailLog, MealPlan
|
||||
from app.services.scraper_service import ScraperService
|
||||
from app.security import require_admin
|
||||
from app.services.scraper_service import ScraperService, enqueue_scrape
|
||||
from typing import List, Optional
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
router = APIRouter()
|
||||
router = APIRouter(dependencies=[Depends(require_admin)])
|
||||
|
||||
|
||||
@router.post("/scrape")
|
||||
def trigger_scrape(source: str = "lucky_california", scrape_type: str = "weekly_ad", db: Session = Depends(get_db)):
|
||||
scraper_service = ScraperService(db)
|
||||
result = scraper_service.run_scrape(source=source, scrape_type=scrape_type)
|
||||
|
||||
return result
|
||||
@router.post("/scrape", status_code=202)
|
||||
def trigger_scrape(
|
||||
background_tasks: BackgroundTasks,
|
||||
source: str = "lucky_california",
|
||||
scrape_type: str = "weekly_ad",
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
log = enqueue_scrape(
|
||||
db,
|
||||
source=source,
|
||||
scrape_type=scrape_type,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
return {"status": "queued", "scrape_log_id": str(log.id)}
|
||||
|
||||
|
||||
@router.get("/logs")
|
||||
|
||||
Reference in New Issue
Block a user