Public Access
26 lines
1.6 KiB
Markdown
26 lines
1.6 KiB
Markdown
# Recipe Catalog Import Phase 2
|
|
|
|
## Scope
|
|
|
|
Added a dedicated synchronous Wikibooks staging service and offline tests without changing recipe ingestion, recipe APIs, or existing dirty recipe tests.
|
|
|
|
## Implementation
|
|
|
|
- Fixed `https://en.wikibooks.org/w/api.php` client with bounded category continuation, required caller User-Agent, sequential rate limiting, timeouts, streamed payload cap, HTTP `Retry-After` retries, and MediaWiki `maxlag`/`ratelimited` exponential backoff.
|
|
- Validated MediaWiki response shapes and restricted page fetches to bounded `Cookbook:` titles.
|
|
- Normalized rendered HTML into text-only recipe candidates: intro, infobox yields/times, ingredient and ordered instruction subsections, quantity/unit/source fields, and no images.
|
|
- Added deterministic validation and quarantine for invalid servings, insufficient ingredients/instructions, invalid quantities, placeholders, and distinctive title-token ingredient mismatches.
|
|
- Staged exact API `parse` payloads with canonical checksums, immutable provenance inputs, CC BY-SA 4.0 attribution/history, and source/page/revision deduplication. The service flushes only.
|
|
|
|
## Verification
|
|
|
|
- Offline focused run: `22 passed, 2 deselected in 1.60s`.
|
|
- Full run against disposable PostgreSQL: `24 passed in 3.08s`, including the two-session concurrent deduplication test.
|
|
- The disposable database was removed after verification.
|
|
- `backend/venv/bin/python -m py_compile backend/app/services/wikibooks_import.py backend/tests/test_wikibooks_import.py`: passed.
|
|
- Scoped `git diff --check`: clean for the service, test, and two fixtures.
|
|
|
|
## Status
|
|
|
|
Phase 2 is verified. Phase 3 has not started.
|