Public Access
feat: LLM-powered second-pass ingredient matcher + matcher improvements
Matcher improvements (matcher.py): - Plural normalization: 'tortillas'→'tortilla', 'thighs'→'thigh' so subset recall check works without stemmer - Precision floor lowered 0.45→0.30: allows 'Bacon'→'Wright Brand Bacon' (1/3=0.33) while exclusion words still block category contaminants - _EXCLUSION_WORDS now normalized through same singularizer for consistency LLM second-pass (llm_matcher.py): - run_llm_match_job(): for each still-unmatched ingredient, collects top-12 candidates from grocery catalog ranked by fuzzy×precision (same metric as AUTO matcher), then asks Ollama to pick the best match - Candidate scoring: combined = (partial_token_sort_ratio/100) × precision ensures "McCormick Black Pepper" outranks "Dr Pepper" for 'Black Pepper' - Stores picks as source='auto_llm' (confidence=0.750) - Ollama Cloud endpoint: https://ollama.com/v1, model: kimi-k2.6:cloud Migration 0010: adds 'auto_llm' to ingredient_match_source_enum Config: OLLAMA_BASE_URL / OLLAMA_API_KEY / OLLAMA_MODEL settings Docker-compose: wires all three Ollama + Spoonacular env vars to backend/scheduler Scraper service: calls run_llm_match_job after run_match_job on every scrape Results: AUTO matcher went from 36→25 unmatched (plural normalization fix), LLM added 3 more (Black Pepper, Zucchini, Chicken Thighs). Remaining 22 are genuine Lucky CA catalog gaps (standalone olive oil, dried spices, etc. not in Swiftly weekly ad). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,16 +39,29 @@ _STOP_WORDS = frozenset({
|
||||
"and", "with", "for", "the",
|
||||
})
|
||||
|
||||
def _singularize(word: str) -> str:
|
||||
"""Best-effort English singularization for grocery product names.
|
||||
|
||||
Handles 'tortillas'→'tortilla', 'thighs'→'thigh', 'chickpeas'→'chickpea'.
|
||||
Requires length > 3 to avoid mangling short words like 'has', 'was'.
|
||||
"""
|
||||
if len(word) > 4 and word.endswith("ies"):
|
||||
return word[:-3] + "y" # 'berries' → 'berry'
|
||||
if len(word) > 3 and word.endswith("s") and not word.endswith("ss"):
|
||||
return word[:-1] # 'tortillas' → 'tortilla'
|
||||
return word
|
||||
|
||||
|
||||
# If any of these words appear in a grocery item's sig-words but NOT in the
|
||||
# ingredient's sig-words, the match is rejected outright. Prevents category
|
||||
# cross-contamination: "Garlic" must not match "Garlic Bread", "Lime" must not
|
||||
# match "Lime Margarita", etc.
|
||||
_EXCLUSION_WORDS = frozenset({
|
||||
_EXCLUSION_WORDS_RAW = frozenset({
|
||||
# Baked goods / bread products
|
||||
"bread", "loaf", "rolls", "bun", "buns", "croissant",
|
||||
"bread", "loaf", "roll", "rolls", "bun", "buns", "croissant",
|
||||
"cracker", "crackers", "cookie", "cookies", "cake", "cupcake", "muffin", "bagel",
|
||||
# Chips / snack foods
|
||||
"chips",
|
||||
"chip", "chips",
|
||||
# Pasta / noodles
|
||||
"pasta", "noodle", "noodles", "vermicelli", "spaghetti", "linguine",
|
||||
"fettuccine", "penne", "rigatoni", "macaroni", "rotini", "orzo",
|
||||
@@ -64,8 +77,10 @@ _EXCLUSION_WORDS = frozenset({
|
||||
# Prepared poultry (prevents "Garlic Herb Rotisserie Chicken" matching "Garlic")
|
||||
"rotisserie",
|
||||
# Baby / personal care (belt-and-suspenders after stop-word rework)
|
||||
"baby", "wipes", "diaper",
|
||||
"baby", "wipe", "wipes", "diaper",
|
||||
})
|
||||
# Pre-normalize exclusion words so they match the singularized sig-word sets.
|
||||
_EXCLUSION_WORDS = frozenset(_singularize(w) for w in _EXCLUSION_WORDS_RAW)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -76,9 +91,13 @@ class MatchResult:
|
||||
|
||||
|
||||
def _sig_words(text: str) -> frozenset:
|
||||
"""Lowercase alpha tokens >2 chars, stop-words removed."""
|
||||
"""Lowercase alpha tokens >2 chars, stop-words removed, plurals normalized."""
|
||||
tokens = re.sub(r"[^a-z ]", " ", text.lower()).split()
|
||||
return frozenset(t for t in tokens if len(t) > 2 and t not in _STOP_WORDS)
|
||||
return frozenset(
|
||||
_singularize(t)
|
||||
for t in tokens
|
||||
if len(t) > 2 and t not in _STOP_WORDS
|
||||
)
|
||||
|
||||
|
||||
def run_match_job(
|
||||
@@ -185,10 +204,10 @@ def run_match_job(
|
||||
continue
|
||||
# Precision penalises grocery items with many extra words.
|
||||
precision = len(ing_sig) / len(gsig)
|
||||
# Hard floor: grocery must not have >2× the sig-words of the ingredient.
|
||||
# Catches long branded products that sneak past exclusion words, e.g.
|
||||
# "Garlic Herb Rotisserie Chicken" for "Garlic".
|
||||
if precision < 0.45:
|
||||
# Hard floor: grocery must not dwarf the ingredient in sig-word count.
|
||||
# 0.30 allows "Bacon" (1 sig-word) → "Wright Brand Bacon" (3 sig-words)
|
||||
# while exclusion words still block "Garlic Herb Rotisserie Chicken".
|
||||
if precision < 0.30:
|
||||
continue
|
||||
combined = (score / 100.0) * precision
|
||||
if combined > best_combined:
|
||||
|
||||
Reference in New Issue
Block a user