diff --git a/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md b/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md index c7107e8..68a9316 100644 --- a/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md +++ b/recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md @@ -406,7 +406,7 @@ Not complete until G1–G7 pass on the device path (Tier 3), not just offline. | F-08 | DINO unfit as classifier and retriever here | major | VERIFIED | P1 | open-world | | F-09 | Confirmed labels never re-evaluated; first accept locks | major | HIGH CONFIDENCE | P1 | open-world | | F-10 | Uncommitted live diff: unbounded blur retries; rationale invalidated | major | HIGH CONFIDENCE | P1 | open-world | -| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | open-world | +| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | **fixed 2026-09-29** | | F-13 | Stale handoff test status; swallowed callback exception in a test fixture | polish | VERIFIED | P3 | open-world | Counts: 11 VERIFIED / 2 HIGH CONFIDENCE / 0 HYPOTHESIS (hypotheses listed @@ -519,11 +519,20 @@ covers a recognition backend. The product-level floor is Van's system (G1–G4). invalidated 69/66 numbers. - **Change:** revert per §8. -### F-12 — l06 package version — P2 · major · VERIFIED -- **Evidence:** the packshot shows "内臓脂肪ケア フィッテ"; the shelf shows +### F-12 — l06 package version — P2 · major · **FIXED 2026-09-29** +- **Evidence:** the packshot showed "内臓脂肪ケア フィッテ"; the shelf shows "内臓脂肪を減らす キユーピーフィッテ". Improved matcher 4/4 at full, 1/4 at live; 3/3 with a shelf reference. -- **Change:** enroll the current package; adopt the packshot-OCR check. +- **Fix:** the current package is enrolled. It was recorded as blocked on a + capture because Kewpie's own site serves the superseded design for that JAN; + an independent retailer listing carries the current one, and it matches the + shelf facings exactly (checked by eye against a crop of `add/image6.jpg` + before enrolling). Measured: live decision 60% → 64%, end-to-end 59% → 62%, + full decision 81% → 83%. +- **Still open:** the packshot-OCR check that would have caught this. It is + implemented (`rebuild_catalog` refuses a view whose name/OCR score is below + 0.70) but it is a false negative on l04 and l07, so it cannot yet be trusted + as a gate - see the catalog corrections section. ### F-13 — stale test status — P3 · polish · VERIFIED - **Evidence:** 49/49 pass on both trees. An `AttributeError: 'str' object has @@ -572,7 +581,7 @@ running the real server against `add/image6.jpg`. | F-07 evaluator defects | Done. Four separate ceilings, honest precision denominator, catalog-derived candidate set, `sku_of` grouping, leak signature. | | F-08 DINO fenced off | Done. Not the default, not in the live path, refuses to start without weights. | | F-10 uncommitted live experiments | Reverted. `crop_first_attempt`, the blur early return, `vote_agree` and `max_attempts 4` are gone. | -| F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. | +| F-12 l06 current package | **Fixed.** The current package is enrolled from an independent retailer listing and matches the shelf facings exactly. Live decision 60% → 64%, end-to-end 59% → 62%. | | F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. | | F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. | | Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and each view's bytes, not just the view names: names alone let a different packshot be swapped in under the same name with the hash unchanged. | @@ -826,6 +835,48 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット claim about the contents. Measured on the corrected labels, full resolution: precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%. +### External benchmark: 20 images, 0 false accepts + +The image6 shelf is one photo with one label set, so every number above is +measured on data the design was tuned against. Twenty images for eight SKUs +were collected from independent sources (six retailer and marketplace domains) +to test the pipeline on data it had no part in building. None is Tier 2; all +are `external_proxy` and every manifest says so. + +Leak-tested first: the highest template match of any of the 20 against +`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo. + +Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against +`stock-lopia-clean` (`data/external-eval/*/eval_result.json`): + +| Set | Images | Correct | Abstain | False accepts | +|---|---|---|---|---| +| `kewpie/` | 12 | 9 | 3 | **0** | +| `other-mayo/` | 8 | 4 | 4 | **0** | +| **Total** | **20** | **13** | **7** | **0** | +- **`sku_l07` is answered with no enrolled image.** Both its images accept + through the text lane alone - one reads ラッキー フヨネーズ, the other the + rear label's 製造者：丸和油脂株式会社 with 内容量：200g, which together + name the row 丸和油脂 ラッキーマヨネーズ 200g. This is the text lane's + coverage claim holding on data that was not used to build it. `sku_l02` + also accepts one image on text alone, but it does have an enrolled view. +- **The l04 abstentions are correct behaviour.** l04 is also unenrolled and its + reads are weak: one reads only コと旨 タ 味 (every row scores 0.0), another + scores 0.379 against a 0.45 accept line. Abstaining on a weak read is the + designed outcome, not a miss. + +The seven abstentions are all unenrolled or weakly-read SKUs. No image was +given a wrong name, which is the property the decision optimises for. + +### Tier-3 sources are staged, not satisfied + +Two reusable pan clips were found and saved (`data/external-eval/tier3-sources/`): +a 1920x1080 CC BY 3.0 supermarket pan and a 360p CC BY mayonnaise-bay pan, with +11 rejected candidates recorded. Neither gives per-SKU identity for this +catalog, and the only LOPIA store walk found carries the standard YouTube +licence and was therefore not downloaded. Tier 3 still needs the operator's own +recorded walks. + Still open, and not implementable from here: Tier 2 held-out labels (F-11), the remaining E-1/E-4..E-9 experiments, and the completion gate G1-G7, all of which need a second independent shelf visit.