tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

docs(decision): record the external benchmark and the F-12 fix

Browse source 
Twenty independent-source images across eight SKUs, leak-tested (max template
match against add/image6.jpg is 0.675) and scored through the shipped lane:
13 correct, 7 abstain, 0 false accepts. The seven abstentions are all
unenrolled or weakly-read SKUs.

The result worth keeping is sku_l07: no enrolled image, both images accepted on
text alone. That is the text lane's coverage claim holding on data it had no
part in building.

F-12 is marked fixed with its measured effect, and the packshot-OCR check that
would have caught it is recorded as implemented but not yet trustworthy - it is
a false negative on l04 and l07.
This commit is contained in:
kiyo9w 2026-09-29 03:23:31 +09:00
parent 6a45fb990d
commit dfdaef4e61
1 changed files with 56 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

61
recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
View file

@ -406,7 +406,7 @@ Not complete until G1–G7 pass on the device path (Tier 3), not just offline.
| F-08 | DINO unfit as classifier and retriever here | major | VERIFIED | P1 | open-world |
| F-09 | Confirmed labels never re-evaluated; first accept locks | major | HIGH CONFIDENCE | P1 | open-world |
| F-10 | Uncommitted live diff: unbounded blur retries; rationale invalidated | major | HIGH CONFIDENCE | P1 | open-world |
| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | open-world |
| F-12 | l06 packshot is a superseded package design | major | VERIFIED | P2 | **fixed 2026-09-29** |
| F-13 | Stale handoff test status; swallowed callback exception in a test fixture | polish | VERIFIED | P3 | open-world |
Counts: 11 VERIFIED / 2 HIGH CONFIDENCE / 0 HYPOTHESIS (hypotheses listed
@ -519,11 +519,20 @@ covers a recognition backend. The product-level floor is Van's system (G1–G4).
invalidated 69/66 numbers.
- **Change:** revert per §8.
### F-12 — l06 package version — P2 · major · VERIFIED
- **Evidence:** the packshot shows "内臓脂肪ケア フィッテ"; the shelf shows
### F-12 — l06 package version — P2 · major · **FIXED 2026-09-29**
- **Evidence:** the packshot showed "内臓脂肪ケア フィッテ"; the shelf shows
"内臓脂肪を減らす キユーピーフィッテ". Improved matcher 4/4 at full, 1/4 at
live; 3/3 with a shelf reference.
- **Change:** enroll the current package; adopt the packshot-OCR check.
- **Fix:** the current package is enrolled. It was recorded as blocked on a
capture because Kewpie's own site serves the superseded design for that JAN;
an independent retailer listing carries the current one, and it matches the
shelf facings exactly (checked by eye against a crop of `add/image6.jpg`
before enrolling). Measured: live decision 60% → 64%, end-to-end 59% → 62%,
full decision 81% → 83%.
- **Still open:** the packshot-OCR check that would have caught this. It is
implemented (`rebuild_catalog` refuses a view whose name/OCR score is below
0.70) but it is a false negative on l04 and l07, so it cannot yet be trusted
as a gate - see the catalog corrections section.
### F-13 — stale test status — P3 · polish · VERIFIED
- **Evidence:** 49/49 pass on both trees. An `AttributeError: 'str' object has
@ -572,7 +581,7 @@ running the real server against `add/image6.jpg`.
| F-07 evaluator defects | Done. Four separate ceilings, honest precision denominator, catalog-derived candidate set, `sku_of` grouping, leak signature. |
| F-08 DINO fenced off | Done. Not the default, not in the live path, refuses to start without weights. |
| F-10 uncommitted live experiments | Reverted. `crop_first_attempt`, the blur early return, `vote_agree` and `max_attempts 4` are gone. |
| F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. |
| F-12 l06 current package | **Fixed.** The current package is enrolled from an independent retailer listing and matches the shelf facings exactly. Live decision 60% → 64%, end-to-end 59% → 62%. |
| F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. |
| F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. |
| Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and each view's bytes, not just the view names: names alone let a different packshot be swapped in under the same name with the hash unchanged. |
@ -826,6 +835,48 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### External benchmark: 20 images, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is
measured on data the design was tuned against. Twenty images for eight SKUs
were collected from independent sources (six retailer and marketplace domains)
to test the pipeline on data it had no part in building. None is Tier 2; all
are `external_proxy` and every manifest says so.
Leak-tested first: the highest template match of any of the 20 against
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
| Set | Images | Correct | Abstain | False accepts |
|---|---|---|---|---|
| `kewpie/` | 12 | 9 | 3 | **0** |
| `other-mayo/` | 8 | 4 | 4 | **0** |
| **Total** | **20** | **13** | **7** | **0** |
- **`sku_l07` is answered with no enrolled image.** Both its images accept
through the text lane alone - one reads ラッキー フヨネーズ, the other the
rear label's 製造者：丸和油脂株式会社 with 内容量：200g, which together
name the row 丸和油脂 ラッキーマヨネーズ 200g. This is the text lane's
coverage claim holding on data that was not used to build it. `sku_l02`
also accepts one image on text alone, but it does have an enrolled view.
- **The l04 abstentions are correct behaviour.** l04 is also unenrolled and its
reads are weak: one reads only コと旨 タ 味 (every row scores 0.0), another
scores 0.379 against a 0.45 accept line. Abstaining on a weak read is the
designed outcome, not a miss.
The seven abstentions are all unenrolled or weakly-read SKUs. No image was
given a wrong name, which is the property the decision optimises for.
### Tier-3 sources are staged, not satisfied
Two reusable pan clips were found and saved (`data/external-eval/tier3-sources/`):
a 1920x1080 CC BY 3.0 supermarket pan and a 360p CC BY mayonnaise-bay pan, with
11 rejected candidates recorded. Neither gives per-SKU identity for this
catalog, and the only LOPIA store walk found carries the standard YouTube
licence and was therefore not downloaded. Tier 3 still needs the operator's own
recorded walks.
Still open, and not implementable from here: Tier 2 held-out labels (F-11), the
remaining E-1/E-4..E-9 experiments, and the completion gate G1-G7, all of which
need a second independent shelf visit.