tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity
product-recognition-demo/recognization/data/external-eval
History
kiyo9w 11826cc43c feat(eval): report the external benchmark per SKU 
A set total hides which products the lane can and cannot name. Every SKU in
these sets has images from two independent source domains, so a SKU that scores
the same on both is not one lucky photograph.

The breakdown is what makes the two interesting results visible: l01, l02, l03,
l07 and l09 are 2/2 correct on both their domains, l07 with no enrolled image
at all, while l11 abstains on both and l04 abstains on all four of its images.
2026-09-29 03:37:04 +09:00
..
kewpie feat(eval): report the external benchmark per SKU 2026-09-29 03:37:04 +09:00
l12-reconciliation test(external): score 8 independent-source images, 0 false accepts 2026-09-29 03:13:16 +09:00
other-mayo feat(eval): report the external benchmark per SKU 2026-09-29 03:37:04 +09:00
tier3-sources test(external): score 8 independent-source images, 0 false accepts 2026-09-29 03:13:16 +09:00
README.md docs(decision): report the external benchmark on the corrected basis 2026-09-29 03:30:16 +09:00

README.md

External evidence (2026-09-29)

Independent-source media used to stress the identity pipeline. None of this is Tier 2. DECISION.md's Tier 2 is a second labelled store visit; these are external_proxy artifacts, and every manifest here says so.

Directory What it holds
other-mayo/ 8 accepted images for sku_l04, sku_l07, sku_l09 from 6 source domains, plus the scored result
l12-reconciliation/ What sku_l12 actually is, from the shelf tag and package pixels
kewpie/ 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring.
tier3-sources/ Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay

Leak checks

Two, both in code/eval_external.py, both run before any number is reported:

  • Against the evaluation photo. Every image is template-matched against add/image6.jpg. The highest score across all 20 is 0.675 (sku_l07_1), and the 0.95+ band that identifies a crop of the evaluation photo is empty.
  • Against the catalog. An image that has since been enrolled is refused rather than scored. kewpie/sku_l06_1.jpg is one: it is now the enrolled l06 view, so scoring it would be training accuracy wearing a held-out label.

Result: 19 scored, 0 false accepts

.venv/bin/python code/eval_external.py --stock stock-lopia-clean \
    data/external-eval/kewpie data/external-eval/other-mayo
Set Scored Correct Abstain False accepts Leaked
kewpie/ 11 8 3 0 1
other-mayo/ 8 4 4 0 0
Total 19 12 7 0 1

Two things this establishes that the image6 shelf cannot:

  • sku_l07 is answered with no enrolled image. Both l07 images accept through the text lane alone: sku_l07_1 reads ラッキー フヨネーズ, and sku_l07_2 reads the rear label's 製造者：丸和油脂株式会社 with 内容量：200g. The catalog row is 丸和油脂 ラッキーマヨネーズ 200g, so the manufacturer and the size together name it. This is the text lane's coverage claim - that a product with no reference image can still be named - holding on data that was not used to build it.
  • The l04 abstentions are correct behaviour, not a miss. l04 has no enrolled image either, and its reads are weak: sku_l04_2 reads only コと旨 タ 味 (all rows score 0.0), and sku_l04_4 reads the rear label with やまや garbled to まや, scoring 0.379 - below the accept line. Abstaining on a weak read is the designed outcome.

sku_l07_2 accepts with a negative local score (-0.55) on text evidence alone. That is the lane doing its job, but it is worth naming: the accept rests on two matched tokens (manufacturer + size), not on geometry.

What is still missing

No external source gives per-SKU identity for the whole catalog, and no reusable clip combines the target category with shelf-distance text resolution. The Tier-2 and Tier-3 gates in DECISION.md still need the operator's own recorded walks.