tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

docs(decision): report the external benchmark on the corrected basis

Browse source 
19 scored rather than 20: sku_l06_1.jpg is now the enrolled l06 view, so it is
reported as leaked instead of as a held-out correct accept. 12 correct,
7 abstain, 0 false accepts.

Both leak checks are named in the record and in the external README, since the
enrollment that closed F-12 is exactly the kind of change that silently
converts a held-out row into training data.
This commit is contained in:
kiyo9w 2026-09-29 03:30:16 +09:00
parent e25fcda67d
commit 56cff09292
2 changed files with 34 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

24
recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
View file

@ -835,7 +835,7 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
claim about the contents. Measured on the corrected labels, full resolution:
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### External benchmark: 20 images, 0 false accepts
### External benchmark: 19 scored, 0 false accepts
The image6 shelf is one photo with one label set, so every number above is
measured on data the design was tuned against. Twenty images for eight SKUs
@ -843,17 +843,25 @@ were collected from independent sources (six retailer and marketplace domains)
to test the pipeline on data it had no part in building. None is Tier 2; all
are `external_proxy` and every manifest says so.
Leak-tested first: the highest template match of any of the 20 against
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo.
Two leak checks run before any number is reported, both in
`code/eval_external.py`:
- Against the evaluation photo: the highest template match of any image against
`add/image6.jpg` is 0.675, so none is a crop of it.
- Against the catalog: an image that has since been enrolled is refused rather
than scored. `sku_l06_1.jpg` is one - it is now the enrolled l06 view, so
scoring it would be training accuracy wearing a held-out label. It is
reported as `leaked`, not as a correct accept.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
| Set | Images | Correct | Abstain | False accepts |
|---|---|---|---|---|
| `kewpie/` | 12 | 9 | 3 | **0** |
| `other-mayo/` | 8 | 4 | 4 | **0** |
| **Total** | **20** | **13** | **7** | **0** |
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|---|---|---|---|---|---|
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **19** | **12** | **7** | **0** | **1** |
- **`sku_l07` is answered with no enrolled image.** Both its images accept
through the text lane alone - one reads ラッキー フヨネーズ, the other the
rear label's 製造者：丸和油脂株式会社 with 内容量：200g, which together