tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

docs(decision): report the external benchmark on the corrected basis

Browse source 
19 scored rather than 20: sku_l06_1.jpg is now the enrolled l06 view, so it is
reported as leaked instead of as a held-out correct accept. 12 correct,
7 abstain, 0 false accepts.

Both leak checks are named in the record and in the external README, since the
enrollment that closed F-12 is exactly the kind of change that silently
converts a held-out row into training data.
This commit is contained in:
kiyo9w 2026-09-29 03:30:16 +09:00
parent e25fcda67d
commit 56cff09292
2 changed files with 34 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

31
recognization/data/external-eval/README.md
View file

@ -8,25 +8,30 @@ Tier 2.** DECISION.md's Tier 2 is a second labelled store visit; these are
|---|---|
| `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result |
| `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels |
| `kewpie/` | Kewpie 450 g research (the near-unknown case) |
| `kewpie/` | 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring. |
| `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay |
## Leak test
## Leak checks
Every accepted image was template-matched against `add/image6.jpg`. Highest
score across all 8 is 0.675 (`sku_l07_1`), and the 0.95+ band that identifies a
crop of the evaluation photo is empty. The images are genuinely external.
Two, both in `code/eval_external.py`, both run before any number is reported:
## Result: 8 images, 0 false accepts
- **Against the evaluation photo.** Every image is template-matched against
`add/image6.jpg`. The highest score across all 20 is 0.675 (`sku_l07_1`), and
the 0.95+ band that identifies a crop of the evaluation photo is empty.
- **Against the catalog.** An image that has since been enrolled is refused
rather than scored. `kewpie/sku_l06_1.jpg` is one: it is now the enrolled l06
view, so scoring it would be training accuracy wearing a held-out label.
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
`stock-lopia-clean` (`other-mayo/eval_result.json`):
## Result: 19 scored, 0 false accepts
| Verdict | Count | Files |
|---|---|---|
| Correct accept | 4 | `sku_l07_1`, `sku_l07_2`, `sku_l09_1`, `sku_l09_2` |
| Abstain | 4 | all `sku_l04_*` |
| False accept | **0** | — |
.venv/bin/python code/eval_external.py --stock stock-lopia-clean \
data/external-eval/kewpie data/external-eval/other-mayo
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|---|---|---|---|---|---|
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
| **Total** | **19** | **12** | **7** | **0** | **1** |
Two things this establishes that the image6 shelf cannot: