docs(decision): report the external benchmark on the corrected basis
19 scored rather than 20: sku_l06_1.jpg is now the enrolled l06 view, so it is reported as leaked instead of as a held-out correct accept. 12 correct, 7 abstain, 0 false accepts. Both leak checks are named in the record and in the external README, since the enrollment that closed F-12 is exactly the kind of change that silently converts a held-out row into training data.
This commit is contained in:
parent
e25fcda67d
commit
56cff09292
2 changed files with 34 additions and 21 deletions
|
|
@ -8,25 +8,30 @@ Tier 2.** DECISION.md's Tier 2 is a second labelled store visit; these are
|
|||
|---|---|
|
||||
| `other-mayo/` | 8 accepted images for `sku_l04`, `sku_l07`, `sku_l09` from 6 source domains, plus the scored result |
|
||||
| `l12-reconciliation/` | What `sku_l12` actually is, from the shelf tag and package pixels |
|
||||
| `kewpie/` | Kewpie 450 g research (the near-unknown case) |
|
||||
| `kewpie/` | 12 accepted images for six SKUs from 6 source domains, plus the scored result. One is now enrolled, so it is excluded from scoring. |
|
||||
| `tier3-sources/` | Two reusable pan clips and 11 rejected candidates for a future Tier-3 replay |
|
||||
|
||||
## Leak test
|
||||
## Leak checks
|
||||
|
||||
Every accepted image was template-matched against `add/image6.jpg`. Highest
|
||||
score across all 8 is 0.675 (`sku_l07_1`), and the 0.95+ band that identifies a
|
||||
crop of the evaluation photo is empty. The images are genuinely external.
|
||||
Two, both in `code/eval_external.py`, both run before any number is reported:
|
||||
|
||||
## Result: 8 images, 0 false accepts
|
||||
- **Against the evaluation photo.** Every image is template-matched against
|
||||
`add/image6.jpg`. The highest score across all 20 is 0.675 (`sku_l07_1`), and
|
||||
the 0.95+ band that identifies a crop of the evaluation photo is empty.
|
||||
- **Against the catalog.** An image that has since been enrolled is refused
|
||||
rather than scored. `kewpie/sku_l06_1.jpg` is one: it is now the enrolled l06
|
||||
view, so scoring it would be training accuracy wearing a held-out label.
|
||||
|
||||
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
|
||||
`stock-lopia-clean` (`other-mayo/eval_result.json`):
|
||||
## Result: 19 scored, 0 false accepts
|
||||
|
||||
| Verdict | Count | Files |
|
||||
|---|---|---|
|
||||
| Correct accept | 4 | `sku_l07_1`, `sku_l07_2`, `sku_l09_1`, `sku_l09_2` |
|
||||
| Abstain | 4 | all `sku_l04_*` |
|
||||
| False accept | **0** | — |
|
||||
.venv/bin/python code/eval_external.py --stock stock-lopia-clean \
|
||||
data/external-eval/kewpie data/external-eval/other-mayo
|
||||
|
||||
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|
||||
|---|---|---|---|---|---|
|
||||
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
|
||||
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
|
||||
| **Total** | **19** | **12** | **7** | **0** | **1** |
|
||||
|
||||
Two things this establishes that the image6 shelf cannot:
|
||||
|
||||
|
|
|
|||
|
|
@ -835,7 +835,7 @@ out of "210g"), and percentages are excluded because "カロリー 50% カット
|
|||
claim about the contents. Measured on the corrected labels, full resolution:
|
||||
precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
|
||||
|
||||
### External benchmark: 20 images, 0 false accepts
|
||||
### External benchmark: 19 scored, 0 false accepts
|
||||
|
||||
The image6 shelf is one photo with one label set, so every number above is
|
||||
measured on data the design was tuned against. Twenty images for eight SKUs
|
||||
|
|
@ -843,17 +843,25 @@ were collected from independent sources (six retailer and marketplace domains)
|
|||
to test the pipeline on data it had no part in building. None is Tier 2; all
|
||||
are `external_proxy` and every manifest says so.
|
||||
|
||||
Leak-tested first: the highest template match of any of the 20 against
|
||||
`add/image6.jpg` is 0.675, so none is a crop of the evaluation photo.
|
||||
Two leak checks run before any number is reported, both in
|
||||
`code/eval_external.py`:
|
||||
|
||||
- Against the evaluation photo: the highest template match of any image against
|
||||
`add/image6.jpg` is 0.675, so none is a crop of it.
|
||||
- Against the catalog: an image that has since been enrolled is refused rather
|
||||
than scored. `sku_l06_1.jpg` is one - it is now the enrolled l06 view, so
|
||||
scoring it would be training accuracy wearing a held-out label. It is
|
||||
reported as `leaked`, not as a correct accept.
|
||||
|
||||
Scored through the shipped `LocalLaneAdapter` + `DecisionPolicy` against
|
||||
`stock-lopia-clean` (`data/external-eval/*/eval_result.json`):
|
||||
|
||||
| Set | Images | Correct | Abstain | False accepts |
|
||||
|---|---|---|---|---|
|
||||
| `kewpie/` | 12 | 9 | 3 | **0** |
|
||||
| `other-mayo/` | 8 | 4 | 4 | **0** |
|
||||
| **Total** | **20** | **13** | **7** | **0** |
|
||||
| Set | Scored | Correct | Abstain | False accepts | Leaked |
|
||||
|---|---|---|---|---|---|
|
||||
| `kewpie/` | 11 | 8 | 3 | **0** | 1 |
|
||||
| `other-mayo/` | 8 | 4 | 4 | **0** | 0 |
|
||||
| **Total** | **19** | **12** | **7** | **0** | **1** |
|
||||
|
||||
- **`sku_l07` is answered with no enrolled image.** Both its images accept
|
||||
through the text lane alone - one reads ラッキー フヨネーズ, the other the
|
||||
rear label's 製造者:丸和油脂株式会社 with 内容量:200g, which together
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue