tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

chore(eval): reconcile the distractor sets and refresh the sweep numbers

Browse source 
The Wikimedia collector truncated the whole basename at 60 characters, so 13
files lost their extension and cv2 could not read them by name - the distractor
sets were quietly scoring 13 images fewer than they held. It now truncates the
stem and keeps the extension, and the files on disk are renamed to match.

The search terms also pull in .djvu volumes whose page 1 Commons serves as a
JPEG thumbnail. Twelve book scans were in the sets; they contributed 0 false
accepts, and they are not shelves, so the collector filters them and the
manifests no longer list them.

With all 2650 crops readable the sweep is re-measured end to end: 3.09% at the
old lines, 0.30% at the shipped 4.0/4, image6 44 shown / 42 correct / 95.5%.
The accept sweep table and the verdict's far-unknown sentence now carry those
numbers instead of the pre-calibration ones.

Also: the fetched images are gitignored (manifests and results are not), the
l10/l11 shelf-tag crops that back the catalog corrections are committed, and a
stale pre-gate eval.json and a duplicate of eval/labels/image6.json are gone.
This commit is contained in:
kiyo9w 2026-09-29 04:17:54 +09:00
parent 4bdad66bbe
commit 5eb4bdf6c8
19 changed files with 646 additions and 253 deletions
Download patch file Download diff file Expand all files Collapse all files

55
recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
View file

@ -57,11 +57,12 @@ store-scoped, enrolled catalog, with live-video evidence accumulation**.
becomes retrieval plus verification (§6).
- **Open set is the dominant difficulty, and it comes in two kinds.** *Far*
unknowns (other categories) are rejected easily **once the accept lines are
calibrated against them**: 0.37 % and 0.24 % false accepts on 2297 crops from
56 unrelated Wikimedia supermarket photos, at a threshold that keeps 42 of 48
correct image6 labels. The audit's 0.2-0.7 % was measured on 602 crops from
the `add/` photos with the audit's own variant matcher; the shipped lane at
the shipped lines was 3.61 % until 2026-09-29 (§ Re-measured, far unknowns).
calibrated against them**: 3.09 % at the old lines, 0.30 % at the shipped
ones, on 2650 crops from 60 unrelated Wikimedia supermarket photos, at a
threshold that keeps 42 of 48 correct image6 labels. The audit's 0.2-0.7 %
was measured on 602 crops from the `add/` photos with the audit's own variant
matcher; the shipped lane at the shipped lines was 3.09 % until 2026-09-29
(§ Re-measured, far unknowns).
*Near*
unknowns (siblings of enrolled products: Kewpie 450 g next to Kewpie からし,
Pure Select plain next to Pure Select コクうま) produce genuine,
@ -396,7 +397,7 @@ with always-argmax, run on that same catalog.
Status 2026-09-29, offline on image6 plus the Wikimedia distractor sets:
**G2 passes** (95 % precision, full resolution). **G3's far-unknown half
passes** (0.37 % / 0.24 % on 2297 unrelated crops); its near-unknown half is
passes** (0.30 % on 2650 unrelated crops); its near-unknown half is
measured on image6 only, where the size gate rejects the documented 450 g case.
**G8 passes** (identity p95 well inside 60 ms). **G1, G4, G5, G6, G7 are not
yet measured on held-out data** - Tier 2 does not exist, and G4/G5 need a
@ -656,13 +657,16 @@ facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75 1.00 and
1.66, so 2.0 is the boundary in both. Sweeping it at full resolution, through
the evaluator (`--accept`, columns as it defines them: end-to-end is
accepted-and-correct over the 64 in-catalog facings, precision is over the
facings it answered):
facings it answered). Re-measured 2026-09-29 after the accept lines were
calibrated against unrelated shelves (§ Far unknowns); the 2.0 row is what the
audit measured, the 4.0 row is the shipped line:
accept end-to-end precision false accepts
2.0 75% 91% 5
3.0 67% 91% 4
4.0 67% 93% 3
6.0 62% 95% 2
2.0 75% 92% 4
3.0 66% 93% 3
4.0 66% 95% 2
5.0 62% 98% 1
6.0 59% 97% 1
The line is still provisional - it is tuned on Tier 1 and needs Tier-2
calibration - but it does not need to differ by input path, and the design
@ -853,12 +857,9 @@ precision 86% -> 91%, false accepts 8 -> 5, coverage unchanged at 79%.
### Far unknowns: the accept lines were tuned on a shelf that has no unknowns
Every threshold in this document was fitted on image6, where each crop is a
facing of a product the shelf actually holds. That is a closed-set tuning set,
and it hid the far-unknown rate completely: the shipped lane accepted 83 of
2297 crops from 56 Wikimedia Commons supermarket photos of milk, soy and snack
shelves - 3.61 %, against G3's 1 %.
and it hid the far-unknown rate completely: the shipped lane accepted 82 of
2650 crops from 60 Wikimedia Commons supermarket photos of milk, soy and snack
shelves - 3.09 %, against G3's 1 %.
Those photos share nothing with the LOPIA catalog, so every accept is a false
accept by construction. Two mechanisms produced them:
@ -867,24 +868,24 @@ accept by construction. Two mechanisms produced them:
drops bigrams no catalog name contains, so a short garbage read can end up
with a perfect ratio on two pairs. 20 of 26 text-lane answers rested on
exactly two: `避くて おいしい` named `sku_l12` through おい/いし.
- **The local lane accepted pure geometry.** The residual 25 were matches with
no text overlap at all, scoring 2.26-4.81 against an accept line of 2.0.
- **The local lane accepted pure geometry.** The residual accepts were matches
with no text overlap at all, scoring 2.26-4.81 against an accept line of 2.0.
Both lines moved: `TEXT_MIN_SHARED` 2 -> 4, `LOCAL_ACCEPT` 2.0 -> 4.0. Measured
on 2297 distractor crops and 109 image6 facings:
on 2650 distractor crops and 109 image6 facings:
| `LOCAL_ACCEPT` | `TEXT_MIN_SHARED` | image6 shown / correct | precision | false accepts |
|---|---|---|---|---|
| 2.0 | 2 | 54 / 49 | 90.7 % | 83 (3.61 %) |
| 2.0 | 4 | 52 / 48 | 92.3 % | 30 (1.31 %) |
| **4.0** | **4** | **44 / 42** | **95.5 %** | **8 (0.35 %)** |
| 5.0 | 6 | 36 / 36 | 100.0 % | 0 (0.00 %) |
| 2.0 | 2 | 46 / 43 | 93.5 % | 82 (3.09 %) |
| 2.0 | 4 | 44 / 42 | 95.5 % | 8 (0.30 %) |
| **4.0** | **4** | **44 / 42** | **95.5 %** | **8 (0.30 %)** |
| 5.0 | 6 | 39 / 38 | 97.4 % | 3 (0.11 %) |
4.0/4 is the only region passing both G2 (>= 95 % precision) and G3 (<= 1 %
far-unknown) with coverage left. On the shipped evaluator that is 66 %
end-to-end at 95 % precision with 2 false accepts, against 77 % / 91 % / 5
before - the coverage is what the precision costs, and G2 is the gate that says
to pay it.
end-to-end at 95 % precision with 2 false accepts, against 75 % / 92 % / 4 at
the old 2.0 line - the coverage is what the precision costs, and G2 is the gate
that says to pay it.
Reproduce: `.venv/bin/python code/eval_distractors.py --stock stock-lopia-clean
--write data/external-shelves/wikimedia-commons data/external-shelves/wikimedia-commons-2`.