f74366d6a4
Everything else here is measured on two photographs. This collects real store shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY, CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on them. 46 of 54 images carry a shelf; the rest are book scans and a shop front. What it found, and it is not what I expected: - **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - tight boxes on images the model has never seen, and correctly nothing on the book scans. - **Identity transfers when the product is genuinely there.** On a 2011 milk-case photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and R-1 elsewhere. The matcher does reach new photographs of products it has templates for. - **Precision is the failure, and rejection is missing.** In that same photograph it labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 - they are all white milk cartons. Across 24 shelves whose products are not in the catalog it accepted 451 identities, and the score cannot be thresholded to fix it: the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and 189.9 (deer cookies). A threshold that catches the foreigners also throws away the real matches. That confirms on real data what the literature and the review predicted: the matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection - is the fix, not a refinement. docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the images themselves are not redistributed. 175 tests pass. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
4 KiB
Evaluation on real store photos from Wikimedia Commons
Everything else in this repository is measured on two photographs. This evaluates the pipeline on 54 real store photos it has never seen, collected from Wikimedia Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a manifest.
Reproduce:
python code/collect_shelf_images.py ~/dorayaki-data/shelf-images # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images
1. Detection transfers, and well
| Images with at least one detection | 46 of 54 |
| Total detections | 2,258 |
| Book scans, illustrations, a photo of a shop front | 0 detections (correct) |
The detector is the teammate's single-class YOLO trained on one shelf, and it found individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with tight boxes. Detection is not the weak link.
2. Identity transfers when the product is genuinely present
The original 21-SKU catalog contains 明治 おいしい牛乳. On
File:2011年4月の牛乳 (5588310773).jpg - a milk case photographed in 2011, entirely
unseen - the pipeline found 6 facings of 明治 おいしい牛乳 and labelled them
correctly, and on File:Kawasaki-Inryo-1.jpg it found 野菜生活100 and
プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it
has templates for.
3. Precision is the failure, and rejection is missing entirely
The same run labelled these as 明治 おいしい牛乳, in the same photograph:
- 低脂肪殺菌牛乳 (a タカナシ carton, top left)
- 有機豆乳 (top left corner)
- 特濃4.5 (three cartons, top middle)
- 無調整豆乳 (top right)
- 北海道牛乳 (several, bottom right)
They are all white milk cartons. The matcher has no way to say "unknown", so every look-alike becomes a claim.
Measured over 24 foreign shelves whose products are not in the catalog: 451 facings accepted as recognised products, with the score distribution overlapping the real one and its extremes landing on the wrong images:
| accepted | median score | p10 | p90 | |
|---|---|---|---|---|
The shelf the catalog describes (image6) |
71 | 18.2 | 5.2 | 372.4 |
| Foreign shelves (24 images) | 451 | 6.0 | — | — |
| A bento (「198円 中華弁当」) | 1 | 50.5 | ||
| A supermarket interior | 1 | 80.6 | ||
| Deer cookies at Nara | 6 | 33.1 | 189.9 |
No threshold on this score separates the two: it would have to sit above 5 (to catch the foreign p10) and below 6 (to keep the real median), and the worst offenders score higher than any correct match on the demo shelf.
4. What this means for the plan
It confirms, on real data, the two conclusions the literature and the review pointed at - and it makes them concrete:
- The matcher's raw inlier score is not a confidence. It cannot be used for rejection or for ranking, and it is the reason packaging recognition cannot name products on its own.
- Fine-grained look-alikes are the failure mode: milk cartons, soy-milk cartons, and the twenty near-identical mayonnaise packs from the earlier benchmark. The GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a 17.5-point drop at Recall@1): global embeddings cluster categories well and rank visually similar SKUs badly.
So step 4 of the plan is not a refinement, it is the fix: a shortlist followed by local re-ranking (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a calibrated rejection on an embedding distance where same-product and different-product are actually separable (measured earlier: above 0.6 and below 0.5 respectively, unlike this score).
Attribution
The images are from Wikimedia Commons under their individual licences (CC0, CC BY,
CC BY-SA, public domain); manifest.json records the author, licence and source URL
of each one. They are evaluation material and are not redistributed here.