f74366d6a4
Everything else here is measured on two photographs. This collects real store shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY, CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on them. 46 of 54 images carry a shelf; the rest are book scans and a shop front. What it found, and it is not what I expected: - **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves, spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - tight boxes on images the model has never seen, and correctly nothing on the book scans. - **Identity transfers when the product is genuinely there.** On a 2011 milk-case photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and R-1 elsewhere. The matcher does reach new photographs of products it has templates for. - **Precision is the failure, and rejection is missing.** In that same photograph it labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 - they are all white milk cartons. Across 24 shelves whose products are not in the catalog it accepted 451 identities, and the score cannot be thresholded to fix it: the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and 189.9 (deer cookies). A threshold that catches the foreigners also throws away the real matches. That confirms on real data what the literature and the review predicted: the matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection - is the fix, not a refinement. docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the images themselves are not redistributed. 175 tests pass. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
90 lines
4 KiB
Markdown
90 lines
4 KiB
Markdown
# Evaluation on real store photos from Wikimedia Commons
|
|
|
|
Everything else in this repository is measured on two photographs. This evaluates the
|
|
pipeline on **54 real store photos it has never seen**, collected from Wikimedia
|
|
Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a
|
|
manifest.
|
|
|
|
Reproduce:
|
|
|
|
```bash
|
|
python code/collect_shelf_images.py ~/dorayaki-data/shelf-images # needs network
|
|
python code/eval_external_images.py ~/dorayaki-data/shelf-images
|
|
```
|
|
|
|
## 1. Detection transfers, and well
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| Images with at least one detection | **46 of 54** |
|
|
| Total detections | **2,258** |
|
|
| Book scans, illustrations, a photo of a shop front | 0 detections (correct) |
|
|
|
|
The detector is the teammate's single-class YOLO trained on one shelf, and it found
|
|
individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice
|
|
racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with
|
|
tight boxes. Detection is not the weak link.
|
|
|
|
## 2. Identity transfers when the product is genuinely present
|
|
|
|
The original 21-SKU catalog contains 明治 おいしい牛乳. On
|
|
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case photographed in 2011, entirely
|
|
unseen - the pipeline found **6 facings of 明治 おいしい牛乳 and labelled them
|
|
correctly**, and on `File:Kawasaki-Inryo-1.jpg` it found 野菜生活100 and
|
|
プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it
|
|
has templates for.
|
|
|
|
## 3. Precision is the failure, and rejection is missing entirely
|
|
|
|
The same run labelled these as 明治 おいしい牛乳, in the same photograph:
|
|
|
|
* 低脂肪殺菌牛乳 (a タカナシ carton, top left)
|
|
* 有機豆乳 (top left corner)
|
|
* 特濃4.5 (three cartons, top middle)
|
|
* 無調整豆乳 (top right)
|
|
* 北海道牛乳 (several, bottom right)
|
|
|
|
They are all white milk cartons. The matcher has no way to say "unknown", so every
|
|
look-alike becomes a claim.
|
|
|
|
Measured over 24 foreign shelves whose products are *not* in the catalog: **451
|
|
facings accepted as recognised products**, with the score distribution overlapping
|
|
the real one and its extremes landing on the wrong images:
|
|
|
|
| | accepted | median score | p10 | p90 |
|
|
| --- | --- | --- | --- | --- |
|
|
| The shelf the catalog describes (`image6`) | 71 | 18.2 | 5.2 | 372.4 |
|
|
| Foreign shelves (24 images) | 451 | 6.0 | — | — |
|
|
| A bento (「198円 中華弁当」) | 1 | 50.5 | | |
|
|
| A supermarket interior | 1 | 80.6 | | |
|
|
| Deer cookies at Nara | 6 | 33.1 | | 189.9 |
|
|
|
|
**No threshold on this score separates the two**: it would have to sit above 5 (to
|
|
catch the foreign p10) and below 6 (to keep the real median), and the worst offenders
|
|
score higher than any correct match on the demo shelf.
|
|
|
|
## 4. What this means for the plan
|
|
|
|
It confirms, on real data, the two conclusions the literature and the review pointed
|
|
at - and it makes them concrete:
|
|
|
|
1. **The matcher's raw inlier score is not a confidence.** It cannot be used for
|
|
rejection or for ranking, and it is the reason packaging recognition cannot name
|
|
products on its own.
|
|
2. **Fine-grained look-alikes are the failure mode**: milk cartons, soy-milk cartons,
|
|
and the twenty near-identical mayonnaise packs from the earlier benchmark. The
|
|
GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a
|
|
17.5-point drop at Recall@1): global embeddings cluster categories well and rank
|
|
visually similar SKUs badly.
|
|
|
|
So step 4 of the plan is not a refinement, it is the fix: a **shortlist followed by
|
|
local re-ranking** (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a
|
|
**calibrated rejection** on an embedding distance where same-product and
|
|
different-product are actually separable (measured earlier: above 0.6 and below 0.5
|
|
respectively, unlike this score).
|
|
|
|
## Attribution
|
|
|
|
The images are from Wikimedia Commons under their individual licences (CC0, CC BY,
|
|
CC BY-SA, public domain); `manifest.json` records the author, licence and source URL
|
|
of each one. They are evaluation material and are not redistributed here.
|