# Evaluation on real store photos from Wikimedia Commons

Everything else in this repository is measured on two photographs. This evaluates the
pipeline on **54 real store photos it has never seen**, collected from Wikimedia
Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a
manifest.

Reproduce:

```bash
python code/collect_shelf_images.py ~/dorayaki-data/shelf-images   # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images
```

## 1. Detection transfers, and well

| | |
| --- | --- |
| Images with at least one detection | **46 of 54** |
| Total detections | **2,258** |
| Book scans, illustrations, a photo of a shop front | 0 detections (correct) |

The detector is the teammate's single-class YOLO trained on one shelf, and it found
individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice
racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with
tight boxes. Detection is not the weak link.

## 2. Identity transfers when the product is genuinely present

The original 21-SKU catalog contains 明治 おいしい牛乳. On
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case photographed in 2011, entirely
unseen - the pipeline found **6 facings of 明治 おいしい牛乳 and labelled them
correctly**, and on `File:Kawasaki-Inryo-1.jpg` it found 野菜生活100 and
プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it
has templates for.

## 3. Precision is the failure, and rejection is missing entirely

The same run labelled these as 明治 おいしい牛乳, in the same photograph:

* 低脂肪殺菌牛乳 (a タカナシ carton, top left)
* 有機豆乳 (top left corner)
* 特濃4.5 (three cartons, top middle)
* 無調整豆乳 (top right)
* 北海道牛乳 (several, bottom right)

They are all white milk cartons. The matcher has no way to say "unknown", so every
look-alike becomes a claim.

Measured over 24 foreign shelves whose products are *not* in the catalog: **451
facings accepted as recognised products**, with the score distribution overlapping
the real one and its extremes landing on the wrong images:

| | accepted | median score | p10 | p90 |
| --- | --- | --- | --- | --- |
| The shelf the catalog describes (`image6`) | 71 | 18.2 | 5.2 | 372.4 |
| Foreign shelves (24 images) | 451 | 6.0 | — | — |
| A bento (「198円 中華弁当」) | 1 | 50.5 | | |
| A supermarket interior | 1 | 80.6 | | |
| Deer cookies at Nara | 6 | 33.1 | | 189.9 |

**No threshold on this score separates the two**: it would have to sit above 5 (to
catch the foreign p10) and below 6 (to keep the real median), and the worst offenders
score higher than any correct match on the demo shelf.

## 4. What this means for the plan

It confirms, on real data, the two conclusions the literature and the review pointed
at - and it makes them concrete:

1. **The matcher's raw inlier score is not a confidence.** It cannot be used for
   rejection or for ranking, and it is the reason packaging recognition cannot name
   products on its own.
2. **Fine-grained look-alikes are the failure mode**: milk cartons, soy-milk cartons,
   and the twenty near-identical mayonnaise packs from the earlier benchmark. The
   GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a
   17.5-point drop at Recall@1): global embeddings cluster categories well and rank
   visually similar SKUs badly.

So step 4 of the plan is not a refinement, it is the fix: a **shortlist followed by
local re-ranking** (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a
**calibrated rejection** on an embedding distance where same-product and
different-product are actually separable (measured earlier: above 0.6 and below 0.5
respectively, unlike this score).

## Attribution

The images are from Wikimedia Commons under their individual licences (CC0, CC BY,
CC BY-SA, public domain); `manifest.json` records the author, licence and source URL
of each one. They are evaluation material and are not redistributed here.