product-recognition-demo/recognization/docs/EXTERNAL-EVAL.md

# Evaluation on real store photos from Wikimedia Commons

Everything else in this repository is measured on two photographs. This evaluates the
pipeline on **54 real store photos it has never seen**, collected from Wikimedia
Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a
manifest.

Reproduce:

```bash
python code/collect_shelf_images.py ~/dorayaki-data/shelf-images   # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images
```

## 1. Detection transfers, and well

| | |
| --- | --- |
| Images with at least one detection | **46 of 54** |
| Total detections | **2,258** |
| Book scans, illustrations, a photo of a shop front | 0 detections (correct) |

The detector is the teammate's single-class YOLO trained on one shelf, and it found
individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice
racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with
tight boxes. Detection is not the weak link.

## 2. Identity transfers when the product is genuinely present

The original 21-SKU catalog contains 明治 おいしい牛乳. On
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case photographed in 2011, entirely
unseen - the pipeline found **6 facings of 明治 おいしい牛乳 and labelled them
correctly**, and on `File:Kawasaki-Inryo-1.jpg` it found 野菜生活100 and
プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it
has templates for.

## 3. Precision is the failure, and rejection is missing entirely

The same run labelled these as 明治 おいしい牛乳, in the same photograph:

* 低脂肪殺菌牛乳 (a タカナシ carton, top left)
* 有機豆乳 (top left corner)
* 特濃4.5 (three cartons, top middle)
* 無調整豆乳 (top right)
* 北海道牛乳 (several, bottom right)

They are all white milk cartons. The matcher has no way to say "unknown", so every
look-alike becomes a claim.

Measured over 24 foreign shelves whose products are *not* in the catalog: **451
facings accepted as recognised products**, with the score distribution overlapping
the real one and its extremes landing on the wrong images:

| | accepted | median score | p10 | p90 |
| --- | --- | --- | --- | --- |
| The shelf the catalog describes (`image6`) | 71 | 18.2 | 5.2 | 372.4 |
| Foreign shelves (24 images) | 451 | 6.0 | — | — |
| A bento (「198円 中華弁当」) | 1 | 50.5 | | |
| A supermarket interior | 1 | 80.6 | | |
| Deer cookies at Nara | 6 | 33.1 | | 189.9 |

**No threshold on this score separates the two**: it would have to sit above 5 (to
catch the foreign p10) and below 6 (to keep the real median), and the worst offenders
score higher than any correct match on the demo shelf.

## 4. What this means for the plan

It confirms, on real data, the two conclusions the literature and the review pointed
at - and it makes them concrete:

1. **The matcher's raw inlier score is not a confidence.** It cannot be used for
   rejection or for ranking, and it is the reason packaging recognition cannot name
   products on its own.
2. **Fine-grained look-alikes are the failure mode**: milk cartons, soy-milk cartons,
   and the twenty near-identical mayonnaise packs from the earlier benchmark. The
   GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a
   17.5-point drop at Recall@1): global embeddings cluster categories well and rank
   visually similar SKUs badly.

So step 4 of the plan is not a refinement, it is the fix: a **shortlist followed by
local re-ranking** (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a
**calibrated rejection** on an embedding distance where same-product and
different-product are actually separable (measured earlier: above 0.6 and below 0.5
respectively, unlike this score).

## Attribution

The images are from Wikimedia Commons under their individual licences (CC0, CC BY,
CC BY-SA, public domain); `manifest.json` records the author, licence and source URL
of each one. They are evaluation material and are not redistributed here.


---

# Part 2: the identity stack, measured on a shelf where the product IS present

Part 1 measured shelves whose products are absent. This part uses
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case where **14 of 107 facings are
明治 おいしい牛乳**, a product the original catalog holds - so precision and recall can
be computed instead of just false accepts. The 107 crops were labelled by eye from a
contact sheet.

## The shipped matcher

| | claims | correct | precision | recall |
| --- | --- | --- | --- | --- |
| matcher | 31 | 12 | **39 %** | **86 %** |

It finds almost every true facing and makes 19 false claims, on cartons of 特濃4.5,
低脂肪殺菌牛乳, 明治ラブ and 北海道牛乳. Correct where the product is, wrong on the
look-alikes: the same shape as Part 1.

## The embedding gallery: rejection, but not precision

| threshold | false accepts on foreign shelves | milk shelf: precision | recall |
| --- | --- | --- | --- |
| 0.45 | 29 % | 28 % | 79 % |
| 0.50 | 18 % | 29 % | 64 % |
| 0.55 | 11 % | 35 % | 50 % |
| **0.60** | **5 %** | 45 % | 36 % |
| 0.65 | 2 % | 0 % | 0 % |

The gallery's value is the **controllable rejection**: on shelves where nothing
matches, false accepts fall from the matcher's 47.8 % to **5 %** at 0.60, and the
curve is smooth enough to pick an operating point. But on the shelf where the product
*is* present its precision (28-45 %) is no better than the matcher's 39 % - it buys
honesty about the unknown, not finer discrimination.

## Enrolling a shelf from itself does not fix it

The flywheel's first step is to enroll a store's own shelf. Measured by taking 1-3
views per product from the labelled milk shelf and testing on the rest:

| views per SKU | top-1 | wrong | rejected |
| --- | --- | --- | --- |
| 1 | **26 %** | 30 | 29 |
| 2 | 25 % | 27 | 27 |
| 3 | 23 % | 24 | 26 |

Flat at ~25 %, and far below the 72-76 % measured on the demo shelf with the same
number of views. The demo shelf's facings are near-identical crops of the same
photograph; real milk cartons differ by small text and a colour band. **Off-the-shelf
DINOv2 is not discriminative enough for fine-grained SKUs**, which is what the
literature says (RP2K and RPC both fine-tune; GroceryVision reports a 17.5-point drop
from Recall@5 to Recall@1). Fine-tuning is not a refinement of the plan, it is a
requirement - and its labels are what the tag channel and the confirmation loop
produce.

## The packaging's own text is a real channel, and resolution-bound

The cartons differ by the text printed on them, so OCR was tested as an identity
channel - first single-scale, then with the tag reader's trick (two upscales, the red
channel, pooled):

| product | single scale | multi-scale + red |
| --- | --- | --- |
| MEGMILK (large Latin text) | 85 % | **85 %** |
| 北海道牛乳 (large kanji) | 65 % | **65 %** |
| 明治ラブ | 0 % | **75 %** |
| 特濃4.5 | 0 % | **50 %** |
| 明治 おいしい牛乳 (small vertical text) | 7 % | **28 %** |
| 低脂肪殺菌牛乳 / 無調整豆乳 / 酪農3.6 | 0 % | 0 % |
| **overall** | 51 % | **60 %** |

Text works where it is large and fails where it is small - the same resolution wall as
the price tags. This is a capture-spec question, not a model question.

## The identity architecture this implies

| channel | what it is good for | measured |
| --- | --- | --- |
| Detection | finding facings on unseen shelves | 46/54 images, tight boxes |
| Tag (corroborated) | exact identity and price, where readable | 15/15 fields on the demo shelf; sparse in the field |
| Packaging text | brand-level discrimination | 60 % recall, higher when text is large |
| Embedding gallery | **rejection** - saying "unknown" | 5 % false accepts at 0.60 |
| Fine-tuned embedding | fine-grained SKUs (milk cartons) | not built; off-the-shelf is ~25 % |
| Confirmation loop | everything the above cannot settle | not built |

So the honest design is: **reject by default**, name the product from the tag when it
is readable, from the packaging text when it is legible, from a fine-tuned embedding
otherwise - and hand the remainder to the operator, whose answers become the next
enrollment.