product-recognition-demo/recognization/docs/EXTERNAL-EVAL.md

# Evaluation on real store photos from Wikimedia Commons

Everything else in this repository is measured on two photographs. This evaluates the
pipeline on **54 real store photos it has never seen**, collected from Wikimedia
Commons with their licences (CC0 / CC BY / CC BY-SA / public domain) recorded in a
manifest.

Reproduce:

```bash
python code/collect_shelf_images.py ~/dorayaki-data/shelf-images   # needs network
python code/eval_external_images.py ~/dorayaki-data/shelf-images
```

## 1. Detection transfers, and well

| | |
| --- | --- |
| Images with at least one detection | **46 of 54** |
| Total detections | **2,258** |
| Book scans, illustrations, a photo of a shop front | 0 detections (correct) |

The detector is the teammate's single-class YOLO trained on one shelf, and it found
individual cartons, bottles and packets on dairy cases, soy-sauce shelves, spice
racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong grocery - with
tight boxes. Detection is not the weak link.

## 2. Identity transfers when the product is genuinely present

The original 21-SKU catalog contains 明治 おいしい牛乳. On
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case photographed in 2011, entirely
unseen - the pipeline found **6 facings of 明治 おいしい牛乳 and labelled them
correctly**, and on `File:Kawasaki-Inryo-1.jpg` it found 野菜生活100 and
プロビオヨーグルト R-1. So the matcher does transfer to new photographs of products it
has templates for.

## 3. Precision is the failure, and rejection is missing entirely

The same run labelled these as 明治 おいしい牛乳, in the same photograph:

* 低脂肪殺菌牛乳 (a タカナシ carton, top left)
* 有機豆乳 (top left corner)
* 特濃4.5 (three cartons, top middle)
* 無調整豆乳 (top right)
* 北海道牛乳 (several, bottom right)

They are all white milk cartons. The matcher has no way to say "unknown", so every
look-alike becomes a claim.

Measured over 24 foreign shelves whose products are *not* in the catalog: **451
facings accepted as recognised products**, with the score distribution overlapping
the real one and its extremes landing on the wrong images:

| | accepted | median score | p10 | p90 |
| --- | --- | --- | --- | --- |
| The shelf the catalog describes (`image6`) | 71 | 18.2 | 5.2 | 372.4 |
| Foreign shelves (24 images) | 451 | 6.0 | — | — |
| A bento (「198円 中華弁当」) | 1 | 50.5 | | |
| A supermarket interior | 1 | 80.6 | | |
| Deer cookies at Nara | 6 | 33.1 | | 189.9 |

**No threshold on this score separates the two**: it would have to sit above 5 (to
catch the foreign p10) and below 6 (to keep the real median), and the worst offenders
score higher than any correct match on the demo shelf.

## 4. What this means for the plan

It confirms, on real data, the two conclusions the literature and the review pointed
at - and it makes them concrete:

1. **The matcher's raw inlier score is not a confidence.** It cannot be used for
   rejection or for ranking, and it is the reason packaging recognition cannot name
   products on its own.
2. **Fine-grained look-alikes are the failure mode**: milk cartons, soy-milk cartons,
   and the twenty near-identical mayonnaise packs from the earlier benchmark. The
   GroceryVision study of 190 open VLMs found the same shape (94.5 % Recall@5, a
   17.5-point drop at Recall@1): global embeddings cluster categories well and rank
   visually similar SKUs badly.

So step 4 of the plan is not a refinement, it is the fix: a **shortlist followed by
local re-ranking** (PRISM's shape: SigLIP shortlist, then LightGlue verification) and a
**calibrated rejection** on an embedding distance where same-product and
different-product are actually separable (measured earlier: above 0.6 and below 0.5
respectively, unlike this score).

## Attribution

The images are from Wikimedia Commons under their individual licences (CC0, CC BY,
CC BY-SA, public domain); `manifest.json` records the author, licence and source URL
of each one. They are evaluation material and are not redistributed here.


---

# Part 2: the identity stack, measured on a shelf where the product IS present

Part 1 measured shelves whose products are absent. This part uses
`File:2011年4月の牛乳 (5588310773).jpg` - a milk case where **14 of 107 facings are
明治 おいしい牛乳**, a product the original catalog holds - so precision and recall can
be computed instead of just false accepts. The 107 crops were labelled by eye from a
contact sheet.

## The shipped matcher

| | claims | correct | precision | recall |
| --- | --- | --- | --- | --- |
| matcher | 31 | 12 | **39 %** | **86 %** |

It finds almost every true facing and makes 19 false claims, on cartons of 特濃4.5,
低脂肪殺菌牛乳, 明治ラブ and 北海道牛乳. Correct where the product is, wrong on the
look-alikes: the same shape as Part 1.

## The embedding gallery: rejection, but not precision

| threshold | false accepts on foreign shelves | milk shelf: precision | recall |
| --- | --- | --- | --- |
| 0.45 | 29 % | 28 % | 79 % |
| 0.50 | 18 % | 29 % | 64 % |
| 0.55 | 11 % | 35 % | 50 % |
| **0.60** | **5 %** | 45 % | 36 % |
| 0.65 | 2 % | 0 % | 0 % |

The gallery's value is the **controllable rejection**: on shelves where nothing
matches, false accepts fall from the matcher's 47.8 % to **5 %** at 0.60, and the
curve is smooth enough to pick an operating point. But on the shelf where the product
*is* present its precision (28-45 %) is no better than the matcher's 39 % - it buys
honesty about the unknown, not finer discrimination.

## Enrolling a shelf from itself does not fix it

The flywheel's first step is to enroll a store's own shelf. Measured by taking 1-3
views per product from the labelled milk shelf and testing on the rest:

| views per SKU | top-1 | wrong | rejected |
| --- | --- | --- | --- |
| 1 | **26 %** | 30 | 29 |
| 2 | 25 % | 27 | 27 |
| 3 | 23 % | 24 | 26 |

Flat at ~25 %, and far below the 72-76 % measured on the demo shelf with the same
number of views. The demo shelf's facings are near-identical crops of the same
photograph; real milk cartons differ by small text and a colour band. **Off-the-shelf
DINOv2 is not discriminative enough for fine-grained SKUs**, which is what the
literature says (RP2K and RPC both fine-tune; GroceryVision reports a 17.5-point drop
from Recall@5 to Recall@1). Fine-tuning is not a refinement of the plan, it is a
requirement - and its labels are what the tag channel and the confirmation loop
produce.

## The packaging's own text is a real channel, and resolution-bound

The cartons differ by the text printed on them, so OCR was tested as an identity
channel - first single-scale, then with the tag reader's trick (two upscales, the red
channel, pooled):

| product | single scale | multi-scale + red |
| --- | --- | --- |
| MEGMILK (large Latin text) | 85 % | **85 %** |
| 北海道牛乳 (large kanji) | 65 % | **65 %** |
| 明治ラブ | 0 % | **75 %** |
| 特濃4.5 | 0 % | **50 %** |
| 明治 おいしい牛乳 (small vertical text) | 7 % | **28 %** |
| 低脂肪殺菌牛乳 / 無調整豆乳 / 酪農3.6 | 0 % | 0 % |
| **overall** | 51 % | **60 %** |

Text works where it is large and fails where it is small - the same resolution wall as
the price tags. This is a capture-spec question, not a model question.

## The identity architecture this implies

| channel | what it is good for | measured |
| --- | --- | --- |
| Detection | finding facings on unseen shelves | 46/54 images, tight boxes |
| Tag (corroborated) | exact identity and price, where readable | 15/15 fields on the demo shelf; sparse in the field |
| Packaging text | brand-level discrimination | 60 % recall, higher when text is large |
| Embedding gallery | **rejection** - saying "unknown" | 5 % false accepts at 0.60 |
| Fine-tuned embedding | fine-grained SKUs (milk cartons) | not built; off-the-shelf is ~25 % |
| Confirmation loop | everything the above cannot settle | not built |

So the honest design is: **reject by default**, name the product from the tag when it
is readable, from the packaging text when it is legible, from a fine-tuned embedding
otherwise - and hand the remainder to the operator, whose answers become the next
enrollment.


---

# Part 3: fine-tuning the embedding, and what it buys

Part 2 ended on the number that matters: enrolling a real shelf from itself leaves
off-the-shelf DINOv2 at ~25 % top-1, because real milk cartons differ by small text
and a colour band. The literature's answer is fine-tuning, so this measures it.

`code/finetune_identity.py` trains a small projection head (384 -> 128, cosine
classifier, angular margin) on frozen DINOv2 features, with augmentation standing in
for the second visit we do not have. Two enrolment views per product, twelve
augmented copies each, 72 held-out facings, 8 products. Balanced accuracy is reported
because the shelf is unbalanced (MEGMILK has 34 facings, 酪農3.6 has two - a system
that always answered "MEGMILK" would look respectable on the raw rate).

| | top-1 | balanced | answered | of those right |
| --- | --- | --- | --- | --- |
| raw DINOv2 cosine | 2/72 (2 %) | 1 % | 72 | 3 % |
| raw, rejecting below 0.55 | 2/72 (2 %) | 1 % | 44 | 5 % |
| **fine-tuned + margin** | 28/72 (38 %) | **73 %** | 72 | 39 % |
| **fine-tuned, rejecting below 0.4** | 17/72 (23 %) | 58 % | 18 | **94 %** |

Three things follow.

1. **Fine-tuning is not a refinement, it is the difference between working and not.**
   Balanced accuracy goes from **1 % to 73 %** on the same held-out facings. Off-the-
   shelf features cannot separate these products; a head trained on sixteen crops can.
2. **With a rejection threshold it becomes precise**: answering only 18 of 72 facings
   and being right on 94 % of them is exactly the operating point a store wants -
   name what you are sure of, and hand the rest to the operator. The raw model cannot
   do this at any threshold (5 % of answered).
3. **Where it fails is where the labels are coarse.** Per class, with rejection:
   明治ラブ 2/2, おいしい無調整豆乳 2/2, 低脂肪 1/1, 特濃4.5 1/2, 北海道牛乳 8/21,
   明治 おいしい牛乳 2/12, and **MEGMILK 1/32**. MEGMILK is the class I labelled most
   coarsely - the contact sheet shows what are probably several different products
   under one brand - so the class is internally inconsistent and the model correctly
   refuses most of it. Better labels, not a better model, is what that needs.

## The honest limit

Enrolment and held-out facings come from **one photograph**. Augmentation stands in
for a second visit, which is not the same thing: a real second visit changes the
light, the angle and the stock. So 73 % balanced and 94 % precision are **upper
bounds**, and the only way to know the real number is to record a second visit
(`docs/DATA.md`).

What is settled is the mechanism: **a head trained on labels the pipeline can produce
itself** - tag-named facings, plus operator confirmations - moves fine-grained
identity from useless to usable, and a rejection threshold turns it into something a
store can act on.


---

# Part 4: making it usable, and the trap that came with it

The head is now part of a store: `python code/finetune_identity.py --stock stock-lopia`
trains on that store's enrolled views and writes `<stock>/identity_head.pt`, which
`ProductGallery` loads automatically. The first attempt exposed a trap worth writing
down.

## A trained head destroys rejection

Measured on the same two sets (the home shelf it was enrolled from, and 20 shelves
whose products are absent):

| | home shelf accepted | foreign shelves accepted |
| --- | --- | --- |
| raw features, threshold 0.55 | 78 % | **5 %** |
| trained head, threshold 0.40 | 58 % | **44 %** |

A discriminative head has no "none of the above": trained with an angular margin to
separate seven products, it maps *everything* confidently onto its seven prototypes,
including cartons it has never seen. Rejection, which was the gallery's whole point,
disappears.

## The fix is two stages, and it needs no retraining

The two mechanisms fail in opposite directions, so each does what it is good at:

* **raw features decide whether to answer** (they generalise: 5 % false accepts on
  strangers),
* **the head decides what to answer** (it discriminates: 73 % balanced on
  fine-grained products against 1 % raw).

| | home shelf accepted | foreign shelves accepted |
| --- | --- | --- |
| **raw gate, then head ranking** | 47 % | **5 %** |

The home-shelf acceptance is lower than raw-only because the gate is strict, which is
the honest trade: fewer claims, and the claims are worth more.

## Still open

* A joint accuracy number on the LOPIA store needs held-out labels there; its
  enrolment views came from the facings the tags named, so scoring on those would be
  scoring on the training set.
* Training the head *with* out-of-distribution negatives (crops from other stores)
  should let it keep rejecting on its own, which would remove the need for the gate.
* Everything above is still one photograph per store. The second visit remains the
  measurement that settles it.