The external evaluation showed the matcher cannot say "unknown": on 20 shelves whose
products are absent it claimed 47.8% of facings as recognised, and its raw inlier
score cannot be thresholded (foreign shelves' median 6.0 overlaps the real shelf's
5.2 p10, and the worst offenders score 50-190). So the gallery becomes a selectable
identity backend:
--identity gallery (or DORAYAKI_IDENTITY=gallery)
Measured on the same 20 foreign shelves: false accepts fall from 47.8% to 13.8% at
the default threshold and to **5%** at 0.60, with a smooth curve to pick an operating
point from. Detection is unchanged (the gallery only names boxes).
What it does NOT fix, measured on a milk shelf where 14 of 107 facings really are
明治 おいしい牛乳 (labelled by eye): precision is 28-45%, no better than the matcher's
39%, and enrolling the shelf from itself (1-3 views per product) is flat at ~25%
top-1. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs - the
same conclusion the literature reaches, now measured here.
The packaging's own text is a real channel and resolution-bound: 51% recall
single-scale, 60% with the tag reader's multi-scale + red-channel trick, 85% where
the text is large (MEGMILK) and 28% where it is small vertical Japanese.
docs/EXTERNAL-EVAL.md records all of it, including the identity architecture it
implies: reject by default, name from the tag when readable, from the packaging text
when legible, from a fine-tuned embedding otherwise, and hand the rest to the
operator.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Everything else here is measured on two photographs. This collects real store
shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY,
CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on
them. 46 of 54 images carry a shelf; the rest are book scans and a shop front.
What it found, and it is not what I expected:
- **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves,
spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong
grocery - tight boxes on images the model has never seen, and correctly nothing on
the book scans.
- **Identity transfers when the product is genuinely there.** On a 2011 milk-case
photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and
R-1 elsewhere. The matcher does reach new photographs of products it has templates
for.
- **Precision is the failure, and rejection is missing.** In that same photograph it
labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 -
they are all white milk cartons. Across 24 shelves whose products are not in the
catalog it accepted 451 identities, and the score cannot be thresholded to fix it:
the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median
is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and
189.9 (deer cookies). A threshold that catches the foreigners also throws away the
real matches.
That confirms on real data what the literature and the review predicted: the
matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the
failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection -
is the fix, not a refinement.
docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the
images themselves are not redistributed. 175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
