The live session scored crops with the matcher regardless of the configured identity
backend, so a live scan used the old mechanism while the still path used the enrolled
gallery - measured, the matcher accepts look-alikes and cannot say "unknown", which is
exactly what a real shelf needs it to do. score_crop now routes to the gallery, and
the decision uses a similarity scale (accept at 0.60 with a 0.04 margin) rather than
inlier-count thresholds that mean nothing on a cosine.
The synthetic walk is gone: demo.py, demo_replay.py, the /v1/demo endpoints and the
walk benchmark are deleted, and the app's frame source is the camera on a device (the
single-image replay stays for the simulator, which has no camera).
stock-demo merges both demo shelves (27 products, 62 views, trained head) so one
server recognises either image, and run-demo-server.sh starts it bound to 0.0.0.0 and
prints the exact URL to put in Config.xcconfig.
Verified over the LAN with 1280x960 frames: 40 frames in 5.3s, 9 products recognised
on demo image 1.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Everything else here is measured on two photographs. This collects real store
shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY,
CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on
them. 46 of 54 images carry a shelf; the rest are book scans and a shop front.
What it found, and it is not what I expected:
- **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves,
spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong
grocery - tight boxes on images the model has never seen, and correctly nothing on
the book scans.
- **Identity transfers when the product is genuinely there.** On a 2011 milk-case
photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and
R-1 elsewhere. The matcher does reach new photographs of products it has templates
for.
- **Precision is the failure, and rejection is missing.** In that same photograph it
labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 -
they are all white milk cartons. Across 24 shelves whose products are not in the
catalog it accepted 451 identities, and the score cannot be thresholded to fix it:
the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median
is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and
189.9 (deer cookies). A threshold that catches the foreigners also throws away the
real matches.
That confirms on real data what the literature and the review predicted: the
matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the
failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection -
is the fix, not a refinement.
docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the
images themselves are not redistributed. 175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
