product-recognition-demo/recognization/docs/DATA.md

Capturing real data

Everything measured in this project so far comes from one or two photographs. That is its largest weakness: the identity benchmark takes its gallery and its queries from the same photo, so every accuracy number is an in-distribution number.

This is the procedure for replacing that with data from a real store, recorded on the device that will do the scanning.

1. Record a walk

From recognization, run ./run-demo-server.sh stock-lopia once to provision the ignored config/local.env token and the native apps' local LAN settings, then stop that server. Rebuild the app after provisioning. The token is a secret: keep .demo-token, config/local.env, and app-local settings out of commits and shared logs. For a recorded walk, start the same gallery backend on the launcher's port; code/app.py reads the token from config/local.env:

cd product-recognition-demo/recognization  # from the app repository root
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
  --port 8099 --host 0.0.0.0 --stock-dir stock-lopia --identity gallery \
  --live-conf 0.15 --classify-workers 8 \
  --record-frames ~/dorayaki-data/storeA-visit1 \
  --record-every 3

This requires the local pinned DINOv2 file described in ../README.md (weights/dinov2_vits14_lvd142m.safetensors, or DORAYAKI_DINOV2_WEIGHTS). A LAN bind without a token is refused. Record the second visit to a separate directory with the same catalog and model.

Then run the app and scan. Each scan session becomes its own directory:

~/dorayaki-data/storeA-visit1/<session>/
    000000.jpg        the frame, exactly as the client sent it
    000001.jpg
    index.jsonl       one line per recorded frame:
                      {"frame": 123, "path": "000001.jpg", "at": 1699...,
                       "identity_backend": "gallery",
                       "summary": {...},
                       "tracks": [{"box": [...], "identity": {...},
                                   "tag_identity": {...}, "tag": {...}}]}

Frames are sampled, not continuous, and the index keeps each frame's own number so the sampling is visible rather than silent. /v1/health reports how many frames have been written.

2. What to capture, and why

Two visits to the same shelf are worth more than one long visit, because the benchmark needs a cross-visit split: the gallery comes from visit 1 and the test set from visit 2. That is the difference between "it works on this photo" and "it works on this shelf".

Per shelf, capture:

• a slow walk along each rail at working distance (about 1 m) — the live loop's real input;
• a close pass (about 30 cm) over a few tags — the only distance where the barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
• the same shelf again on a second visit, ideally with something changed: a gap, a moved facing, a new product.

3. Turn a recording into labels

The tag channel seeds the labels. A facing whose tag named a product, with a price the tag's own arithmetic corroborates, is a candidate label rather than a guess:

python -c '
import json, sys
sys.path.insert(0, "code")
from record import labelling_candidates
seeds = labelling_candidates(sys.argv[1])
print(len(seeds), "candidate labels")
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
' "$HOME/dorayaki-data/storeA-visit1/REPLACE_WITH_SESSION/index.jsonl"

A human then confirms or corrects them in the app's confirmation screen — the same screen that closes the flywheel. Nothing here is a training set on its own; it is the raw material for one.

4. Replay a cross-visit benchmark

Enroll the gallery from visit 1 only, then freeze that catalog. Confirm labels for visit 2 independently; labels.json beside its index.jsonl (or supplied with --labels) is a JSON list, or {"labels": [...]}. Each entry has frame (the number in index.jsonl), box (pixel coordinates [x1, y1, x2, y2]), and sku_id (a catalog SKU). These are confirmed annotations, not the server's predictions. For the second visit, run from recognization with the same frozen stock-lopia catalog:

SESSION="$HOME/dorayaki-data/storeA-visit2/REPLACE_WITH_SESSION"
YOLO_OFFLINE=1 .venv/bin/python code/eval_live.py "$SESSION" \
  --stock-dir stock-lopia --labels "$SESSION/labels.json" \
  --replay --device cpu

A recording written by the current server names identity_backend on each index line, and the evaluator uses that. --identity gallery (or matcher) is required when the index omits it, including recordings from before that field existed, and must match the recorded value when both are present. It does not silently score the matcher.

Replace REPLACE_WITH_SESSION with the actual session name. The evaluator replays recorded JPEGs through LiveSession, writes eval_live.json in that directory, and reports backend, loaded head, weight/catalog fingerprints, decision counts, and held-out labelled top-1 when valid labels exist. It rejects missing or unpinned DINOv2 weights rather than substituting matcher scores; labels copied from catalog templates are excluded as leakage. Without confirmed held-out visit-2 labels, counts are diagnostic only, not gallery walk accuracy. No held-out cross-visit result is available here yet.