product-recognition-demo/recognization/docs/DATA.md

Capturing real data

Everything measured in this project so far comes from one or two photographs. That is its largest weakness: the identity benchmark takes its gallery and its queries from the same photo, so every accuracy number is an in-distribution number.

This is the procedure for replacing that with data from a real store, recorded on the device that will do the scanning.

1. Record a walk

Start the server with recording on. Frames are written every Nth processed frame, and each one is written with a line describing what the server saw in it.

cd recognization
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
  --port 8090 --host 0.0.0.0 \
  --classify-workers 8 \
  --record-frames ~/dorayaki-data/2026-09-25-storeA \
  --record-every 3

Then run the app and scan. Each scan session becomes its own directory:

~/dorayaki-data/2026-09-25-storeA/<session>/
    000000.jpg        the frame, exactly as the client sent it
    000001.jpg
    index.jsonl       one line per recorded frame:
                      {"frame": 123, "path": "000001.jpg", "at": 1699...,
                       "summary": {...},
                       "tracks": [{"box": [...], "identity": {...},
                                   "tag_identity": {...}, "tag": {...}}]}

Frames are sampled, not continuous, and the index keeps each frame's own number so the sampling is visible rather than silent. /v1/health reports how many frames have been written.

2. What to capture, and why

Two visits to the same shelf are worth more than one long visit, because the benchmark needs a cross-visit split: the gallery comes from visit 1 and the test set from visit 2. That is the difference between "it works on this photo" and "it works on this shelf".

Per shelf, capture:

• a slow walk along each rail at working distance (about 1 m) — the live loop's real input;
• a close pass (about 30 cm) over a few tags — the only distance where the barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
• the same shelf again on a second visit, ideally with something changed: a gap, a moved facing, a new product.

3. Turn a recording into labels

The tag channel seeds the labels. A facing whose tag named a product, with a price the tag's own arithmetic corroborates, is a candidate label rather than a guess:

python -c "
import json, sys
sys.path.insert(0, 'code')
from record import labelling_candidates
seeds = labelling_candidates('~/dorayaki-data/2026-09-25-storeA/<session>/index.jsonl')
print(len(seeds), 'candidate labels')
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
"

A human then confirms or corrects them in the app's confirmation screen — the same screen that closes the flywheel. Nothing here is a training set on its own; it is the raw material for one.

4. Freeze a benchmark

Write the confirmed labels into a manifest that the evaluation reads, and keep it in the repository so results are reproducible:

benchmarks/<store>-<date>/
    visit1/<sku_id>/<view>.jpg      the gallery: 2-3 views per SKU
    visit2/*.jpg                    the test frames
    labels.json                     {"visit2/0007.jpg": [{"box": [...], "sku_id": "..."}]}

Then the identity benchmark (see code/bench_identity.py) can be run with a cross-visit split, and every number it reports is a number worth acting on. Until that exists, treat the current figures as what they are: measurements on a photograph.