kiyo9w a718acdc15 feat: record a live session so it can become data

The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.

`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.

Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.

`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.

167 tests pass.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

2026-09-25 01:34:15 +09:00

3.4 KiB

Raw Blame History

Capturing real data

Everything measured in this project so far comes from one or two photographs. That is its largest weakness: the identity benchmark takes its gallery and its queries from the same photo, so every accuracy number is an in-distribution number.

This is the procedure for replacing that with data from a real store, recorded on the device that will do the scanning.

1. Record a walk

Start the server with recording on. Frames are written every Nth processed frame, and each one is written with a line describing what the server saw in it.

cd recognization
YOLO_OFFLINE=1 .venv/bin/python code/app.py \
  --port 8090 --host 0.0.0.0 \
  --classify-workers 8 \
  --record-frames ~/dorayaki-data/2026-09-25-storeA \
  --record-every 3

Then run the app and scan. Each scan session becomes its own directory:

~/dorayaki-data/2026-09-25-storeA/<session>/
    000000.jpg        the frame, exactly as the client sent it
    000001.jpg
    index.jsonl       one line per recorded frame:
                      {"frame": 123, "path": "000001.jpg", "at": 1699...,
                       "summary": {...},
                       "tracks": [{"box": [...], "identity": {...},
                                   "tag_identity": {...}, "tag": {...}}]}

Frames are sampled, not continuous, and the index keeps each frame's own number so the sampling is visible rather than silent. /v1/health reports how many frames have been written.

2. What to capture, and why

Two visits to the same shelf are worth more than one long visit, because the benchmark needs a cross-visit split: the gallery comes from visit 1 and the test set from visit 2. That is the difference between "it works on this photo" and "it works on this shelf".

Per shelf, capture:

a slow walk along each rail at working distance (about 1 m) — the live loop's real input;
a close pass (about 30 cm) over a few tags — the only distance where the barcode is decodable (measured: 0.63 px per module at 1 m needs ~3× more pixels);
the same shelf again on a second visit, ideally with something changed: a gap, a moved facing, a new product.

3. Turn a recording into labels

The tag channel seeds the labels. A facing whose tag named a product, with a price the tag's own arithmetic corroborates, is a candidate label rather than a guess:

python -c "
import json, sys
sys.path.insert(0, 'code')
from record import labelling_candidates
seeds = labelling_candidates('~/dorayaki-data/2026-09-25-storeA/<session>/index.jsonl')
print(len(seeds), 'candidate labels')
print(json.dumps(seeds[:3], ensure_ascii=False, indent=1))
"

A human then confirms or corrects them in the app's confirmation screen — the same screen that closes the flywheel. Nothing here is a training set on its own; it is the raw material for one.

4. Freeze a benchmark

Write the confirmed labels into a manifest that the evaluation reads, and keep it in the repository so results are reproducible:

benchmarks/<store>-<date>/
    visit1/<sku_id>/<view>.jpg      the gallery: 2-3 views per SKU
    visit2/*.jpg                    the test frames
    labels.json                     {"visit2/0007.jpg": [{"box": [...], "sku_id": "..."}]}

Then the identity benchmark (see code/bench_identity.py) can be run with a cross-visit split, and every number it reports is a number worth acting on. Until that exists, treat the current figures as what they are: measurements on a photograph.

3.4 KiB Raw Blame History Unescape Escape