The client is a pure API consumer: live frames go out over /v1/stream, stills
over /v1/recognize, and it can be pointed at another server with
?server=&token=. It keeps the team's visual identity (green palette, logo, EN/JA
toggle) and adds the live surface: state-coloured boxes with name/price chips,
interpolated track boxes, a results/s + latency HUD, quality hints, and an
in-view list that highlights a SKU's boxes.
Photo mode recognises the whole shelf at once instead of one click per product:
boxes paint immediately, identities fill in, tapping a product inspects it
(crop + feature-match visualization), the algorithm selector re-classifies, and
dispense updates stock.
Fixes found by browser verification with a synthetic camera feed:
- The WebSocket handler referenced its processing loop before definition, so the
socket died with NameError and live mode never received a result. Helpers are
now defined inside the handler.
- The client never kept a reference to the video element, so frame capture
returned early and nothing was ever sent.
- /v1/images/{id}/classify passed detection dicts where the engine expected box
dicts (KeyError: 'width').
- Placeholder chrome (thumb, dashes, hidden groups) leaked on first paint.
Verified in Chrome with a fake camera: live 8-12 results/s, ~60 products in
view, product card and feature matching populated, photo 105/170 recognised on
image.jpg and 59/111 on add/image6.jpg, 0 console errors.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
app.py becomes the recognition server: /v1 REST (health, catalog + template
images, samples, recognize, classify, inspect, stock) plus /v1/stream over
WebSocket, with token auth, size limits, one consistent error shape, and the
legacy /api/sessions routes kept but now backed by the real engine instead of
the fake adapter. Default bind is 127.0.0.1; LAN and TLS are opt-in.
stream_client.py is the reference non-browser client (camera, video file, or a
pan across a still) and prints results/s, RTT, detect time and label stability.
Live quality work driven by measurements:
- Labels are remembered per place: a fresh track covering a recently recognized
product inherits the label instead of queueing for classification again. On a
continuous pan this raised recognized products in view from 9/66 to 17-21/66
and cut time-to-first-label to ~1.6 s.
- Classification priority is newest-first, because sorting by area spends the
budget on products that are already leaving the frame.
- Per-session classification concurrency raised to 12.
hybrid_fast (ORB shortlist then hybrid) is implemented but NOT the default: it
is 3.4x faster (13 ms vs 44 ms p50) yet agreed with hybrid on only 47% of
accepted crops across 553 crops, far below the 98% adoption rule. It stays
available for experiments; the numbers are in the commit and the benchmark.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
One LiveSession per streaming client. Each frame runs detection and tracking,
then classification is scheduled for products that are new, big enough, fully in
frame and not already settled; results are cached on the track so detection
never waits for classification. Empty-slot gaps are smoothed over frames and
quality hints (glare, blur, too far) come from a 320 px downscale.
Correctness work that the acceptance run forced out:
- Classification submissions happen OUTSIDE the session lock. add_done_callback
runs the callback inline when the future already completed, so attaching it
while holding the lock deadlocked the session on the second frame.
- A label is dropped only when its box has drifted off the product for two
consecutive frames (panning slides boxes onto neighbours), and the anchor is
the box the crop came from, not wherever the track moved by the time
classification finished.
- An id that disappears and comes back is a restart, not the same object: the
old label is cleared instead of silently labelling a neighbour. relabels and
restarts counters are monotonic per id so clients can tell a re-verification
from a silent flip.
- Track ids are tracked per session (tracking.py) and counters survive pruning.
Measured on the synthetic pan (30 frames, 248 label observations): 20
re-verifications, 0 unexplained label changes, detection 20 ms/frame.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
engine.py is the process-wide, thread-safe core the server is built on: one
YOLO instance behind a lock, thread-local matcher access, a classification pool
so detection is never starved, the stock catalog under its own lock, empty-slot
geometry normalized to a reference width (the detector's gap thresholds are
absolute pixels), and a small LRU of recent still images for later
classify/inspect calls. Detection output is normalized so every client sees the
same box shape.
tracking.py wraps ultralytics BYTETracker. The important part: ultralytics
mints track ids from a process-global counter and resets it when a tracker is
constructed, so with two live clients one client's ids would be re-issued while
the other still held cached labels. SessionTracker gives each tracker its own
counter and never touches the global one.
Measured defaults (bench_engine.py, M5): detection p50 26-29 ms on CPU vs
112 ms on MPS with byte-identical boxes, so device defaults to cpu; hybrid
classifies at ~51 ms/crop, 92 crops/s across 8 threads.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps the model's uncertainty out of the UI: decide_identity() separates raw
matcher scores into accepted / ambiguous / unknown / rejected, and only an
accepted result can name a SKU. Zero evidence is rejected, never a product.
observe.py adds frame-local observations and one-zone scan sessions (draft,
finalize, cancel, rescan) with replay, staleness and backpressure rules.
pipeline.process_click() now reports the identity level instead of failing
when the match is not accepted.
Incorporates earlier uncommitted session work in this repository
(identity.py, observe.py, yolo_adapter.py, test_observe.py, and the pipeline
hunk those needed); yolo_adapter keeps its decoded frame per thread so
concurrent sessions cannot swap images.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The recognition server will classify crops from several clients at once. Two
things blocked that:
- OpenCV Feature2D objects were shared across calls, so parallel
classify_crop() calls could corrupt each other. Each thread now gets its own
extractor set; the template cache is untouched.
- score_crop() is extracted from classify_crop() so callers that need every SKU
score (live track aggregation) do not have to re-derive it from the top-5
list. classify_crop() output is byte-identical to before (verified against the
pre-change module on 82 real crops x 4 methods).
- ShelfProductDetector accepts device/imgsz so the server can pin inference.
Dependencies: pin ultralytics to the version that wrote the checkpoint and that
the session tracker targets (8.4.153), add lap so ultralytics never
pip-installs it mid-request, and add websockets for the streaming endpoint.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
