The client is a pure API consumer: live frames go out over /v1/stream, stills
over /v1/recognize, and it can be pointed at another server with
?server=&token=. It keeps the team's visual identity (green palette, logo, EN/JA
toggle) and adds the live surface: state-coloured boxes with name/price chips,
interpolated track boxes, a results/s + latency HUD, quality hints, and an
in-view list that highlights a SKU's boxes.
Photo mode recognises the whole shelf at once instead of one click per product:
boxes paint immediately, identities fill in, tapping a product inspects it
(crop + feature-match visualization), the algorithm selector re-classifies, and
dispense updates stock.
Fixes found by browser verification with a synthetic camera feed:
- The WebSocket handler referenced its processing loop before definition, so the
socket died with NameError and live mode never received a result. Helpers are
now defined inside the handler.
- The client never kept a reference to the video element, so frame capture
returned early and nothing was ever sent.
- /v1/images/{id}/classify passed detection dicts where the engine expected box
dicts (KeyError: 'width').
- Placeholder chrome (thumb, dashes, hidden groups) leaked on first paint.
Verified in Chrome with a fake camera: live 8-12 results/s, ~60 products in
view, product card and feature matching populated, photo 105/170 recognised on
image.jpg and 59/111 on add/image6.jpg, 0 console errors.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
app.py becomes the recognition server: /v1 REST (health, catalog + template
images, samples, recognize, classify, inspect, stock) plus /v1/stream over
WebSocket, with token auth, size limits, one consistent error shape, and the
legacy /api/sessions routes kept but now backed by the real engine instead of
the fake adapter. Default bind is 127.0.0.1; LAN and TLS are opt-in.
stream_client.py is the reference non-browser client (camera, video file, or a
pan across a still) and prints results/s, RTT, detect time and label stability.
Live quality work driven by measurements:
- Labels are remembered per place: a fresh track covering a recently recognized
product inherits the label instead of queueing for classification again. On a
continuous pan this raised recognized products in view from 9/66 to 17-21/66
and cut time-to-first-label to ~1.6 s.
- Classification priority is newest-first, because sorting by area spends the
budget on products that are already leaving the frame.
- Per-session classification concurrency raised to 12.
hybrid_fast (ORB shortlist then hybrid) is implemented but NOT the default: it
is 3.4x faster (13 ms vs 44 ms p50) yet agreed with hybrid on only 47% of
accepted crops across 553 crops, far below the 98% adoption rule. It stays
available for experiments; the numbers are in the commit and the benchmark.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
engine.py is the process-wide, thread-safe core the server is built on: one
YOLO instance behind a lock, thread-local matcher access, a classification pool
so detection is never starved, the stock catalog under its own lock, empty-slot
geometry normalized to a reference width (the detector's gap thresholds are
absolute pixels), and a small LRU of recent still images for later
classify/inspect calls. Detection output is normalized so every client sees the
same box shape.
tracking.py wraps ultralytics BYTETracker. The important part: ultralytics
mints track ids from a process-global counter and resets it when a tracker is
constructed, so with two live clients one client's ids would be re-issued while
the other still held cached labels. SessionTracker gives each tracker its own
counter and never touches the global one.
Measured defaults (bench_engine.py, M5): detection p50 26-29 ms on CPU vs
112 ms on MPS with byte-identical boxes, so device defaults to cpu; hybrid
classifies at ~51 ms/crop, 92 crops/s across 8 threads.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
