product-recognition-demo/recognization/docs/API.md

Dorayaki recognition server API

The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture clients talk to it over REST and one WebSocket. Any number of clients can connect at once; no client shares state with another.

• Web client (served by the server itself): http://127.0.0.1:8080/
• Reference non-browser client: code/stream_client.py
• Interactive API docs: http://127.0.0.1:8080/docs

Running it

cd recognization
uv pip install --python .venv/bin/python -r requirements-dev.txt   # first time
.venv/bin/python code/app.py --port 8080

Flag	Default	Meaning
--host	127.0.0.1	Bind address. 0.0.0.0 exposes it on the LAN (set a token).
--port	8080	Port.
--device	cpu	cpu, mps, cuda, cuda:0. CPU is the measured default here.
--imgsz	640	Detector input size.
--weights	runs/detect/train/weights/best.pt	Model file.
--stock-dir	stock	Catalog directory (images + stock.csv).
--classify-workers	cpu_count - 2	Classification thread pool size.
--live-method	hybrid	Default method for live sessions.
--live-conf	0.1	Detection threshold in live mode (the tracker gates).
--max-sessions	4	Concurrent live sessions.
--ssl-certfile / --ssl-keyfile	–	HTTPS (needed for a phone camera).
--log-level	info	Logging level.

Environment: DORAYAKI_* equivalents (DORAYAKI_API_TOKEN, DORAYAKI_DEVICE, DORAYAKI_MAX_SESSIONS, DORAYAKI_CORS_ORIGINS, DORAYAKI_LIVE_INFLIGHT, ...). Command-line flags win over environment.

The model never phones home: the server sets YOLO_OFFLINE=1 and YOLO_AUTOINSTALL=false before importing ultralytics, and lap is pinned in requirements.txt so the tracker never pip-installs anything mid-request.

Authentication

When DORAYAKI_API_TOKEN is set, every /v1 route except GET /v1/health requires Authorization: Bearer <token>, and /v1/stream takes ?token=<token> (browsers cannot set WebSocket headers). The token is deliberately environment-only so it never lands in shell history, and the access log is off because it would print the query string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.

Coordinates

Boxes are [x1, y1, x2, y2] integers, origin top-left, in the pixel space of the image or frame the client sent. Responses echo "coordinate_convention": "pixel-xyxy-origin-top-left".

Identity states

One vocabulary for REST and WebSocket. A product name is only ever attached to a recognized result - the model's uncertainty is never presented as certainty.

state	Meaning	Has sku_id?
detected	Product found, not classified yet (photo mode with classify=false)	no
unconfirmed	Live: tracked, not enough frames to classify yet	no
identifying	Live: classification in flight (reserved)	no
recognized	Accepted unique catalog match	yes
needs_confirmation	Ambiguous or weak match; candidates lists the top 3	no
unrecognized	Not in the catalog (or no matching evidence)	no

identity_level carries the raw decision: accepted, ambiguous, unknown, rejected.

Errors

Every failure uses one shape:

{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}

Code	HTTP	When
BAD_REQUEST	400	Malformed request, unknown algorithm, empty body
UNAUTHORIZED	401	Missing or wrong bearer token
NOT_FOUND	404	Unknown SKU, sample, or route
IMAGE_EXPIRED	404	image_id left the still-image store (15 min / 16 images)
PAYLOAD_TOO_LARGE	413	Upload > 20 MB, live frame > 4 MB
UNSUPPORTED_MEDIA_TYPE	415	Body is not an image content type
UNDECODABLE_IMAGE	422	Bytes are not a readable image
VALIDATION_ERROR	422	Body failed validation
INTERNAL	500	Server fault (details only in the server log)

REST

GET /v1/health (no auth)

{"status": "ok",
 "model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
 "catalog": {"version": "408a658680ed", "sku_count": 21},
 "sessions_active": 0, "uptime_s": 21.1}

GET /v1/catalog

Products with stock and an image_url for the template photo. Local file paths are never exposed.

GET /v1/catalog/{sku_id}/image

Template image bytes. The path is resolved through the catalog map only, so a crafted sku_id cannot reach the filesystem.

POST /v1/catalog/{sku_id}/stock

{"delta": -1}

-100..100, non-zero. Stock is in memory and resets on restart, as before.

GET /v1/samples and GET /v1/samples/{sample_id}

The demo shelf images (image.jpg, add/*, the training mosaic) by id, from an allowlist built at startup.

POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true

Body: raw image bytes (Content-Type: image/jpeg|image/png|image/webp) or multipart with a file field.

{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
 "catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
 "coordinate_convention": "pixel-xyxy-origin-top-left",
 "detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
                 "identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
                              "confidence": 0.8948, "score": 47.71,
                              "candidates": [{"sku_id": "sku_005", "score": 47.71}],
                              "attempts": 1, "method": "hybrid"}}],
 "empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
                  "missing_items_est": 2, "center": [860, 476], "row_index": 3}],
 "summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
             "detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
 "timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}

classify=false returns detected identities immediately so a client can paint boxes before identities arrive. The image_id stays valid for 15 minutes for classify/inspect.

POST /v1/images/{image_id}/classify

{"method": "hybrid", "detection_ids": [0, 3]}

detection_ids optional. Returns identities for the whole image (or the selected detections) plus a summary. A method change re-runs it.

POST /v1/images/{image_id}/inspect

{"box": [1435, 383, 1493, 520], "method": "hybrid"}

Returns the full decision (identity, per-method inlier sub_details, product), the cropped product as a data URL, and a feature-match visualization. When the result is not accepted, the visualization is drawn against the top candidate and is_candidate is true.

WebSocket /v1/stream

Live frames in, tracked products out. One session per connection.

On connect

{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
 "model": {"version": "...", "device": "cpu", "imgsz": 640},
 "config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
 "limits": {"max_frame_bytes": 4194304}}

Frames (binary): 4-byte big-endian frame id + JPEG bytes. The id is echoed back so the client can pair a result with the frame it sent.

Control (text JSON)

Client sends	Server replies
{"type":"config","method":"orb","conf":0.15,"empty_slots":false}	{"type":"config_ack","config":{...}}
{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}	{"type":"inspect_result","request_id":"r1",...} or an error
{"type":"reset"}	{"type":"reset_ack"}
{"type":"ping","t":1}	{"type":"pong","t":1}

Per processed frame

{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
 "tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
             "first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
             "identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
                          "score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
             "product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
 "empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
                                   "too_far": false, "hint": null},
 "summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
             "unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
             "by_sku": [{"sku_id": "sku_005", "facings": 3}]},
 "timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
 "stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
           "server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}

Backpressure: the server keeps one pending frame per session and processes the newest, counting the ones it replaced in frames_dropped. A client should keep one frame in flight (stream_client.py --pace result does exactly that); sending faster just drops frames and breaks the pan's continuity for the tracker.

Recoverable errors (the socket stays open): BAD_FRAME, UNDECODABLE_IMAGE, PAYLOAD_TOO_LARGE, BAD_MESSAGE, TRACK_NOT_IN_VIEW, INTERNAL.

Closes: 1008 wrong/missing token, 1013 session limit reached, 120 s idle.

What the client can rely on

• A label is only shown for an accepted match. Weak or ambiguous results are needs_confirmation with candidates; unknown products are unrecognized.
• A label never changes silently. A settled label is dropped only when the box has drifted off the product for two consecutive frames, or when the track id was restarted. Both increment relabels/restarts, so a client can audit every change.
• One product is classified once. While the camera pans, a fresh track that covers a recently recognized product inherits that label (inherited: true) instead of queueing for classification again.
• Detection never waits for classification. Frames come back at detection speed (~25-45 ms on this machine); labels fill in as the classification pool finishes.

Design decisions

Decision	Why	Rejected
Server lives in this repo, entry point python code/app.py	It owns the weights, templates, and the team's workflow	A separate service repo (split the model from its data)
WebSocket carrying JPEG frames for live, REST for stills	Browsers and embedded clients both speak it; no media stack	WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python)
Latest-frame-wins with one pending slot	Lowest latency and self-throttling on any network	Queueing every frame (adds latency, never catches up)
Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids	Kalman prediction plus low-score second association is proven; ids must not be shared	No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model)
Classify once per track, aggregate up to 3 attempts, lock on accept	Keeps detection real-time and labels stable	Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes)
Label inheritance by place	The same product gets several ids while panning; re-classifying it is waste	Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance)
hybrid as the live default	Most accurate (21/21 SKUs on image.jpg)	hybrid_fast (ORB shortlist): 3.4x faster but only 47% agreement with hybrid on 553 crops, so it is not the default
CPU as the default device	Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes	MPS (slower here), CUDA (no NVIDIA GPU on this machine)
Identity states instead of raw scores	"Never turn model uncertainty into business certainty" (project master)	Showing a best-guess SKU with a confidence number
Token via env var, access log off	Keeps credentials out of shell history and logs	A --token flag, default uvicorn access log (prints the query string)
Legacy /api/sessions* kept, engine-backed	The earlier session model is still useful and was already tested	Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing)

Measured performance

Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (code/bench_engine.py, code/stream_client.py --pace result):

Measurement	Value
Model load / first predict	50 ms / 985 ms (warm-up covers this at startup)
Detection, 1280x853 frame	p50 28.7 ms, p95 31.0 ms (169 boxes)
Detection, native 2048x1365	p50 28.7 ms, p95 87.6 ms (170 boxes)
Detection under live load	p50 43-66 ms (classification pool running)
Classification per crop (hybrid)	p50 44 ms, p95 95 ms
Classification throughput	92 crops/s at 8 threads
Still recognition, whole shelf	170 detections, 105 recognized, ~2.0 s total
Live results	8-14 per second, round-trip p50 58-78 ms
Time to first label on a pan	~1.6-2.7 s (includes waiting for the product to enter)

The slide's "約28~30ms per image" matches the measured detection time (28.7 ms); the full detect-and-classify of a whole 170-product shelf is ~2 s because each product is matched against the catalog separately.

Known limitations

• No session totals. Facings are counted only for products currently in view. A product that leaves and re-enters gets a new track id, so cumulative counts would double count.
• Identity thresholds are uncalibrated closed-set defaults (identity.py), not measured accuracy.
• surf is SIFT in this OpenCV build (no nonfree SURF); the label stays for continuity with the paper.
• Fast pans classify less. A very fast pan creates more new tracks than the classification pool can settle; a slower, natural scan labels nearly everything.
• Coverage guidance is partial: glare/blur/too-far hints exist; "which part of the fixture was scanned" does not (that needs mosaicking).
• Stock is in memory and resets when the server restarts.
• hybrid_fast is experimental and off by default.

Mobile client notes (future lane)

The protocol is capture-source agnostic; an iPad/phone client needs no server changes.

• WebSocket client (Ktor WebSockets plugin), one frame in flight.
• Capture JPEG frames from AVCaptureVideoDataOutput (iOS) or CameraX ImageAnalysis (Android), sent upright; map server boxes into the view with the same object-fit transform the web client uses.
• iOS ATS: allow local networking (NSAllowsLocalNetworking) or use HTTPS; Android needs cleartext permission for plain http://.
• The dorayaki app repo requires an ADR before adding a second API base URL, and its scan screen is a timer-driven placeholder today - wiring it to this server is a separate lane.

Tests

cd recognization/code
YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_observe test_engine test_tracking test_live test_api -v
DORAYAKI_REAL_MODEL=1 YOLO_OFFLINE=1 ../.venv/bin/python -m unittest test_real_model -v
cd .. && YOLO_OFFLINE=1 .venv/bin/python code/test_suite.py     # the original CLI suite

Run tests with an explicit module list: unittest discover would import test.py/fast_test.py, which are scratch scripts with hardcoded paths from another machine and a blocking OpenCV window.