product-recognition-demo/recognization/docs/API.md

Dorayaki recognition server API

The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture clients talk to it over REST and one WebSocket. Any number of clients can connect at once; no client shares state with another.

• Web client (served by the server itself): http://127.0.0.1:8080/
• Reference non-browser client: code/stream_client.py
• Interactive API docs: http://127.0.0.1:8080/docs
• Planned extensions (price-tag OCR, reconciliation, compliance, mobile scan): docs/design-price-tags-and-compliance.md

Running it

cd recognization
uv pip install --python .venv/bin/python -r requirements-dev.txt   # first time
.venv/bin/python code/app.py --port 8080

Flag	Default	Meaning
--host	127.0.0.1	Bind address. 0.0.0.0 exposes it on the LAN (set a token).
--port	8080	Port.
--device	cpu	cpu, mps, cuda, cuda:0. CPU is the measured default here.
--imgsz	640	Detector input size.
--weights	runs/detect/train/weights/best.pt	Model file.
--stock-dir	stock	Catalog directory (images + stock.csv).
--classify-workers	5	Classification thread pool size (measured default).
--live-method	hybrid	Default method for live sessions.
--live-conf	0.1	Detection threshold in live mode (the tracker gates).
--no-tags	off	Disable price-tag OCR entirely (DORAYAKI_LIVE_TAGS=0).
DORAYAKI_OCR	auto	OCR backend: auto (Vision on macOS), vision, fake, none.
DORAYAKI_TORCH_THREADS	4	torch CPU threads; 0 leaves torch's default.
--max-sessions	4	Concurrent live sessions.
--ssl-certfile / --ssl-keyfile	–	HTTPS (needed for a phone camera).
--log-level	info	Logging level.

Environment: DORAYAKI_* equivalents (DORAYAKI_API_TOKEN, DORAYAKI_DEVICE, DORAYAKI_MAX_SESSIONS, DORAYAKI_CORS_ORIGINS, DORAYAKI_LIVE_INFLIGHT, ...). Command-line flags win over environment.

The model never phones home: the server sets YOLO_OFFLINE=1 and YOLO_AUTOINSTALL=false before importing ultralytics, and lap is pinned in requirements.txt so the tracker never pip-installs anything mid-request.

Authentication

When DORAYAKI_API_TOKEN is set, every /v1 route except GET /v1/health requires Authorization: Bearer <token>, and /v1/stream takes ?token=<token> (browsers cannot set WebSocket headers). The token is deliberately environment-only so it never lands in shell history, and the access log is off because it would print the query string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.

Coordinates

Boxes are [x1, y1, x2, y2] integers, origin top-left, in the pixel space of the image or frame the client sent. Responses echo "coordinate_convention": "pixel-xyxy-origin-top-left".

Identity states

One vocabulary for REST and WebSocket. A product name is only ever attached to a recognized result - the model's uncertainty is never presented as certainty.

state	Meaning	Has sku_id?
detected	Product found, not classified yet (photo mode with classify=false)	no
unconfirmed	Live: tracked, not enough frames to classify yet	no
identifying	Live: classification in flight (reserved)	no
recognized	Accepted unique catalog match	yes
needs_confirmation	Ambiguous or weak match; candidates lists the top 3	no
unrecognized	Not in the catalog (or no matching evidence)	no

identity_level carries the raw decision: accepted, ambiguous, unknown, rejected.

Errors

Every failure uses one shape:

{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}

Code	HTTP	When
BAD_REQUEST	400	Malformed request, unknown algorithm, empty body
UNAUTHORIZED	401	Missing or wrong bearer token
NOT_FOUND	404	Unknown SKU, sample, or route
IMAGE_EXPIRED	404	image_id left the still-image store (15 min / 16 images)
PAYLOAD_TOO_LARGE	413	Upload > 20 MB, live frame > 4 MB
UNSUPPORTED_MEDIA_TYPE	415	Body is not an image content type
UNDECODABLE_IMAGE	422	Bytes are not a readable image
VALIDATION_ERROR	422	Body failed validation
INTERNAL	500	Server fault (details only in the server log)

REST

GET /v1/health (no auth)

{"status": "ok",
 "model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
 "catalog": {"version": "408a658680ed", "sku_count": 21},
 "sessions_active": 0, "uptime_s": 21.1}

GET /v1/catalog

Products with stock and an image_url for the template photo. Local file paths are never exposed.

GET /v1/catalog/{sku_id}/image

Template image bytes. The path is resolved through the catalog map only, so a crafted sku_id cannot reach the filesystem.

POST /v1/catalog/{sku_id}/stock

{"delta": -1}

-100..100, non-zero. Stock is in memory and resets on restart, as before.

GET /v1/samples and GET /v1/samples/{sample_id}

The demo shelf images (image.jpg, add/*, the training mosaic) by id, from an allowlist built at startup.

POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true

Body: raw image bytes (Content-Type: image/jpeg|image/png|image/webp) or multipart with a file field.

{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
 "catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
 "coordinate_convention": "pixel-xyxy-origin-top-left",
 "detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
                 "identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
                              "confidence": 0.8948, "score": 47.71,
                              "candidates": [{"sku_id": "sku_005", "score": 47.71}],
                              "attempts": 1, "method": "hybrid"}}],
 "empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
                  "missing_items_est": 2, "center": [860, 476], "row_index": 3}],
 "summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
             "detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
 "timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}

classify=false returns detected identities immediately so a client can paint boxes before identities arrive. The image_id stays valid for 15 minutes for classify/inspect.

Price tags (OCR)

When an OCR backend is available (macOS Vision; see the design doc), a still can carry tag evidence. ?tags=true runs the tag pass synchronously (measured ~0.9 s for a six-band shelf) and adds, per detection:

"price_tag": {"id": "tag_3f2a", "box": [91, 657, 250, 744], "state": "readable",
              "price": 357, "regular_price": null, "discount": 0.24, "unit_price": 290.52,
              "name": "ゼロノンコレステロール", "flags": ["sale", "discounted"],
              "confidence": 0.82, "observations": 2, "settled": false},
"reconciliation": {"state": "confirmed | mismatch | unverified | no_tag | tag_only",
                   "signals": [{"kind": "price", "agrees": false, "item": 168, "tag": 269}],
                   "message": "tag says 269, catalog says 168"}

price_tag.state is readable (text parsed), unreadable (a tag was found but its text was not read) or absent (no tag under this facing). The summary gains price_tags counts and compliance:

"price_tags": {"readable": 8, "unreadable": 18, "absent": 85},
"compliance": {"violations": 7, "by_rule": {"price_mismatch": 7},
               "rows": [{"row_index": 2, "rule": "price_mismatch", "box": [...], "missing": 1}],
               "hints": [{"rule": "unreadable_tags", "detail": "tags detected for 5 of 16 facings in this row"}]}

Compliance fails closed on evidence quality. A row is only judged when at least 70% of its facings carry tag evidence (readable or unreadable); otherwise it produces a hint such as move closer: the price tags in this row could not be read, never a violation. Findings must also persist for 3 of the last 5 frames before they publish. This is deliberate: the first implementation reported 106 missing-tag violations on a shelf where only 8 tags were readable, which was our detection being incomplete, not the shelf.

POST /v1/images/{image_id}/classify re-reconciles against the cached tag bundle and returns reconciliation (and price_tag) on each identity row, because in photo mode the tags are read before the identities exist.

POST /v1/images/{image_id}/classify

{"method": "hybrid", "detection_ids": [0, 3]}

detection_ids optional. Returns identities for the whole image (or the selected detections) plus a summary. A method change re-runs it.

POST /v1/images/{image_id}/inspect

{"box": [1435, 383, 1493, 520], "method": "hybrid"}

Returns the full decision (identity, per-method inlier sub_details, product), the cropped product as a data URL, and a feature-match visualization. When the result is not accepted, the visualization is drawn against the top candidate and is_candidate is true.

WebSocket /v1/stream

Live frames in, tracked products out. One session per connection.

On connect

{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
 "model": {"version": "...", "device": "cpu", "imgsz": 640},
 "config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
 "limits": {"max_frame_bytes": 4194304, "max_crop_bytes": 1048576}}

Frames (binary): 4-byte big-endian frame id + JPEG bytes. The id is echoed back so the client can pair a result with the frame it sent.

Control (text JSON)

Client sends	Server replies
{"type":"config","method":"orb","conf":0.15,"empty_slots":false}	{"type":"config_ack","config":{...}}
{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}	{"type":"inspect_result","request_id":"r1",...} or an error
{"type":"capabilities","max_crop_bytes":131072}	{"type":"capabilities_ack","capabilities":{"crop":true,"max_crop_bytes":131072}}
{"type":"crop","request_id":"c1","track_id":12,"frame_id":41,"jpeg":"<base64 JPEG>"}	{"type":"crop_ack","request_id":"c1","track_id":12,"frame_id":41} or crop_error
{"type":"reset"}	{"type":"reset_ack"}
{"type":"ping","t":1}	{"type":"pong","t":1}

Per processed frame

{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
 "tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
             "first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
             "identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
                          "score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
             "product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
 "empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
                                   "too_far": false, "hint": null},
 "summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
             "unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
             "by_sku": [{"sku_id": "sku_005", "facings": 3}]},
 "timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
 "stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
           "server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}

The optional crop lane is additive to protocol 1. Without capabilities, the server sends no crop_request and existing clients keep their frame/control flow. After opting in, a result may include "crop_request":{"request_id":"c1", "track_id":12,"frame_id":41,"box":[812,404,850,460]} on an eligible track. The box uses the sent frame's pixel coordinates (the raw detection), not the camera's native resolution. Keep that exact frame's full-resolution capture briefly, map its box into the capture, encode the crop as JPEG, and reply in a text JSON crop message. Binary frames remain 4-byte id + JPEG; the crop does not take a frame id of its own. crop_ack means accepted for asynchronous classification, not recognized; subsequent results carry the updated identity and identity.source (frame, crop, or inherited). stats.crops has requested, accepted, and rejected counts.

The hello limit is 1 MiB of decoded crop bytes. The optional max_crop_bytes in capabilities states the client's limit; the acknowledged session limit is clamped to 1 KiB–1 MiB (default 128 KiB when omitted). Use the acknowledged limit, not only hello's ceiling. At most four requests are outstanding per session and each track object is requested at most once. A request expires after more than six processed frames; clients keep a bounded recent-frame buffer and may miss a crop if its capture has already been evicted. A late response cannot label a recycled track. Changing method or resetting invalidates requests.

Crop failures are recoverable text messages with type: "crop_error", code, and message. Pre-decode failures also echo request_id; session-level refusals do not currently echo it. Codes: CROP_BAD_MESSAGE (missing/invalid base64 JPEG or missing IDs), CROP_TOO_LARGE (empty or oversized decoded crop, or excessive encoded length), CROP_UNKNOWN_REQUEST (no outstanding request, including one already expired), CROP_MISMATCH (track/frame differs), CROP_STALE (late or invalidated track/session), CROP_UNDECODABLE (unreadable image), and CROP_UNAVAILABLE (classifier shutting down). A request is single-use, including when its response is rejected; continue streaming frames rather than retrying it.

Backpressure: the server keeps one pending frame per session and processes the newest, counting the ones it replaced in frames_dropped. A client should keep one frame in flight (stream_client.py --pace result does exactly that); sending faster just drops frames and breaks the pan's continuity for the tracker.

Recoverable errors (the socket stays open): BAD_FRAME, UNDECODABLE_IMAGE, PAYLOAD_TOO_LARGE, BAD_MESSAGE, TRACK_NOT_IN_VIEW, INTERNAL.

Closes: 1008 wrong/missing token, 1013 session limit reached, 120 s idle.

What the client can rely on

• A label is only shown for an accepted match. Weak or ambiguous results are needs_confirmation with candidates; unknown products are unrecognized.
• A label never changes silently. A settled label is dropped only when the box has drifted off the product for two consecutive frames, or when the track id was restarted. Both increment relabels/restarts, so a client can audit every change.
• One product is classified once. While the camera pans, a fresh track that covers a recently recognized product inherits that label (inherited: true) instead of queueing for classification again.
• Detection never waits for classification. Frames come back at detection speed (~25-45 ms on this machine); labels fill in as the classification pool finishes.

Design decisions

Decision	Why	Rejected
Server lives in this repo, entry point python code/app.py	It owns the weights, templates, and the team's workflow	A separate service repo (split the model from its data)
WebSocket carrying JPEG frames for live, REST for stills	Browsers and embedded clients both speak it; no media stack	WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python)
Latest-frame-wins with one pending slot	Lowest latency and self-throttling on any network	Queueing every frame (adds latency, never catches up)
Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids	Kalman prediction plus low-score second association is proven; ids must not be shared	No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model)
Classify once per track, aggregate up to 3 attempts, lock on accept	Keeps detection real-time and labels stable	Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes)
Label inheritance by place	The same product gets several ids while panning; re-classifying it is waste	Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance)
hybrid as the live default	Most accurate (21/21 SKUs on image.jpg)	hybrid_fast (ORB shortlist): 3.4x faster but only 47% agreement with hybrid on 553 crops, so it is not the default
CPU as the default device	Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes	MPS (slower here), CUDA (no NVIDIA GPU on this machine)
Identity states instead of raw scores	"Never turn model uncertainty into business certainty" (project master)	Showing a best-guess SKU with a confidence number
Token via env var, access log off	Keeps credentials out of shell history and logs	A --token flag, default uvicorn access log (prints the query string)
Legacy /api/sessions* kept, engine-backed	The earlier session model is still useful and was already tested	Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing)

Measured performance

Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (code/bench_engine.py, code/stream_client.py --pace result):

Measurement	Value
Model load / first predict	50 ms / 985 ms (startup warm-up covers this)
Detection, 1280x853 frame	p50 28.7 ms, p95 31.0 ms (169 boxes)
Detection, native 2048x1365	p50 28.7 ms, p95 87.6 ms (170 boxes)
Detection during a live pan	p50 46-48 ms (classification pool running)
Classification per crop (hybrid)	p50 44 ms, p95 95 ms
Classification throughput	92 crops/s at 8 threads
Still recognition, whole shelf	170 detections, 105 recognized, ~2.0 s total
Live results, one client	10.5-12.2 per second, round-trip p50 58-78 ms
Live results, two clients	8.5 per second each, detection p50 75 ms
Time to first label on a pan	~1.0-2.6 s (includes waiting for the product to enter)

The live numbers use the shipped defaults: --classify-workers 5 and torch capped at 4 threads (DORAYAKI_TORCH_THREADS=4). Uncapped torch and 8 workers measured slower (11.1 results/s single-client) because detection and classification fight over the CPU. Detection is the part that must stay real-time; classification fills in behind it.

The slide's "約28~30ms per image" matches the measured detection time (28.7 ms); the full detect-and-classify of a whole 170-product shelf is ~2 s because each product is matched against the catalog separately.

Price-tag OCR: what it can and cannot do (measured)

• Capture distance decides everything. At add/image6.jpg's ~1 m (tags ~300x80 px, digits ~22 px) macOS Vision reads a tag's printed fields reliably and the tag pass finds 8 readable tags with names and prices in ~0.9 s. At image.jpg's ~2 m (digits ~14 px) it reads none of them - the pass correctly reports nothing readable.
• Tag detection is the weak link, not the OCR. On a shelf with ~100 facings the pass currently finds tags for a minority of them at this distance; the compliance hints say so instead of inventing violations.
• A single read can misread a digit, and a consistent misread survives the vote. The vote needs independent reads (live mode accumulates them across frames; stills read each band twice at different upscales) and a dead heat stays unread rather than guessing, but when one wrong reading wins most frames it settles as wrong - measured once in a 40-frame live run (3609 for a 360 tag, 10 observations against 8). The safeguard is downstream: a wrong tag price shows as a mismatch with both values and is never applied to the product identity. Cross-checking the sale price against the regular price and discount is the recorded next improvement.
• The demo catalog is fictional, so tag prices legitimately disagree with it. That is shown as mismatch with both values; it is not an error.

Known limitations

• No session totals. Facings are counted only for products currently in view. A product that leaves and re-enters gets a new track id, so cumulative counts would double count.
• Identity thresholds are uncalibrated closed-set defaults (identity.py), not measured accuracy.
• surf is SIFT in this OpenCV build (no nonfree SURF); the label stays for continuity with the paper.
• Fast pans classify less. A very fast pan creates more new tracks than the classification pool can settle; a slower, natural scan labels nearly everything.
• Coverage guidance is partial: glare/blur/too-far hints exist; "which part of the fixture was scanned" does not (that needs mosaicking).
• Stock is in memory and resets when the server restarts.
• hybrid_fast is experimental and off by default.

Mobile client notes (future lane)

The protocol is capture-source agnostic; an iPad/phone client needs no server changes.

• WebSocket client (Ktor WebSockets plugin), one frame in flight.
• Capture JPEG frames from AVCaptureVideoDataOutput (iOS) or CameraX ImageAnalysis (Android), sent upright; map server boxes into the view with the same object-fit transform the web client uses.
• iOS ATS: allow local networking (NSAllowsLocalNetworking) or use HTTPS; Android needs cleartext permission for plain http://.
• The dorayaki app repo requires an ADR before adding a second API base URL, and its scan screen is a timer-driven placeholder today - wiring it to this server is a separate lane.

Tests

cd recognization
YOLO_OFFLINE=1 YOLO_AUTOINSTALL=false python -m unittest discover -s code -p 'test_*.py' -v

test_real_model is discovered and skipped unless DORAYAKI_REAL_MODEL=1. CI installs requirements.lock and runs the same command.