tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity
product-recognition-demo/recognization/docs/API.md

418 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Dorayaki recognition server API
The computer-vision model (YOLO26n product detector + the SURF/SIFT, ORB, BRISK and
Hybrid feature-matching classifier + the stock catalog) runs as a service. Capture
clients talk to it over REST and one WebSocket. Any number of clients can connect at
once; no client shares state with another.
- Web client (served by the server itself): `http://127.0.0.1:8080/`
- Reference non-browser client: `code/stream_client.py`
- Interactive API docs: `http://127.0.0.1:8080/docs`
- Planned extensions (price-tag OCR, reconciliation, compliance, mobile scan):
`docs/design-price-tags-and-compliance.md`
## Running it
```bash
cd recognization
uv pip install --python .venv/bin/python -r requirements-dev.txt # first time
.venv/bin/python code/app.py --port 8080
```
| Flag | Default | Meaning |
| --- | --- | --- |
| `--host` | `127.0.0.1` | Bind address. `0.0.0.0` exposes it on the LAN (set a token). |
| `--port` | `8080` | Port. |
| `--device` | `cpu` | `cpu`, `mps`, `cuda`, `cuda:0`. CPU is the measured default here. |
| `--imgsz` | `640` | Detector input size. |
| `--weights` | `runs/detect/train/weights/best.pt` | Model file. |
| `--stock-dir` | `stock` | Catalog directory (images + `stock.csv`). |
| `--classify-workers` | `5` | Classification thread pool size (measured default). |
| `--live-method` | `hybrid` | Default method for live sessions. |
| `--live-conf` | `0.1` | Detection threshold in live mode (the tracker gates). |
| `--no-tags` | off | Disable price-tag OCR entirely (`DORAYAKI_LIVE_TAGS=0`). |
| `DORAYAKI_OCR` | `auto` | OCR backend: `auto` (Vision on macOS), `vision`, `fake`, `none`. |
| `DORAYAKI_TORCH_THREADS` | `4` | torch CPU threads; 0 leaves torch's default. |
| `--max-sessions` | `4` | Concurrent live sessions. |
| `--ssl-certfile` / `--ssl-keyfile` | – | HTTPS (needed for a phone camera). |
| `--log-level` | `info` | Logging level. |
Environment: `DORAYAKI_*` equivalents (`DORAYAKI_API_TOKEN`, `DORAYAKI_DEVICE`,
`DORAYAKI_MAX_SESSIONS`, `DORAYAKI_CORS_ORIGINS`, `DORAYAKI_LIVE_INFLIGHT`, ...).
Command-line flags win over environment.
The model never phones home: the server sets `YOLO_OFFLINE=1` and
`YOLO_AUTOINSTALL=false` before importing ultralytics, and `lap` is pinned in
`requirements.txt` so the tracker never pip-installs anything mid-request.
## Authentication
When `DORAYAKI_API_TOKEN` is set, every `/v1` route except `GET /v1/health` requires
`Authorization: Bearer <token>`, and `/v1/stream` takes `?token=<token>` (browsers
cannot set WebSocket headers). The token is deliberately environment-only so it never
lands in shell history, and the access log is off because it would print the query
string. A token on a LAN bind is a speed bump, not security: prefer HTTPS.
## Coordinates
Boxes are `[x1, y1, x2, y2]` integers, origin top-left, in the pixel space of the
image or frame the client sent. Responses echo
`"coordinate_convention": "pixel-xyxy-origin-top-left"`.
## Identity states
One vocabulary for REST and WebSocket. **A product name is only ever attached to a
`recognized` result** - the model's uncertainty is never presented as certainty.
| `state` | Meaning | Has `sku_id`? |
| --- | --- | --- |
| `detected` | Product found, not classified yet (photo mode with `classify=false`) | no |
| `unconfirmed` | Live: tracked, not enough frames to classify yet | no |
| `identifying` | Live: classification in flight (reserved) | no |
| `recognized` | Accepted unique catalog match | **yes** |
| `needs_confirmation` | Ambiguous or weak match; `candidates` lists the top 3 | no |
| `unrecognized` | Not in the catalog (or no matching evidence) | no |
`identity_level` carries the raw decision: `accepted`, `ambiguous`, `unknown`,
`rejected`.
## Errors
Every failure uses one shape:
```json
{"error": {"code": "IMAGE_EXPIRED", "message": "image 'img_x' is no longer available"}}
```
| Code | HTTP | When |
| --- | --- | --- |
| `BAD_REQUEST` | 400 | Malformed request, unknown algorithm, empty body |
| `UNAUTHORIZED` | 401 | Missing or wrong bearer token |
| `NOT_FOUND` | 404 | Unknown SKU, sample, or route |
| `IMAGE_EXPIRED` | 404 | `image_id` left the still-image store (15 min / 16 images) |
| `PAYLOAD_TOO_LARGE` | 413 | Upload > 20 MB, live frame > 4 MB |
| `UNSUPPORTED_MEDIA_TYPE` | 415 | Body is not an image content type |
| `UNDECODABLE_IMAGE` | 422 | Bytes are not a readable image |
| `VALIDATION_ERROR` | 422 | Body failed validation |
| `INTERNAL` | 500 | Server fault (details only in the server log) |
## REST
### `GET /v1/health` (no auth)
```json
{"status": "ok",
"model": {"version": "best.pt@ultralytics-8.4.153", "device": "cpu", "imgsz": 640, "warm": true},
"catalog": {"version": "408a658680ed", "sku_count": 21},
"sessions_active": 0, "uptime_s": 21.1}
```
### `GET /v1/catalog`
Products with stock and an `image_url` for the template photo. Local file paths are
never exposed.
### `GET /v1/catalog/{sku_id}/image`
Template image bytes. The path is resolved through the catalog map only, so a crafted
`sku_id` cannot reach the filesystem.
### `POST /v1/catalog/{sku_id}/stock`
```json
{"delta": -1}
```
`-100..100`, non-zero. Stock is in memory and resets on restart, as before.
### `GET /v1/samples` and `GET /v1/samples/{sample_id}`
The demo shelf images (`image.jpg`, `add/*`, the training mosaic) by id, from an
allowlist built at startup.
### `POST /v1/recognize?classify=true&method=hybrid&conf=0.25&empty_slots=true`
Body: raw image bytes (`Content-Type: image/jpeg|image/png|image/webp`) or multipart
with a `file` field.
```json
{"image_id": "img_11d072958c35bc35", "width": 2048, "height": 1365,
"catalog_version": "408a658680ed", "model_version": "best.pt@ultralytics-8.4.153",
"coordinate_convention": "pixel-xyxy-origin-top-left",
"detections": [{"id": 0, "box": [1435, 383, 1493, 520], "confidence": 0.8258,
"identity": {"state": "recognized", "sku_id": "sku_005", "identity_level": "accepted",
"confidence": 0.8948, "score": 47.71,
"candidates": [{"sku_id": "sku_005", "score": 47.71}],
"attempts": 1, "method": "hybrid"}}],
"empty_slots": [{"id": "empty_1", "box": [812, 404, 909, 549], "width": 97, "height": 145,
"missing_items_est": 2, "center": [860, 476], "row_index": 3}],
"summary": {"in_view": 170, "recognized": 105, "needs_confirmation": 64, "unrecognized": 1,
"detected": 0, "empty_slots": 4, "by_sku": [{"sku_id": "sku_003", "facings": 14}]},
"timing_ms": {"detect": 24.77, "classify": 1498.88, "total": 1523.65}}
```
`classify=false` returns `detected` identities immediately so a client can paint boxes
before identities arrive. The `image_id` stays valid for 15 minutes for
`classify`/`inspect`.
### Price tags (OCR)
When an OCR backend is available (macOS Vision; see the design doc), a still can
carry tag evidence. `?tags=true` runs the tag pass synchronously (measured ~0.9 s
for a six-band shelf) and adds, per detection:
```json
"price_tag": {"id": "tag_3f2a", "box": [91, 657, 250, 744], "state": "readable",
"price": 357, "regular_price": null, "discount": 0.24, "unit_price": 290.52,
"name": "ゼロノンコレステロール", "flags": ["sale", "discounted"],
"confidence": 0.82, "observations": 2, "settled": false},
"reconciliation": {"state": "confirmed | mismatch | unverified | no_tag | tag_only",
"signals": [{"kind": "price", "agrees": false, "item": 168, "tag": 269}],
"message": "tag says 269, catalog says 168"}
```
`price_tag.state` is `readable` (text parsed), `unreadable` (a tag was found but
its text was not read) or `absent` (no tag under this facing). The summary gains
`price_tags` counts and `compliance`:
```json
"price_tags": {"readable": 8, "unreadable": 18, "absent": 85},
"compliance": {"violations": 7, "by_rule": {"price_mismatch": 7},
"rows": [{"row_index": 2, "rule": "price_mismatch", "box": [...], "missing": 1}],
"hints": [{"rule": "unreadable_tags", "detail": "tags detected for 5 of 16 facings in this row"}]}
```
**Compliance fails closed on evidence quality.** A row is only judged when at
least 70% of its facings carry tag evidence (readable or unreadable); otherwise it
produces a hint such as `move closer: the price tags in this row could not be
read`, never a violation. Findings must also persist for 3 of the last 5 frames
before they publish. This is deliberate: the first implementation reported 106
missing-tag violations on a shelf where only 8 tags were readable, which was our
detection being incomplete, not the shelf.
`POST /v1/images/{image_id}/classify` re-reconciles against the cached tag bundle
and returns `reconciliation` (and `price_tag`) on each identity row, because in
photo mode the tags are read before the identities exist.
### `POST /v1/images/{image_id}/classify`
```json
{"method": "hybrid", "detection_ids": [0, 3]}
```
`detection_ids` optional. Returns identities for the whole image (or the selected
detections) plus a summary. A method change re-runs it.
### `POST /v1/images/{image_id}/inspect`
```json
{"box": [1435, 383, 1493, 520], "method": "hybrid"}
```
Returns the full decision (`identity`, per-method inlier `sub_details`, `product`), the
cropped product as a data URL, and a feature-match visualization. When the result is
not accepted, the visualization is drawn against the top candidate and
`is_candidate` is `true`.
## WebSocket `/v1/stream`
Live frames in, tracked products out. One session per connection.
**On connect**
```json
{"type": "hello", "protocol": 1, "session_id": "...", "catalog_version": "...",
"model": {"version": "...", "device": "cpu", "imgsz": 640},
"config": {"method": "hybrid", "conf": 0.1, "empty_slots": true},
"limits": {"max_frame_bytes": 4194304, "max_crop_bytes": 1048576}}
```
**Frames (binary):** `4-byte big-endian frame id` + JPEG bytes. The id is echoed back
so the client can pair a result with the frame it sent.
**Control (text JSON)**
| Client sends | Server replies |
| --- | --- |
| `{"type":"config","method":"orb","conf":0.15,"empty_slots":false}` | `{"type":"config_ack","config":{...}}` |
| `{"type":"inspect","track_id":12,"method":"hybrid","request_id":"r1"}` | `{"type":"inspect_result","request_id":"r1",...}` or an error |
| `{"type":"capabilities","max_crop_bytes":131072}` | `{"type":"capabilities_ack","capabilities":{"crop":true,"max_crop_bytes":131072}}` |
| `{"type":"crop","request_id":"c1","track_id":12,"frame_id":41,"jpeg":"<base64 JPEG>"}` | `{"type":"crop_ack","request_id":"c1","track_id":12,"frame_id":41}` or `crop_error` |
| `{"type":"reset"}` | `{"type":"reset_ack"}` |
| `{"type":"ping","t":1}` | `{"type":"pong","t":1}` |
**Per processed frame**
```json
{"type": "result", "frame_id": 41, "seq": 41, "width": 1280, "height": 720,
"tracks": [{"track_id": 12, "box": [812, 404, 909, 549], "raw_box": [...], "confidence": 0.83,
"first_seen_seq": 30, "relabels": 0, "restarts": 0, "inherited": false,
"identity": {"state": "recognized", "sku_id": "sku_005", "confidence": 0.89,
"score": 47.71, "candidates": [...], "attempts": 1, "method": "hybrid"},
"product": {"sku_id": "sku_005", "product_name": "...", "price": 179.0}}],
"empty_slots": [...], "quality": {"sharpness": 812.4, "blur": false, "glare": 0.004,
"too_far": false, "hint": null},
"summary": {"in_view": 61, "recognized": 8, "identifying": 0, "needs_confirmation": 1,
"unrecognized": 0, "detected": 22, "unconfirmed": 30, "empty_slots": 0,
"by_sku": [{"sku_id": "sku_005", "facings": 3}]},
"timing_ms": {"decode": 3.1, "detect": 21.4, "track": 0.8, "total": 25.3},
"stats": {"frames_received": 264, "frames_processed": 264, "frames_dropped": 0,
"server_fps": 13.2, "classify_pending": 3, "classify_ms_p50": 88.9}}
```
The optional crop lane is additive to protocol `1`. Without `capabilities`, the
server sends no `crop_request` and existing clients keep their frame/control flow.
After opting in, a `result` may include `"crop_request":{"request_id":"c1",
"track_id":12,"frame_id":41,"box":[812,404,850,460]}` on an eligible
track. The box uses the **sent frame's** pixel coordinates (the raw detection),
not the camera's native resolution. Keep that exact frame's full-resolution
capture briefly, map its box into the capture, encode the crop as JPEG, and reply
in a text JSON `crop` message. Binary frames remain `4-byte id + JPEG`; the crop
does not take a frame id of its own. `crop_ack` means accepted for asynchronous
classification, not recognized; subsequent results carry the updated identity and
`identity.source` (`frame`, `crop`, or `inherited`). `stats.crops` has `requested`,
`accepted`, and `rejected` counts.
The hello limit is 1 MiB of **decoded crop bytes**. The optional
`max_crop_bytes` in `capabilities` states the client's limit; the acknowledged
session limit is clamped to 1 KiB–1 MiB (default 128 KiB when omitted). Use the
acknowledged limit, not only hello's ceiling. At most four requests are outstanding
per session and each track object is requested at most once. A request expires
after more than six processed frames; clients keep a bounded recent-frame buffer
and may miss a crop if its capture has already been evicted. A late response
cannot label a recycled track. Changing method or resetting invalidates requests.
Crop failures are recoverable text messages with `type: "crop_error"`, `code`,
and `message`. Pre-decode failures also echo `request_id`; session-level refusals
do not currently echo it. Codes: `CROP_BAD_MESSAGE` (missing/invalid base64 JPEG
or missing IDs), `CROP_TOO_LARGE` (empty or oversized decoded crop, or excessive
encoded length), `CROP_UNKNOWN_REQUEST` (no outstanding request, including one
already expired), `CROP_MISMATCH` (track/frame differs), `CROP_STALE` (late or
invalidated track/session), `CROP_UNDECODABLE` (unreadable image), and
`CROP_UNAVAILABLE` (classifier shutting down). A request is single-use, including
when its response is rejected; continue streaming frames rather than retrying it.
**Backpressure:** the server keeps one pending frame per session and processes the
newest, counting the ones it replaced in `frames_dropped`. A client should keep one
frame in flight (`stream_client.py --pace result` does exactly that); sending faster
just drops frames and breaks the pan's continuity for the tracker.
**Recoverable errors** (the socket stays open): `BAD_FRAME`, `UNDECODABLE_IMAGE`,
`PAYLOAD_TOO_LARGE`, `BAD_MESSAGE`, `TRACK_NOT_IN_VIEW`, `INTERNAL`.
**Closes:** `1008` wrong/missing token, `1013` session limit reached, `120 s` idle.
## What the client can rely on
- **A label is only shown for an accepted match.** Weak or ambiguous results are
`needs_confirmation` with candidates; unknown products are `unrecognized`.
- **A label never changes silently.** A settled label is dropped only when the box has
drifted off the product for two consecutive frames, or when the track id was
restarted. Both increment `relabels`/`restarts`, so a client can audit every change.
- **One product is classified once.** While the camera pans, a fresh track that covers
a recently recognized product inherits that label (`inherited: true`) instead of
queueing for classification again.
- **Detection never waits for classification.** Frames come back at detection speed
(~25-45 ms on this machine); labels fill in as the classification pool finishes.
## Design decisions
| Decision | Why | Rejected |
| --- | --- | --- |
| Server lives in this repo, entry point `python code/app.py` | It owns the weights, templates, and the team's workflow | A separate service repo (split the model from its data) |
| WebSocket carrying JPEG frames for live, REST for stills | Browsers and embedded clients both speak it; no media stack | WebRTC (heavy dependency, only wins on lossy WAN), gRPC (browsers), in-browser inference (the matcher is OpenCV Python) |
| Latest-frame-wins with one pending slot | Lowest latency and self-throttling on any network | Queueing every frame (adds latency, never catches up) |
| Tracking-by-detection (ultralytics ByteTrack) wrapped for per-session ids | Kalman prediction plus low-score second association is proven; ids must not be shared | No tracking (re-classify everything, far too slow), a hand-rolled tracker (no motion model) |
| Classify once per track, aggregate up to 3 attempts, lock on accept | Keeps detection real-time and labels stable | Classifying every box every frame (measured as impossible: ~44 ms per crop x 50-170 boxes) |
| Label inheritance by place | The same product gets several ids while panning; re-classifying it is waste | Re-classify on every id (measured 9/66 recognized vs 17-21/66 with inheritance) |
| `hybrid` as the live default | Most accurate (21/21 SKUs on `image.jpg`) | `hybrid_fast` (ORB shortlist): 3.4x faster but only 47% agreement with `hybrid` on 553 crops, so it is not the default |
| CPU as the default device | Measured: 26-29 ms p50 vs 112 ms on MPS, with byte-identical boxes | MPS (slower here), CUDA (no NVIDIA GPU on this machine) |
| Identity states instead of raw scores | "Never turn model uncertainty into business certainty" (project master) | Showing a best-guess SKU with a confidence number |
| Token via env var, access log off | Keeps credentials out of shell history and logs | A `--token` flag, default uvicorn access log (prints the query string) |
| Legacy `/api/sessions*` kept, engine-backed | The earlier session model is still useful and was already tested | Deleting it (loses the work), leaving it on the fake adapter (silently returned nothing) |
## Measured performance
Apple M5, 10 cores, 24 GB, CPU device, 2026-09-24 (`code/bench_engine.py`,
`code/stream_client.py --pace result`):
| Measurement | Value |
| --- | --- |
| Model load / first predict | 50 ms / 985 ms (startup warm-up covers this) |
| Detection, 1280x853 frame | p50 28.7 ms, p95 31.0 ms (169 boxes) |
| Detection, native 2048x1365 | p50 28.7 ms, p95 87.6 ms (170 boxes) |
| Detection during a live pan | p50 46-48 ms (classification pool running) |
| Classification per crop (hybrid) | p50 44 ms, p95 95 ms |
| Classification throughput | 92 crops/s at 8 threads |
| Still recognition, whole shelf | 170 detections, 105 recognized, ~2.0 s total |
| Live results, one client | 10.5-12.2 per second, round-trip p50 58-78 ms |
| Live results, two clients | 8.5 per second each, detection p50 75 ms |
| Time to first label on a pan | ~1.0-2.6 s (includes waiting for the product to enter) |
The live numbers use the shipped defaults: `--classify-workers 5` and torch capped at
4 threads (`DORAYAKI_TORCH_THREADS=4`). Uncapped torch and 8 workers measured slower
(11.1 results/s single-client) because detection and classification fight over the
CPU. Detection is the part that must stay real-time; classification fills in behind it.
The slide's "約28~30ms per image" matches the measured **detection** time (28.7 ms);
the full detect-and-classify of a whole 170-product shelf is ~2 s because each product
is matched against the catalog separately.
## Price-tag OCR: what it can and cannot do (measured)
- **Capture distance decides everything.** At `add/image6.jpg`'s ~1 m (tags ~300x80 px,
digits ~22 px) macOS Vision reads a tag's printed fields reliably and the tag pass
finds 8 readable tags with names and prices in ~0.9 s. At `image.jpg`'s ~2 m (digits
~14 px) it reads none of them - the pass correctly reports nothing readable.
- **Tag detection is the weak link, not the OCR.** On a shelf with ~100 facings the
pass currently finds tags for a minority of them at this distance; the compliance
hints say so instead of inventing violations.
- **A single read can misread a digit, and a consistent misread survives the vote.**
The vote needs independent reads (live mode accumulates them across frames; stills
read each band twice at different upscales) and a dead heat stays unread rather than
guessing, but when one wrong reading wins most frames it settles as wrong - measured
once in a 40-frame live run (`3609` for a `360` tag, 10 observations against 8). The
safeguard is downstream: a wrong tag price shows as a `mismatch` with both values and
is never applied to the product identity. Cross-checking the sale price against the
regular price and discount is the recorded next improvement.
- **The demo catalog is fictional**, so tag prices legitimately disagree with it. That
is shown as `mismatch` with both values; it is not an error.
## Known limitations
- **No session totals.** Facings are counted only for products currently in view. A
product that leaves and re-enters gets a new track id, so cumulative counts would
double count.
- **Identity thresholds are uncalibrated** closed-set defaults (`identity.py`), not
measured accuracy.
- **`surf` is SIFT** in this OpenCV build (no nonfree SURF); the label stays for
continuity with the paper.
- **Fast pans classify less.** A very fast pan creates more new tracks than the
classification pool can settle; a slower, natural scan labels nearly everything.
- **Coverage guidance is partial**: glare/blur/too-far hints exist; "which part of the
fixture was scanned" does not (that needs mosaicking).
- **Stock is in memory** and resets when the server restarts.
- **`hybrid_fast` is experimental** and off by default.
## Mobile client notes (future lane)
The protocol is capture-source agnostic; an iPad/phone client needs no server changes.
- WebSocket client (Ktor `WebSockets` plugin), one frame in flight.
- Capture JPEG frames from `AVCaptureVideoDataOutput` (iOS) or CameraX
`ImageAnalysis` (Android), sent upright; map server boxes into the view with the
same object-fit transform the web client uses.
- iOS ATS: allow local networking (`NSAllowsLocalNetworking`) or use HTTPS; Android
needs cleartext permission for plain `http://`.
- The dorayaki app repo requires an ADR before adding a second API base URL, and its
scan screen is a timer-driven placeholder today - wiring it to this server is a
separate lane.
## Tests
```bash
cd recognization
YOLO_OFFLINE=1 YOLO_AUTOINSTALL=false python -m unittest discover -s code -p 'test_*.py' -v
```
`test_real_model` is discovered and skipped unless `DORAYAKI_REAL_MODEL=1`.
CI installs `requirements.lock` and runs the same command.
View git blame Copy permalink