/v1/demo/plan + /v1/demo/frame give a demo the one thing a still image cannot:
motion. The virtual camera pans a shelf row left to right, steps up, and pans
back - a store employee sweeping a shelf - over a real shelf photo, ~1 ms per
frame, deterministic, and wrapping so an unattended demo loops. The client drives
only `step`, so the walk's speed follows whatever frame rate it can sustain.
Three real defects came out of measuring the walk rather than a still:
- Bands were keyed by on-screen position, so a walking camera minted a new key
every few pixels of pan and re-read the same shelf continuously. Keyed by row
now; OCR stops chasing the window.
- The classification budget was consumed by retries of products already leaving
the frame. Retries now skip anything near the edge, and a track gets 2 attempts.
- The pan advance was 0.22 of the window per step: a facing was in view for one
or two frames, so nothing could ever be classified (9 labelled tracks in 20 s).
0.06 keeps a product in view for about a second.
Measured on the walk (20 s, 1280x720 frames, one client): orb labels 296 tracks
across 21 SKUs with the first label at 0.2 s, while hybrid manages 13 tracks /
4 SKUs at 5.4 s - so the live default for a demo is the fast method, with hybrid
kept for tap-to-inspect.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Web client: price-tag boxes in a distinct style (teal, chip with the parsed
price), an OCR section in the product card showing the tag's price, name,
discount and the reconciliation verdict with both values, and a compliance strip
under the stage. The photo stage now draws the image into the canvas and uses the
same containment as the media, so boxes cannot drift from the product (they did:
the canvas stretched to the stage while the photo was contained).
Server fixes the browser runs forced out:
- Photo mode reconciled before identities existed, so every tag read "tag only".
The tag bundle is cached with the still and classify re-reconciles against the
identities it just produced, returning them per detection.
- The still path only ever read the first four bands (plan hands out two per
call); it now drains every band, twice at different upscales so the vote has
independent reads.
- Voting is confidence-weighted and a dead heat stays unread; the plausible
shelf-price range is 10-9999, which stops a misread from becoming a ¥26,300
price. The compliance coverage bar moved to 0.7 after 0.5 produced a false
"group without price tag" on a shelf where every facing has one.
Measured on add/image6.jpg: 8 readable tags with names and prices, 942 ms for
the whole still, 7 real price mismatches, no false violations, and hints that
say plainly when a row could not be read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The engine owns the OCR backend, a shared OCR lock and a dedicated tag worker, so
a tag pass can never occupy a classification worker or slow a frame. A live
session plans rail bands on the frame loop (cheap), runs them on that worker, and
attaches whatever the cache already knows; a still runs one synchronous pass.
Tracks and detections now carry price_tag and reconciliation, and the summary
carries price_tags plus compliance. Config can turn tags off per session.
Compliance now fails closed on evidence quality. The first real run produced 106
"missing tag" violations on a shelf where only 8 tags were readable - our tag
*detection* was incomplete, not the shelf. A row is only judged when at least
half its facings carry tag evidence; otherwise it reports "tags detected for 14
of 91 facings in this row" as a hint. Missing-tag findings are one per row with a
count, not one per facing, and price_mismatch still reports every disagreement.
On add/image6.jpg the result is now 7 real price mismatches and no false
violations.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
reconcile() compares one accepted identity with its tag and the catalog entry and
publishes a state plus the individual signals (JAN exact, name by longest common
substring, price against either the sale or the regular price). It never changes
the product identity: a disagreement is a second opinion shown next to the first,
and only a checksummed barcode outranks the matcher.
evaluate_compliance() is the first rule family: missing_price_tag, the owner's
missing_tag_group (>= 3 consecutive facings without a tag), orphan_tag,
price_mismatch, plus an unreadable_tags hint. Every rule fails closed on evidence
quality - a row whose tags could not be read produces "move closer", never a
violation. Findings must appear in 3 of the last 5 frames before they publish, so
the UI never flickers.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ocr.py: one interface, backends by availability - macOS Vision through pyobjc
(measured 182-246 ms per rail band), a fake engine for tests, and an explicit
"none". The owner chose the macOS backend; the interface keeps a cross-platform
engine a drop-in, and the module says so.
tags.py: rail bands from the detector's own row clustering (no new geometry
model), clustering of OCR lines into tag-shaped regions, deterministic Japanese
price-tag parsing (sale price by printed area, regular price, discount, unit
price, name, tax/size artifacts rejected), and TagService - a per-session cache
with multi-pass voting so a static tag becomes one settled reading instead of
thirty noisy ones, and so the frame loop never waits for OCR.
Measured on add/image6.jpg: 6 bands, 12 tags, 8 readable with real prices and
names (305 エッグケア, 357 ゼロノンコレステロール, 400 フィッテ), 39-254 ms per
band. On image.jpg (2 m) it correctly finds nothing readable - the tags are too
small there, which is the capture requirement recorded in the design.
Tests found four real bugs: unit prices leaking into the price field, packaging
sizes like <310g> read as prices, cluster_tags dropping the line boxes (so the
big sale price was never preferred), and brittle tag-to-product association when
a cluster spans facings.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The owner's brainstorm thought through to an implementable design, backed by
measurements taken against the real demo images rather than assumptions:
- Tag readability is a capture-distance problem first. At image.jpg's ~2 m the
tags are ~14 px digits and no engine reads them; at add/image6.jpg's ~1 m
(22 px digits) macOS Vision reads all five tag fields including the large red
sale price, and RapidOCR reads the printed fields at 0.97-1.00 confidence but
misses the stylised one. Tesseract jpn is unusable on tags.
- Engine choice is an interface with backends: macOS Vision (182 ms/rail band)
as the Mac fast path, RapidOCR (0.75-1.3 s/band, cross-platform) as the
default elsewhere, fake for tests, PaddleOCR held in reserve. Cloud OCR is
rejected: offline demo, and no owner decision to ship shelf images out.
- OCR must never sit in the frame loop: a TagService runs rail-band passes on a
cadence with a spatial cache and multi-pass voting (tags are static), which
also fixes digit misreads.
- Packaging text confirms 67% of the catalog templates (measured) but only
10%/3% on the demo shelves, because the demo catalog is fictional and the
photos are a real store. Absence is therefore neutral, never a contradiction,
and the demo needs one shelf whose tags and catalog describe the same store.
- Compliance rules fail closed on evidence quality: a row whose tags are merely
unreadable yields a capture hint, never a violation. The owner's "group of
items missing price tags" is the missing_tag_group rule (>= 3 consecutive
facings without a tag).
- Barcode/JAN is the exact-identity path, currently undecodable at demo
distance; the catalog needs optional jan and regular_price columns.
- The mobile lane can start now against protocol v1: tap-to-inspect works with
today's schema, and the tag fields are additive when they land.
Also records the five open questions for the owner (capture assets, engine
default, mobile order, catalog columns, parallel start).
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
STrack lives in ultralytics.trackers.byte_tracker and the YAML loader is
ultralytics.utils.YAML; the basetrack/yaml_load names this module used are gone
in the pinned version. Caught by running the tracker tests against the real
dependency.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Detection and classification share the CPU, and leaving both uncapped made them
fight: uncapped torch plus 8 workers measured 11.1 results/s single-client and
dropped to 6.3/s each with two clients. Capping torch at 4 threads and
classifying with 5 workers gives 12.2 results/s single-client (46 ms detection)
and 8.5/s each for two clients, with zero unexplained label changes.
Detection is the part that must stay real-time; classification fills in behind
it, so the defaults now protect detection's share of the CPU. Both remain
overridable (--classify-workers, DORAYAKI_TORCH_THREADS).
docs/API.md records the shipped numbers and says which configuration produced
them, including the honest gap: detection during a live pan is ~46-48 ms, not
the 28 ms of an idle measurement.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
docs/API.md is the full contract: run flags and environment, auth, coordinate
convention, the identity vocabulary, every REST endpoint and WebSocket message
with examples, the backpressure rule, what a client may rely on, the design
decisions with their rejected alternatives, measured performance, known
limitations, and notes for the future mobile client.
README.md (Vietnamese) now documents the client-server shape, both modes, the
flags, the API, the test commands, and the source layout. It also repairs the
PowerShell escape damage in the original file (backtick-r had become a line
break, backtick-b a backspace, backtick-a a bell), which had left the code
fences and file paths mangled.
test_real_model.py is the opt-in integration test (DORAYAKI_REAL_MODEL=1): it
guards the measured baseline (170 detections, 105 recognized on image.jpg) and
proves a pan keeps labels stable with no unexplained changes.
run_app.sh / run_app.bat pass arguments through to the server.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_live: a product is unconfirmed, then labelled, without blocking frames; a
weak match never names a SKU and ends as needs_confirmation; a restarted id is
re-verified rather than inheriting its old label; inherited labels are flagged;
latest-frame-wins accounting; payload shape; glare hint; per-session
classification cap; a method change clears labels and drops late results.
test_api: health, catalog shape (no local paths), template-image 404s including
traversal-looking ids, stock validation, recognize over raw bytes and multipart,
415/422/400 with one error shape, classify/inspect roundtrip, IMAGE_EXPIRED,
token auth on REST and the socket, the WebSocket protocol (hello, frames,
config/reset/ping, recoverable BAD_FRAME and UNDECODABLE_IMAGE, inspect and
TRACK_NOT_IN_VIEW), the session limit, and the legacy session route.
Server fixes the tests forced out: FastAPI validation errors now use the same
{"error": {...}} envelope as every other failure, and the legacy session routes
accept a missing JSON body instead of returning 500.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_engine: detection normalization, one-caller-at-a-time detection, method
validation, undecodable payloads, crop isolation, classify_boxes accepting the
detection dicts the API actually hands it, summary counting, empty-slot
normalization across frame sizes, the still-image store (roundtrip, TTL, bound),
catalog/stock behaviour, and template-image path resolution that refuses
anything outside the catalog.
test_tracking: per-session id numbering, the case that matters (constructing a
second tracker must not reset a running session's ids), independent sessions,
pan continuity, buffer expiry producing a new id, detection index mapping, and
reset scoping.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The client is a pure API consumer: live frames go out over /v1/stream, stills
over /v1/recognize, and it can be pointed at another server with
?server=&token=. It keeps the team's visual identity (green palette, logo, EN/JA
toggle) and adds the live surface: state-coloured boxes with name/price chips,
interpolated track boxes, a results/s + latency HUD, quality hints, and an
in-view list that highlights a SKU's boxes.
Photo mode recognises the whole shelf at once instead of one click per product:
boxes paint immediately, identities fill in, tapping a product inspects it
(crop + feature-match visualization), the algorithm selector re-classifies, and
dispense updates stock.
Fixes found by browser verification with a synthetic camera feed:
- The WebSocket handler referenced its processing loop before definition, so the
socket died with NameError and live mode never received a result. Helpers are
now defined inside the handler.
- The client never kept a reference to the video element, so frame capture
returned early and nothing was ever sent.
- /v1/images/{id}/classify passed detection dicts where the engine expected box
dicts (KeyError: 'width').
- Placeholder chrome (thumb, dashes, hidden groups) leaked on first paint.
Verified in Chrome with a fake camera: live 8-12 results/s, ~60 products in
view, product card and feature matching populated, photo 105/170 recognised on
image.jpg and 59/111 on add/image6.jpg, 0 console errors.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
app.py becomes the recognition server: /v1 REST (health, catalog + template
images, samples, recognize, classify, inspect, stock) plus /v1/stream over
WebSocket, with token auth, size limits, one consistent error shape, and the
legacy /api/sessions routes kept but now backed by the real engine instead of
the fake adapter. Default bind is 127.0.0.1; LAN and TLS are opt-in.
stream_client.py is the reference non-browser client (camera, video file, or a
pan across a still) and prints results/s, RTT, detect time and label stability.
Live quality work driven by measurements:
- Labels are remembered per place: a fresh track covering a recently recognized
product inherits the label instead of queueing for classification again. On a
continuous pan this raised recognized products in view from 9/66 to 17-21/66
and cut time-to-first-label to ~1.6 s.
- Classification priority is newest-first, because sorting by area spends the
budget on products that are already leaving the frame.
- Per-session classification concurrency raised to 12.
hybrid_fast (ORB shortlist then hybrid) is implemented but NOT the default: it
is 3.4x faster (13 ms vs 44 ms p50) yet agreed with hybrid on only 47% of
accepted crops across 553 crops, far below the 98% adoption rule. It stays
available for experiments; the numbers are in the commit and the benchmark.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
One LiveSession per streaming client. Each frame runs detection and tracking,
then classification is scheduled for products that are new, big enough, fully in
frame and not already settled; results are cached on the track so detection
never waits for classification. Empty-slot gaps are smoothed over frames and
quality hints (glare, blur, too far) come from a 320 px downscale.
Correctness work that the acceptance run forced out:
- Classification submissions happen OUTSIDE the session lock. add_done_callback
runs the callback inline when the future already completed, so attaching it
while holding the lock deadlocked the session on the second frame.
- A label is dropped only when its box has drifted off the product for two
consecutive frames (panning slides boxes onto neighbours), and the anchor is
the box the crop came from, not wherever the track moved by the time
classification finished.
- An id that disappears and comes back is a restart, not the same object: the
old label is cleared instead of silently labelling a neighbour. relabels and
restarts counters are monotonic per id so clients can tell a re-verification
from a silent flip.
- Track ids are tracked per session (tracking.py) and counters survive pruning.
Measured on the synthetic pan (30 frames, 248 label observations): 20
re-verifications, 0 unexplained label changes, detection 20 ms/frame.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
engine.py is the process-wide, thread-safe core the server is built on: one
YOLO instance behind a lock, thread-local matcher access, a classification pool
so detection is never starved, the stock catalog under its own lock, empty-slot
geometry normalized to a reference width (the detector's gap thresholds are
absolute pixels), and a small LRU of recent still images for later
classify/inspect calls. Detection output is normalized so every client sees the
same box shape.
tracking.py wraps ultralytics BYTETracker. The important part: ultralytics
mints track ids from a process-global counter and resets it when a tracker is
constructed, so with two live clients one client's ids would be re-issued while
the other still held cached labels. SessionTracker gives each tracker its own
counter and never touches the global one.
Measured defaults (bench_engine.py, M5): detection p50 26-29 ms on CPU vs
112 ms on MPS with byte-identical boxes, so device defaults to cpu; hybrid
classifies at ~51 ms/crop, 92 crops/s across 8 threads.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps the model's uncertainty out of the UI: decide_identity() separates raw
matcher scores into accepted / ambiguous / unknown / rejected, and only an
accepted result can name a SKU. Zero evidence is rejected, never a product.
observe.py adds frame-local observations and one-zone scan sessions (draft,
finalize, cancel, rescan) with replay, staleness and backpressure rules.
pipeline.process_click() now reports the identity level instead of failing
when the match is not accepted.
Incorporates earlier uncommitted session work in this repository
(identity.py, observe.py, yolo_adapter.py, test_observe.py, and the pipeline
hunk those needed); yolo_adapter keeps its decoded frame per thread so
concurrent sessions cannot swap images.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The recognition server will classify crops from several clients at once. Two
things blocked that:
- OpenCV Feature2D objects were shared across calls, so parallel
classify_crop() calls could corrupt each other. Each thread now gets its own
extractor set; the template cache is untouched.
- score_crop() is extracted from classify_crop() so callers that need every SKU
score (live track aggregation) do not have to re-derive it from the top-5
list. classify_crop() output is byte-identical to before (verified against the
pre-change module on 82 real crops x 4 methods).
- ShelfProductDetector accepts device/imgsz so the server can pin inference.
Dependencies: pin ultralytics to the version that wrote the checkpoint and that
the session tracker targets (8.4.153), add lap so ultralytics never
pip-installs it mid-request, and add websockets for the streaming endpoint.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
