Vision took a crop from 5 ms to 25 ms, so re-reading the same package on every
attempt was the measured cause of the live coverage collapse: 789 -> 60 crops/s
at 8 workers, and 12/8/2/6 labels on the same clip at four pacings.
The reading is now cached on the track and later attempts re-run only the local
lane. Measured through the real server on add/image6.jpg:
saturated pool 15.9 -> 49.4 crops/s (8 workers, 64 crops)
per crop 142 -> 92 ms on a 300x400 crop
3 s clip, send 19.7 mean labels (15-24, unstable) -> 24.0 (23-25, stable)
12 s clip, send 48/131 -> 53/131; result pacing 53/134 -> 50/135
The 12 s numbers are within run-to-run noise; the short clip is where the
pacing sensitivity lived and it is now flat across pacings.
An empty read is not cached - a crop too small or too blurry to read says
nothing about the next one - and the cache is cleared with the rest of the
evidence on a method change or a drift relabel, so a reading never outlives the
product it described. The native crop lane skips OCR on the same rule.
Test: OCR runs once per track while the local lane keeps running. Disabling the
skip makes it fail 2 != 1. Suite 280 -> 281.
Two review findings, both confirmed by measurement.
1. The evaluator had no always-argmax lane, so every "vs Van" claim compared
against the *gated* matcher, which is not Van's contract - the original demo
answered every crop with its best-scoring SKU and had no accept line. The new
'van' lane is that contract: same scores, no gate. Against it the local lane
wins both axes at both resolutions (full 61%/77% vs 52%/29%; live 59%/95% vs
57%/32%), but the live coverage margin is one facing, not seven, and the 95%
figure is live-resolution only - at full resolution the same lane reads 77%
with 11 false accepts and a hub alarm.
2. A crop reading only "400g" scored 0.667 on three SKUs, above TEXT_ACCEPT, so
the text lane named a product from a size line. The fill-in job now requires
two non-digit bigrams; the veto is unchanged, so "65%" still contradicts a
local answer. The raw OCR read travels with the ranking through
score_crop_lanes for that check.
Tests: 3 for the size gate (policy + live) and the van lane's own row. Removing
the gate fails all three. Suite 276 -> 280.
load_shelf_skus ran after write_catalog, so a missing path or a malformed list
escaped as a raw traceback over a catalog that had already been replaced. It is
now read inside the try, before anything is written, so both cases fail with the
tool's own message and the destination is untouched.
Adds BadShelfListFailsBeforeTheSwapTests, which asserts the clean message and
that the output path does not exist. Re-injecting the ordering fails both tests
with the raw FileNotFoundError. Suite 274 -> 276.
The protocol asks for a source at least 800px tall so a view can be stored as a
200/400/800px pyramid. Nothing enforced it, and the clean catalog's l09 view is
305px - the documented defect where its inlier scores dwarf every other
reference (85-129 against 10-40) and it wins facings it should not.
Reported, not refused: the view is usable and dropping it would remove the SKU
from the catalog entirely, which is worse than an incomparable score. The fix is
re-sourcing the packshot, which is a data task. The check names the view and its
height so the re-sourcing has a target. Suite 272 -> 274.
Three items from DECISION §8's enrollment protocol were unimplemented and none
of them needed Tier-2 data:
- Eval reports now record provenance: catalog hash, label-file hash, image hash
and the code revision, so a number can be traced to the inputs that produced
it. The catalog hash covers the CSV bytes and the view names, not image bytes:
a re-encoded view of the same product is the same catalog.
- rebuild_catalog flags cross-SKU near-duplicates as text-discriminated
siblings. These are the near-unknown case no threshold rejects - the geometry
is real because the sibling shares the brand's artwork - so they are reported
rather than refused.
- --shelf-tags turns the completeness rule into a live-ready verdict. Without
the shelf's SKU list the verdict is 'unknown', not a guess.
Adds test_rebuild_catalog.py (11 tests; the tool had no test file). Suite 261
-> 272. Also corrects score_crop_lanes' docstring, which still described the
live path as averaging local scores when the local scale decides on the newest
crop.
The shipped default scale had no test: make_session only builds matcher-backend
engines, so scale == "local" in _on_classified never ran in the suite - and
that is the branch that was wrong twice (the rank re-sort, and the early return
that made the text lane's fill-in job unreachable).
Adds LocalScaleEngine, which injects the lanes' evidence into the live path, and
two live tests: a text-only read still labels the track, and a text veto blocks a
local accept. Adds DecisionPolicyTests for the veto/fill-in boundary, including
the lane-order-vs-value case that the rank re-sort broke.
Both regressions were re-injected to confirm the tests fail for the intended
reason: restoring the early return fails the fill-in test, and re-sorting the
lane output by value fails the order test. Suite 251 -> 261.
The record said '2 false accepts of 48 unknowns', which conflated two fields:
the 2 are 0 unknown accepts plus 1 unenrolled in-catalog facing and 1 enrolled
misidentification. Also restores the hybrid-verify line the lane vocabulary
edit dropped, and reframes the l09 note: its hubness ratio is 1.76 full /
1.36 live against G7's limit of 2.0, so the gate passes and the 30% internal
share heuristic is what raises the alarm.
The default backend could not be scored by the repo's own evaluator, so the
number that justifies the architecture lived only in an ad-hoc script.
- eval_identity.py: `--lanes local` wraps identity_lanes.LocalLane and
TextLane behind the same four ceilings as every other lane. `Lane.decide`
is now the seam: the default runs the shared gate over the lane's ranking,
and the fused local+text lane overrides it, so the production decision is
what gets scored rather than a re-implementation of it.
- rebuild_catalog.py: one-directional image/CSV match (an enrolled image with
no row refuses; a row with no image is a text-only coverage gap), packshot
OCR against the catalog name, capture-session refusal from an evaluation
manifest, and the 160 px floor. The CSV is copied byte-for-byte so all 12
rows survive a rebuild, and unmanaged files are carried over.
Reproduced from the repo, live resolution: 59% end-to-end at 95% precision,
2 false accepts of 48 unknowns, 20.6 ms per crop.
Replaces the DINO gallery as the recognition path with the measured design
from docs/research/2026-09-28-recognition-architecture/DECISION.md: RootSIFT
over a per-SKU reference pyramid, similarity-constrained RANSAC and a hue
check, fused with an OCR text lane that vetoes a match whose package text
names a different catalog row and answers on its own when geometry finds
nothing.
- identity_lanes.py: LocalLane, TextLane, DecisionPolicy. The text lane
builds each row's signature from the catalog name plus its packshot OCR, so
a row with no enrolled image is still answerable.
- engine.py: every backend groups scores by canonical SKU, so no
view-suffixed id reaches the wire and a name and price always resolve.
decide_crop fuses the lanes; score_crop_lanes feeds the live path. Both
lanes are built in warm_up (non-fatally) so the first frame is not the slow
one, and the text lane shares the engine's OCR lock.
- live.py: decides per crop on the lane's own rank order. The lane's order
comes from its rank key and its value is only the gate, so re-sorting by
value named a different SKU than the lane chose; averaging in an empty
observation also diluted a good confidence below the accept line.
- identity.py: a `local` score scale whose gate is one absolute line, because
the value is already a margin over the runner-up.
- app.py, run-demo-server.sh: `--identity local` is the default; the demo
server no longer starts on the leaked `stock-lopia` catalog.
- eval_live.py: the replay vocabulary accepts `local`, so a recording from
the shipped server can be scored.
- README.md: documents the three backends and what each is for.
Measured on the corrected image6 labels at live resolution: 36/61 in-catalog
facings named at 95% precision, 0 of 48 unknowns accepted, 11 labels sourced
from the text lane alone. Through the real server: /v1/recognize 90%
precision, /v1/stream 100% precision, no view-suffixed ids on either.
Two session models shared one process. `/v1/*` and `/v1/stream` are the live
one; `/api/sessions*` and `/api/compare_live_vs_still` were the earlier model,
still constructed on every `create_app()` through `build_legacy_scans` even
though nothing called them. The web client and `stream_client.py` speak only
`/v1`.
Deleted: `observe.py` (ScanService/PerceptionService/PerceptionAdapter),
`yolo_adapter.py` (imported only by the legacy builder), `pipeline.py` and
`test_suite.py` (the old single-image CLI, reachable only from each other),
`test_observe.py` (its Session* tests), the eight legacy route handlers,
`build_legacy_scans` with its two facades, `create_fastapi_app`, and the
`scans` parameter on `create_app`.
`IdentityRankingTests` moved to `test_identity.py` before `test_observe.py` was
deleted: those five ranking rules are the live path's contract, not the legacy
model's. `test_api.py` keeps every `/v1` and WebSocket test.
Verified: 246 tests OK (2 skipped), and a route dump of a freshly built app
shows no `/api/*` path with all nine `/v1` routes and the WebSocket intact.
The previous commit stopped the live path re-gating gallery scores at
COSINE_ACCEPT, but it passed zero accept lines. `aggregate_votes` divides by the
whole observation window, so one rejected crop beside one accepted 0.45 averages
to 0.225 - and a zero gate labelled that, at a score the gallery itself calls
unknown.
The aggregate now clears the gallery's own line: `head_threshold` (0.40) when a
fine-tuned head is loaded, `threshold` (0.55) otherwise. The margin gate is
still the only extra gate, so a head match between 0.40 and 0.60 is labelled.
Regression test: a rejected crop followed by an accepted 0.45 must not label the
track (fails on the zero-gate version).
Verified: `python -m unittest discover -s code -p 'test_*.py'` - 258 tests, OK,
2 skipped.
`score_crop` returns only scores the gallery already accepted: the raw 0.55
line, or the fine-tuned head's own 0.40. The live path then ranked that
aggregate with `decide_identity(scale="cosine")`, whose defaults rewrite to
COSINE_ACCEPT 0.60 / COSINE_AMBIGUOUS 0.50 - a second, higher line. Every head
match between 0.40 and 0.60 was therefore refused or downgraded to ambiguous,
which is the live regression: the head's own training views score 0.57-0.77.
`engine.inspect` and `_classify_with_gallery` already avoid this by ranking
through `decision_for_match`, which applies only the published margin. The live
path now does the same: on the cosine scale it passes explicit zero accept
lines, so the margin gate is the only extra gate.
Regression tests: a 0.45 head-scale match is labelled (fails on the previous
code), and a 0.45/0.44 pair inside the margin stays unnamed.
Verified: `python -m unittest discover -s code -p 'test_*.py'` - 257 tests, OK,
2 skipped.
The server segfaulted a few seconds into every live scan that used the crop
lane. The crash report put the fault in libtorch's Metal shader library
(`MetalShaderLibrary::exec_unary_kernel` inside an unordered_map insert),
reached from `copy_cast_kernel_mps` under `clone`/`reshape`.
Reproduced deterministically: eight threads calling `embed()` on a fresh
process crash it, while the same eight threads after one warm forward are
fine. Metal's shader library is process-global and is not thread-safe while
kernels are still being compiled into it, which is exactly what the first
forward does. The detector already serializes its predict for this reason;
the gallery did not.
Two changes:
* A module-level `_MPS_INFER_LOCK` covers the whole MPS region of a forward -
model load, `.to(device)`, the forward, and the normalize that follows it.
Normalize dispatches its own Metal kernels, so leaving it outside the lock
left the same first-compile race open. Module-level because the state being
protected is the process's, not one gallery's.
* A cache hit skips the embedding forward, so the model used to be built and
the kernels compiled by whichever classify worker asked first - up to eight
racing into one process-wide cache. The load path now warms the backbone
single-threaded under the existing lock, on the cache-hit branch too. The
readiness path loads the gallery before the socket accepts, so the warmup
is free. It is Metal-only and best-effort: a CPU gallery has no shader
library to compile, and a cache hit is served without a usable backbone by
contract.
The regression test records overlap rather than timing: a fake backbone that
sleeps and counts concurrent entries, run from four threads, asserting the
count never exceeds one. It fails on the previous gallery.py and needs no MPS.
The crop lane asked the phone for a full-resolution crop of a facing the
frame could not decide. Four defects made that unsafe or useless:
* A capability was half-applied. `crop: true` without a usable
`max_crop_bytes` enabled the lane on the client's say-so, leaving the two
sides believing different limits. A capability is now believed whole or
refused with BAD_MESSAGE, and the limit must be an integer in 1..1048576
(bool is an int in Python and is not a limit).
* A small JPEG could claim enormous dimensions. The header is now read
without decoding and anything over 4 megapixels is refused before the
decode allocates it.
* A request was single-use, so an expired one stranded a facing only the
crop lane could decide. The lane now spends the same max_attempts budget
the frame path spends, refilled on method change and on settled-label
drift, and a phone that never answers still cannot loop.
* A candidate could be asked about from a box the frame path would refuse:
clipped by the frame edge, carried over from an earlier frame, or measured
at a score the tracker does not trust. Both lanes now share one edge gate,
and only a track whose box this frame actually measured is asked about.
Drift and method change also cleared `identity` but left `confirmed = True`,
which permanently blocked both the crop lane and label inheritance for that
track; both now clear it, without which the budget reset was dead code.
Blur and glare are refused as whole-capture faults, while `too_far` is
exempt on purpose: a distant shelf is the one frame a native-resolution crop
exists to decide.
A cache key proves the store, not the rows inside the file. The old check
asked only that every cached row appear somewhere in the enrolled list, so a
row moved across SKUs still passed: the matrix that travelled with it would
then be filed under the wrong product, and a valid key would rename a SKU.
Rows now walk the enrolled list forward and never backwards, so a moved,
repeated or foreign row has nowhere left to match. Views that could not be
decoded stay absent without disturbing the order of the ones after them,
which is what keeps a shorter cache a hit rather than a re-embed.
The live session scored crops with the matcher regardless of the configured identity
backend, so a live scan used the old mechanism while the still path used the enrolled
gallery - measured, the matcher accepts look-alikes and cannot say "unknown", which is
exactly what a real shelf needs it to do. score_crop now routes to the gallery, and
the decision uses a similarity scale (accept at 0.60 with a 0.04 margin) rather than
inlier-count thresholds that mean nothing on a cosine.
The synthetic walk is gone: demo.py, demo_replay.py, the /v1/demo endpoints and the
walk benchmark are deleted, and the app's frame source is the camera on a device (the
single-image replay stays for the simulator, which has no camera).
stock-demo merges both demo shelves (27 products, 62 views, trained head) so one
server recognises either image, and run-demo-server.sh starts it bound to 0.0.0.0 and
prints the exact URL to put in Config.xcconfig.
Verified over the LAN with 1280x960 frames: 40 frames in 5.3s, 9 products recognised
on demo image 1.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two real store photographs, chosen for density and for products that look different
from each other, each with its own enrolled store:
demo-images/demo-image-1-mediya-dairy.jpg CC0 MEDIYA dairy case
21 products kept (13 price cards dropped), 16 recognised, all 14 products named
demo-images/demo-image-2-milk-case.jpg CC BY 2.0 supermarket milk case
56 detections, 49 recognised, all 13 products named
Both were enrolled the way a store would be: the products were read off a contact
sheet of the detections, their crops became the store's gallery (stock-mediya,
stock-milk150), and a head was trained on them. `build_store_from_shelf.py` is that
flow.
The filter came out of the first image. Every Japanese supermarket photo has the same
artifact: the detector boxes the shelf-edge price cards as products, because a card is
roughly the shape of a pack - 13 of 34 detections on image 1 were cards. Their text
separates them cleanly: a card's characters are 30-44% digits and it prints the
tax-excluded price (本体), while a product's are 3-19% digits. Cards are now dropped
before classification, and the count is reported as `price_cards_dropped` rather than
hidden.
demo-images/README.md records how to run both, what the demo shows, and the honest
caveat: both shelves are enrolled, so the recognition rate is in-distribution - what
the app does on a shelf it has been onboarded to, not on one it has never seen.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The head is now part of a store rather than a script: `finetune_identity.py --stock
stock-lopia` trains on that store's enrolled views (27 views over 7 products, 12
augmented copies each) and writes `<stock>/identity_head.pt`, which ProductGallery
loads automatically.
The first attempt exposed a trap worth recording. A discriminative head has no "none
of the above": trained with an angular margin to separate seven products, it maps
everything confidently onto those seven, including cartons it has never seen.
raw features, threshold 0.55 home shelf 78% accepted foreign shelves 5% accepted
trained head, threshold 0.40 home shelf 58% accepted foreign shelves 44% accepted
Rejection - the gallery's whole point - disappears. The fix needs no retraining: the
two mechanisms fail in opposite directions, so raw features decide *whether* to answer
(they generalise) and the head decides *what* to answer (it discriminates). Measured:
47% accepted on the home shelf, 5% on foreign ones. Lower coverage than raw-only,
which is the honest trade.
The head also needs its own threshold (0.40 against the raw 0.55): a trained head's
similarity scale is its own, and reusing the raw line rejected almost everything it
said.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Part 2 ended on the number that matters: enrolling a real shelf from itself leaves
off-the-shelf DINOv2 at ~25% top-1, because real milk cartons differ by small text
and a colour band. This trains a small projection head on frozen features - 384 -> 128,
cosine classifier, angular margin - with augmentation standing in for the second visit
we do not have, and measures it on 72 held-out facings over 8 products.
raw DINOv2 cosine top-1 2/72 (2%) balanced 1%
raw, rejecting below 0.55 top-1 2/72 (2%) balanced 1% (5% of answered right)
fine-tuned + margin top-1 28/72 (38%) balanced 73%
fine-tuned, rejecting below 0.4 top-1 17/72 (23%) balanced 58% (94% of answered right)
So fine-tuning is the difference between working and not (balanced 1% -> 73%), and
with a rejection threshold the model becomes precise - it answers 18 of 72 facings and
is right on 94% of them, which is the operating point a store wants. The raw model
cannot do that at any threshold.
Where it fails is where my labels are coarse: MEGMILK, which the contact sheet shows
is probably several products under one brand, scores 1/32 because the class is
internally inconsistent and the model correctly refuses most of it.
Honest limit, stated in the doc: enrolment and held-out facings come from one
photograph, so these are upper bounds; only a recorded second visit settles the real
number. What is settled is the mechanism - a head trained on labels the pipeline can
produce itself moves fine-grained identity from useless to usable.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The external evaluation showed the matcher cannot say "unknown": on 20 shelves whose
products are absent it claimed 47.8% of facings as recognised, and its raw inlier
score cannot be thresholded (foreign shelves' median 6.0 overlaps the real shelf's
5.2 p10, and the worst offenders score 50-190). So the gallery becomes a selectable
identity backend:
--identity gallery (or DORAYAKI_IDENTITY=gallery)
Measured on the same 20 foreign shelves: false accepts fall from 47.8% to 13.8% at
the default threshold and to **5%** at 0.60, with a smooth curve to pick an operating
point from. Detection is unchanged (the gallery only names boxes).
What it does NOT fix, measured on a milk shelf where 14 of 107 facings really are
明治 おいしい牛乳 (labelled by eye): precision is 28-45%, no better than the matcher's
39%, and enrolling the shelf from itself (1-3 views per product) is flat at ~25%
top-1. Off-the-shelf DINOv2 is not discriminative enough for fine-grained SKUs - the
same conclusion the literature reaches, now measured here.
The packaging's own text is a real channel and resolution-bound: 51% recall
single-scale, 60% with the tag reader's multi-scale + red-channel trick, 85% where
the text is large (MEGMILK) and 28% where it is small vertical Japanese.
docs/EXTERNAL-EVAL.md records all of it, including the identity architecture it
implies: reject by default, name from the tag when readable, from the packaging text
when legible, from a fine-tuned embedding otherwise, and hand the rest to the
operator.
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Everything else here is measured on two photographs. This collects real store
shelves from Wikimedia Commons (scriptable API, explicit licences: CC0, CC BY,
CC BY-SA, public domain, each recorded in a manifest) and evaluates the pipeline on
them. 46 of 54 images carry a shelf; the rest are book scans and a shop front.
What it found, and it is not what I expected:
- **Detection transfers.** 2,258 detections across dairy cases, soy-sauce shelves,
spice racks, a confectionery aisle, an Okinawan supermarket and a Hong Kong
grocery - tight boxes on images the model has never seen, and correctly nothing on
the book scans.
- **Identity transfers when the product is genuinely there.** On a 2011 milk-case
photo it labelled six facings of 明治 おいしい牛乳 correctly, and 野菜生活100 and
R-1 elsewhere. The matcher does reach new photographs of products it has templates
for.
- **Precision is the failure, and rejection is missing.** In that same photograph it
labelled 低脂肪殺菌牛乳, 有機豆乳, 特濃4.5, 無調整豆乳 and 北海道牛乳 as おいしい牛乳 -
they are all white milk cartons. Across 24 shelves whose products are not in the
catalog it accepted 451 identities, and the score cannot be thresholded to fix it:
the real shelf's median is 18.2 with a p10 of 5.2, while the foreign shelves' median
is 6.0 and the worst offenders score 50.5 (a bento), 80.6 (a shop interior) and
189.9 (deer cookies). A threshold that catches the foreigners also throws away the
real matches.
That confirms on real data what the literature and the review predicted: the
matcher's raw inlier score is not a confidence, and fine-grained look-alikes are the
failure mode. Step 4 of the plan - shortlist, local re-rank, calibrated rejection -
is the fix, not a refinement.
docs/EXTERNAL-EVAL.md records the numbers, the method and the attribution; the
images themselves are not redistributed. 175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The camera moves and the shelf does not, but every measurement so far treats a
frame as the world: a facing is "a track id", and that id dies the moment the camera
pans away and comes back. Label inheritance, drift checks and restart counters were
each correct fixes for that, and all of them are symptoms.
`shelf_map.py` keeps a per-session map of the shelf plane. Frames are registered by
their content shift, detections are placed in shelf coordinates, and a detection that
lands where a facing already is merges into it - so identity votes and tag readings
pool on the physical facing.
Measured, and reported honestly:
- Registration is exact on the synthetic walk: 0.01 px median error against the demo
camera's own offsets, over 23 consecutive pairs, with both phase correlation and
ORB + RANSAC. That validates the harness, not the method - the walk is a pure
translation of one photograph. Real video needs step 1's footage.
- Pooling works: a facing seen 15 times across 2 track ids reports one identity at
1.00 agreement.
- De-duplication is NOT yet better than the tracker: 179 facings against 156 track
ids, where the shelf holds roughly 70. Both over-count, the map slightly more.
Two bugs were found by measuring, and both were silent: the content shift is the
negation of the camera's motion (getting it backwards turned ~60 facings into 857),
and matching purely by box overlap lets jitter split a facing (210 facings, 103 of
them seen once; matching by track continuity first brought it to 179 with 38).
175 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The deep review's first recommendation was real data, because every accuracy number
so far comes from one or two photographs - the identity benchmark even takes its
gallery and its queries from the same photo. This is the mechanism for replacing
that.
`--record-frames DIR` writes each processed frame, sampled every Nth, with a JSON
line describing what the server saw in it: the boxes, the identity channels and the
tag reading. The lines are the labelling input - `labelling_candidates()` returns
the facings the tag channel named with a corroborated price, which are seeds for the
confirmation screen rather than guesses. Nothing here is a training set on its own.
Sampling keeps each frame's own number in the index, so a sampled recording says so
instead of hiding it, and `/v1/health` reports how much has been written. Measured
on a 163-frame walk: 32 frames and their lines.
`docs/DATA.md` is the runbook: what to capture (a slow walk per rail, a close pass
for the barcode, and the same shelf on a second visit), how to turn it into labels,
and how to freeze a benchmark with a cross-visit split.
167 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A tag is only readable when its own arithmetic corroborates the price, so the
fixture needs the 162.00 that belongs with its 150 (150 x 1.08 = 162.00). Without
it the fixture asserted the old behaviour.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
From the deep review's hygiene list:
- The demo replay (a still recognition replayed as if it were live tracking) moves
out of app.py into demo_replay.py, and every payload it produces is marked
`synthetic: true`. It had been committed inside an unrelated feature commit, which
made a film-making device look like part of the server.
- `/v1/health` reports the OCR engine and whether it is available. Without it an
operator could not tell a broken OCR from a shelf with no readable tags.
- A failed OCR pass is counted and logged instead of swallowed: `TagService.stats()`
now carries `passes_failed` and `last_pass_error`.
- The live classification budget goes back to the measured production defaults
(16 in flight, 2 attempts); a demo walk over a nearly static shelf asks for more
through `--live-inflight` / `--live-attempts` rather than silently changing the
default for everyone.
162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Three findings from the deep review, all about the tag channel claiming more than it
can support.
1. The exact signal is a barcode. The reconciliation compared `tag.jan` against the
catalog's JAN, but that field held the regular price - a number compared against
a product code. It now reads a `barcode` field (populated only by a close-up
decode) and the printed number is compared as a price.
2. A price-mismatch violation required "an anchor", which was `tag.jan` - a price.
The anchor is now the tag's own arithmetic agreeing, or a name that actually
resembles the product, and the name signal carries its score as a field instead
of being parsed back out of a human-readable note string.
3. Tag identity could reach anywhere along a row: a facing whose own tag was not
detected inherited whichever readable tag was nearest, which can be a different
product several columns away. Assignment is now capped at the product's own run
(2.5 facing widths, at least 200 px), and a tag may only overrule the packaging
channel when it is corroborated or its name matches at 0.6 rather than 0.4.
162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two corrections, both found by the deep review.
1. The number printed large beside the sale price is NOT the JAN's last three
digits. Checked against the manufacturer, キユーピー ゼロ ノンコレステロール 310g
is JAN 4901577092381 - it ends in 381, not the 357 the tag prints. The number is
the regular (pre-discount) price, and all three tags on the demo shelf confirm it
against their printed discount: (357-269)/357 = 24.6 % -> 2.4割引,
(400-379)/400 = 5.25 % -> 0.5割引, (248-199)/248 = 19.8 % -> 1.9割引. The field
is renamed `regular_price`, the barcode keeps its own name, and the discount is
now a second arithmetic check alongside the tax-included price - so a tag can be
checked against itself twice.
2. A price is only readable when that arithmetic corroborates it. Measured on a live
walk before this: 90 tracks carried a tag marked readable, with prices of 20, 23,
36, 50, 72, 2002 and 4508. The gate has three strengths - an exact tax pair, an
agreeing printed discount, or both plus agreeing digits - and `TagRecord.state`
now consults it, so the wire cannot call an uncorroborated read readable.
Also fixed while measuring: voting picks each field independently, so a record could
publish a price from one frame beside a tax-included price from another (measured:
price 149 with unit 214.92, which never appeared together - 214.92/1.08 = 199). The
arithmetic is now re-run on the values actually being published, stale flags are
dropped, and an implausible regular price (3050 beside 349) is suppressed rather
than published with a discount derived from it.
Measured through the real still path: readable tags 9 -> 5, every published price
now satisfies its own tax-included price, and the live walk publishes none instead
of publishing garbage. 162 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The user's objection was right and it changed the design: the tag channel is
sparse in the real world (measured: 0 tags readable at ~2 m on the main shelf), so
packaging recognition has to carry identity. Measured on a 12-class, 57-facing
evaluation set read off the demo shelf, the packaging channel as shipped cannot:
views per SKU embedding top-1 shipped matcher top-1
1 57 % 33 %
2 72 % 27 %
3 75 % 20 %
4 76 % 5 %
Two things follow. First, two views per SKU is where most of the gain is - that is
the enrollment spec (a first scan already produces several facings per product).
Second, the matcher gets *worse* with more views, because its raw inlier score is
not calibrated to be aggregated and it cannot express two views of one SKU at all:
it keys templates by filename.
So identity becomes retrieval: `gallery.py` embeds every enrolled view with DINOv2
ViT-S/14, scores a SKU by its best view, and rejects when nothing is close enough
(an unknown product must come back unknown, not as the nearest stranger). Measured
with held-out views - the gallery never contains its own queries - 2 views per SKU
over 12 SKUs gives 69 % top-1 with rejection working, and the threshold moves the
rejection line.
It is affordable in the live loop only on the GPU: 159 ms per crop on CPU versus
3.6 ms on the M5's MPS, and a live frame only needs its new tracks embedded.
157 tests pass; test_gallery.py covers the retrieval contract (best view wins,
several views are one answer, strangers are rejected) with injected embeddings so
it needs no torch.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Stage 2 of the plan is "identity as retrieval", so the first thing measured was
whether retrieval can even work here. It cannot, on its own: an off-the-shelf
DINOv2 embedding does group identical products (two orange juices, two apple
juices, two almond drinks) but the fine-grained cases collapse - 32 cartons into
one cluster, and nine different bottled drinks grouped by *container shape*. That
matches the literature (RP2K, RPC all fine-tune; nobody ships off-the-shelf
features for SKU identity).
Meanwhile the shelf has been carrying the answer. Every product has a tag printed
under it, and the tag spells the product's name out in clean text, once per
product - the store's own label. This adds that as an identity channel:
- a tag with a readable name is matched against the catalog by name similarity;
- facings are assigned to the *nearest tag along their row* (the tags mark where
products sit), not by geometric grouping, which was measured to merge different
products into a 945 px "product" that then swallowed the wrong tag;
- rows are clustered first, because the same column on six shelves would
otherwise become one product.
Then the demo catalog itself had to be fixed: it listed another store's products
(senbei, Tropicana) while the shelf holds mayonnaise, so nothing could match. The
catalog for this shelf was read off the tags - 12 products with names and prices -
and the products were enrolled as packaging templates by cropping the facings the
tags had just named.
Measured through the real still path on add/image6.jpg (111 detections):
tag named the product 28 facings (all 28 verified correct by eye)
packaging named the product 74 facings
both spoke 23 facings, agreeing 16 (69 %)
facings with any identity 79
So packaging recognition makes a ~30 % error rate on the facings the tag can
check, and every disagreement carried a low packaging score. The tag therefore
wins a conflict, and the conflict is recorded (`identity_conflict`) instead of
hidden - a product whose tag says one thing and whose packaging says another is a
real finding, not noise to suppress.
144 tests pass; test_tag_identity.py adds 6.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two false positives came out of running the rules against a real shelf, and both
had the same shape: the rules were arguing with our own detector instead of with
the shelf.
1. One tag per product, not per facing. Measured: a row carries 16-18 facings but
only 4-5 tags, because four facings of a mayonnaise share a single tag. Counting
facings invents missing tags on a shelf that is perfectly labelled. The rules
now group a row's facings into products - by identity when the facings were
recognised, by size and spacing otherwise - and count products.
2. A product's tag may sit anywhere along its run. Measured: a run of
ピュアセレクト facings was split by geometry because one facing was detected
narrower than its neighbours (54 px beside 150 px), and the fragment was then
reported as missing the tag that was sitting further along its own run. A
product now counts as missing only when no tag exists for its SKU anywhere in
the row, and a fragment far narrower than the row's typical product is not
judged at all.
3. A mismatch needs one trustworthy anchor. Measured noise: a wrong identity plus
a garbled tag read (name match 0.11, no JAN) produced a confident "tag says 90,
catalog says 188" that was really two bad reads disagreeing. Without the tag's
JAN or a name that resembles the product, that is now a hint, not a violation.
Measured on add/image6.jpg through the real still path: 0 false violations without
identities (was 1), and with identities the only finding left is a genuine
tag-versus-catalog price difference. Coverage hints now read "tags detected for 2
of 4 products in this row". 143 tests pass.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
