feat: add Van's real baseline lane and stop size tokens naming products
Some checks failed
tests / model-free (pull_request) Has been cancelled
Some checks failed
tests / model-free (pull_request) Has been cancelled
Two review findings, both confirmed by measurement. 1. The evaluator had no always-argmax lane, so every "vs Van" claim compared against the *gated* matcher, which is not Van's contract - the original demo answered every crop with its best-scoring SKU and had no accept line. The new 'van' lane is that contract: same scores, no gate. Against it the local lane wins both axes at both resolutions (full 61%/77% vs 52%/29%; live 59%/95% vs 57%/32%), but the live coverage margin is one facing, not seven, and the 95% figure is live-resolution only - at full resolution the same lane reads 77% with 11 false accepts and a hub alarm. 2. A crop reading only "400g" scored 0.667 on three SKUs, above TEXT_ACCEPT, so the text lane named a product from a size line. The fill-in job now requires two non-digit bigrams; the veto is unchanged, so "65%" still contradicts a local answer. The raw OCR read travels with the ranking through score_crop_lanes for that check. Tests: 3 for the size gate (policy + live) and the van lane's own row. Removing the gate fails all three. Suite 276 -> 280.
This commit is contained in:
parent
13ee7426e2
commit
fd72251160
7 changed files with 180 additions and 22 deletions
|
|
@ -589,17 +589,57 @@ own" job unreachable. Live now decides per crop on the lane's own order.
|
|||
Measured after the change, through the real server: `/v1/recognize` names 41 of
|
||||
112 detections at 90% precision with no view-suffixed ids; the live WebSocket
|
||||
names 12 at 100% precision with 0 unknown accepts, first label on frame 3.
|
||||
Offline on the corrected labels: 36/61 (59%) at 95% precision, 0/48 unknowns,
|
||||
11 of those labels sourced from the text lane alone.
|
||||
Those are detector-box figures, not label-matched ones: the 41 are detections
|
||||
whose box overlaps a labelled facing, so they measure the endpoint rather than
|
||||
the identity lane. The label-matched number is the evaluator's, below.
|
||||
|
||||
## The baseline is Van's contract, not the gated matcher
|
||||
|
||||
The `matcher` lane applies `decide_identity`, which is not what the original
|
||||
demo did - it answered every crop with its best-scoring SKU and had no accept
|
||||
line. Comparing against the gated matcher understates the baseline's coverage
|
||||
and overstates its precision, so the evaluator now has a `van` lane: the same
|
||||
scores, always answered. Every "vs Van" claim in this record uses that row.
|
||||
|
||||
.venv/bin/python code/eval_identity.py --stock stock-lopia-clean \
|
||||
--labels docs/research/2026-09-28-recognition-architecture/evidence/image6_v2.json \
|
||||
--image add/image6.jpg --lanes van,matcher,local --live
|
||||
|
||||
full resolution — 109 scored facings (61 in-catalog, 48 unknown)
|
||||
lane catalog identity decision end-to-end precision false acc
|
||||
van 69% 76% 76% 52% 29% 77
|
||||
matcher 69% 76% 71% 49% 39% 46 HUB!
|
||||
local 69% 95% 88% 61% 77% 11 HUB!
|
||||
|
||||
live resolution — 109 scored facings (61 in-catalog, 48 unknown)
|
||||
lane catalog identity decision end-to-end precision false acc
|
||||
van 69% 83% 83% 57% 32% 74
|
||||
matcher 69% 83% 76% 52% 43% 43
|
||||
local 69% 86% 86% 59% 95% 2
|
||||
|
||||
Against Van's real contract the local lane wins both axes at both resolutions:
|
||||
61% at 77% against 52% at 29% (full), and 59% at 95% against 57% at 32% (live).
|
||||
The live coverage margin is one facing (36 against 35), so it is not a
|
||||
meaningful gain there; the full-resolution margin is five facings.
|
||||
|
||||
**The 95% figure is live-resolution only.** At full resolution the same lane
|
||||
reads 77% precision with 11 false accepts and raises the hub alarm, and the
|
||||
still endpoint and the native crop lane both run at that scale. Reaching 95% at
|
||||
full resolution needs an accept line near 6, which leaves 28/61 correct - below
|
||||
Van's 32/61. So the design does not yet beat Van on both axes at full
|
||||
resolution, and the accept line is provisional: it is tuned on Tier 1 and needs
|
||||
per-input-path calibration on Tier 2.
|
||||
|
||||
The live count is pacing-dependent, not a fixed property: a track needs two
|
||||
attempts before it settles, so a harness that pushes frames faster than the
|
||||
classifier drains them reports fewer labels. Re-run at four pacings on the same
|
||||
12-frame clip: 12, 8, 2 and 6 named at full speed, 14 named with 0.1 s between
|
||||
frames, and 38 named of 30 frames at 0.1 s. Precision stays 100% throughout and
|
||||
0 unknowns are accepted at every pacing, which is the property that matters;
|
||||
coverage is a function of how long the phone holds still, so it belongs to the
|
||||
device measurement (G4), not to this record.
|
||||
0 unknowns are accepted at every pacing. Coverage is a gate (G1), not a
|
||||
footnote: 12 of 61 streamed is a regression against Van's 57% until device data
|
||||
says otherwise, and the likely cause is measurable - OCR runs on every crop at
|
||||
every attempt, taking a crop from 5 ms to 25 ms and throughput from 789 to 60
|
||||
crops/s with 8 workers.
|
||||
|
||||
The shipped default scale now has test coverage. `make_session` builds only
|
||||
`matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue