vnguyen/product-recognition-demo
1
0
Fork 1
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(decision): measure G1 against Van on the same stream and catalog

Browse source 
The G1 row compared against a van run whose server had loaded the legacy
stock-lopia catalog, so its ids were template-level (sku_005) and every answer
scored wrong. Re-run with both servers on stock-lopia-clean, 20 s pan 12:

  van    17/27 (63%) coverage   53% precision   7 wrong   6 unenrolled accepts
  local  19-20/27 (70-74%)      77-85%          0 wrong   0 unenrolled accepts

G1 asks the new system to be no worse than Van on coverage; it is better on
both axes, so G1 now reads as passing on the stream rather than unmeasured.
Both recordings are committed under stream-gated/.
This commit is contained in:
kiyo9w 2026-09-29 04:28:20 +09:00
parent 01d9a5e614
commit 81a2ab10e7
3 changed files with 2913 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

25
recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
View file

@ -396,12 +396,14 @@ with always-argmax, run on that same catalog.
| G8 runtime | server identity latency p95 per crop; server memory | ≤ 60 ms; within the demo host's budget |
Status 2026-09-29, offline on image6 plus the Wikimedia distractor sets:
**G2 passes** (95 % precision, full resolution). **G3's far-unknown half
passes** (0.30 % on 2650 unrelated crops); its near-unknown half is
measured on image6 only, where the size gate rejects the documented 450 g case.
**G8 passes** (identity p95 well inside 60 ms). **G1, G4, G5, G6, G7 are not
yet measured on held-out data** - Tier 2 does not exist, and G4/G5 need a
recorded walk. Nothing here is a device-path result.
**G1 passes on the stream** (70-74 % coverage against Van's 63 % on the same
stream and catalog, 0 wrong against Van's 7). **G2 passes** (95 % precision
full, 97 % live). **G3's far-unknown half passes** (0.30 % on 2650 unrelated
crops); its near-unknown half is measured on image6 only, where the size gate
rejects the documented 450 g case. **G8 passes** (identity p95 well inside
60 ms). **G4, G5, G6, G7 are not yet measured on held-out data** - Tier 2 does
not exist, and G4/G5 need a recorded walk. Nothing here is a device-path
result.
Not complete until G1–G7 pass on the device path (Tier 3), not just offline.
@ -709,6 +711,17 @@ backend against the `van` lane on the same stream:
12 van 11/24 (46%) 28%
12 local 16/24 (67%) 83%
Re-measured 2026-09-29 with both servers on `stock-lopia-clean`, 20 s pan 12,
`stream-gated/van_tracks.json` against `stream-gated/tracks_*.json`:
lane facings named precision wrong unenrolled false accepts
van 17/27 (63%) 53% 7 6
local 19-20/27 (70-74%) 77-85% 0 0
Van names fewer facings than the local lane and is wrong on 7 of the 53 it
does name, plus 6 accepts of products the catalog does not hold. G1 asks the
new system to be no worse than Van on coverage; it is better on both axes.
Two live-path bugs surfaced while measuring this, both fixed:
- **Inherited labels carried frame id 0.** `_inherit_labels` set `label_box`