vnguyen/product-recognition-demo
1
0
Fork 1
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(decision): refresh every number the corrected labels moved

Browse source 
Five blocks still printed pre-correction figures. Each was re-run through the
evaluator and replaced with what it actually outputs now:

- The lane comparison: local 75% at 91% against Van 64% at 38% (full), and
  59% at 93% against 66% at 39% (live). At live resolution Van now answers more
  facings correctly and is wrong on 67 of them to the local lane's 3 - stated
  plainly rather than left as a win on both axes.
- The accept sweep: 75/67/67/62% end-to-end at 91/91/93/95% precision, with the
  evaluator's own column definitions spelled out, since my first re-derivation
  used the wrong denominator.
- The G1 stream claim: four pan replays give 63-70% coverage at 76-84%
  precision, not 75% at 100%.
- The upscale paragraph: the guard is intact, but sku_l09.jpg is now a
  1000x1000 packshot, so the current run no longer exercises it.
- catalog_hash and the pyramid floor: the hash covers view bytes, not just
  names, and every view is now above the 800 px floor.

Also refreshes the server smoke against the renamed catalog: 53 named in 2.58 s,
no view-suffixed ids, no missing product info. l12 is not among them - it has
no enrolled view, which is the documented gap.
This commit is contained in:
kiyo9w 2026-09-29 03:18:30 +09:00
parent 306ea9b3e8
commit 8754738dc1
3 changed files with 50 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

1
recognization/data/evaluations/2026-09-29-session/server-smoke/recognize-after-l12-rename.json Normal file
View file

File diff suppressed because one or more lines are too long

5
recognization/data/evaluations/2026-09-29-session/server-smoke/summary.json
View file

@ -2,10 +2,12 @@
"endpoint": "/v1/recognize",
"image": "add/image6.jpg",
"catalog": "stock-lopia-clean",
"catalog_version": "d0f835fc7826",
"detections": 111,
"named": 53,
"named_without_product_info": 0,
"view_suffixed_ids": [],
"elapsed_s": 2.58,
"by_sku": {
"sku_l10": 9,
"sku_l11": 5,
@ -16,5 +18,6 @@
"sku_l02": 3,
"sku_l03": 5,
"sku_l05": 3
}
},
"note": "After the l12 rename. l12 is not named: it has no enrolled view, which is the documented catalog gap."
}

77
recognization/docs/research/2026-09-28-recognition-architecture/DECISION.md
View file

@ -95,7 +95,7 @@ verification → calibrated open-set decision → temporal accumulation.
| 8 | Bug #1, template IDs vs SKU IDs | R, and **still live in production** | Fixed only in the evaluator. The live path accepts `sku_l01_1`, then finds no catalog row for it (F-02). |
| 9 | Hybrid 80 % > matcher 70 % | X | About one facing, on wrong labels. |
| 10 | Bug #2, unscaled boxes caused the live collapse | R | Reproduced on *both* catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse. |
| 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12); the l09 packshot is only 305 px tall. |
| 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12). The l09 packshot was 305 px tall at audit time and has since been replaced by a 1000x1000 one, so that caveat no longer applies to the shipped catalog. |
| 12 | Bug #3, the reject gate inside the shortlist | R | Hybrid-49 %-vs-20 % result is invalid. |
| 13 | DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher | C | On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %. |
| 14 | Matcher ~8 ms vs hybrid ~45 ms | R, with a correction | The hybrid lane scores the **full** catalog and then filters (`eval_identity.py` `HybridLane.rank`), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design. |
@ -575,10 +575,10 @@ running the real server against `add/image6.jpg`.
| F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. |
| F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. |
| F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. |
| Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and the view names, not image bytes. |
| Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and each view's bytes, not just the view names: names alone let a different packshot be swapped in under the same name with the hash unchanged. |
| Enrollment protocol: sibling flag | Done. `rebuild_catalog` reports cross-SKU near-duplicates as text-discriminated siblings (cosine ≥ 0.98, same shape). Reported, not refused: they are legitimately different products, and only package text separates them. |
| Enrollment protocol: completeness rule | Done. `--shelf-tags` turns it into a `live_ready` verdict; without the shelf's SKU list the verdict is `unknown`, not a guess. |
| Enrollment protocol: pyramid floor | Done. `rebuild_catalog` reports views under 800 px tall. The clean catalog's l09 view is 305 px, which is the documented defect; the check names it so the re-sourcing has a target. |
| Enrollment protocol: pyramid floor | Done. `rebuild_catalog` reports views under 800 px tall. Every view in `stock-lopia-clean` is now 1000 px tall, so the check is clean on the shipped catalog; it still names any view that regresses. |
Two defects found while wiring the lanes into the live path, both fixed:
`live` re-sorted the local lane's scores by value, which discards the lane's own
@ -605,36 +605,40 @@ scores, always answered. Every "vs Van" claim in this record uses that row.
--labels eval/labels/image6.json \
--image add/image6.jpg --lanes van,matcher,local --live
full resolution — 109 scored facings (61 in-catalog, 48 unknown)
full resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc
van 69% 76% 76% 52% 29% 77
matcher 69% 76% 71% 49% 39% 46 HUB!
local 69% 93% 86% 59% 88% 5
van 83% 77% 77% 64% 38% 68
matcher 83% 77% 74% 61% 51% 38
local 83% 92% 81% 75% 91% 5
live resolution — 109 scored facings (61 in-catalog, 48 unknown)
live resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc
van 69% 83% 83% 57% 32% 74
matcher 69% 83% 76% 52% 43% 43
local 69% 86% 81% 56% 97% 1
van 83% 79% 79% 66% 39% 67
matcher 83% 79% 74% 61% 54% 33
local 83% 85% 60% 59% 93% 3
Against Van's real contract the local lane wins both axes at both resolutions:
59% at 88% against 52% at 29% (full), and 56% at 97% against 57% at 32% (live).
The live coverage margin is one facing against Van, so it is not a meaningful
gain there; the full-resolution margin is four facings at three times the
precision.
75% at 91% against 64% at 38% (full), and 59% at 93% against 66% at 39%
(live). At live resolution Van answers more facings correctly - 66% against
59% - and is wrong on 67 of them to the local lane's 3. That is the trade the
decision makes deliberately: coverage is a gate (G1) and precision is not
negotiable, so the lane abstains where Van guesses.
**One accept line serves both resolutions.** An earlier draft of this section
claimed full resolution needed a line near 6 to reach 95%, leaving 28/61
correct. That was measured before the reference-upscaling and text-signature
fixes and no longer holds. The separation is clean at both scales: correct
facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75 1.00 and
1.66, so 2.0 is the boundary in both. Sweeping it at full resolution:
1.66, so 2.0 is the boundary in both. Sweeping it at full resolution, through
the evaluator (`--accept`, columns as it defines them: end-to-end is
accepted-and-correct over the 64 in-catalog facings, precision is over the
facings it answered):
accept correct precision false accepts
2.0 59% 88% 5
3.0 52% 91% 3
4.0 49% 94% 2
6.0 43% 96% 1
accept end-to-end precision false accepts
2.0 75% 91% 5
3.0 67% 91% 4
4.0 67% 93% 3
6.0 62% 95% 2
The line is still provisional - it is tuned on Tier 1 and needs Tier-2
calibration - but it does not need to differ by input path, and the design
@ -685,15 +689,21 @@ Two live-path bugs surfaced while measuring this, both fixed:
whole pan-distance from the product it names. It now stamps the current
frame and anchors to the raw detection box rather than the smoothed track
box, which lags a pan by about one step.
- **The reference lane upscaled views smaller than a pyramid level.**
`sku_l09.jpg` is 407x305, so its 400 px level was a 305->400 blow-up: a blur
with no high-frequency detail to disagree with, which won 31 of 54 crops on
the shelf it was cut from. A view is now used only at the levels it has.
- **The reference lane upscaled views smaller than a pyramid level.** The
`sku_l09.jpg` enrolled at the time was 407x305, so its 400 px level was a
305->400 blow-up: a blur with no high-frequency detail to disagree with,
which won 31 of 54 crops on the shelf it was cut from. A view is now used
only at the levels it has. The guard is in `LocalLane._load` and still
applies; the file itself has since been replaced by a 1000x1000 packshot,
which is why the current run no longer exercises it.
Coverage is still a gate (G1). The stream now names 75% of the facings a
window can show at 100% precision, against Van's 50% at 45% on the same
stream, so the earlier "53 of 131 tracks" figure was a harness artifact rather
than a recognition limit.
Coverage is still a gate (G1). Four 20 s pan replays of `add/image6.jpg`
through the shipped lane name 63-70% of the in-catalog facings a window can
show, at 76-84% precision (`score_stream.py` over
`data/evaluations/2026-09-29-session/stream/fin_*.json`). Van's contract on the
same stream answers more and is wrong far more often, which is the trade the
decision makes. The earlier "53 of 131 tracks" figure was a harness artifact
rather than a recognition limit.
The shipped default scale now has test coverage. `make_session` builds only
`matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran
@ -768,9 +778,12 @@ were measured against and are shifted before the IoU test, so a pan no longer
slides a stale box onto its neighbour. The text lane is resolved before the
session lock is taken, removing a lock-ordering hazard against its build lock.
Stream, 20 s pan, four runs: 71-79% coverage at 76-84% precision, against
62-71% at 73-75% before. Still-image endpoint: 54 named detections in 2.2 s,
against 41 before.
Stream, 20 s pan, four runs: 63-70% coverage at 76-84% precision. The earlier
"71-79% at 76-84%" was measured before the tare facings were relabelled from
unknown to `sku_l12` and before the size gate landed; the denominator moved
from 61 in-catalog facings to 64, and the size gate trades a little coverage
for precision. Still-image endpoint: 53 named detections in 2.6 s, against 41
before.
### Catalog corrections 2026-09-29