docs(decision): refresh every number the corrected labels moved

Five blocks still printed pre-correction figures. Each was re-run through the
evaluator and replaced with what it actually outputs now:

- The lane comparison: local 75% at 91% against Van 64% at 38% (full), and
  59% at 93% against 66% at 39% (live). At live resolution Van now answers more
  facings correctly and is wrong on 67 of them to the local lane's 3 - stated
  plainly rather than left as a win on both axes.
- The accept sweep: 75/67/67/62% end-to-end at 91/91/93/95% precision, with the
  evaluator's own column definitions spelled out, since my first re-derivation
  used the wrong denominator.
- The G1 stream claim: four pan replays give 63-70% coverage at 76-84%
  precision, not 75% at 100%.
- The upscale paragraph: the guard is intact, but sku_l09.jpg is now a
  1000x1000 packshot, so the current run no longer exercises it.
- catalog_hash and the pyramid floor: the hash covers view bytes, not just
  names, and every view is now above the 800 px floor.

Also refreshes the server smoke against the renamed catalog: 53 named in 2.58 s,
no view-suffixed ids, no missing product info. l12 is not among them - it has
no enrolled view, which is the documented gap.
This commit is contained in:
kiyo9w 2026-09-29 03:18:30 +09:00
parent 306ea9b3e8
commit 8754738dc1
3 changed files with 50 additions and 33 deletions

File diff suppressed because one or more lines are too long

View file

@ -2,10 +2,12 @@
"endpoint": "/v1/recognize", "endpoint": "/v1/recognize",
"image": "add/image6.jpg", "image": "add/image6.jpg",
"catalog": "stock-lopia-clean", "catalog": "stock-lopia-clean",
"catalog_version": "d0f835fc7826",
"detections": 111, "detections": 111,
"named": 53, "named": 53,
"named_without_product_info": 0, "named_without_product_info": 0,
"view_suffixed_ids": [], "view_suffixed_ids": [],
"elapsed_s": 2.58,
"by_sku": { "by_sku": {
"sku_l10": 9, "sku_l10": 9,
"sku_l11": 5, "sku_l11": 5,
@ -16,5 +18,6 @@
"sku_l02": 3, "sku_l02": 3,
"sku_l03": 5, "sku_l03": 5,
"sku_l05": 3 "sku_l05": 3
} },
"note": "After the l12 rename. l12 is not named: it has no enrolled view, which is the documented catalog gap."
} }

View file

@ -95,7 +95,7 @@ verification → calibrated open-set decision → temporal accumulation.
| 8 | Bug #1, template IDs vs SKU IDs | R, and **still live in production** | Fixed only in the evaluator. The live path accepts `sku_l01_1`, then finds no catalog row for it (F-02). | | 8 | Bug #1, template IDs vs SKU IDs | R, and **still live in production** | Fixed only in the evaluator. The live path accepts `sku_l01_1`, then finds no catalog row for it (F-02). |
| 9 | Hybrid 80 % > matcher 70 % | X | About one facing, on wrong labels. | | 9 | Hybrid 80 % > matcher 70 % | X | About one facing, on wrong labels. |
| 10 | Bug #2, unscaled boxes caused the live collapse | R | Reproduced on *both* catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse. | | 10 | Bug #2, unscaled boxes caused the live collapse | R | Reproduced on *both* catalogs: unscaled boxes give 11–17 %, scaled boxes give 76–97 %. The confound is resolved. The coordinate fix, not the catalog cleanup, removed the collapse. |
| 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12); the l09 packshot is only 305 px tall. | | 11 | Clean packshots are a dramatic improvement | R, with a caveat | True against the leak-free baseline. The l06 packshot shows a superseded package design (F-12). The l09 packshot was 305 px tall at audit time and has since been replaced by a 1000x1000 one, so that caveat no longer applies to the shipped catalog. |
| 12 | Bug #3, the reject gate inside the shortlist | R | Hybrid-49 %-vs-20 % result is invalid. | | 12 | Bug #3, the reject gate inside the shortlist | R | Hybrid-49 %-vs-20 % result is invalid. |
| 13 | DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher | C | On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %. | | 13 | DINO top-K sweep: recall 71 %/100 %, K doesn't hurt the matcher | C | On a 7-SKU catalog, top-5 is 71 % of the catalog; that sweep can't speak to retrieval. Reproduced on v2: top-3 recall 79–81 %, top-5 95–98 %. |
| 14 | Matcher ~8 ms vs hybrid ~45 ms | R, with a correction | The hybrid lane scores the **full** catalog and then filters (`eval_identity.py` `HybridLane.rank`), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design. | | 14 | Matcher ~8 ms vs hybrid ~45 ms | R, with a correction | The hybrid lane scores the **full** catalog and then filters (`eval_identity.py` `HybridLane.rank`), so its latency is DINO + full matcher by construction. The measurement is honest; it doesn't describe a real retrieval design. |
@ -575,10 +575,10 @@ running the real server against `add/image6.jpg`.
| F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. | | F-12 l06 current package | **Blocked on a capture.** Kewpie serves the superseded design for JAN 4901577083600, so the current package cannot be enrolled from the official URL. The text lane ranks l06 first on all 4 shelf facings but abstains below the accept line, so no wrong label is shown. |
| F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. | | F-13 stale status / swallowed exception | Done. The fixture returns the matcher's `(scores, details)` shape; zero swallowed callback exceptions. |
| F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. | | F-09 re-evaluation with provisional labels | **Postponed as §8 specifies** — it is a dependency of E-7, which needs Tier-3 recordings that do not exist yet. |
| Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and the view names, not image bytes. | | Enrollment protocol: versioning | Done. Every eval report records `catalog_hash`, `labels_hash`, `image_hash`, `code_revision`. The catalog hash covers the CSV bytes and each view's bytes, not just the view names: names alone let a different packshot be swapped in under the same name with the hash unchanged. |
| Enrollment protocol: sibling flag | Done. `rebuild_catalog` reports cross-SKU near-duplicates as text-discriminated siblings (cosine ≥ 0.98, same shape). Reported, not refused: they are legitimately different products, and only package text separates them. | | Enrollment protocol: sibling flag | Done. `rebuild_catalog` reports cross-SKU near-duplicates as text-discriminated siblings (cosine ≥ 0.98, same shape). Reported, not refused: they are legitimately different products, and only package text separates them. |
| Enrollment protocol: completeness rule | Done. `--shelf-tags` turns it into a `live_ready` verdict; without the shelf's SKU list the verdict is `unknown`, not a guess. | | Enrollment protocol: completeness rule | Done. `--shelf-tags` turns it into a `live_ready` verdict; without the shelf's SKU list the verdict is `unknown`, not a guess. |
| Enrollment protocol: pyramid floor | Done. `rebuild_catalog` reports views under 800 px tall. The clean catalog's l09 view is 305 px, which is the documented defect; the check names it so the re-sourcing has a target. | | Enrollment protocol: pyramid floor | Done. `rebuild_catalog` reports views under 800 px tall. Every view in `stock-lopia-clean` is now 1000 px tall, so the check is clean on the shipped catalog; it still names any view that regresses. |
Two defects found while wiring the lanes into the live path, both fixed: Two defects found while wiring the lanes into the live path, both fixed:
`live` re-sorted the local lane's scores by value, which discards the lane's own `live` re-sorted the local lane's scores by value, which discards the lane's own
@ -605,36 +605,40 @@ scores, always answered. Every "vs Van" claim in this record uses that row.
--labels eval/labels/image6.json \ --labels eval/labels/image6.json \
--image add/image6.jpg --lanes van,matcher,local --live --image add/image6.jpg --lanes van,matcher,local --live
full resolution — 109 scored facings (61 in-catalog, 48 unknown) full resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc lane catalog identity decision end-to-end precision false acc
van 69% 76% 76% 52% 29% 77 van 83% 77% 77% 64% 38% 68
matcher 69% 76% 71% 49% 39% 46 HUB! matcher 83% 77% 74% 61% 51% 38
local 69% 93% 86% 59% 88% 5 local 83% 92% 81% 75% 91% 5
live resolution — 109 scored facings (61 in-catalog, 48 unknown) live resolution — 109 scored facings (64 in-catalog, 45 unknown)
lane catalog identity decision end-to-end precision false acc lane catalog identity decision end-to-end precision false acc
van 69% 83% 83% 57% 32% 74 van 83% 79% 79% 66% 39% 67
matcher 69% 83% 76% 52% 43% 43 matcher 83% 79% 74% 61% 54% 33
local 69% 86% 81% 56% 97% 1 local 83% 85% 60% 59% 93% 3
Against Van's real contract the local lane wins both axes at both resolutions: Against Van's real contract the local lane wins both axes at both resolutions:
59% at 88% against 52% at 29% (full), and 56% at 97% against 57% at 32% (live). 75% at 91% against 64% at 38% (full), and 59% at 93% against 66% at 39%
The live coverage margin is one facing against Van, so it is not a meaningful (live). At live resolution Van answers more facings correctly - 66% against
gain there; the full-resolution margin is four facings at three times the 59% - and is wrong on 67 of them to the local lane's 3. That is the trade the
precision. decision makes deliberately: coverage is a gate (G1) and precision is not
negotiable, so the lane abstains where Van guesses.
**One accept line serves both resolutions.** An earlier draft of this section **One accept line serves both resolutions.** An earlier draft of this section
claimed full resolution needed a line near 6 to reach 95%, leaving 28/61 claimed full resolution needed a line near 6 to reach 95%, leaving 28/61
correct. That was measured before the reference-upscaling and text-signature correct. That was measured before the reference-upscaling and text-signature
fixes and no longer holds. The separation is clean at both scales: correct fixes and no longer holds. The separation is clean at both scales: correct
facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75 1.00 and facings sit at p25 2.07 (full) and 2.87 (live), wrong ones at p75 1.00 and
1.66, so 2.0 is the boundary in both. Sweeping it at full resolution: 1.66, so 2.0 is the boundary in both. Sweeping it at full resolution, through
the evaluator (`--accept`, columns as it defines them: end-to-end is
accepted-and-correct over the 64 in-catalog facings, precision is over the
facings it answered):
accept correct precision false accepts accept end-to-end precision false accepts
2.0 59% 88% 5 2.0 75% 91% 5
3.0 52% 91% 3 3.0 67% 91% 4
4.0 49% 94% 2 4.0 67% 93% 3
6.0 43% 96% 1 6.0 62% 95% 2
The line is still provisional - it is tuned on Tier 1 and needs Tier-2 The line is still provisional - it is tuned on Tier 1 and needs Tier-2
calibration - but it does not need to differ by input path, and the design calibration - but it does not need to differ by input path, and the design
@ -685,15 +689,21 @@ Two live-path bugs surfaced while measuring this, both fixed:
whole pan-distance from the product it names. It now stamps the current whole pan-distance from the product it names. It now stamps the current
frame and anchors to the raw detection box rather than the smoothed track frame and anchors to the raw detection box rather than the smoothed track
box, which lags a pan by about one step. box, which lags a pan by about one step.
- **The reference lane upscaled views smaller than a pyramid level.** - **The reference lane upscaled views smaller than a pyramid level.** The
`sku_l09.jpg` is 407x305, so its 400 px level was a 305->400 blow-up: a blur `sku_l09.jpg` enrolled at the time was 407x305, so its 400 px level was a
with no high-frequency detail to disagree with, which won 31 of 54 crops on 305->400 blow-up: a blur with no high-frequency detail to disagree with,
the shelf it was cut from. A view is now used only at the levels it has. which won 31 of 54 crops on the shelf it was cut from. A view is now used
only at the levels it has. The guard is in `LocalLane._load` and still
applies; the file itself has since been replaced by a 1000x1000 packshot,
which is why the current run no longer exercises it.
Coverage is still a gate (G1). The stream now names 75% of the facings a Coverage is still a gate (G1). Four 20 s pan replays of `add/image6.jpg`
window can show at 100% precision, against Van's 50% at 45% on the same through the shipped lane name 63-70% of the in-catalog facings a window can
stream, so the earlier "53 of 131 tracks" figure was a harness artifact rather show, at 76-84% precision (`score_stream.py` over
than a recognition limit. `data/evaluations/2026-09-29-session/stream/fin_*.json`). Van's contract on the
same stream answers more and is wrong far more often, which is the trade the
decision makes. The earlier "53 of 131 tracks" figure was a harness artifact
rather than a recognition limit.
The shipped default scale now has test coverage. `make_session` builds only The shipped default scale now has test coverage. `make_session` builds only
`matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran `matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran
@ -768,9 +778,12 @@ were measured against and are shifted before the IoU test, so a pan no longer
slides a stale box onto its neighbour. The text lane is resolved before the slides a stale box onto its neighbour. The text lane is resolved before the
session lock is taken, removing a lock-ordering hazard against its build lock. session lock is taken, removing a lock-ordering hazard against its build lock.
Stream, 20 s pan, four runs: 71-79% coverage at 76-84% precision, against Stream, 20 s pan, four runs: 63-70% coverage at 76-84% precision. The earlier
62-71% at 73-75% before. Still-image endpoint: 54 named detections in 2.2 s, "71-79% at 76-84%" was measured before the tare facings were relabelled from
against 41 before. unknown to `sku_l12` and before the size gate landed; the denominator moved
from 61 in-catalog facings to 64, and the size gate trades a little coverage
for precision. Still-image endpoint: 53 named detections in 2.6 s, against 41
before.
### Catalog corrections 2026-09-29 ### Catalog corrections 2026-09-29