feat: add Van's real baseline lane and stop size tokens naming products
Some checks failed
tests / model-free (pull_request) Has been cancelled

Two review findings, both confirmed by measurement.

1. The evaluator had no always-argmax lane, so every "vs Van" claim compared
   against the *gated* matcher, which is not Van's contract - the original demo
   answered every crop with its best-scoring SKU and had no accept line. The new
   'van' lane is that contract: same scores, no gate. Against it the local lane
   wins both axes at both resolutions (full 61%/77% vs 52%/29%; live 59%/95% vs
   57%/32%), but the live coverage margin is one facing, not seven, and the 95%
   figure is live-resolution only - at full resolution the same lane reads 77%
   with 11 false accepts and a hub alarm.

2. A crop reading only "400g" scored 0.667 on three SKUs, above TEXT_ACCEPT, so
   the text lane named a product from a size line. The fill-in job now requires
   two non-digit bigrams; the veto is unchanged, so "65%" still contradicts a
   local answer. The raw OCR read travels with the ranking through
   score_crop_lanes for that check.

Tests: 3 for the size gate (policy + live) and the van lane's own row. Removing
the gate fails all three. Suite 276 -> 280.
This commit is contained in:
tngo 2026-09-28 23:12:44 +09:00
parent 13ee7426e2
commit fd72251160
7 changed files with 180 additions and 22 deletions

View file

@ -514,17 +514,18 @@ class RecognitionEngine:
return decide_identity({}, method="local", scale="local")
text_lane = self.identity_text_lane()
text_rank: Any = None
read = ""
if text_lane is not None:
read = text_lane.read(crop)
text_rank = sorted(
text_lane.scores_from_text(read).items(), key=lambda kv: kv[1], reverse=True
)
return DecisionPolicy().decide(lane.rank(crop), text_rank)
return DecisionPolicy().decide(lane.rank(crop), text_rank, text_read=read)
def score_crop_lanes(
self, crop: Any, method: Optional[str] = None
) -> Tuple[Dict[str, float], List[Tuple[str, float]]]:
"""Both lanes' evidence for one crop: local scores and text ranking.
) -> Tuple[Dict[str, float], List[Tuple[str, float]], str]:
"""Both lanes' evidence for one crop: local scores, text ranking, raw read.
The live path accumulates evidence across looks, so it needs the lanes
separately rather than one fused decision. How it combines them depends
@ -532,16 +533,18 @@ class RecognitionEngine:
margin over the runner-up, and averaging in an empty observation halves
it below the accept line), while the cosine and inlier scales average
the scores over the track's window. The text reading is always the
newest. The still path calls `decide_crop`, which fuses a single crop.
newest. The raw read travels with the ranking because the decision needs
it to tell a product name from a size line. The still path calls
`decide_crop`, which fuses a single crop.
"""
scores = self.score_crop(crop, method)
text_lane = self.identity_text_lane()
if text_lane is None:
return scores, []
return scores, [], ""
read = text_lane.read(crop)
return scores, sorted(
text_lane.scores_from_text(read).items(), key=lambda kv: kv[1], reverse=True
)
), read
def identity_lane(self) -> Any:
"""The local lane, built once. None when the backend is not `local`."""

View file

@ -59,7 +59,7 @@ UNKNOWN = "unknown"
# Two independent label passes disagreed here, so the row is not ground truth.
# Scoring it would be scoring a coin flip.
EXCLUDE = "exclude"
LANES = ("local", "matcher", "gallery", "hybrid-verify")
LANES = ("local", "matcher", "van", "gallery", "hybrid-verify")
# A single SKU above this share of accepts is the failure the last audit found:
# every dark package collapsing onto one enrolled product.
HUB_SHARE = 0.30
@ -307,6 +307,40 @@ class MatcherLane(Lane):
return sorted(group_by_sku(scores).items(), key=lambda kv: kv[1], reverse=True)
class VanLane(MatcherLane):
"""Van's contract: the matcher's top SKU, always answered.
The `matcher` lane applies `decide_identity`, which is not what the original
demo did - it answered every crop with its best-scoring SKU and had no
accept line at all. Comparing the new system against the gated matcher
understates the baseline's coverage and overstates its precision, so the
completion gate's "Van" row is this lane: same scores, no gate.
"""
name = "van"
def decide(self, crop: Any, ranked: List[Tuple[str, float]], *, accept: float,
margin: float, scale: str) -> Dict[str, Any]:
from identity import IDENTITY_ACCEPTED
if not ranked:
return {
"status": IDENTITY_ACCEPTED, "method": self.name, "best_sku": None,
"score": 0.0, "confidence": 0.0, "identity_level": IDENTITY_ACCEPTED,
"ranked_candidates": [], "details": {}, "message": "no candidate",
"source": None,
}
best, score = ranked[0]
return {
"status": IDENTITY_ACCEPTED, "method": self.name, "best_sku": best,
"score": round(float(score), 2), "confidence": round(float(score), 4),
"identity_level": IDENTITY_ACCEPTED,
"ranked_candidates": [{"sku_id": sku, "score": round(float(v), 2)}
for sku, v in ranked[:5]],
"details": {}, "message": "Always-argmax baseline", "source": "van",
}
class LocalLaneAdapter(Lane):
"""The shipped default: RootSIFT pyramid plus the OCR text lane.
@ -333,11 +367,12 @@ class LocalLaneAdapter(Lane):
margin: float, scale: str) -> Dict[str, Any]:
from identity_lanes import DecisionPolicy
read = self.text_lane.read(crop)
text_rank = sorted(
self.text_lane.scores_from_text(self.text_lane.read(crop)).items(),
self.text_lane.scores_from_text(read).items(),
key=lambda kv: kv[1], reverse=True,
)
return DecisionPolicy(local_accept=accept).decide(ranked, text_rank)
return DecisionPolicy(local_accept=accept).decide(ranked, text_rank, text_read=read)
def _catalog_rows(stock_dir: str) -> List[Dict[str, Any]]:
@ -571,8 +606,10 @@ def main() -> int:
unenrolled = Counter(f["label"] for f in held if f["label"] != UNKNOWN and f["label"] not in enrolled_skus)
lanes: Dict[str, Lane] = {}
if "matcher" in wanted or "hybrid-verify" in wanted:
if "matcher" in wanted or "hybrid-verify" in wanted or "van" in wanted:
lanes["matcher"] = MatcherLane(args.stock_dir)
if "van" in wanted:
lanes["van"] = VanLane(args.stock_dir)
if "local" in wanted:
lanes["local"] = LocalLaneAdapter(args.stock_dir)
if "gallery" in wanted or "hybrid-verify" in wanted:

View file

@ -263,6 +263,31 @@ def bigrams(text: str) -> set:
return {text[i:i + 2] for i in range(len(text) - 1)}
def identity_bigrams(text: str) -> set:
"""Bigrams carrying a non-digit character.
A crop reading only "400g" matches every 400g product's size line, so size
and number tokens must not name a product on their own. They still count
for the veto, where the local lane has already supplied the evidence and
the text only has to contradict it.
"""
return {gram for gram in bigrams(text) if not gram.isdigit()}
def carries_identity(read: str) -> bool:
"""Whether a read holds enough non-numeric text to name a product.
A crop reading only "400g" matches every 400g product's size line, so size
and number tokens must not answer on their own. Two non-digit bigrams is
the floor: "400g" has one ("0g") and is refused, while "コクうま 65%" has
five and still names l09. The veto is unaffected - there the local lane has
already supplied the evidence and the text only has to contradict it, so
"65%" stays usable.
"""
return len(identity_bigrams(normalize_text(read))) >= 2
def text_similarity(read: str, signature: str) -> float:
"""Share of the read text's bigrams that the signature explains.
@ -352,6 +377,18 @@ class TextLane:
"""Bigram similarity per catalog row. Empty when nothing was read."""
return self.scores_from_text(self.read(crop))
def carries_identity(self, read: str) -> bool:
"""Whether a read holds enough non-numeric text to name a product.
A crop reading only "400g" matches every 400g product's size line, so
size and number tokens must not answer on their own. Two non-digit
bigrams is the floor: "400g" has one ("0g") and is refused, while
"コクうま 65%" has five and still names l09. The veto is unaffected -
there the local lane has already supplied the evidence and the text
only has to contradict it, so "65%" stays usable.
"""
return len(identity_bigrams(normalize_text(read))) >= 2
def scores_from_text(self, read: str) -> Dict[str, float]:
if len(normalize_text(read)) < 4:
return {}
@ -412,8 +449,14 @@ class DecisionPolicy:
text_rank: Any = None,
*,
details: Optional[Dict[str, Any]] = None,
text_read: Optional[str] = None,
) -> Dict[str, Any]:
"""One decision, in the shape `decide_identity` already returns."""
"""One decision, in the shape `decide_identity` already returns.
`text_read` is the raw OCR text behind `text_rank`. It gates the
fill-in job only: a read that is nothing but a size or a number cannot
name a product, but it can still veto one.
"""
from identity import (
IDENTITY_ACCEPTED,
IDENTITY_AMBIGUOUS,
@ -430,6 +473,7 @@ class DecisionPolicy:
if text_top
and text_top[1] >= self.text_accept
and (text_top[1] - text_runner) >= self.text_margin
and (text_read is None or carries_identity(text_read))
else None
)

View file

@ -1094,9 +1094,9 @@ class LiveSession:
source: str = SOURCE_FRAME,
) -> None:
try:
scores, text_rank = future.result()
scores, text_rank, text_read = future.result()
except Exception:
scores, text_rank = {}, []
scores, text_rank, text_read = {}, [], ""
finished_at = time.perf_counter()
with self._lock:
track = self._tracks.get(track_id)
@ -1173,7 +1173,9 @@ class LiveSession:
# chose. The newest crop is the evidence, not the mean: averaging
# in an empty observation halves a good confidence below the
# accept line, which is the dilution the cosine branch guards too.
decision = DecisionPolicy().decide(list(scores.items()), text_rank)
decision = DecisionPolicy().decide(
list(scores.items()), text_rank, text_read=text_read
)
else:
decision = decide_identity(mean_scores, method=method, scale=scale)
track.candidates = [

View file

@ -453,6 +453,22 @@ class DecisionPolicyTests(unittest.TestCase):
decision = self.policy().decide([("sku_a", 3.0), ("sku_b", 5.0)], [])
self.assertEqual(decision["best_sku"], "sku_a")
def test_a_size_only_read_does_not_fill_in(self):
"""Size and number tokens are not identity text."""
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="400g")
self.assertIsNone(decision["best_sku"])
def test_a_size_only_read_still_vetoes(self):
"""The veto keeps "65%" usable: the local lane supplied the evidence."""
decision = self.policy().decide(
[("sku_a", 5.0)], [("sku_b", 0.9)], text_read="65%"
)
self.assertIsNone(decision["best_sku"])
def test_a_named_read_with_a_size_still_fills_in(self):
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="コクうま 65%")
self.assertEqual(decision["best_sku"], "sku_b")
def test_text_fills_in_when_local_has_nothing(self):
decision = self.policy().decide([], [("sku_b", 0.9)])
self.assertEqual(decision["best_sku"], "sku_b")

View file

@ -115,7 +115,7 @@ class LocalScaleEngine(RecognitionEngine):
def score_crop_lanes(self, crop, method=None):
if self.lane_answers:
return self.lane_answers.pop(0)
return {}, []
return {}, [], ""
class FakeInventory:
@ -331,7 +331,7 @@ class LocalScaleTests(unittest.TestCase):
dropping that path silently removed the lane's fill-in job.
"""
session, engine, _, _ = make_session(engine_cls=LocalScaleEngine)
engine.lane_answers = [({}, [("sku_a", 0.9)])]
engine.lane_answers = [({}, [("sku_a", 0.9)], "コクうま 65%")]
result = process(session, 1)
self.assertTrue(
wait_for(lambda: any(
@ -355,7 +355,7 @@ class LocalScaleTests(unittest.TestCase):
wrong reason.
"""
session, engine, _, _ = make_session(engine_cls=LocalScaleEngine)
engine.lane_answers = [({"sku_a": 5.0}, [("other", 0.3)])] * 4
engine.lane_answers = [({"sku_a": 5.0}, [("other", 0.3)], "other マヨネーズ")] * 4
for frame_id in range(1, 14):
result = process(session, frame_id)
for track in result["tracks"]:
@ -365,6 +365,22 @@ class LocalScaleTests(unittest.TestCase):
)
engine.shutdown()
def test_a_size_only_read_cannot_name_a_product(self):
"""A crop reading "400g" matches every 400g product's size line.
Size and number tokens are not identity text: without this gate the
text lane answered a distractor crop reading only "400g" as a product.
"""
session, engine, _, _ = make_session(engine_cls=LocalScaleEngine)
engine.lane_answers = [({}, [("sku_a", 0.9)], "400g")] * 4
for frame_id in range(1, 14):
for track in process(session, frame_id)["tracks"]:
self.assertIsNone(
track["identity"]["sku_id"],
"a size-only read must not name a product",
)
engine.shutdown()
class TagPassIntegrationTests(unittest.TestCase):
"""The tag pass must reach the live result without blocking frames."""

View file

@ -589,17 +589,57 @@ own" job unreachable. Live now decides per crop on the lane's own order.
Measured after the change, through the real server: `/v1/recognize` names 41 of
112 detections at 90% precision with no view-suffixed ids; the live WebSocket
names 12 at 100% precision with 0 unknown accepts, first label on frame 3.
Offline on the corrected labels: 36/61 (59%) at 95% precision, 0/48 unknowns,
11 of those labels sourced from the text lane alone.
Those are detector-box figures, not label-matched ones: the 41 are detections
whose box overlaps a labelled facing, so they measure the endpoint rather than
the identity lane. The label-matched number is the evaluator's, below.
## The baseline is Van's contract, not the gated matcher
The `matcher` lane applies `decide_identity`, which is not what the original
demo did - it answered every crop with its best-scoring SKU and had no accept
line. Comparing against the gated matcher understates the baseline's coverage
and overstates its precision, so the evaluator now has a `van` lane: the same
scores, always answered. Every "vs Van" claim in this record uses that row.
.venv/bin/python code/eval_identity.py --stock stock-lopia-clean \
--labels docs/research/2026-09-28-recognition-architecture/evidence/image6_v2.json \
--image add/image6.jpg --lanes van,matcher,local --live
full resolution — 109 scored facings (61 in-catalog, 48 unknown)
lane catalog identity decision end-to-end precision false acc
van 69% 76% 76% 52% 29% 77
matcher 69% 76% 71% 49% 39% 46 HUB!
local 69% 95% 88% 61% 77% 11 HUB!
live resolution — 109 scored facings (61 in-catalog, 48 unknown)
lane catalog identity decision end-to-end precision false acc
van 69% 83% 83% 57% 32% 74
matcher 69% 83% 76% 52% 43% 43
local 69% 86% 86% 59% 95% 2
Against Van's real contract the local lane wins both axes at both resolutions:
61% at 77% against 52% at 29% (full), and 59% at 95% against 57% at 32% (live).
The live coverage margin is one facing (36 against 35), so it is not a
meaningful gain there; the full-resolution margin is five facings.
**The 95% figure is live-resolution only.** At full resolution the same lane
reads 77% precision with 11 false accepts and raises the hub alarm, and the
still endpoint and the native crop lane both run at that scale. Reaching 95% at
full resolution needs an accept line near 6, which leaves 28/61 correct - below
Van's 32/61. So the design does not yet beat Van on both axes at full
resolution, and the accept line is provisional: it is tuned on Tier 1 and needs
per-input-path calibration on Tier 2.
The live count is pacing-dependent, not a fixed property: a track needs two
attempts before it settles, so a harness that pushes frames faster than the
classifier drains them reports fewer labels. Re-run at four pacings on the same
12-frame clip: 12, 8, 2 and 6 named at full speed, 14 named with 0.1 s between
frames, and 38 named of 30 frames at 0.1 s. Precision stays 100% throughout and
0 unknowns are accepted at every pacing, which is the property that matters;
coverage is a function of how long the phone holds still, so it belongs to the
device measurement (G4), not to this record.
0 unknowns are accepted at every pacing. Coverage is a gate (G1), not a
footnote: 12 of 61 streamed is a regression against Van's 57% until device data
says otherwise, and the likely cause is measurable - OCR runs on every crop at
every attempt, taking a crop from 5 ms to 25 ms and throughput from 789 to 60
crops/s with 8 workers.
The shipped default scale now has test coverage. `make_session` builds only
`matcher`-backend engines, so `scale == "local"` in `_on_classified` never ran