feat: add Van's real baseline lane and stop size tokens naming products

Two review findings, both confirmed by measurement.

1. The evaluator had no always-argmax lane, so every "vs Van" claim compared
   against the *gated* matcher, which is not Van's contract - the original demo
   answered every crop with its best-scoring SKU and had no accept line. The new
   'van' lane is that contract: same scores, no gate. Against it the local lane
   wins both axes at both resolutions (full 61%/77% vs 52%/29%; live 59%/95% vs
   57%/32%), but the live coverage margin is one facing, not seven, and the 95%
   figure is live-resolution only - at full resolution the same lane reads 77%
   with 11 false accepts and a hub alarm.

2. A crop reading only "400g" scored 0.667 on three SKUs, above TEXT_ACCEPT, so
   the text lane named a product from a size line. The fill-in job now requires
   two non-digit bigrams; the veto is unchanged, so "65%" still contradicts a
   local answer. The raw OCR read travels with the ranking through
   score_crop_lanes for that check.

Tests: 3 for the size gate (policy + live) and the van lane's own row. Removing
the gate fails all three. Suite 276 -> 280.
This commit is contained in:
tngo 2026-09-28 23:12:44 +09:00
parent 13ee7426e2
commit fd72251160
7 changed files with 180 additions and 22 deletions

View file

@ -263,6 +263,31 @@ def bigrams(text: str) -> set:
return {text[i:i + 2] for i in range(len(text) - 1)}
def identity_bigrams(text: str) -> set:
"""Bigrams carrying a non-digit character.
A crop reading only "400g" matches every 400g product's size line, so size
and number tokens must not name a product on their own. They still count
for the veto, where the local lane has already supplied the evidence and
the text only has to contradict it.
"""
return {gram for gram in bigrams(text) if not gram.isdigit()}
def carries_identity(read: str) -> bool:
"""Whether a read holds enough non-numeric text to name a product.
A crop reading only "400g" matches every 400g product's size line, so size
and number tokens must not answer on their own. Two non-digit bigrams is
the floor: "400g" has one ("0g") and is refused, while "コクうま 65%" has
five and still names l09. The veto is unaffected - there the local lane has
already supplied the evidence and the text only has to contradict it, so
"65%" stays usable.
"""
return len(identity_bigrams(normalize_text(read))) >= 2
def text_similarity(read: str, signature: str) -> float:
"""Share of the read text's bigrams that the signature explains.
@ -352,6 +377,18 @@ class TextLane:
"""Bigram similarity per catalog row. Empty when nothing was read."""
return self.scores_from_text(self.read(crop))
def carries_identity(self, read: str) -> bool:
"""Whether a read holds enough non-numeric text to name a product.
A crop reading only "400g" matches every 400g product's size line, so
size and number tokens must not answer on their own. Two non-digit
bigrams is the floor: "400g" has one ("0g") and is refused, while
"コクうま 65%" has five and still names l09. The veto is unaffected -
there the local lane has already supplied the evidence and the text
only has to contradict it, so "65%" stays usable.
"""
return len(identity_bigrams(normalize_text(read))) >= 2
def scores_from_text(self, read: str) -> Dict[str, float]:
if len(normalize_text(read)) < 4:
return {}
@ -412,8 +449,14 @@ class DecisionPolicy:
text_rank: Any = None,
*,
details: Optional[Dict[str, Any]] = None,
text_read: Optional[str] = None,
) -> Dict[str, Any]:
"""One decision, in the shape `decide_identity` already returns."""
"""One decision, in the shape `decide_identity` already returns.
`text_read` is the raw OCR text behind `text_rank`. It gates the
fill-in job only: a read that is nothing but a size or a number cannot
name a product, but it can still veto one.
"""
from identity import (
IDENTITY_ACCEPTED,
IDENTITY_AMBIGUOUS,
@ -430,6 +473,7 @@ class DecisionPolicy:
if text_top
and text_top[1] >= self.text_accept
and (text_top[1] - text_runner) >= self.text_margin
and (text_read is None or carries_identity(text_read))
else None
)