fix: cut whole quantity tokens, and gate the fill-in by default
Some checks failed
tests / model-free (pull_request) Has been cancelled

Review 2, item 2. The first version of this gate counted bigrams with a
non-digit character, which let the unit's own letters through: "400g 50%"
left "0G" and "0%", "450g カロリー50%" left "0%", and "500ml" left
"0M". All three still named products.

The quantity token is now cut whole - digits plus unit, in either script -
before counting, and the floor is three bigrams rather than two, because
"カロリ" is two and it is a descriptor every low-calorie package prints.

DecisionPolicy.decide gated the fill-in only when a read was passed, so the
unsafe behaviour was the default. Omitting the read now means no fill-in: a
caller with a ranking and no read has not shown the text can name anything.
TextLane.carries_identity was a dead duplicate of the module function; deleted.

Tests: the five cases from the review plus the default-gate case. Suite 284 ->
286. Evaluator unchanged: local 61%/77% full, 59%/95% live.
This commit is contained in:
tngo 2026-09-28 23:46:03 +09:00
parent 80e69e74df
commit ff11529476
2 changed files with 59 additions and 32 deletions

View file

@ -427,7 +427,9 @@ class DecisionPolicyTests(unittest.TestCase):
row - the two jobs compose, so the result is the text's SKU, never the
vetoed local one.
"""
decision = self.policy().decide([("sku_a", 5.0)], [("sku_b", 0.9)])
decision = self.policy().decide(
[("sku_a", 5.0)], [("sku_b", 0.9)], text_read="コクうま 65%"
)
self.assertEqual(decision["best_sku"], "sku_b")
self.assertEqual(decision["source"], "text")
@ -470,10 +472,38 @@ class DecisionPolicyTests(unittest.TestCase):
self.assertEqual(decision["best_sku"], "sku_b")
def test_text_fills_in_when_local_has_nothing(self):
decision = self.policy().decide([], [("sku_b", 0.9)])
decision = self.policy().decide([], [("sku_b", 0.9)], text_read="コクうま 65%")
self.assertEqual(decision["best_sku"], "sku_b")
self.assertEqual(decision["source"], "text")
def test_a_ranking_without_a_read_does_not_fill_in(self):
"""Omitting the read is the safe default, not an ungated one.
A caller that has a ranking and no read has not shown the text can name
anything, so the fill-in job stays closed.
"""
decision = self.policy().decide([], [("sku_b", 0.9)])
self.assertIsNone(decision["best_sku"])
def test_quantity_strings_are_not_names(self):
"""Every pure quantity string, including the ones a digits-only rule let through.
"400g 50%" and "450g カロリー50%" survived the first version of this
gate: it counted non-digit bigrams, and the unit's own letters ("0G",
"0%") are non-digits. Cutting the whole quantity token is what closes
it. "450g カロリー50%" still passes, and correctly so - "カロリー" is
four characters of real text, and it is the text lane's job to find
that no catalog row is named by it.
"""
from identity_lanes import carries_identity
for read in ("400g", "400g 50%", "500ml", "65%", "3日", "中辛"):
with self.subTest(read=read):
self.assertFalse(carries_identity(read), f"{read!r} is not a name")
for read in ("コクうま 65%", "450g カロリー50%"):
with self.subTest(read=read):
self.assertTrue(carries_identity(read), f"{read!r} carries name text")
def test_text_below_its_own_line_does_not_fill_in(self):
decision = self.policy().decide([], [("sku_b", 0.3)])
self.assertIsNone(decision["best_sku"])