tngo/product-recognition-demo
1
0
Fork 0
forked from vnguyen/product-recognition-demo
Code Pull requests Activity

fix: bound and gate the crop evidence lane

Browse source 
The crop lane asked the phone for a full-resolution crop of a facing the
frame could not decide. Four defects made that unsafe or useless:

* A capability was half-applied. `crop: true` without a usable
  `max_crop_bytes` enabled the lane on the client's say-so, leaving the two
  sides believing different limits. A capability is now believed whole or
  refused with BAD_MESSAGE, and the limit must be an integer in 1..1048576
  (bool is an int in Python and is not a limit).
* A small JPEG could claim enormous dimensions. The header is now read
  without decoding and anything over 4 megapixels is refused before the
  decode allocates it.
* A request was single-use, so an expired one stranded a facing only the
  crop lane could decide. The lane now spends the same max_attempts budget
  the frame path spends, refilled on method change and on settled-label
  drift, and a phone that never answers still cannot loop.
* A candidate could be asked about from a box the frame path would refuse:
  clipped by the frame edge, carried over from an earlier frame, or measured
  at a score the tracker does not trust. Both lanes now share one edge gate,
  and only a track whose box this frame actually measured is asked about.

Drift and method change also cleared `identity` but left `confirmed = True`,
which permanently blocked both the crop lane and label inheritance for that
track; both now clear it, without which the budget reset was dead code.

Blur and glare are refused as whole-capture faults, while `too_far` is
exempt on purpose: a distant shelf is the one frame a native-resolution crop
exists to decide.
This commit is contained in:
kiyo9w 2026-09-28 17:22:52 +09:00
parent 02e69acac7
commit aa25f99699
5 changed files with 504 additions and 46 deletions
Download patch file Download diff file Expand all files Collapse all files

165
recognization/code/live.py
View file

@ -25,7 +25,7 @@ from concurrent.futures import Future, ThreadPoolExecutor
from dataclasses import dataclass, field
LOGGER = logging.getLogger("dorayaki.live")
from typing import Any, Dict, List, Optional, Sequence, Tuple
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
current_dir = os.path.dirname(os.path.abspath(__file__))
if current_dir not in sys.path:
@ -66,6 +66,11 @@ SOURCE_INHERITED = "inherited"
# A crop the client returns is base64 inside a text control message, so it is
# bounded on its own rather than riding the binary frame path.
MAX_CROP_BYTES = 1024 * 1024
# Frame faults a full-resolution crop cannot undo. Blur and glare are optical
# faults of the whole capture, so the phone would photograph the same bad
# scene. "too_far" is exempt on purpose: a distant shelf is the one frame a
# native-resolution crop exists to decide.
CROP_BLIND_QUALITY = ("blur", "glare")
@dataclass
@ -135,9 +140,11 @@ class TrackState:
# or pruned then re-created). Restarting resets the label, so it is always a
# re-verification rather than an unexplained label change.
restarts: int = 0
# A crop is asked for at most once per object. A recycled id gets a fresh
# TrackState, so the new object may be asked again.
crop_requested: bool = False
# How many full-resolution crops have been asked for on this object. It is
# the budget the frame path spends on attempts, so a phone that never
# answers cannot hold a small facing in the lane for ever. A recycled id
# gets a fresh TrackState, so a new object starts with a fresh budget.
crop_attempts: int = 0
# Which evidence produced the newest observation: frame or crop.
last_source: str = SOURCE_FRAME
@ -162,6 +169,21 @@ def _iou(a: Sequence[int], b: Sequence[int]) -> float:
return inter / union if union > 0 else 0.0
def _clear_of_edge(box: Sequence[int], frame_shape: Sequence[int], margin_ratio: float) -> bool:
"""True when a box sits clear of the frame edge.
A facing the frame clips is only part of a product, so neither lane may
spend a decision on it. Both lanes ask this question, so a candidate the
frame path would refuse cannot be asked about through the crop lane
instead.
"""
frame_h, frame_w = frame_shape[:2]
margin_x = int(frame_w * margin_ratio)
margin_y = int(frame_h * margin_ratio)
x1, y1, x2, y2 = box
return x1 > margin_x and y1 > margin_y and x2 < frame_w - margin_x and y2 < frame_h - margin_y
def _crop_off_product(requested: Sequence[int], current: Sequence[int]) -> bool:
"""True when a track no longer covers the box a crop was taken from.
@ -175,6 +197,21 @@ def _crop_off_product(requested: Sequence[int], current: Sequence[int]) -> bool:
return _iou(current, requested) < DRIFT_IOU
def _validate_crop_limit(value: Any) -> int:
"""The crop limit a client stated, or ValueError.
The limit is a memory bound, so a value the server cannot honour is refused
rather than clamped: quietly handing a client that asked for 512 B a 1 KiB
floor, or one that asked for 4 GiB a 1 MiB ceiling, leaves the two sides
believing different limits. bool is an int in Python and is not a limit.
"""
if isinstance(value, bool) or not isinstance(value, int):
raise ValueError("max_crop_bytes must be an integer")
if value <= 0 or value > MAX_CROP_BYTES:
raise ValueError("max_crop_bytes must be 1..%d" % MAX_CROP_BYTES)
return value
class _CropRejected(Exception):
"""A returned crop that may not be used, carrying the reply to send back."""
@ -376,6 +413,10 @@ class LiveSession:
self._generation += 1
for track in self._tracks.values():
track.state = STATE_DETECTED
# Nothing has decided this track under the new method yet,
# so it is unconfirmed again rather than a settled track
# that happens to carry no label.
track.confirmed = False
track.identity = None
track.candidates = []
track.score_sums = {}
@ -383,6 +424,10 @@ class LiveSession:
track.attempts = 0
track.in_flight = 0
track.last_attempt_seq = 0
# A crop asked about under the old method is evidence measured
# under settings the session no longer holds, so the new method
# gets a budget of its own.
track.crop_attempts = 0
track.last_attempt_area = 0
# Outstanding crops were minted against the old method and the
# old generation; answering one now would be evidence measured
@ -452,12 +497,18 @@ class LiveSession:
self._process_times.append(tracked_at)
by_id = {box["id"]: box for box in boxes}
# Ids whose box this frame actually measured. A track the tracker
# is carrying without a detection keeps its last measured box, and
# that box must not be handed to the phone as if it described the
# frame the request points at.
measured: Set[int] = set()
active: Dict[int, TrackState] = {}
for track in tracks:
state = self._tracks.get(track.track_id)
raw_box = None
confidence = track.score
if track.det_index is not None and track.det_index in by_id:
measured_now = track.det_index is not None and track.det_index in by_id
if measured_now:
detection = by_id[track.det_index]
raw_box = list(detection["box"])
confidence = float(detection["confidence"])
@ -513,6 +564,10 @@ class LiveSession:
if raw_box and not low_score:
state.raw_box = raw_box
state.confidence = confidence
if measured_now:
# A detection in this frame's list, at a score the tracker
# trusts: the box describes the frame the request names.
measured.add(track.track_id)
elif not state.raw_box and raw_box:
state.raw_box = raw_box
state.confidence = confidence
@ -556,7 +611,7 @@ class LiveSession:
# releasing it. add_done_callback can run the callback inline when the
# future already finished, and the callback takes this same lock.
submissions = self._plan_classification(image, seq)
crop_requests = self._plan_crop_requests(seq, int(frame_id))
crop_requests = self._plan_crop_requests(image.shape, seq, int(frame_id), measured)
summary = self._summary_locked()
stats = self._stats_locked()
@ -669,9 +724,19 @@ class LiveSession:
if track.drift_frames < 2:
return
track.drift_frames = 0
# A crop asked about the product this box has now left says nothing
# about the neighbour in its place, so the answer goes with the label
# and the new product is asked about on a budget of its own.
for request_id in [rid for rid, rec in self._crop_requests.items()
if rec["track_id"] == track.track_id]:
self._crops_rejected += 1
self._release_crop_slot(self._crop_requests.pop(request_id))
self._relabel_count[track.track_id] = self._relabel_count.get(track.track_id, 0) + 1
track.relabels = self._relabel_count[track.track_id]
track.state = STATE_DETECTED
# The label went with the product, so the track is undecided again:
# a settled track that carries no label can never be asked about.
track.confirmed = False
track.identity = None
track.candidates = []
track.score_sums = {}
@ -681,6 +746,7 @@ class LiveSession:
track.last_attempt_seq = 0
track.last_attempt_area = 0
track.label_box = []
track.crop_attempts = 0
# ------------------------------------------------------------------ classification
def _plan_classification(self, image: Any, seq: int) -> List[Tuple[TrackState, Any, int]]:
@ -690,7 +756,6 @@ class LiveSession:
after the lock is released.
"""
frame_h, frame_w = image.shape[:2]
margin_x = int(frame_w * self.settings.edge_margin_ratio)
margin_y = int(frame_h * self.settings.edge_margin_ratio)
inflight = sum(track.in_flight for track in self._tracks.values())
capacity = max(0, self.settings.max_inflight_per_session - inflight)
@ -711,7 +776,7 @@ class LiveSession:
x1, y1, x2, y2 = track.raw_box
if min(x2 - x1, y2 - y1) < self.settings.min_crop_px:
continue
if x1 <= margin_x or y1 <= margin_y or x2 >= frame_w - margin_x or y2 >= frame_h - margin_y:
if not _clear_of_edge(track.raw_box, image.shape, self.settings.edge_margin_ratio):
continue
if track.attempts >= self.settings.max_attempts:
continue
@ -775,24 +840,27 @@ class LiveSession:
def enable_crops(self, max_crop_bytes: Optional[int] = None) -> Dict[str, Any]:
"""Turn the lane on for a client that advertised it. The client states the
largest crop it will produce; the server keeps the lower of the two."""
largest crop it will produce; a limit that is not one is refused and the
lane stays where it was, so a bad statement enables nothing."""
stated = None if max_crop_bytes is None else _validate_crop_limit(max_crop_bytes)
with self._lock:
self.settings.crops = True
if max_crop_bytes is not None:
try:
stated = int(max_crop_bytes)
except (TypeError, ValueError):
stated = 0
self.settings.max_crop_bytes = max(1024, min(stated, MAX_CROP_BYTES))
if stated is not None:
self.settings.max_crop_bytes = stated
return self.capabilities()
def _plan_crop_requests(self, seq: int, frame_id: int) -> Dict[int, Dict[str, Any]]:
def _plan_crop_requests(
self, frame_shape: Tuple[int, int], seq: int, frame_id: int, measured: Set[int]
) -> Dict[int, Dict[str, Any]]:
"""Ask the phone for a fuller crop of tracks the frame crop could not use.
The box is in the pixel space of `frame_id`, the frame the client just
sent, so the phone can answer from the buffer behind that exact frame
rather than from whatever is on screen now. One request per track object,
at most `max_crop_inflight` outstanding, and only for a client that
rather than from whatever is on screen now. That is also why the box
has to be one this frame measured and has to sit clear of the edge: the
phone crops by these numbers, so a carried-over or clipped box buys a
photograph of the wrong thing. At most `max_attempts` requests per
object, `max_crop_inflight` outstanding, and only for a client that
advertised the lane.
"""
self._expire_crop_requests(seq)
@ -801,16 +869,27 @@ class LiveSession:
room = self.settings.max_crop_inflight - len(self._crop_requests)
if room <= 0:
return {}
if (self._last_quality or {}).get("hint") in CROP_BLIND_QUALITY:
# A blurred or glared frame photographs the same bad scene at the
# camera's own resolution, so there is nothing to ask the phone for.
return {}
planned: Dict[int, Dict[str, Any]] = {}
for track in self._tracks.values():
if len(planned) >= room:
break
if track.crop_requested or track.in_flight > 0:
if track.crop_attempts >= self.settings.max_attempts or track.in_flight > 0:
continue
if track.state != STATE_DETECTED or track.confirmed:
continue
if track.hits < self.settings.min_hits or len(track.raw_box) != 4:
continue
if track.track_id not in measured:
# The box is a carry-over: either no detection matched this
# frame, or the one that did was too low a score to move it.
# Either way it points at where the product used to be.
continue
if not _clear_of_edge(track.raw_box, frame_shape, self.settings.edge_margin_ratio):
continue
x1, y1, x2, y2 = track.raw_box
# The frame crop is too small to say anything about this product. At
# the camera's own resolution it may still be worth a decision, and
@ -832,7 +911,7 @@ class LiveSession:
"seq": seq,
"method": self.settings.method,
}
track.crop_requested = True
track.crop_attempts += 1
# Hold the track while the phone answers, so the frame path does not
# spend a second attempt on the same box at the same time.
track.in_flight += 1
@ -899,14 +978,22 @@ class LiveSession:
generation = record["generation"]
method = record["method"]
object_generation = record["object_generation"]
try:
future = self._executor.submit(self.engine.score_crop, crop, method)
except RuntimeError: # executor already shut down
with self._lock:
with self._lock:
# Decoding is the slow part and the session keeps running through it,
# so the binding checked before the decode is re-checked here. The
# check and the submit share one critical section: an ack has to mean
# this crop was measured against the track object it answers.
stale = self._crop_stale_reason_locked(record)
if stale is not None:
self._release_crop_slot(record)
self._crops_rejected += 1
return _crop_error("CROP_UNAVAILABLE", "classifier is shutting down")
with self._lock:
return _crop_error("CROP_STALE", stale)
try:
future = self._executor.submit(self.engine.score_crop, crop, method)
except RuntimeError: # executor already shut down
self._release_crop_slot(record)
self._crops_rejected += 1
return _crop_error("CROP_UNAVAILABLE", "classifier is shutting down")
self._crops_accepted += 1
future.add_done_callback(
lambda fut, tid=track_id_out, gen=generation, m=method, started=submitted_at,
@ -975,6 +1062,26 @@ class LiveSession:
# the track is by the time the answer lands.
return record, list(record["box"]), payload
def _crop_stale_reason_locked(self, record: Dict[str, Any]) -> Optional[str]:
"""Why a decoded crop can no longer speak for its track, or None.
Run after the decode, under the lock that submits the classification.
A reset or a method change, a recycled track id, a pan that carried the
track off the photographed product, or another attempt that already
settled the track can all land while the JPEG is being decoded, and an
ack is a promise that this crop decided that track. Lock must be held.
"""
if record["generation"] != self._generation:
return "the session changed while the crop was decoded"
track = self._tracks.get(record["track_id"])
if track is None or track.object_generation != record["object_generation"]:
return "the track id was recycled while the crop was decoded"
if _crop_off_product(record["box"], track.raw_box):
return "the track moved off the product while the crop was decoded"
if track.state != STATE_DETECTED or track.confirmed:
return "the track is no longer waiting to be decided"
return None
def _on_classified(
self,
future: Future,
@ -1000,6 +1107,12 @@ class LiveSession:
track.in_flight = max(0, track.in_flight - 1)
return
track.in_flight = max(0, track.in_flight - 1)
if track.state != STATE_DETECTED or track.confirmed:
# Other evidence settled this track while the answer sat in the
# classifier - a label inherited from a labelled neighbour, say.
# A late answer must not repaint a label the client is already
# showing, so the track keeps the one it settled on.
return
if source == SOURCE_CROP and _crop_off_product(submitted_box, track.raw_box):
# The answer was measured on the box the track has since left.
# Dropping it leaves the track to be decided from a later frame