RoomSighterModel comparison
Loading benchmark data…
Two detection models, RT-DETR-L and YOLO26-l, run together and a small learned scorer merges and re-scores their detections. Every number below is measured on the same 8,694 held-out test images, which were never used for training or for fitting the scorer.
Share of real objects of the navigation-critical classes covering at least 10% of the frame (close to you) for which a detection is announced. The layer lowers the bar for these objects and announces them as “possible”. On average: 0.68 without it, 0.83 at the 80% setting, 0.91 at 90%, at the price of about 0.6 (80%) or 2.0 (90%) extra, mostly wrong, “possible” detections per image.
| Model | Browser (WebGPU) | On-device (CoreML) |
|---|---|---|
| YOLO26s | 109 ms | 76 ms |
| RT-DETR-L | 319 ms | 100 ms |
| YOLO26-l | — | 20 ms |
| Fusion (this app) | 547 ms | 123 ms |
Median per frame, measured on a MacBook Air, Apple M1 (one 640 px frame; not an iPhone). Two networks run on every frame, so the fusion is slower than a single model; phones differ.
Measured offline on a held-out research test set. Real-world accuracy, especially with phone photos in poor light, can be lower.
The charts below come from an earlier training run of one model (exp-5) and are kept for reference. They are not the model in this app.
| Model | Split | Precision | Recall | F1 | mAP@0.5 | mAP@0.5:0.95 | FPS | Inference (ms) | Device |
|---|---|---|---|---|---|---|---|---|---|
| YOLO26n (exp-12) -- NYU domain-transfer | test | 0.388 | 0.428 | 0.407 | 0.388 | 0.280 | 22.0 | 45.5 | cpu (onnxruntime) |
| YOLO26n (exp-4) -- superseded by exp-5 | val | 0.512 | 0.399 | 0.449 | 0.398 | 0.278 | — | 0.2 | NVIDIA RTX PRO 6000 Blackwell Server Edition (training-time benchmark, not representative of on-device/browser inference) |
| YOLO26n (exp-5) -- official validation | val | 0.714 | 0.579 | 0.639 | 0.642 | 0.461 | — | 0.3 | NVIDIA RTX PRO 6000 Blackwell Server Edition (training-time benchmark, not representative of on-device/browser inference) |
YOLO26n (exp-12) -- NYU domain-transfer: exp-12: fine-tuned from a COCO-pretrained YOLO26n checkpoint (not NYU-only from-scratch, superseding exp-8, see script docstring). Evaluated here against the NYU test split for a domain-transfer measurement -- exp-12 was not trained on NYU Depth V2, so this is a generalization number, not a same-distribution one.
YOLO26n (exp-4) -- superseded by exp-5: The real, official validation from exp-4's own training run (100 epochs, Ultralytics 8.4.164) -- an IN-DISTRIBUTION measurement against its own val split. exp-4 replaces exp-12 (docs/11_decision_log.md, 2026-09-27/28 entries): scope expanded from 14 classes to the full 59-class insighter-indoor-balanced dataset (door rebuilt from clean indoor photos after the original source turned out to be mislabeled street/locker photos; stairs kept; window added as a genuinely new class). The lower overall mAP50 vs. exp-12's 14-class number (0.398 vs 0.626) reflects scope breadth diluting the instance-weighted average with harder, lower-support long-tail classes (book, spoon, knife, backpack, etc.) -- not a regression on any class exp-12 already covered; door in particular improved dramatically (mAP50 0.703 -> 0.932) after the source-data fix. Per-class real-world confidence thresholds (frontend/src/inference/detector.ts, ios/InSighter/Detection/DetectorService.swift) come from a SEPARATE, more rigorous held-out test-split sweep (ml/scripts/eval_test_thresholds.py, 24,790 test images never used in training or checkpoint selection), not from this val table.
YOLO26n (exp-5) -- official validation: The real, official validation from exp-5's own training run (100 epochs, Ultralytics 8.4.171, Platform-hosted RTX PRO 6000 Blackwell) -- an IN-DISTRIBUTION measurement against its own val split. exp-5 supersedes exp-4 as this project's deployed detection model (same 59-class scope and class order, dataset rebuilt as insighter-indoor-balanced-v3 vs exp-4's v2) and is a real, substantial improvement, not a rerun: mAP50 0.398 -> 0.642, mAP50-95 0.278 -> 0.461, precision 0.512 -> 0.714, recall 0.399 -> 0.579. No early stopping -- trained the full 100 epochs, best.pt == epoch 100. val_box_loss/val_cls_loss/val_l1_loss/lr are only known for epoch 100 (the one per-epoch row available from the Platform's own metrics export); log.txt's console output only carries train box/cls/l1 loss and val precision/recall/mAP per epoch, so those three curves show a single point. confusion_matrix and class_distribution are deliberately omitted here (not carried over from exp-4) -- they haven't been recomputed for exp-5 yet and reusing exp-4's would misrepresent a different model's actual behavior. Per-class confidence thresholds (frontend/src/inference/detector.ts, ios/InSighter/Detection/DetectorService.swift) are being re-measured against exp-5 via a separate held-out test-split sweep (ml/scripts/eval_test_thresholds.py), not carried over from exp-4's sweep.
Sorted by this app's accuracy. The last column is the change against RT-DETR-L alone. A few classes have few test examples, so single numbers are noisy.
The guarantee is approximate: it holds on average, but door, table and stairs fall short of the target on this test set (calibrated on different images).