Skip to content

Benchmark

Fixed splits, one command, one leaderboard. Numbers come from roofsight eval and nowhere else.

Metrics

Metric What it measures Why
Mask AP, AP50 COCO segm AP per category and overall the standard number
Small-obstacle recall recall @ mask IoU 0.5 of obstacle instances < 32² px, any score the number that matters for PV: a missed vent is a module you cannot place
Boundary F F-score of the roof-plane boundary, 1 px tolerance at 640 px, scaled with image size plane edges become module rows; AP does not see a 3 px error
Latency median ms at 640 px: PyTorch CUDA, ONNX Runtime CPU, Core ML on iPhone it has to run on site
Pitch / azimuth MAE degrees, own-photo subset only, against ARKit-derived ground truth what the installer actually uses

Small-obstacle recall does not consume predictions: two small ground-truth instances covered by one prediction both count. The question is "would the planner have seen it", not AP.

Running

uv run roofsight train --config configs/train/rfdetr-s.yaml          # → runs/<id>/
uv run roofsight predict --run runs/<id> --split test                # → runs/<id>/predictions-test.json
uv run roofsight eval --run runs/<id> --split test                   # → runs/<id>-eval/metrics.json
uv run roofsight leaderboard                                          # → docs/leaderboard.md, paper/results.tex

Every run directory has a run.json with git sha, config hash, dataset version, seed and hardware. The leaderboard lists a run only if run.json and metrics.json are both present.

iPhone latency comes from the RoofGeometryBench app target, which writes a latency.json in the same format as roofsight.eval.latency; copy it into the run directory before eval.

Baselines (paper)

  • SAM 3 zero-shot with the labeling prompts: what you get for free, and an upper bound on the auto labels
  • YOLO26-seg n/s: reported, not shipped (AGPL)
  • Mask2Former-Swin-T: accuracy reference

Submitting

Open a pull request with the run directory minus weights (run.json, metrics.json, latency.json, the config). The leaderboard is regenerated by CI; do not edit it by hand.