Metrics, Thresholds, and Robustness
Understand what a score measures, how a criterion becomes a decision, and how to read a robustness comparison.
A useful result answers three questions: what was measured, on which samples, and against which criterion. Read these together before deciding whether a model is suitable for your intended use.
Performance measures the selected task
For object detection, a prediction must identify both a class and a box. A correct class with a poorly placed box may still fail to match the annotation.
| Term | Meaning | What to check |
|---|---|---|
| Precision | Of the detections predicted as positive, how many are correct | False positives and the confidence threshold |
| Recall | Of the annotated objects, how many are found | Missed objects, including low-confidence predictions that were filtered out |
| Intersection over Union (IoU) | Box overlap area divided by their union area | Whether predicted and annotated boxes align |
| Average Precision (AP) | A summary of the precision–recall curve | The class, matching rule, and evaluated samples |
| Detection mAP 50-95 | Mean AP across classes and IoU matching thresholds from 0.50 to 0.95 in steps of 0.05 | Higher is better; inspect sample coverage before comparing runs |
The Duplicate-box overlap setting used during inference controls duplicate suppression. It is separate from the IoU matching thresholds used to score AP.
A threshold turns a measurement into a decision
Choose criteria before reviewing results, based on the intended use of the model. A completed run can still fail its criterion. A metric without a criterion can still provide a useful measurement.
For Robustness, Maximum allowed degradation (%) is an upper limit: the comparison passes when degradation is less than or equal to the limit. Leaving it blank measures the change without a Pass/Fail criterion.
For example, a 20% limit accepts a 12% degradation and rejects a 25% degradation. It is a relative change limit, not a 20-point drop in the metric. See Status and Decisions for N/A and aggregate decision rules.
Robustness compares clean and transformed inputs
The baseline is the score on the original inputs. The transformed score is measured after applying the chosen transform, such as Fog. Compare the same sample population and metric in both branches.
For a nonzero baseline, Knotest calculates:
| Metric direction | Relative degradation (%) |
|---|---|
| Higher is better | (baseline − transformed) / abs(baseline) × 100 |
| Lower is better | (transformed − baseline) / abs(baseline) × 100 |
Positive degradation means the score became worse. Negative degradation means it improved in this comparison. A negative value is therefore not an error by itself.
If both values are zero, degradation is zero. If the baseline is zero and the transformed value is nonzero, a relative percentage is unavailable. Missing transformed measurements or mismatched sample counts also make the comparison unavailable; do not interpret them as a pass.
Read the Helmet example carefully
The tutorial shows a baseline of 0.024, Fog severity 3 of 0.008, and recorded degradation of 66.2226%. The displayed scores are rounded, so calculating from those displayed values may not reproduce the exact percentage. The comparison exceeds its 20% limit: Criteria not met.
100 random Validation samples were used (selection seed 42). This demonstrates the workflow; it does not establish deployment performance or establish behavior under all weather conditions. Use representative held-out data and examine additional severities and failure cases before drawing broader conclusions.
Continue with Evaluate Robustness for the screen recording, or Evaluation Settings to look up individual controls.