Benchmark
Same logs, same rules. Scores in the open.
kshana-bench is a fixed set of tasks, splits and metrics for GNSS interference detection and timing holdover, so that different methods can be compared on the same data. A submission is a file; nothing runs on the submitter's behalf. Baselines are the engine's own monitors, with bars written down before the first scored run.
01The board
Result files, scored by the engine.
kshana-bench leaderboard
Result files scored under kshana-bench/2.0.0. A row says this file scored this number on this split. It does not mean any product has been rated or approved by anyone, and no organisation is named as an endorser.
Advisory: modelled or measured against stated references, not an approval, certification or statement of conformance. A score describes a log or corpus, never a product.
Task (a): detection on receiver logs
| System | AUC | P_d at P_fa cap (upper bound) | Provenance | Truth SHA-256 |
|---|---|---|---|---|
| kshana receiver-trust agc_k_sigma (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.981 | 0.875 | scored | 334946332bb0 |
| kshana receiver-trust cn0_drop_db (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.952 | 0.904 | scored | 334946332bb0 |
| kshana receiver-trust jam_ind_threshold (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.909 | 0.147 | scored | 334946332bb0 |
| System | AUC | P_d at P_fa cap (upper bound) | Provenance | Truth SHA-256 |
|---|---|---|---|---|
| kshana receiver-trust agc_k_sigma (0.35 default monitor statistic) kshana-bench/2.0.0 | 1.000 | 0.000 | scored | 38b7d5e8b158 |
| kshana receiver-trust jam_ind_threshold (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.610 | 0.148 | scored | 38b7d5e8b158 |
| kshana receiver-trust cn0_drop_db (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.564 | 0.562 | scored | 38b7d5e8b158 |
| System | AUC | P_d at P_fa cap (upper bound) | Provenance | Truth SHA-256 |
|---|---|---|---|---|
| kshana receiver-trust cn0_drop_db (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.864 | 0.522 | scored | ee721e0bf7dc |
| kshana receiver-trust agc_k_sigma (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.766 | 0.000 | scored | ee721e0bf7dc |
| kshana receiver-trust jam_ind_threshold (0.35 default monitor statistic) kshana-bench/2.0.0 | 0.556 | 0.104 | scored | ee721e0bf7dc |
Task (b): detection on IQ
| System | AUC | P_d at P_fa cap (upper bound) | Provenance | Truth SHA-256 |
|---|---|---|---|---|
| kshana iq monitors: cn0_drop (CUSUM) kshana-bench/2.0.0 | 0.785 | n/a | carried: docs/design/DETECTION-ROC.md section 3: the R1 campaign of tests/iq_campaign_roc.rs, run 2026-10-13; not re-run for bench v1. Bar R1 (AUC >= 0.95) was NOT MET. | 000000000000 |
Task (c): timing holdover prediction
| System | RMSE 3600 s (ns) | RMSE 300 s (ns) | RMSE 7200 s (ns) | Provenance | Truth SHA-256 |
|---|---|---|---|---|---|
| kshana-bench baseline: linear kshana-bench/2.0.0 | 0.128 | 0.019 | 0.198 | scored | 2b71fa325bff |
| kshana-bench baseline: quadratic kshana-bench/2.0.0 | 0.135 | 0.019 | 0.235 | scored | 2b71fa325bff |
| kshana-bench baseline: hold kshana-bench/2.0.0 | 26.876 | 2.238 | 53.778 | scored | 2b71fa325bff |
| System | RMSE 3600 s (ns) | RMSE 300 s (ns) | RMSE 7200 s (ns) | Provenance | Truth SHA-256 |
|---|---|---|---|---|---|
| kshana-bench baseline: linear kshana-bench/2.0.0 | 0.162 | 0.023 | 0.243 | scored | b2ed58864e98 |
| kshana-bench baseline: quadratic kshana-bench/2.0.0 | 0.183 | 0.024 | 0.290 | scored | b2ed58864e98 |
| kshana-bench baseline: hold kshana-bench/2.0.0 | 24.612 | 2.050 | 49.314 | scored | b2ed58864e98 |
Task (d): out-of-distribution optimism gap
| System | AUC in | AUC out | Optimism gap | Flag | Provenance | Truth SHA-256 |
|---|---|---|---|---|---|---|
| impairment-eval baseline: fused(max-z) kshana-bench/2.0.0 | 0.998 | 0.982 | 0.015 | within tolerance | scored | f69917c70201 |
| impairment-eval baseline: sqm-imbalance kshana-bench/2.0.0 | 0.867 | 0.866 | 0.001 | within tolerance | scored | f69917c70201 |
| impairment-eval baseline: agc-excess kshana-bench/2.0.0 | 0.854 | 0.828 | 0.026 | within tolerance | scored | f69917c70201 |
| impairment-eval baseline: raim-parity kshana-bench/2.0.0 | 0.655 | 0.642 | 0.013 | within tolerance | scored | f69917c70201 |
| impairment-eval baseline: energy(cn0-drop) kshana-bench/2.0.0 | 0.610 | 0.572 | 0.038 | within tolerance | scored | f69917c70201 |
| System | AUC in | AUC out | Optimism gap | Flag | Provenance | Truth SHA-256 |
|---|---|---|---|---|---|---|
| impairment-eval baseline: fused(max-z) kshana-bench/2.0.0 | 0.998 | 0.983 | 0.015 | within tolerance | scored | 1dddb8f6f0d5 |
| impairment-eval baseline: sqm-imbalance kshana-bench/2.0.0 | 0.882 | 0.882 | 0.001 | within tolerance | scored | 1dddb8f6f0d5 |
| impairment-eval baseline: agc-excess kshana-bench/2.0.0 | 0.848 | 0.819 | 0.030 | within tolerance | scored | 1dddb8f6f0d5 |
| impairment-eval baseline: raim-parity kshana-bench/2.0.0 | 0.682 | 0.672 | 0.010 | within tolerance | scored | 1dddb8f6f0d5 |
| impairment-eval baseline: energy(cn0-drop) kshana-bench/2.0.0 | 0.595 | 0.565 | 0.030 | within tolerance | scored | 1dddb8f6f0d5 |
Every score is modelled unless a row says an external oracle agreed. Licence of the specification and manifests: CC-BY-4.0 / CC0 proposed, owner decision pending. The real-data holdout is not secret: see the specification, section 4.
02What a row means
A score describes a log, not a product.
- A row says: this file scored X on split S under specification version V. It does not say a product is certified, conformant, approved or endorsed, and no organisation is named as an endorser.
- The data are recorded or synthetic and named in the specification. Third-party data is never copied into the repository: manifests and fetch scripts pin each file by SHA-256, and each licence is recorded before the data is used.
- The truth for a recorded log is the test plan's scheduled slots, not recorded on-air times, so a detection probability is a lower bound. The specification says so beside each task.
- A sealed holdout is scored only in a leaderboard round, and its seed is revealed after it. Bars for any baseline that claims a number are committed before the scored run and never loosened afterwards; a miss is recorded as NOT MET.
- Advisory only: Kshana results are modelled or measured against stated references, not an approval or a statement of conformance.
Run it yourself: kshana bench score turns a submission and a split's truth into a result file; kshana bench leaderboard renders result files into this page.