Kshana Connect: mapping files
A mapping file (kshana.connect/1, TOML) tells Kshana how to read an export it has no reader
for, and turns it into a kshana.obs/1 observation file. No code is written:
the file names the columns, their units and the time format.
Step by step, for a monitoring station: guides/MONITORING-STATION.md. Step by step, for scoring an interference classifier on the benchmark: guides/BENCHMARK-CLASSES.md.
kshana connect map export.csv --mapping export.connect.toml --station station.toml --out site.obs.jsonl
kshana obs check site.obs.jsonl
Three worked examples are in examples/connect/: an Android raw
measurement CSV (one row per satellite, satellites from a constellation code and a number), a
timing receiver's semicolon-separated monitoring export (one column per satellite, GPS week and
time of week, height in feet) and a monitoring service's nested JSON (position in radians).
tests/connect_examples.rs maps all three.
The file#
schema = "kshana.connect/1"
[source]
format = "csv" # csv, tsv, json or jsonl
delimiter = ";" # csv only; one character, default ","
skip_lines = 0 # lines before the header (csv, tsv)
comment = "#" # lines starting with this are ignored (csv, tsv)
records = "/data/epochs" # json only: JSON pointer to the array of records
filter = { column = "type", equals = ["Raw"] } # keep only these rows
[time]
column = "tow"
format = "gps-week-tow" # rfc3339, unix-s, unix-ms, gps-s, gps-week-tow or pattern
week_column = "week" # gps-week-tow: the full GPS week
pattern = "%d/%m/%Y %H:%M:%S" # pattern: %Y %m %d %H %M %S (seconds may have a fraction)
scale = "utc" # rfc3339, unix-* and pattern: utc (default), gps or tai
group_tolerance_s = 0.0005 # rows this close in time are one epoch
[fix]
lat = { column = "lat", unit = "deg" } # deg or rad
lon = { column = "lon", unit = "deg" }
height = { column = "alt", unit = "ft" } # m, ft or km
n_used = { column = "nsat" }
[signals]
# One row per satellite ...
sat = { column = "prn" } # G05, G5 or "G 5"; or number_column with system / system_column + system_map
cn0 = { column = "cn0", unit = "dB-Hz" }
band_column = "signal" # or band = "L1" (the default)
# ... or one column per satellite:
wide = { G05 = "cn0_g05", E11 = "cn0_e11" }
[receiver]
agc = { column = "agc" } # native units
jam_ind = { column = "jamming_indicator" }
spoofing = { column = "spoof" } # receiver's own flag: 0 unknown, 1 none, 2 indicated, 3 affirmed
jamming = { column = "jam" }
pdop = { column = "pdop" }
hdop = { column = "hdop" }
vdop = { column = "vdop" }
Every field takes scale and offset (applied after the unit: value × scale + offset), and
in JSON a column starting with / is a JSON pointer into nested objects.
Rules#
- Rows with the same time (within
group_tolerance_s) are one epoch, so long and wide exports both work, and position rows interleaved with satellite rows join their epoch. - An empty cell (or a JSON
null) is a missing value; anything else that is not a number is an error naming the line and the column. Nothing is guessed. - Times must not go backwards; a row that does is an error naming it, never a silent reorder.
- GPS and TAI times are converted to UTC with the leap-second table; the observation file's
time_labelis UTC. - A CSV header without a column the mapping names is a mapping error.
- The output header's
source_formatisconnect:<format>.
Conformance vectors#
conformance/connect/ holds one folder per adapter with a vector.toml (input, mapping file or
built-in reader, station, expected output). kshana connect check conformance/connect runs them
all and exits 1 on any byte difference, naming the first differing line; --bless records the
expected outputs after a reviewed change. CI runs the folder (tests/connect_conformance.rs).
Every new adapter comes with its vector.
Not in this version: Parquet and HDF5 inputs (they need large dependencies; a conversion to CSV or JSON Lines is the route for now) and live inputs (A3a).
Live: watching a directory (library)#
kshana::connect_watch::step follows the files a logger writes or rotates into a directory and returns
the kshana.obs/1 records that became complete since the last step, through the same mapping file.
- The files whose names end with the chosen suffix are read in name order, as one continuous log: a
later file's comment lines and header row are dropped. So an epoch whose rows a rotation split is
still one epoch, and every
t_scounts from the directory's first epoch. - Only complete lines are read; a line still being written waits for the next step.
- The last epoch is held back until rows after it arrive, because more rows of it may still be written.
flushreleases it at the end of a run. - Files must be in time order by name. A single JSON document is refused (use
jsonl). - It polls, with no operating-system file events, so it behaves the same on every platform and on network file systems. The directory is re-read on each step, so move processed files away from time to time.
- Checked by replaying a log written in pieces and rotated part-way through an epoch: the result is the
same as
kshana connect mapon the whole log (tests/connect_watch.rs). - From the command line:
kshana connect watch <dir> --mapping <connect.toml> --station <station.toml>prints new records as JSON lines every--intervalseconds (default 2).--suffixpicks the files (default: the mapping's format, for example.csv).--oncereads what is there, releases the last epoch and exits.
Live MQTT subscription (kshana::connect_mqtt)#
kshana::connect_mqtt subscribes to a topic on an MQTT broker and maps the rows that arrive, with the
same mapping file, into kshana.obs/1 records.
- MQTT 3.1.1 over TCP: clean session, optional user name and password, one topic filter (
+and#work), QoS 0 or 1. A QoS 1 message is acknowledged. A ping is sent when the line has been quiet for half the keep-alive. - Each message is one or more rows of the mapping's format (
jsonl,csvortsv). The messages are read in arrival order as one continuous log, asconnect watchreads a directory, and the last epoch is held back until a later row arrives. Publishers send rows without a header, so a CSV or TSV mapping needs the header row given (--header). - Rows already emitted are dropped as soon as their epoch is emitted, and
t_sstill counts from the first epoch of the run. Each message therefore costs the same however long the subscription has run, and memory stays bounded. - TLS is not built in, and the default build adds no crate for MQTT. For a broker that only accepts TLS, run a local TLS relay (for example stunnel, or a local broker bridged to it) and subscribe to that.
- Checked against a loopback broker that publishes a log one row per message, alternating QoS 0 and 1:
the result is the same as
kshana connect mapon the whole log, with the trimming path exercised (tests/connect_mqtt.rs). - From the command line:
kshana connect mqtt <host:port> --topic <filter> --mapping <connect.toml> --station <station.toml>prints records as JSON lines as they complete.--userwith--password-env <VAR>reads the password from an environment variable, never from the command line.--max-messages <n>ends the run after n messages and releases the last epoch. - Many stations on one broker: subscribe with a wildcard (
--topic 'site/+/raw') and add--per-topic. Each topic then gets its own feed, and its records carry the station id<station.id>/<topic>, with one header per station before its first epoch.
Live NATS input (kshana::connect_nats)#
kshana connect nats <host:port> --subject <s> --mapping <connect.toml> --station <station.toml>
subscribes to a NATS subject and maps the rows that arrive, through the same feeds as MQTT.
- NATS client protocol over TCP:
CONNECT(with--userand--password-env <VAR>, or--token-env <VAR>; secrets come from environment variables only),SUB, thenMSGandHMSG(headers are skipped). A serverPINGis answered withPONG, and a-ERRis reported. A lost connection is made again, as for MQTT. --per-topicgives each subject its own feed (station id<station.id>/<subject>), for a wildcard subject such asgnss.*.raw.- TLS is not built in: for a server that requires it, use a local TLS relay.
- Checked against a loopback server that publishes a log one row per message on two subjects, with a
header-carrying
HMSGand aPINGin between. Each subject's epochs equalkshana connect mapon that log (tests/connect_nats.rs).
Kafka is not read yet. A Kafka topic can reach Kshana today through a bridge to MQTT or NATS.
Live WebSocket input (kshana::connect_ws)#
kshana::connect_ws reads rows streamed over a WebSocket (RFC 6455) and maps them like an MQTT
subscription, through the same continuous-log feed.
ws://over TCP. The upgrade key is random and the server'sSec-WebSocket-Acceptis checked. Text and binary messages are read, fragments are joined, a ping is answered with a pong, and a close ends the stream (the held last epoch is released then). Frames the client sends are masked. Messages over 16 MiB are refused.wss://is not built in. Run a local TLS relay and connect to it withws://.- Checked against a loopback server that sends one message in two fragments, a ping, one row per
message and a close: the result is the same as
kshana connect mapon the whole log; the accept key matches the example in RFC 6455 (tests/connect_ws.rs). - From the command line:
kshana connect ws <ws://host:port/path> --mapping <connect.toml> --station <station.toml> [--header <row>] [--max-messages <n>].
Load test (kshana connect load-test)#
kshana connect load-test runs many synthetic stations through the live feeds: by default 1,000
stations at 1 Hz for 10 s, each sending one CSV row per epoch in the layout of
examples/connect/timing-monitor.csv. It passes when the feeds keep up with the offered load, 99 % of
messages take at most 250 ms, and every epoch comes back.
--via memory(default) puts the rows straight into the feeds as fast as they are taken, and measures how many messages one core maps a second.--via mqtthas a loopback broker publish every station's rows on its own topic, paced at the real rate. One wildcard subscription splits them per topic, and the test measures the time from a message being sent to its records being returned.
Measured with a release build on one core of a cloud machine on 6 Nov 2026 (1,000 stations, 1 Hz, 10 s; your machine will differ, so run it there):
| Via | Capacity | Time per message (median / 99th percentile / max) |
|---|---|---|
| memory | 40,659 messages/s against 1,000 offered | 0.022 / 0.042 / 0.276 ms |
| mqtt | (as memory) | 12.9 / 28.5 / 29.7 ms |
Over MQTT, all 1,000 stations send in the same instant each second, so most of that time is a message
waiting behind the others of its second. tests/connect_load.rs runs a smaller load in both modes on
every change. It also checks that per-topic feeds give each station exactly the epochs of
kshana connect map on its own log.
Live conformance kit (kshana connect check-live)#
kshana connect check-live <vector.toml|dir> replays the conformance vectors over MQTT, WebSocket and
the directory watch and checks ordering, holding, reconnection and latency. The rules are in
conformance/connect/README.md. On the bundled vectors every path
passes, with a 99th-percentile latency of a few milliseconds on a laptop.
Grid timing: IEEE C37.118.2 PMU streams (library, MODELLED)#
kshana::c37118 reads a capture of synchrophasor frames (a TCP stream or a dump of UDP payloads) and
reports each PMU's time health: the sync-error bit, the time-quality and unlocked-time classes in the
STAT word, and the message time quality and leap-second flags in FRACSEC. sync_losses groups
consecutive sync-error reports into intervals. Every frame's CRC is checked. A data frame is read only
after a configuration frame (CFG-2) has given each PMU's channel layout. Phasor values are stepped over,
not decoded.
It is not yet a kshana obs convert reader, because the kshana.obs/1 schema has no field for time
quality. That needs a schema minor version. It is checked against a stream from an independent encoder
(tests/c37118.rs); no public PMU capture has been checked yet.
A binary sensor is first decoded by a device file (kshana.device/1, DEVICES.md) into JSON
Lines, which a mapping file then reads like any other JSON Lines export.