Konekäännös, jota äidinkielinen puhuja ei ole tarkistanut. Viitteenä on englanninkielinen versio.

87 osaa tästä sivusta on muuttunut käännöksen jälkeen, ja ne näytetään englanniksi.

Vastaanottimen luottamus ja meriliikenneMuokkaa GitHubissaSource: docs/CONNECT.md

Kshana Connect: mapping files

A mapping file (kshana.connect/1, TOML) tells Kshana how to read an export it has no reader for, and turns it into a kshana.obs/1 observation file. No code is written: the file names the columns, their units and the time format.

Step by step, for a monitoring station: guides/MONITORING-STATION.md. Step by step, for scoring an interference classifier on the benchmark: guides/BENCHMARK-CLASSES.md.

kshana connect map export.csv --mapping export.connect.toml --station station.toml --out site.obs.jsonl
kshana obs check site.obs.jsonl

Three worked examples are in examples/connect/: an Android raw measurement CSV (one row per satellite, satellites from a constellation code and a number), a timing receiver's semicolon-separated monitoring export (one column per satellite, GPS week and time of week, height in feet) and a monitoring service's nested JSON (position in radians). tests/connect_examples.rs maps all three.

The file#

schema = "kshana.connect/1"

[source]
format = "csv"          # csv, tsv, json or jsonl
delimiter = ";"         # csv only; one character, default ","
skip_lines = 0          # lines before the header (csv, tsv)
comment = "#"           # lines starting with this are ignored (csv, tsv)
records = "/data/epochs" # json only: JSON pointer to the array of records
filter = { column = "type", equals = ["Raw"] }  # keep only these rows

[time]
column = "tow"
format = "gps-week-tow" # rfc3339, unix-s, unix-ms, gps-s, gps-week-tow or pattern
week_column = "week"    # gps-week-tow: the full GPS week
pattern = "%d/%m/%Y %H:%M:%S"  # pattern: %Y %m %d %H %M %S (seconds may have a fraction)
scale = "utc"           # rfc3339, unix-* and pattern: utc (default), gps or tai
group_tolerance_s = 0.0005  # rows this close in time are one epoch

[fix]
lat = { column = "lat", unit = "deg" }   # deg or rad
lon = { column = "lon", unit = "deg" }
height = { column = "alt", unit = "ft" } # m, ft or km
n_used = { column = "nsat" }

[signals]
# One row per satellite ...
sat = { column = "prn" }  # G05, G5 or "G 5"; or number_column with system / system_column + system_map
cn0 = { column = "cn0", unit = "dB-Hz" }
band_column = "signal"    # or band = "L1" (the default)
# ... or one column per satellite:
wide = { G05 = "cn0_g05", E11 = "cn0_e11" }

[receiver]
agc = { column = "agc" }          # native units
jam_ind = { column = "jamming_indicator" }
spoofing = { column = "spoof" }   # receiver's own flag: 0 unknown, 1 none, 2 indicated, 3 affirmed
jamming = { column = "jam" }
pdop = { column = "pdop" }
hdop = { column = "hdop" }
vdop = { column = "vdop" }

Every field takes scale and offset (applied after the unit: value × scale + offset), and in JSON a column starting with / is a JSON pointer into nested objects.

Rules#

  • Rows with the same time (within group_tolerance_s) are one epoch, so long and wide exports both work, and position rows interleaved with satellite rows join their epoch.
  • An empty cell (or a JSON null) is a missing value; anything else that is not a number is an error naming the line and the column. Nothing is guessed.
  • Times must not go backwards; a row that does is an error naming it, never a silent reorder.
  • GPS and TAI times are converted to UTC with the leap-second table; the observation file's time_label is UTC.
  • A CSV header without a column the mapping names is a mapping error.
  • The output header's source_format is connect:<format>.

Conformance vectors#

conformance/connect/ holds one folder per adapter with a vector.toml (input, mapping file or built-in reader, station, expected output). kshana connect check conformance/connect runs them all and exits 1 on any byte difference, naming the first differing line; --bless records the expected outputs after a reviewed change. CI runs the folder (tests/connect_conformance.rs). Every new adapter comes with its vector.

Not in this version: Parquet and HDF5 inputs (they need large dependencies; a conversion to CSV or JSON Lines is the route for now) and live inputs (A3a).

Live: watching a directory (library)#

kshana::connect_watch::step follows the files a logger writes or rotates into a directory and returns the kshana.obs/1 records that became complete since the last step, through the same mapping file.

  • The files whose names end with the chosen suffix are read in name order, as one continuous log: a later file's comment lines and header row are dropped. So an epoch whose rows a rotation split is still one epoch, and every t_s counts from the directory's first epoch.
  • Only complete lines are read; a line still being written waits for the next step.
  • The last epoch is held back until rows after it arrive, because more rows of it may still be written. flush releases it at the end of a run.
  • Files must be in time order by name. A single JSON document is refused (use jsonl).
  • It polls, with no operating-system file events, so it behaves the same on every platform and on network file systems. The directory is re-read on each step, so move processed files away from time to time.
  • Checked by replaying a log written in pieces and rotated part-way through an epoch: the result is the same as kshana connect map on the whole log (tests/connect_watch.rs).
  • From the command line: kshana connect watch <dir> --mapping <connect.toml> --station <station.toml> prints new records as JSON lines every --interval seconds (default 2). --suffix picks the files (default: the mapping's format, for example .csv). --once reads what is there, releases the last epoch and exits.

Live MQTT subscription (kshana::connect_mqtt)#

kshana::connect_mqtt subscribes to a topic on an MQTT broker and maps the rows that arrive, with the same mapping file, into kshana.obs/1 records.

  • MQTT 3.1.1 over TCP: clean session, optional user name and password, one topic filter (+ and # work), QoS 0 or 1. A QoS 1 message is acknowledged. A ping is sent when the line has been quiet for half the keep-alive.
  • Each message is one or more rows of the mapping's format (jsonl, csv or tsv). The messages are read in arrival order as one continuous log, as connect watch reads a directory, and the last epoch is held back until a later row arrives. Publishers send rows without a header, so a CSV or TSV mapping needs the header row given (--header).
  • Rows already emitted are dropped as soon as their epoch is emitted, and t_s still counts from the first epoch of the run. Each message therefore costs the same however long the subscription has run, and memory stays bounded.
  • TLS is not built in, and the default build adds no crate for MQTT. For a broker that only accepts TLS, run a local TLS relay (for example stunnel, or a local broker bridged to it) and subscribe to that.
  • Checked against a loopback broker that publishes a log one row per message, alternating QoS 0 and 1: the result is the same as kshana connect map on the whole log, with the trimming path exercised (tests/connect_mqtt.rs).
  • From the command line: kshana connect mqtt <host:port> --topic <filter> --mapping <connect.toml> --station <station.toml> prints records as JSON lines as they complete. --user with --password-env <VAR> reads the password from an environment variable, never from the command line. --max-messages <n> ends the run after n messages and releases the last epoch.
  • Many stations on one broker: subscribe with a wildcard (--topic 'site/+/raw') and add --per-topic. Each topic then gets its own feed, and its records carry the station id <station.id>/<topic>, with one header per station before its first epoch.

Live NATS input (kshana::connect_nats)#

kshana connect nats <host:port> --subject <s> --mapping <connect.toml> --station <station.toml> subscribes to a NATS subject and maps the rows that arrive, through the same feeds as MQTT.

  • NATS client protocol over TCP: CONNECT (with --user and --password-env <VAR>, or --token-env <VAR>; secrets come from environment variables only), SUB, then MSG and HMSG (headers are skipped). A server PING is answered with PONG, and a -ERR is reported. A lost connection is made again, as for MQTT.
  • --per-topic gives each subject its own feed (station id <station.id>/<subject>), for a wildcard subject such as gnss.*.raw.
  • TLS is not built in: for a server that requires it, use a local TLS relay.
  • Checked against a loopback server that publishes a log one row per message on two subjects, with a header-carrying HMSG and a PING in between. Each subject's epochs equal kshana connect map on that log (tests/connect_nats.rs).

Kafka is not read yet. A Kafka topic can reach Kshana today through a bridge to MQTT or NATS.

Live WebSocket input (kshana::connect_ws)#

kshana::connect_ws reads rows streamed over a WebSocket (RFC 6455) and maps them like an MQTT subscription, through the same continuous-log feed.

  • ws:// over TCP. The upgrade key is random and the server's Sec-WebSocket-Accept is checked. Text and binary messages are read, fragments are joined, a ping is answered with a pong, and a close ends the stream (the held last epoch is released then). Frames the client sends are masked. Messages over 16 MiB are refused.
  • wss:// is not built in. Run a local TLS relay and connect to it with ws://.
  • Checked against a loopback server that sends one message in two fragments, a ping, one row per message and a close: the result is the same as kshana connect map on the whole log; the accept key matches the example in RFC 6455 (tests/connect_ws.rs).
  • From the command line: kshana connect ws <ws://host:port/path> --mapping <connect.toml> --station <station.toml> [--header <row>] [--max-messages <n>].

Load test (kshana connect load-test)#

kshana connect load-test runs many synthetic stations through the live feeds: by default 1,000 stations at 1 Hz for 10 s, each sending one CSV row per epoch in the layout of examples/connect/timing-monitor.csv. It passes when the feeds keep up with the offered load, 99 % of messages take at most 250 ms, and every epoch comes back.

  • --via memory (default) puts the rows straight into the feeds as fast as they are taken, and measures how many messages one core maps a second.
  • --via mqtt has a loopback broker publish every station's rows on its own topic, paced at the real rate. One wildcard subscription splits them per topic, and the test measures the time from a message being sent to its records being returned.

Measured with a release build on one core of a cloud machine on 6 Nov 2026 (1,000 stations, 1 Hz, 10 s; your machine will differ, so run it there):

Via Capacity Time per message (median / 99th percentile / max)
memory 40,659 messages/s against 1,000 offered 0.022 / 0.042 / 0.276 ms
mqtt (as memory) 12.9 / 28.5 / 29.7 ms

Over MQTT, all 1,000 stations send in the same instant each second, so most of that time is a message waiting behind the others of its second. tests/connect_load.rs runs a smaller load in both modes on every change. It also checks that per-topic feeds give each station exactly the epochs of kshana connect map on its own log.

Live conformance kit (kshana connect check-live)#

kshana connect check-live <vector.toml|dir> replays the conformance vectors over MQTT, WebSocket and the directory watch and checks ordering, holding, reconnection and latency. The rules are in conformance/connect/README.md. On the bundled vectors every path passes, with a 99th-percentile latency of a few milliseconds on a laptop.

Grid timing: IEEE C37.118.2 PMU streams (library, MODELLED)#

kshana::c37118 reads a capture of synchrophasor frames (a TCP stream or a dump of UDP payloads) and reports each PMU's time health: the sync-error bit, the time-quality and unlocked-time classes in the STAT word, and the message time quality and leap-second flags in FRACSEC. sync_losses groups consecutive sync-error reports into intervals. Every frame's CRC is checked. A data frame is read only after a configuration frame (CFG-2) has given each PMU's channel layout. Phasor values are stepped over, not decoded.

It is not yet a kshana obs convert reader, because the kshana.obs/1 schema has no field for time quality. That needs a schema minor version. It is checked against a stream from an independent encoder (tests/c37118.rs); no public PMU capture has been checked yet.

A binary sensor is first decoded by a device file (kshana.device/1, DEVICES.md) into JSON Lines, which a mapping file then reads like any other JSON Lines export.