Concepts

DC has a small, fixed vocabulary. Using it precisely makes the rest of the documentation — and the configuration files — unambiguous.

Glossary

Relationships between them:

• A Measurement emits Records, optionally gated by one or more Conditions
• A Group merges Records from several Measurements into one Record
• The Bridge forwards every Record to the Shipper
• The Shipper delivers Records to one or more Destinations
• A File is uploaded to an object-storage Destination; its metadata becomes a Record
• A Record carries a Tag; each Tag names a route Destinations subscribe to
• A Trigger fires a FlushEvent; every Measurement listening for it releases its buffered window as one Incident

ROS 2

ROS 2 is the core middleware used for DC. If you are unfamiliar with it, visit the ROS 2 documentation before continuing.

Records

A Record is a single data unit in JSON, published on a ROS topic as a dc_interfaces/msg/StringStamped message. For example, a Record from the Memory Measurement:

{
    "date": "2022-12-04T14:16:06.810999008",
    "flattened": false,
    "memory": {
        "used": 76.007431
    },
    "nested": true
}

flattened and nested are added to every Record unconditionally by the base Measurement class — they report whether nested/flatten were configured for this plugin, not something you opt out of showing.

The StringStamped message carries:

1. header: ROS timestamp as std_msgs/Header
2. data: the Record, as a JSON string
3. group_key: the key this Record is nested under when merged into a Group
4. incident_id: the Incident a released Record belongs to; empty outside an incident

The data string must be valid JSON:

"Robot_X:1.5, Robot_Y:1.8" # Not a valid Record
"{'x': 1.5, 'y': 1.8}"     # Valid Record

Timestamps

The timestamp is the time the Record was created. Measurements convert ROS time to a UTC timestamp. Each Destination normalizes it into one field before delivery, controlled by time_key (default date) and time_format (epoch_nanos, the default, for exact integer nanoseconds since the epoch; iso8601 for a string; double for fractional seconds, which rounds — see Destinations).

JSON validation

Each Record is validated against a JSON schema by default, following the JSON Schema Validation specification. Each Measurement ships its own schema, which can be overridden from a custom package or disabled per Measurement with enable_validator: false. See Data validation.

Bridge and Shipper

DC's data plane is an external Shipper process, Vector, fed by the Bridge (dc_bridge) over a local socket. The Bridge renders the Shipper's entire configuration from plain ROS parameters — see Destinations — spawns and supervises the Shipper process, and forwards every Record it receives on its configured input topics.

DC gets the Shipper's disk buffering, backpressure handling, retries, and native Destination support (PostgreSQL, S3-compatible storage, and many more) without embedding or forking it (ADR-0001, ADR-0002).

Tags

Every Record carries a Tag, derived mechanically from the topic it was published on: the leading / is dropped and the remaining / become ..

Tags are what the Shipper routes on, and the dc.<tag> route names are stable public API that passthrough Destinations consume. A Destination selects what it receives through its own inputs list of topics — Tags are derived from that, not configured directly.

Destinations

A Destination is where data ends up: a file is blessed (rendered from plain ROS parameters), and so is another Shipper, and (only for File uploads, not Records) S3-compatible storage. PostgreSQL, S3-compatible storage for Records, and the console are reached through a passthrough recipe instead of the ROS-param form (per ADR-0003); any other Vector sink is reachable through the passthrough too. See Destinations for the full contract.

dc_bridge:
  ros__parameters:
    destinations: ["records_log"]             # Destination names to enable
    records_log:                              # a name you choose
      type: file                              # a blessed type
      receives: records
      inputs: ["/dc/measurement/uptime"]      # the topics this Destination receives
      path: "/tmp/dc/records.ndjson"

measurement_server:
  ros__parameters:
    measurement_plugins: ["uptime"]
    uptime:
      plugin: "dc_measurements/Uptime"
      topic_output: "/dc/measurement/uptime"  # matched by the Destination's `inputs`

Conditions

A Condition enables or disables one or more Measurements. For example, collect camera images only when the robot is stopped. A Measurement can require that all, any, or none of a set of Conditions are active. See Conditions.

Triggers and incidents

A Condition can only give you data from the moment it turned true. When what matters is the run-up to an event — the seconds before an emergency brake — a Trigger is the mechanism: a Measurement holds its recent output in a rolling buffer instead of publishing it, and a Trigger firing on the false→true edge of a Condition composition broadcasts a FlushEvent that releases that buffer. Everything released by one firing, across every Measurement listening, shares one incident_id. See Triggers.

Lifecycle Nodes and Bond

(Source: Nav2 documentation)

Lifecycle (or Managed, more correctly) nodes are unique to ROS 2. More information can be found here. They are nodes that contain state machine transitions for bringup and teardown of ROS 2 servers. This helps in deterministic behavior of ROS systems in startup and shutdown. It also helps users structure their programs in reasonable ways for commercial uses and debugging.

When a node is started, it is in the unconfigured state, only processing the node's constructor which should not contain any ROS networking setup or parameter reading. By the launch system, or the supplied lifecycle manager, the nodes need to be transitioned to inactive by configuring. After, it is possible to activate the node by transitioning through the activating stage.

This state will allow the node to process information and be fully setup to run. The configuration stage, triggering the on_configure() method, will setup all parameters, ROS networking interfaces, and for safety systems, all dynamically allocated memory. The activation stage, triggering the on_activate() method, will active the ROS networking interfaces and set any states in the program to start processing information.

To shutdown, we transition into deactivating, cleaning up, shutting down and end in the finalized state. The networking interfaces are deactivated and stop processing, deallocate memory, exit cleanly, in those stages, respectively.

The lifecycle node framework is used extensively through out this project and all servers utilize it. It is best convention for all ROS systems to use lifecycle nodes if it is possible.

Within DC, we use a wrapper of LifecycleNodes, nav2_util LifecycleNode from Nav2. This wrapper wraps much of the complexities of LifecycleNodes for typical applications. It also includes a bond connection for the lifecycle manager to ensure that after a server transitions up, it also remains active. If a server crashes, it lets the lifecycle manager know and transition down the system to prevent a critical failure. See Eloquent to Foxy for details.

See Lifecycle Manager for the managed-node list, the bond heartbeat and state-transition diagrams, and the autostart flow that ties into the Bridge readiness gate above.

Buffering and data persistence

The Shipper manages its own disk buffer (shipper.data_dir on the Bridge's parameters) with a documented minimum size (shipper.buffer_max_bytes). This buffer persists across reboots: Records accepted by the Bridge but not yet delivered to a Destination survive an outage of that Destination, a robot reboot, or a Bridge restart, and are delivered — with end-to-end acknowledgements — once the Destination is reachable again.

Scheduling and retries

The Shipper retries delivery to a Destination on failure with its own backoff, independent of DC code; see Vector's documentation for details. The Bridge itself is supervised by DC (launch respawn) and, in turn, supervises its own Shipper child process — including a Linux parent-death signal so the Shipper can never outlive the Bridge across a crash or SIGKILL.