Streaming Intelligence

The Bottom Line

ROS telemetry is the live evidence for autonomous decisions, not just data to store and analyze later. A robot fleet can only justify an automated command if the sensor input, local context, active policy, and final command stay linked the exact millisecond the action happens. Sending raw telemetry to a data lake might help with offline reporting, but it tries to rebuild operational truth after the fact. Safe autonomy requires keeping stream processing, temporal memory, safety checks, and immutable logs right on the edge next to the fleet.

‍

ROS telemetry crosses a different failure boundary

ROS 2's documented quality-of-service model distinguishes a live communication contract from an enterprise retention policy. Its middleware exposes policies for history, depth, reliability, durability, deadline, lifespan, and liveliness, and its sensor-data profile can prefer best-effort delivery where the physical world produces data faster than a system can promise delivery of every sample. The ROS 2 quality-of-service model therefore describes a contract about timeliness and loss, not a generic promise that every message should be retained forever.

That distinction matters once a vehicle, arm, or inspection device leaves a stable local network. A topic such as /amcl_pose describes an estimate that will soon be superseded. A diagnostic event may instead mark the beginning of a fault sequence. A command on /cmd_vel has a short safe lifetime, but its authorization may need to be retained for years. Treating each of those messages as a homogeneous record to bridge, replicate, and export loses the meaning carried by delivery requirements and time.

A direct ROS-to-Kafka bridge is not therefore a neutral integration. It chooses a second control plane, a second set of queues, and a second interpretation of delivery semantics. That can be entirely appropriate for analytics, model development, or enterprise integration. It becomes inadequate when that same path is expected to govern a physical command. The operational question is no longer whether the data arrived in a repository. It is whether the system had the right, current evidence to permit a command when the world could still change.

This shifts the design test. Start with the moment at which a robot must slow, stop, reroute, or reject an instruction. Then ask which records must be local, which may arrive late, which facts expire, and which decision must be reproducible. Architecture follows from those answers. It does not follow from a preference for a familiar messaging product.

‍

Connectivity loss exposes hidden shared state

Intermittent wireless connectivity is ordinary fleet behavior, not an exception to hide behind retries. The risk is often described imprecisely as a robot disconnecting and causing Kafka to rebalance. A producer disconnect alone does not necessarily do that. Kafka reassigns partitions when consumer-group membership or subscriptions change, including when a consumer process fails, joins, or returns. Its own consumer rebalance interface instructs applications to handle partition revocation, assignment, lost partitions, and consumer-side state.

The practical problem is the shared state introduced around the bridge. A disconnected robot can leave an edge bridge buffering data. A downstream consumer can time out. An application may need to flush intermediate state before partition ownership changes. None of those outcomes is universal. They depend on the ROS QoS profile, bridge queue depth, client settings, bandwidth, and what the consumer has kept in memory. But together they create a failure boundary that crosses the fleet rather than remaining local to the machine that lost signal.

The consequences should be assessed by message class:

• Ephemeral observations such as pose estimates need bounded queues and an explicit loss policy.
• Safety-relevant diagnostics need an ordered record, but they also need a route that remains useful during an upstream outage.
• Actuation requests require a local validity window and an authorization check that does not wait for a distant reconstruction of state.

The error is to assume that a durable log resolves all three problems in the same way. It does not. A log can preserve a message after delivery. It cannot make an old pose current, grant a stale command authority, or decide which delayed data should be discarded. Those are control decisions. They belong near the point where the robot can still act.

‍

Retention is a question of evidence rather than volume

Raw camera frames, lidar scans, point clouds, pose updates, and diagnostic traces have very different retention value. It is easy to argue about storage cost in the abstract, but a credible estimate requires a declared resolution, encoding, sample rate, compression ratio, replication policy, retention interval, and fleet schedule. Without those inputs, a fleet-size calculation is theater.

The more useful distinction is between data needed to recreate a decision and data that might later be useful. The former is an evidence set. It should include the observations selected by the policy, derived state, command candidate, applicable rule version, authorization result, and the command or refusal emitted. The latter may include raw sensor payloads whose use is diagnostic or scientific rather than immediate. Both matter, but they should not force the same storage path.

Export-into-the-lake architectures commonly make raw retention the center of gravity. They transmit broad telemetry first, organize it later, and hope that enrichment can recover operational meaning. That approach has a legitimate place in offline investigation. It is a poor substitute for a contemporaneous decision record, because later processing cannot know which signals the live system saw, which it rejected, or which local constraints were in force unless those facts were recorded at action time.

The proposed in-situ design keeps a compact decision record close to the fleet, while routing selected raw artifacts to longer-lived storage under explicit retention rules. That is not an argument against Kafka, object storage, or a lake. It is an argument against making archival transport the place where operational truth is first assembled. The record of authority must be complete before data is moved for other purposes.

‍

Schema contracts should preserve meaning

Schema discipline is necessary. It is also a source of friction when one contract is forced to stand in for another. ROS 2 uses interfaces expressed through IDL-derived types and middleware behavior. Kafka-oriented systems often serialize payloads through Avro, Protobuf, or JSON Schema and check them against a centrally managed subject. Schema Registry compatibility rules make clear that backward, forward, full, and transitive compatibility are explicit policy choices, with allowed changes depending on how fields were originally defined.

That is sound engineering for a shared enterprise event contract. The difficulty begins when a bridge treats a ROS interface change as only a serialization problem. A nested field may change units, a sensor driver may expose a revised covariance structure, or a message may be retained only for a different lifespan. A compatibility check can accept a structurally valid change while the operational interpretation has changed. It can also reject a producer change that is harmless within a controlled robot release.

The answer is not schema permissiveness. It is separating three contracts that are too often collapsed:

1. Transport compatibility asks whether a consumer can deserialize the record.
2. Operational semantics asks what the record means, how long it is valid, and what uncertainty it carries.
3. Command authority asks whether the record may contribute to a physical action.

Each needs versioning, but they should not share one approval mechanism. The first can be managed by a registry. The second needs explicit provenance and temporal rules. The third needs deterministic policy that can be tested independently of a model's interpretation. A bridge that preserves bytes but loses these distinctions has not preserved the system's meaning.

‍

Can AI agents govern a fleet without centralizing its data?

Yes, if the agent receives a bounded, current operational view and cannot bypass deterministic command policy. Centralizing every event in a global ontology is one way to make data queryable. It is not the only way, and it creates a problem of its own: the authoritative state is assembled at a distance from the physical process it is meant to govern.

Operational memory is the time-bounded representation of facts, relationships, and decision-relevant history needed to evaluate what should happen now. It differs from topic history. A topic preserves ordered messages. Operational memory connects a robot, its current location estimate, an active work zone, a thermal trend, a prior command, and the validity periods of those facts. It also lets expired signals disappear from a decision view without deleting the underlying evidence.

That is the gap between asking whether an event exists and asking whether a command is permissible. Consider a request to reduce a robot's speed. The answer may depend on current pose confidence, a recent fault code, the vehicle's assigned path, an exclusion zone, and whether the requested command is still fresh. A downstream data platform may reconstruct part of the answer later, but it cannot recover local state, expiry, or policy conditions that were never recorded. A live system needs those conditions before the command is placed on the wire.

Our Sovereign Decision Fabric is an architecture in which events, local reasoning, policy, and decision records stay in the organization-controlled environment where the data is produced. Lascaris is one concrete expression of that pattern. Its relevance is not a catalog of components. The useful architectural claim is that operational memory should sit beside the stream and feed policy before actuation, rather than becoming a derivative dataset after export.

This arrangement also limits what an AI agent is asked to do. An agent may classify a pattern, propose a response, or retrieve related evidence. It should not invent the conditions under which physical authority is granted. The system should make those conditions inspectable in rules and records.

‍

Command authority must survive an audit

An autonomous system is governed only when it can show why it acted and why it was permitted to act. That requirement gets sharper as connectivity expands and human oversight becomes less direct. A NIST analysis of AI risk management for uncrewed systems identifies reduced human oversight, increased connectivity, and access to mission data as reasons to examine data storage, access, modification rights, failure modes, and the ability to determine why a system made a decision.

The design implication is straightforward. An AI inference is not an authorization. A model can place a command candidate on a governed path, but a deterministic policy should evaluate the candidate against current state, scope, validity period, identity, and applicable constraints. The policy result, including rejection, should become an immutable event. That gives investigation a record of the decision boundary, not merely a record of the command that happened to be emitted.

An adequate decision record should bind together at least these elements:

• the source observations and their timestamps;
• the operational facts used and their validity windows;
• the model or rule version that produced the candidate;
• the policy version, authorization result, and reason;
• the resulting command, refusal, or escalation.

‍

No architecture removes uncertainty from sensor interpretation. Nor should it conceal it. A governed path should express uncertainty as an input to policy, retain the evidence that informed the outcome, and fail within a defined local safety posture when required inputs are absent. Centralized analysis can later improve those rules. It cannot retroactively authorize a command or restore a signal that was already stale.

‍

The architecture should be judged by where truth is assembled

The important comparison is not between brands or between a graph and a log. It is between two sequences of responsibility. One exports events, centralizes them, models them, and then attempts to infer what should have happened. The other keeps the live stream, temporal context, policy check, and decision record together before an action crosses into the physical world.

There are real limits to the second approach. Local systems need operational ownership, software updates must preserve policy provenance, and cross-fleet learning still needs controlled aggregation. Some workloads genuinely require central historical analysis. An in-situ design does not eliminate those needs. It makes a narrower claim: no remote platform should become the sole source of context for a command that must be safe and explainable before the network round trip completes.

The policy question is therefore precise. If a robot must justify a command after an outage, a software update, or an incident, where was the relevant truth assembled, and can that answer be replayed without trusting an after-the-fact reconstruction?