Before writing policy code for Neuralzome, we had to ask what the robot knows about its environment at each moment and how certain that knowledge is. Every downstream decision depends on that answer.
The shortcut is treating sensor data as ground truth and moving directly to decision logic. Building on another robot stack makes that tempting. We learned that the shortcut compounds every later problem.
Why Single Sensor Setups Fail
Robot demos often use one RGB D camera in clean lighting, with a centered, visible object. Production floors rarely preserve those conditions for long.
Lighting changes as fixtures warm, doors open, and nearby machines create heat shimmer. Parts arrive in unpredictable stacks, sometimes buried under two others. Depth cameras misread polished metal, shrink wrap, and wet parts, producing holes or artifacts.
With one sensor modality, its failure becomes the robot's world model failure. Nothing provides a comparison. Sensor fusion does not eliminate sensor failure. It reduces the risk of decisions based on bad data.
Neither response suits production. Stopping wastes cycle time. Continuing risks misplaced picks, incorrect assembly, and mechanical collisions.
The Three Modalities We Use
Neuralzome's perception layer uses three sensor streams as a minimum configuration for manipulation tasks.
RGB-D camera (vision plus depth): A structured-light or time-of-flight camera provides the spatial map of the scene, object pose estimates, and visual texture for object identification. We treat this as the highest-information-density source but also the most fragile one under adverse conditions.
Force/torque sensor at the wrist: A six-axis F/T sensor mounted between the robot flange and the end effector provides contact feedback that vision fundamentally cannot. When the robot makes contact with a surface, changes in force and torque tell us whether the contact is nominal or anomalous before any visual evidence is available. For assembly tasks with tolerances under 0.5mm, this is not optional.
Joint-level state feedback: Encoder readings on each joint give us the robot's kinematic state. We use this to estimate end-effector pose independently from the camera, which provides a cross-check and also helps us catch calibration drift over time. A camera that has shifted two millimeters from its nominal mount position will not self-report the error. Comparing camera-estimated end-effector position against the kinematic estimate will.
Three modalities are not always necessary or sufficient. Acoustic emission sensors can detect seating sounds, while proximity sensors support pre contact detection. Still, one modality is fragile wherever contact uncertainty or partial occlusion can compound into production problems.
The Temporal Alignment Problem
Sensors run at different rates. A depth camera, force/torque sensor, and joint encoder may report at different moments. Combining each latest reading fuses measurements from different instants.
For contact tasks, temporal misalignment between the force sensor and control loop can turn clean seating into a bent pin when detection must trigger within 5ms.
We handle this in two layers. The hardware abstraction layer timestamps every measurement on a monotonic clock shared across sensor interface nodes. The state estimation layer keeps a rolling buffer and interpolates a consistent estimate for any requested timestamp. Safety critical decisions never use extrapolation. We wait for data. The added 10ms latency is preferable to acting beyond the sensor's measurement horizon.
State Estimation Architecture
Sensor fusion produces a state estimate, not a raw reading: a structured summary of what the robot knows about the scene and its configuration, with uncertainty.
Scene state includes object pose in the robot base frame, pose uncertainty as a covariance ellipsoid, contact state and type, and confidence for each estimate. Robot state includes end effector pose, joint positions, and velocity from the encoder fed kinematic model.
Uncertainty makes perception useful to decisions. A pose alone tells a grasping policy little. A pose with covariance showing different positional uncertainty lets the policy judge whether to grasp or request a scan from another angle.
In our early access work through 2024 and 2025, diagnosed policy failures most often came from poor state estimates, not poor policies. Correcting perception recovered more performance than policy retraining.
Feeding Uncertainty Forward
Across the Neuralzome stack, the decision layer always receives uncertainty with the state estimate and must use it.
Many control systems return a hard estimate without uncertainty, then plan as if it were exact. Fixed safety offsets compensate for noisy estimates without reasoning about them. That hides uncertainty rather than exposing it.
Explicit uncertainty enables different decisions. High confidence can trigger a direct pick. Moderate confidence can pause for more visual data. Low confidence can request a scan or alert an operator instead of attempting the pick.
This behavior cannot emerge when uncertainty is hidden. A policy can exploit uncertainty only when it is part of the observation used in training and deployment.
What We Are Still Working Through
Sensor fusion remains unsolved in production robotics, and Neuralzome does not handle every case cleanly. Several open areas deserve mention.
Long term sensor degradation is harder than initial calibration. Dust, vibration, and thermal cycling cause drift. We can detect it, but autonomous recalibration still requires a maintenance window.
Unusual materials still trouble depth sensing. Transparent objects, retroreflective tape, and matte black plastics create artifacts. Our fusion pipeline marks them as high uncertainty and filters or flags them. That is conservative, but it can make the robot decline feasible tasks.
A new modality needs more than a data bus connection. It requires temporal alignment, an uncertainty model, and state estimation integration. We are developing a plug in sensor model, but it is not ready.
If perception feels like a box checked before autonomous manipulation begins, revisit it. In our experience, the difficult work often lives there.