A controlled experiment using NVIDIA Cosmos suggests that explicit behavioural state can materially change how a robot acts around people.
A robot can know where a person is without understanding how an interaction with that person is developing.
Position, direction and speed provide important information, but human movement contains additional structure. A person may slow as they approach a crossing, hesitate briefly, change direction and then commit. Each change is small, but together they describe behaviour unfolding through time.
SpatioTemporal is developing Motion Intelligence to represent this structure explicitly. The approach compresses recent movement into a continuously updated behavioural state that can sit alongside the semantic and geometric representations already used by Physical AI systems.
We recently tested whether adding this behavioural information to an existing robot planning system would change its decisions.
In a controlled crossing experiment using NVIDIA Cosmos-Reason2-8B, the proportion of encounters entering a defined collision envelope fell from 24% to 2% when Motion Intelligence-derived behavioural information was available to the planner.
The experiment is preliminary, but the result provides an encouraging first measure of the value of behavioural state within a Physical AI stack.
Adding behaviour to the world model
Modern autonomy systems already construct several representations of their surroundings.
Computer vision can identify a pedestrian, vehicle or object, while geometric systems establish position, distance, orientation and free space. World models can combine these representations with physical dynamics and possible future states, giving a planner richer information from which to choose an action.
Behaviour adds another dimension.
A pedestrian approaching a robot may be slowing, yielding, hesitating or beginning to accelerate across its path. The geometry may remain almost unchanged while the meaning of the interaction changes substantially.
Motion Intelligence is designed to capture these changes through time. Short, overlapping windows of motion are expressed relative to the observing machine and compressed into motion primitives and discrete representations. As new observations arrive, the representation is recalculated, allowing the interpretation to evolve with the behaviour.
The architecture is deliberately complementary. Motion Intelligence does not replace perception, world models or planning, but provides another form of state for those systems to use.
Semantic state describes what is there. Geometric state describes where it is. Behavioural state describes how its movement is evolving.
The Cosmos experiment
We built a simulation environment called Reading the Room to test whether this additional representation could affect downstream robot behaviour.
Four types of shared-space interaction were evaluated, including perpendicular crossing, hallway passing and increasingly dense crowd scenarios. Each scenario contained 100 deterministic seeds representing variations in initial position, speed and encounter geometry.
Every seed was evaluated twice.
In the baseline condition, the robot planner received geometric and kinematic information about the environment. In the augmented condition, it received the same information together with behavioural state generated by the Motion Intelligence pipeline.
The underlying encounter remained identical within each pair, including the human trajectory and starting conditions, while the information available to the planner changed.
NVIDIA Cosmos-Reason2-8B selected the robot’s action every 0.1 seconds using deterministic decoding. At each step it could keep its lane, yield left, yield right, slow or stop, with the same model, action space and configuration used in both conditions.
This paired structure allowed us to compare how the same physical encounter unfolded with and without explicit behavioural information.
Reduction in near-collisions from 24% to 2%
The perpendicular crossing scenario provided the cleanest controlled interaction.
A person and robot approached a common intersection from different directions, with variations in speed and starting position producing encounters ranging from clear precedence to near-simultaneous arrival.
For the experiment, a collision event was defined as centre-to-centre separation of 0.25 metres or less. The simulated agents are represented as point locations rather than articulated bodies, so this should be understood as a proximity envelope rather than literal physical contact.
Across 100 paired scenarios, 24 baseline runs entered the collision envelope, compared with two augmented runs.
The observed rate therefore fell from 24% to 2%, an absolute reduction of 22 percentage points.
The paired data also allowed us to examine how individual encounters changed. There were 26 seeds in which the collision outcome differed between conditions. In 24 cases, the baseline entered the collision envelope while the augmented condition did not; in two cases, the reverse occurred.
An exact two-sided McNemar test produced a statistically significant difference between conditions, with p = 1.05 × 10⁻⁵. The result therefore reflects a systematic difference across the matched encounters rather than a comparison between two unrelated averages.
Safety came with a cost
The augmented robot also became more conservative.
Mean completion time in the crossing scenario increased from 9.86 seconds to 14.31 seconds, while mean robot speed decreased from 0.77 m/s to 0.53 m/s. Close-proximity exposure fell from 2.13 to 0.60 simulation steps.
This trade-off is important when interpreting the 24% to 2% result. The additional behavioural information did not simply improve one metric while leaving the policy unchanged. It altered how the robot navigated the encounter, generally causing it to yield earlier and maintain greater separation.
A useful Physical AI system will ultimately need to balance safety with progress. A robot that always stops can avoid many conflicts, but it is not an effective solution for a warehouse, hospital, footpath or workplace. The objective is appropriate behaviour: knowing when an interaction requires caution and when the machine can continue normally.
The current experiment shows a shift towards caution. Future work needs to determine how precisely that response can be calibrated.
Hallway passing showed the same direction
The hallway experiment produced a similar pattern, with the collision-envelope rate decreasing from 56% to 25% when behavioural information was available.
Completion time again increased, from approximately 10.3 seconds to 15.7 seconds, reinforcing the observation that behavioural state was changing the policy towards earlier and more conservative responses.
The crowd scenarios were less informative. One produced no recorded collisions in either condition, while the densest stress test produced proximity violations under both conditions.
These results expose limitations in the current simulation. Human trajectories are deterministic and do not respond to the robot, agents do not yet have articulated physical geometry, and Cosmos receives exact simulated state rather than the noisy and incomplete observations available to a physical robot.
The experiment should therefore be read as evidence of representation utility within controlled simulation, rather than a claim of general human-intent understanding or real-world safety performance.
What produced the improvement?
There is an important question still to resolve.
The experiment tested behavioural information generated by the Motion Intelligence pipeline, including downstream estimates of likely behaviour. It did not independently isolate every component of the representation.
The observed improvement could therefore come from several sources, including higher-order kinematic information, temporal structure, semantic behavioural classification, discrete motion tokens, or some combination of these signals.
Our next experiments will separate these components through matched ablations. The same encounters can be tested using raw position and velocity, higher-order kinematics, continuous motion embeddings, semantic behavioural labels, discrete motion tokens and combinations of these representations.
We can also vary temporal order, motion-window length and overlapping recomputation independently.
This will allow us to move from demonstrating that behavioural information changes the planner to determining which representation of behaviour provides the most useful signal, at what computational cost, and under which conditions.
A new layer for Physical AI
The commercial opportunity for Motion Intelligence is not dependent on building another robot brain.
Physical AI is developing as a stack of increasingly capable systems. Perception identifies and tracks the world, world models represent and predict it, while planners and policies determine what the machine should do. Different manufacturers will continue to make different choices at each layer.
Behavioural state can complement those choices.
A compact representation of evolving motion could condition an existing planner, contribute to risk estimation, help a world model predict interaction, direct computation towards agents whose behaviour is changing, or provide behavioural variation within simulation.
The Cosmos experiment tested the simplest version of that proposition. We retained the existing model and planner, held the physical encounters constant, and changed the behavioural information available to the system.
Its decisions changed materially.
There is considerably more work between that result and deployment. The representation needs controlled ablation, testing under sensor noise and partial observability, reciprocal human interaction, broader simulation and ultimately physical robots.
The underlying proposition is now testable.
Physical AI already treats semantics, geometry and physical dynamics as important representations of the world. For machines operating around people, behavioural dynamics may deserve the same status.
A machine needs to know what is around it and where it is. Increasingly, it will also need to recognise how the movement around it is changing, early enough for that information to influence what happens next.
That is the role we see for Motion Intelligence, and a step towards robots that read the room.