What happened when we gave Cosmos 3 Motion Intelligence
NVIDIA approached SpatioTemporal with an invitation to put Cosmos 3 through its paces.
Cosmos 3 is NVIDIA’s latest generation of World Foundation Models for Physical AI, part of a broader effort to give machines an increasingly sophisticated understanding of the physical world.
It’s an impressive piece of technology. Cosmos 3 can interpret complex scenes and reason about physical interactions, generate realistic simulations of environments and events, and predict how those environments might evolve. It can also create enormous volumes of synthetic training data, including unusual or dangerous situations that would be difficult to reproduce in the real world. And increasingly, these capabilities are being used to train robot behaviours, helping machines learn how to act in unfamiliar situations before encountering them for real.
Together, these capabilities are becoming an important foundation for modern robotics. Rather than programming every possible situation, researchers can train, test and refine robot intelligence across countless simulated experiences. In simple terms, NVIDIA is building increasingly capable foundations for robot brains.
SpatioTemporal is working on a different, more specialised part of that problem.
We’re building Motion Intelligence, a compact foundation model that understands how things move through space and time. Rather than concentrating on what objects look like or where they’re located, we’re interested in what their movement tells us about what they’re doing, what they’re likely to do next, and how their behaviour affects everything moving around them.
Where Cosmos 3 brings broad physical-world understanding, SpatioTemporal concentrates on something more specific: the instincts required to understand movement around you. The subtle signals of hesitation, intention, yielding and negotiation that humans interpret almost without thinking.
Through SpatioTemporal’s participation in the NVIDIA Inception Program, the collaboration gave us an opportunity to test a straightforward proposition:
What happens when we give a world model an explicit understanding of movement?
Rather than asking Cosmos to interpret everything from the moving scene itself, we could give it the same scene with Motion Intelligence quietly providing additional information about the relationships developing within it.
So that’s what we did.
TL;DR: Cosmos read the room 8x better
Let’s not bury the lede.
We gave NVIDIA Cosmos 3 a series of complex, moving worlds constructed from real-world motion trajectories. These were controlled experimental environments containing multiple moving participants, with interactions developing and changing over time.
We wanted to measure how well Cosmos could understand what was happening within them. Who matters? What’s happening between them? What happens next? And when?
Then we ran the same experiments again, this time with SpatioTemporal’s Motion Intelligence foundation model providing an additional representation of movement and the relationships developing within each scene.
The difference was substantial.
| Reading the Room Metrics | Cosmos 3 alone | Cosmos 3 + Motion Intelligence | Improvement |
|---|---|---|---|
| Relevance: Who matters? | 8.8% | 71.0% | 8× accuracy |
| Interaction: What’s happening? | 4.71 m/s error | 2.37 m/s | 50% lower error |
| Anticipation: What happens next? | 245m error | 35m | 86% lower error |
| Timing: When will it matter? | 20.9s error | 2.3s | 89% lower error |
These results come from controlled experimental benchmarks, not deployed robots. We constructed challenging encounters from recorded motion trajectories and presented identical underlying worlds to both model configurations.
The results suggest something important.
Cosmos 3, already a highly capable world foundation model, performed substantially better at interpreting physical interactions when augmented with a specialised representation of movement.
We didn’t change the underlying worlds. We didn’t give Cosmos an easier problem. We gave it additional information about how movement was unfolding within the same scenes.
Same worlds. One additional motion representation.
Matched controlled benchmark, Showdown II. Each metric is evaluated independently, with different units and error scales. The 8× improvement refers specifically to relevance accuracy, not a combined performance score.
To understand why this matters, it’s worth considering what robots will increasingly be asked to do.
Our thesis: Seeing isn’t the same as understanding
Walk through a busy railway station and you’ll encounter dozens of people moving in different directions. You don’t consciously calculate the trajectories of everyone around you. Most people barely register, while a handful attract your attention because their movement is beginning to interact with yours.
- Someone ahead is slowing.
- Another person is drifting across your path.
- A third is approaching quickly but will probably pass behind you.
- Someone else has hesitated and looks likely to change direction.
You make these judgements continuously, usually without stopping to reason explicitly about any of them. We put it down to human instinct. Reflexes. Reading the room.
But movement contains information.
Where someone is and how fast they’re moving matters, but so does how they got there. Whether they’re accelerating or slowing. Whether their direction has been stable or is beginning to change. How their movement relates to yours, and to everyone else around them.
Humans interpret this history almost effortlessly. We use it to decide what matters now, anticipate what might happen next, and adjust our own behaviour accordingly.
For robots, the problem is considerably harder. None of this comes pre-installed.
Modern robotic perception systems can identify people, objects and obstacles. Navigation systems can calculate routes around them. Planning models can work towards increasingly complex objectives. But identifying a person and understanding what their movement means are different capabilities.
A pedestrian standing near a crossing is an object in a scene. A pedestrian who has slowed, turned towards the road, hesitated and begun moving towards it is telling you something through motion.
The machine doesn’t need to know exactly what that person is thinking. It needs to recognise that the pattern of movement contains information about what may happen next.
This is the problem SpatioTemporal calls Motion Intelligence.
It’s the missing middle layer between seeing the physical world and deciding how to act within it.
Motion is a language
Our first Motion Intelligence foundation model began with a simple question:
Can movement have words?
Language models learn patterns from sequences of tokens. Words and fragments of words form sentences, sentences form paragraphs, and their relationships provide the structure from which models learn.
We wondered whether movement might have a comparable structure.
Rather than training directly on video, we began with recorded motion tracks. We represented short, overlapping windows of movement as discrete Motion Tokens, creating a vocabulary of 10,000 motion primitives.
A moving object could then be represented as a sequence of tokens, describing how its movement developed through time.
Movement exhibits recurring structure. Past sequences shape what is likely to follow. Changes in speed, direction and acceleration formed recognisable patterns that could be learned without requiring the model to understand the visual appearance of the objects producing them.
We’d begun to identify something resembling a grammar of movement.
Try giving your movement a vocabulary.
Move your mouse in the panel. Turn, pause, speed up. Watch your movement become Motion Tokens.
Waiting for movement
Move inside the panel to begin.
That first model established an important foundation for our work: movement could be represented independently of the pixels describing the scene.
It also exposed the next challenge. The physical world doesn’t contain one trajectory at a time.
From a moving object to a moving world
The physical world doesn’t contain one trajectory at a time. A person or a robot moving through a public space may observe dozens of people, vehicles and other moving objects. Some are relevant. Most aren’t. And relevance changes continuously.
A cyclist fifty metres away may initially be unimportant, then become the most consequential actor in the scene. A pedestrian walking beside the robot may matter for several seconds before moving away. Two people approaching one another may create an interaction that changes the movement of everyone around them.
Understanding any one trajectory isn’t enough.
The machine needs to understand the relationships between them.
This became the central question for the next generation of Motion Intelligence:
Can a compact model maintain an understanding of a moving world, identifying what matters and anticipating how physical interactions develop?
The architecture changed accordingly.
MI:1 & MI:2
First, a note on nomenclature: MI:1 and MI:2 describe generations of our Motion Intelligence foundation models, rather than individual model versions.
- MI:1 refers to the first series, encompassing our v1.x models developed around single-actor motion, Motion Tokens and the grammar of movement.
- MI:2 is the second generation. It extends that foundation from understanding individual movers towards understanding multi-actor scenes, the relationships developing between them, and which of those relationships matter.
MI:1: one mover at a time
MI:1 is ego-centric. It learned the grammar of movement from the perspective of a single moving actor: the ego.
That might be a person, robot, car or another moving machine, but there is always one ego at the centre of the model’s attention.
From movement alone, MI:1 can infer considerably more than position and velocity. It can estimate broad physical characteristics and likely class, distinguishing movement consistent with a pedestrian, cyclist, car or truck. It can also characterise behavioural patterns such as hurried, hesitant, tiring or increasingly erratic, and track how those patterns change through time.
But the ego doesn’t have to be the robot itself.
We can project MI:1’s perspective onto another moving actor and ask the same questions of its movement. Cast that gaze onto the cyclist ahead, the pedestrian approaching from the left or the car moving through an intersection, and MI:1 can temporarily read that actor’s movement as though it were the centre of the world.
This external-ego projection is useful, but it also exposes MI:1’s fundamental limitation: even if we move that gaze rapidly from actor to actor, we’re still understanding the world one mover at a time.
The real world isn’t a collection of independent trajectories. It’s a continuously changing network of relationships between them.
MI:2: from trajectories to relationships
Through our collaboration with NVIDIA, we began developing the second generation of Motion Intelligence foundation models: MI:2. Its guiding principle is simple:
Keep track of everything sufficiently. Understand a few things deeply. Continuously change what deserves attention.
In Physical AI – that is, out in the real world – the number of visible objects is rarely the same as the number of consequential objects.
A robot doesn’t need to reason deeply about every pedestrian on a footpath. It needs to recognise the few whose movement may affect what happens next, while remaining sufficiently aware of everyone else to notice when that changes.
The person walking away can fade from attention. The cyclist beginning to converge can move sharply into focus. Someone previously irrelevant can hesitate, turn, accelerate or cross a path and suddenly become important.
MI:2 is designed around that changing structure of relevance.
Rather than treating every visible actor as equally important, it maintains a broader representation of the moving world while concentrating deeper reasoning on the actors and relationships most likely to matter.
The world keeps moving. Attention has to move with it.
A compact understanding of the moving world
There’s a practical problem with maintaining an understanding of a moving world. You can’t carry the whole world around with you.
Every new observation adds more information. More people. More trajectories. More history. More interactions. Yet much of that information quickly becomes irrelevant. What matters is preserving the useful part: who matters, how things are moving in relation to one another, how those relationships are changing, and what may be developing next.
MI:2 therefore needed a kind of working memory for motion.
We call it the World State Vector. The WSV.
A memory of what matters
The WSV is a compact representation of what Motion Intelligence has learned about the relationships within a moving scene. It isn’t a replacement for perception. It isn’t a map, a complete description of the environment or a general-purpose world model. Those systems answer different questions.
The WSV has a narrower job. It preserves useful information about how the moving world is developing.
As actors enter and leave the scene, trajectories change and relevance shifts, MI:2 continually updates this representation. The WSV carries forward what the model has learned about the motion and relationships that matter, without needing to repeatedly describe everything it has observed.
You can think of it as a compressed memory of the moving world.
In the MI:2 model developed for this research, that memory is represented in just 32 dimensions.
Can another model use it?
Creating a compact representation is interesting – but it isn’t enough. For Motion Intelligence to become useful infrastructure for Physical AI, the information it learns needs to be useful outside Motion Intelligence itself.
Could another model take this compact representation of a moving world and use it to understand that world better? Could Motion Intelligence effectively tell another intelligence: Here’s what I’ve learned about the motion. You take it from here.
That became the real test. And that’s where Cosmos 3 came in.
Adding Motion Intelligence to NVIDIA Cosmos 3
NVIDIA’s invitation gave us the opportunity to test Motion Intelligence alongside one of the world’s most significant Physical AI platforms. For SpatioTemporal, it gave us the opportunity to explore where a specialised Motion Intelligence foundation model might sit within a much broader Physical AI architecture, and whether the intelligence we were developing could complement the work being done at NVIDIA rather than duplicate it.
That distinction is central to how we think about SpatioTemporal – we’re not trying to replace perception systems, planners or world models. We’re building a specialised layer between them: one concerned with movement, relationships, intent and what is likely to matter next. The collaboration gave us a way to test that proposition directly.
Could a specialised understanding of movement help NVIDIA Cosmos 3 interpret the same physical world more accurately?
Building the experiment
To answer that properly, we first needed movement worth understanding. Earlier experiments had exposed a problem with relying too heavily on synthetic kinematics. Procedurally generated movement is useful for controlled testing, but it can also become remarkably predictable.
If nearly every actor travels at constant speed in a straight line, a straightforward physics calculation may already provide an excellent forecast. There isn’t much value in asking a foundation model to rediscover something we can calculate directly.
Real movement is messier. People accelerate and slow down. They hesitate. They drift. They change direction. Their trajectories contain the small irregularities and changes through time that make understanding movement interesting in the first place.
Our experimental worlds were assembled from more than a million real motion tracks, preserving the irregularity and variation present in the source trajectories while allowing us to construct repeatable multi-agent encounters.
There’s an important distinction here. These weren’t jointly recorded scenes. Individual recorded trajectories were brought together into controlled experimental worlds. That allowed us to combine the complexity of real movement with the repeatability required for a useful benchmark, as the example below shows. The resulting encounters retain the irregularity of real-world motion while remaining experimentally repeatable.
Same world. Same observations. One additional representation.
We then compared Cosmos interpreting those worlds from causal motion observations alone with Cosmos receiving the same observations augmented by MI:2’s representation.
The model was asked to interpret the developing physical interactions.
But how do you measure whether a machine understands a moving world?
Measuring what matters
We’ve developed a set of measures called Reading the Room Metrics, or RRMs. Their purpose is to separate physical understanding into questions that humans answer almost automatically when moving around one another.
Relevance: Who matters?
A busy scene may contain dozens of actors, but only a small number are consequential to the developing interaction. Relevance measures whether the model identifies those actors correctly. It’s a question of attention, but also of judgement. The closest person isn’t necessarily the most important, and someone who currently appears irrelevant may become important as their movement changes.
Interaction: What’s happening?
Once an actor matters, the next question is how its movement relates to everything else. Are two trajectories converging or separating? Is one actor crossing another’s path? Is the physical relationship changing in a way that matters? Interaction measures the accuracy of the model’s estimate of that developing relationship.
Anticipation: What happens next?
Understanding the present is only part of the problem. A useful motion model must anticipate how movement is likely to develop, particularly when the current state doesn’t tell the whole story. Anticipation measures the accuracy of the model’s forecast of future movement.
Timing: When will it matter?
A robot may correctly identify a future interaction and still misunderstand its significance if it gets the timing wrong. Two paths may cross eventually, but whether their occupants reach the crossing point simultaneously determines whether there’s a meaningful interaction. Timing measures how accurately the model estimates when the consequential part of the interaction will occur.
Together, these measures help distinguish a machine that detects movement from one that understands the developing relationships within it.
Who matters?
What’s happening?
What happens next?
When will it matter?
Explanatory geometry, not model predictions or benchmark results. All four panels use the same moving actors and clock. The nearest actor need not have the closest future approach; dashed futures reveal the authored continuation.
What happened when Cosmos received Motion Intelligence?
The experiment produced improvements across all four Reading the Room Metrics:
Relevance increased from 8.8% to 71.0%.
Cosmos 3 became roughly eight times more accurate at identifying which actors were consequential to the interaction.
This is important because a robot’s understanding of a scene depends heavily on what it chooses to attend to. A system that concentrates on the wrong actors may reason accurately about irrelevant information while missing the interaction that matters.
Robots are constrained by size, weight and power. The proverbial balancing act that all builders have to work within. To make matters worse, robots also need to be self-sufficient – sending the state of the world around it out to the cloud takes time that the robot may not have. Being able to make millisecond decisions, even without network capabilities, will be a standard requirement for robots working in and around humans.
Interaction error fell from 4.71 to 2.37 metres per second.
With Motion Intelligence available, Cosmos produced substantially more accurate estimates of the developing physical relationships between actors. This is a measure of the evolving relationship between trajectories: approaching, crossing, converging, following, separating, yielding or otherwise interacting through motion.
Anticipation error fell from approximately 245 metres to 35 metres.
That’s an 86% reduction. Cosmos 3’s ability to forecast future movement improved substantially when it received the additional Motion Intelligence representation. This is not ‘predicting the future of the world’, like an LLM is overly simply described as being able to ‘predict the next word’ – this is a difference in error from every guess from timestamp to timestamp.
Timing error fell from approximately 20.9 seconds to 2.3 seconds.
That’s an 89% reduction. For a system operating around moving people, understanding when something will happen is often as important as understanding what will happen. Vitally, we’re showing that it’s not just understanding space, it’s also understanding time: the temporal relationship between trajectories – particularly the predicted time until the relevant encounter, crossing, conflict or closest-approach event.
Six decisions sampled evenly in world/case order from the 1,984 paired decisions, without selecting by performance. Dots are actual saved relative-position outputs at 2, 4, 8 and 16 seconds; dotted connections aid reading and do not assert dense predicted trajectories. Each condition can select a different actor. Aggregate gains do not guarantee every answer improves.
These results don’t establish that a robot using Motion Intelligence would automatically become safer or more capable in the real world.
They establish something more specific, and important in its own right.
A major Physical AI foundation model interpreted physical interactions substantially more accurately when given an additional representation of movement.
That suggests movement contains useful structure that can be made available to broader intelligence systems.
Does Motion Intelligence understand time?
There was another question we wanted to test.
Was MI:2 learning useful information from the history of movement, or was it simply encoding a more convenient description of the current spatial arrangement?
To investigate, we deliberately disrupted the temporal ordering of the movement history supplied through the Motion Intelligence pathway.
We then compared that condition with correctly ordered movement.
Across all five Reading the Room measures in this control experiment, the correctly ordered representation performed better than the shuffled version.
The implication is straightforward.
The history of movement mattered.
MI:2 wasn’t simply describing where actors were at a particular instant. Its representation was making use of how the scene had developed through time.
That is central to our original thesis.
A person’s current position and velocity provide useful physical information. But their recent hesitation, acceleration, change of direction or interaction with another person may tell us something additional about what is likely to happen next.
Motion Intelligence is intended to learn that structure.
Ordered history
Shuffled history
The movement illustration uses the same recorded points reordered for explanation; it is not a reconstruction of the exact experimental shuffle. In Showdown II, ordered history improved all five RRM means and paired intervals relative to shuffled history. 1,984 matched decisions in 992 constructed worlds; exploratory, with repeated source tracks.
Reading the room is not the same as deciding what to do
Our experiments with Cosmo 3 have also taught us something important about where Motion Intelligence belongs.
Better physical understanding doesn’t automatically produce better robot behaviour.
We tested whether the information in MI:2’s World State Vector could improve a small model responsible for simple Conduct decisions, such as continuing, yielding or braking.
A strong physics-based representation was already very effective at that particular task. Adding the WSV directly didn’t improve its overall performance.
That result matters because it helps define the boundaries of the technology.
Physics is extremely good at calculating relationships that follow directly from geometry and motion. There is little value in asking a learned model to replace calculations that are already accurate, fast and inexpensive.
Motion Intelligence should contribute where the history and structure of movement provide information beyond those calculations.
We’re continuing to investigate the interface between motion understanding and action, including whether decoded motion information or a residual approach can expose useful information that a physics-only system misses.
For now, the distinction is clear.
Understanding what is happening and deciding what to do about it are related, but different problems.
And that brings us to the larger challenge facing robotics.
The next challenge for robotics isn’t capability
Robotics is advancing rapidly. Machines are becoming more capable of manipulating objects, navigating environments and performing tasks that previously required direct human control. But completing a task isn’t the same as completing it well around people.
A robot may be able to carry a parcel across a room, navigate a corridor or collect an object from a shelf. Those capabilities are valuable, but they don’t tell us much about how the robot behaves when someone unexpectedly steps into its path.
Does it stop unnecessarily? Does it continue when it should yield? Does it make the person uncomfortable by approaching too closely? Can it recognise hesitation, anticipate a crossing movement or negotiate a shared space without constantly forcing people to accommodate it?
These questions become increasingly important as robots move out of controlled environments and into workplaces, hospitals, transport hubs, public spaces and eventually homes.
In our earlier research on the bottleneck of trust in robotics, we explored the gap between demonstrating a machine’s technical abilities and establishing whether people will be comfortable relying on those abilities in everyday environments.
The distinction becomes clearer when we separate robot performance into three questions.
The Three Cs: Capability, Competence and Conduct
SpatioTemporal is a Founding Partner of The Robot Benchmark and The Robot Range, two initiatives being developed to bring more independent, repeatable evidence to the emerging robotics industry.
The Robot Benchmark is intended to establish common measures for comparing robot performance. The Robot Range provides the physical proving ground where those capabilities can be tested under realistic, repeatable conditions.
The original description of what robots are best designed for was ‘The Three D’s’: Dirty, Dull and Dangerous.
SpatioTemporal is proposing that the future of robotics will instead be built around The Three C’s:
Capability: Can it do it?
Does the robot possess the physical, sensory and functional abilities required to perform the task?
Competence: Can it do it well?
Can the robot perform reliably and repeatedly, including when conditions change, interruptions occur or the environment becomes less predictable?
Conduct: Can we trust how it behaves?
How does the robot behave around people, other machines and unexpected events while carrying out its task?
The Three Cs recognise that a robot can be capable and competent without necessarily behaving appropriately in a shared human environment. A machine that completes its assigned task while repeatedly forcing people out of its way may satisfy a narrow performance measure. It may also be a machine nobody particularly wants to work beside. The commercial implications are considerable.
Robot manufacturers need to demonstrate what their systems can do. Buyers need evidence that those systems will perform reliably in their environments. Insurers, regulators and employers need to understand the risks associated with deploying them. And the people who encounter these machines every day need confidence that the robots will behave predictably and considerately around them.
That’s why we’re involved in building independent testing infrastructure.
The Robot Benchmark defines what matters. The Robot Range helps establish whether a machine can actually deliver it.
Capability
Can it reach, carry and deliver?
Competence
Can it repeat the route through variation?
Conduct
Can it share that route considerately?
The Robot Benchmark and The Robot Range →
An authored delivery illustration of three distinct testing questions. The robot’s route, variation and yielding are explanatory, not a deployed controller or evidence of solved Conduct.
This is also where Motion Intelligence becomes commercially relevant.
We aren’t interested in improving a benchmark number for its own sake.
We’re interested in giving robots the ability to understand movement well enough to behave appropriately around people, and in developing the evidence needed to establish whether that understanding actually improves their Conduct.
The robot brain is splitting into different kinds of intelligence
In an earlier SpatioTemporal research article, we explored how the emerging robot brain is being divided by the speed at which different decisions need to be made.
Some decisions can tolerate substantial reasoning time. Others need to happen almost immediately.
A robot planning how to clean a room may have seconds to consider its next task. A robot adjusting its balance, avoiding a moving obstacle or responding to a person stepping into its path may have only milliseconds.
These are different computational problems, and they don’t necessarily belong in the same model.
We increasingly describe the architecture through four systems.
System 0: Control
The fast, deterministic machinery that keeps the robot physically functioning. Balance, actuation, stability and low-level control belong here.
System 1: Conduct
The continuous interpretation of movement and immediate physical relationships that helps a robot behave appropriately around others.
System 2: Task
The intelligence responsible for accomplishing the robot’s immediate objectives, including task execution and planning.
System 3: Deep reasoning
The slower, more computationally expensive intelligence available for complex reasoning, broader context and difficult decisions.
These systems need to work together, but they don’t need to operate at the same speed or carry the same computational burden.
A robot doesn’t need its largest reasoning model to reconsider every minor adjustment to its movement. Equally, a compact motion model shouldn’t be responsible for deciding the robot’s entire purpose.
We think Motion Intelligence has a natural place in System 1.
It provides a continuous interpretation of the moving world, identifying developing interactions and making that information available to the systems responsible for deciding and acting.
System 1 · Conduct / motion interpretation · tens to hundreds of milliseconds
System 2 · Task · hundreds of milliseconds to seconds
System 3 · Deep reasoning · seconds and beyond
The Robot Brain Is Splitting by Time →
Clock speeds are illustrative, not measured inference latency. Hold deep reasoning and the lower lanes keep running. MI can inform immediate Conduct and broader reasoning; the Cosmos experiment did not integrate or validate a complete controller.
There’s an important distinction here.
System 0 keeps the robot physically functioning. System 2 gives it something useful to accomplish.
System 1 helps determine what it’s like to be around while it does it.
And once robots become sufficiently capable of completing their tasks, that distinction may become one of the most important factors determining whether people accept them into everyday life.
Conduct is what makes a capable robot a good machine
Humans don’t become good citizens simply by being physically capable or competent at accomplishing tasks. We also learn how to behave around one another.
We learn when to give way, when to proceed, how much space to leave, how to recognise hesitation and how to anticipate another person’s movement without requiring them to explain their intentions. Much of this intelligence is expressed through conduct rather than explicit reasoning.
A considerate person doesn’t stop whenever another person appears nearby. They understand when to continue, when to slow down, when to yield and when to change course. Robots will need something comparable. This doesn’t mean giving robots human emotions or consciousness. It means giving them better ways to interpret the physical signals people already use when sharing space.
The objective isn’t a robot that always yields, always stops or always takes the safest-looking action. It’s a robot that can recognise what matters and respond appropriately.
This is why we believe Conduct deserves the same engineering attention as control and task performance. In some respects, it may ultimately matter more to adoption. A robot that can’t maintain its balance won’t be useful. A robot that can’t complete its assigned task won’t be commercially valuable.
But once those problems are sufficiently solved, Conduct may determine whether the machine is welcomed into the environments where people actually live and work.
Our Cosmos results don’t prove that Conduct problem has been solved. They demonstrate progress on an important prerequisite: giving a machine a better understanding of the movement occurring around it.
The next challenge is converting that understanding into behaviour that can be independently measured, tested and trusted.
Small intelligence, continuously available
There’s another reason we’re interested in separating these capabilities. Not every physical interaction should require a large foundation model to reconstruct the situation from first principles.
Consider getting out of bed in the morning and making your first coffee. Dolly Parton put it best: “Tumble out of bed and I stumble to the kitchen. Pour myself a cup of ambition. And yawn and stretch and try to come to life.”
You can move through your home, avoid furniture, navigate around another person and perform familiar actions without devoting your full attention to every physical decision. Your brain isn’t necessarily performing a fresh, elaborate reasoning exercise for every step. Much of that behaviour relies on learned, automatic processes, with more deliberate reasoning available when something unexpected happens.
We think robotics can benefit from a similar division of labour.
- Large models are valuable for broad understanding, complex reasoning and planning.
- Smaller, specialised models may be better suited to the continuous interpretation of movement and other time-sensitive information.
MI:1 began as a roughly 30-million-parameter model – around 30Mb, that you can experience live in your browser. Our subsequent MI:2 research has explored substantially more compact representations, including the 32-dimensional World State Vector used in this experiment.
The deployment architecture is still being qualified, so we aren’t claiming a final production model size from these research results, but the size in tiny, and the direction is deliberate:
Use specialised intelligence continuously. Bring broader intelligence to bear when the situation requires it.
For a robot operating around people, this could mean maintaining an inexpensive, continuous understanding of movement while a larger world model concentrates on broader reasoning and task-level decisions.
It’s a complementary architecture, rather than an attempt to make one enormous model responsible for everything.
What Cosmos 3 taught us
There is another important result in this work, and it belongs as much to NVIDIA’s Physical AI ecosystem as it does to SpatioTemporal:
Cosmos 3 proved sufficiently extensible that a small Australian research company could bring an entirely different representation of physical interaction to it and experimentally measure what happened. Physical AI is unlikely to develop through a single model architecture that solves every problem equally well.
Different forms of intelligence have different computational requirements, different sources of information and different timescales. Perception, physics, motion understanding, world modelling, planning and control all contribute something important.
The opportunity is to make those capabilities work together.
Our experiment suggests that movement can be represented explicitly and provided to a broader Physical AI model in a form that improves its interpretation of physical interactions. That’s the role we’re exploring for SpatioTemporal. We’re not building another robot company, replacing perception or competing with world models. We’re building Motion Intelligence as a specialised layer that existing Physical AI systems can use.
A simple way to describe the architecture is: See → Read → Reason → Act.
Perception sees the world. Motion Intelligence reads the movement within it. World models reason about the broader physical situation. Conduct and planning determine how the machine should respond.
Physics remains essential throughout, providing the deterministic understanding of geometry and mechanics that learned models shouldn’t need to rediscover.
Perception supplies observations directly to the broader stack. MI adds a motion representation. Physics supplies geometry and mechanics across the stack.
Conceptual integration surface. Packets show information flow, not measured processing speed or a completed production integration. Adding motion information augments the existing route.
For NVIDIA, the experiment illustrates how a broader Physical AI model can benefit from a specialised representation developed outside its own architecture.
For SpatioTemporal, it provides evidence that the motion layer we’ve been building can contribute useful information to a major foundation model.
For robotics more broadly, it raises a larger question about how physical intelligence should be organised.
Should movement become a native modality of Physical AI?
Modern AI systems work with increasingly rich representations of language, images, video, depth, geometry, actions and world state.
But movement is often treated as something to be inferred from those representations rather than a source of structured information in its own right.
Our research began with the idea that movement might have a learnable grammar.
MI:1 explored that grammar through Motion Tokens.
MI:2 extended the work towards relationships between multiple moving actors, using a compact World State Vector to represent how a scene develops through time.
The Cosmos experiment has now given us evidence that another Physical AI foundation model can make meaningful use of that representation.
And the results are substantial.
Relevance accuracy increased roughly eightfold. Interaction error fell by half. Anticipation error fell by 86%. Timing error fell by 89%.
We also deliberately disrupted the temporal history supplied through Motion Intelligence and found that correctly ordered movement performed better across all five Reading the Room measures.
That doesn’t establish a new universal modality, nor does it mean the broader problem of robot Conduct has been solved.
But it gives us a compelling reason to continue asking the question.
Should Motion become a native modality of Physical AI?
We believe the emerging generation of robots will need more than increasingly capable hands, better cameras and larger reasoning models.
They’ll need to understand the people and machines moving around them, recognise developing interactions and respond in ways that are appropriate to the shared physical world.
And they’ll need to demonstrate that capability through independent, repeatable testing.
The ultimate measure of success won’t be whether a robot can complete an impressive demonstration in isolation.
It’ll be whether that same robot can do its job reliably, competently and considerately in the environments where people actually need it.
Because humans shouldn’t have to learn how to behave around robots.
Robots need to learn how to behave around us.
That’s the larger purpose behind Motion Intelligence, the Conduct layer we’re working towards, and the evidence we’re beginning to establish through experiments such as this one.
We gave Cosmos 3 an explicit representation of movement.
It became dramatically better at reading the room.
Now we want to understand how far that capability can take Physical AI, and what it will take to turn better understanding into better machines.
Research conducted by SpatioTemporal using NVIDIA Cosmos 3, with compute support from NVIDIA. The results described are from controlled experimental benchmarks using constructed multi-agent worlds assembled from recorded motion trajectories. They aren’t measurements of deployed robot performance, and improvements in physical interpretation shouldn’t be interpreted as demonstrated improvements in real-world robot safety or Conduct.
Related SpatioTemporal research: The Robot Brain Is Splitting by Time explores the separation of robot intelligence by latency and function. The Robot Benchmark and The Robot Range explains the Three Cs and the independent testing infrastructure being developed for robotics. SpatioTemporal Research also includes our earlier work on World State Vectors, attention and trust in Physical AI.