When a humanoid robot reaches out and picks up a cup, you're watching roughly fifty milliseconds of hardware doing something visible. What you're not watching is the several layers of software that decided where the cup was, calculated how the arm should move, sent precise electrical signals to a dozen motors, monitored whether those motors were actually doing what they were asked, and prepared to catch an error if they weren't — all before the fingers made contact.
The hardware gets most of the coverage. The actuators (the motors and mechanical drives that move the robot's joints), the sensors, the chassis — these are what show up in promotional materials and funding announcements. The software stack is harder to photograph and harder to explain. It is also where most of the genuinely difficult problems in humanoid robotics live right now.
Understanding how these layers work — and where they tend to break — is useful context for evaluating almost any humanoid robotics claim you'll read.
The Stack, From the Bottom Up
It's useful to think of a humanoid robot's software as a set of layers, each building on the one below it. The analogy isn't perfect — in practice, these layers communicate in both directions and the boundaries between them are blurry — but it captures the basic structure.
At the bottom is motor control: the software that sends commands directly to each joint and reads back what the joint is actually doing. This layer runs at very high frequency — typically several hundred to several thousand times per second — because the physical system can go wrong quickly. A joint that's moving slightly wrong needs to be corrected within milliseconds, not seconds. Motor control software is usually written in low-level languages close to the hardware, and the engineering tradeoffs here are largely about speed, reliability, and how gracefully the system handles unexpected resistance or failure.
Above that sits whole-body control: the software that coordinates all of the joints together to produce stable, intentional movement. This is where balance lives. Walking on two legs requires constant tiny adjustments across the whole body — shifting weight, compensating for uneven surfaces, adapting to the robot's own momentum. Whole-body control algorithms (sometimes called whole-body controllers, or WBCs) take a high-level instruction like "move the right arm forward" and figure out how every other joint in the body needs to respond so the robot doesn't fall over in the process. This is computationally expensive and genuinely hard to get right, particularly in dynamic environments where the robot is moving quickly or the surface underfoot is unpredictable.
The next layer is motion planning: given a goal (reach point A, avoid obstacle B), what sequence of movements should the robot execute? Motion planning for humanoids is more complex than for a stationary robotic arm because the robot's own body is an obstacle — the arm can collide with the torso, the legs, or the other arm. Planning algorithms have to reason about the geometry of the whole body across the full sequence of a movement, not just the endpoint. In cluttered real-world environments, this becomes a significant computational challenge.
Then there is task execution: the logic that decides what to do and in what order. Should the robot pick up the nearest tote, or the one in the target zone? What happens if the tote isn't where it's supposed to be? Task execution software typically runs as a state machine — a structured set of states (searching, approaching, grasping, carrying, depositing) with rules governing transitions between them. For narrowly-defined, well-structured tasks like warehouse tote handling, this layer can be relatively simple and reliable. For general-purpose tasks in unstructured environments, it is where things go wrong most often.
At the top of the stack is perception: making sense of what the sensors are seeing. Cameras, depth sensors, lidar — all of these produce raw data that needs to be interpreted. Where is the cup? What orientation is it in? Is that object a box or a hand? Is that surface flat enough to step on? Perception is the layer where advances in machine learning have made the most visible impact in recent years. Modern humanoid robots use neural networks trained on large datasets to recognise objects, estimate poses, and segment scenes — essentially the same underlying technology as computer vision systems used in other domains, adapted for the specific requirements of embodied robot perception.
Where the Problems Actually Are
Each layer has characteristic failure modes, and understanding them helps explain why robots that look capable in demonstrations struggle in deployment.
Motor control and whole-body control are the most mature layers. The fundamental physics of balance and locomotion are well-understood, and the engineering of robust controllers for walking on flat and mildly uneven surfaces is largely solved — or at least solved well enough for current deployment contexts. This is why the walking in modern humanoid robot videos looks convincing: that part of the stack is genuinely good.
Perception is improving rapidly, but it remains brittle in ways that matter for deployment. A robot trained to recognise objects in a warehouse environment may fail to recognise the same objects in different lighting conditions, or when they are partially occluded, or when they are oriented in a way the training data didn't include. The failure modes are not random — they're systematic, tied to the distribution of situations the system was trained on — but they're hard to anticipate in advance. This is why robots that perform well in controlled demonstrations can behave unexpectedly when something in the environment changes.
Task execution is where the gap between narrow deployment and general capability is most apparent. For a robot doing exactly one task in a consistent environment, a carefully designed state machine is robust and reliable. For a robot asked to do many different tasks, or the same task in environments that vary significantly, the combinatorial complexity of "what to do when things don't go as expected" grows quickly. This is part of why current commercial deployments are deliberately narrow: not because the hardware can't do more, but because the task execution software is most reliable when the task is tightly defined.
The Role of Learning
The past three years have seen significant interest in replacing or augmenting traditional hand-coded task execution and motion planning with systems trained using machine learning — specifically, techniques that allow robots to learn behaviours from demonstration data or from practice in simulation.
The most discussed approach is imitation learning: recording a human demonstrating a task (by teleoperation — remotely controlling the robot — or by capturing human motion with sensors) and training a neural network to reproduce the demonstrated behaviour. The appeal is that it sidesteps the need to hand-code every decision and edge case. The challenge is that imitation learning systems tend to fail on situations outside their training distribution, and collecting sufficient training data for complex, varied tasks is expensive and time-consuming.
Reinforcement learning — training a robot by having it practice a task repeatedly and learn from success and failure — is most tractable in simulation, where a robot can attempt a task millions of times without wearing out hardware. The challenge is the "sim-to-real gap": behaviours learned in simulation don't always transfer to physical robots, because simulations are imperfect models of the physical world. Reducing this gap is an active area of research, and progress is real, but it remains a meaningful constraint on how much simulation-trained behaviour translates reliably to deployment.
Several companies — Physical Intelligence, which raised $400 million in late 2024, is the most prominent — are working on foundation models for robot behaviour: large neural networks trained on diverse robot data that can be adapted to specific tasks with relatively little additional training. This is the most ambitious bet in the field right now. Whether it produces the kind of general-purpose robotic capability that the demos suggest is a genuinely open question, and the answer will likely take several more years to become clear.
Why This Matters for Evaluating Claims
When a company releases a video of a humanoid robot performing a task fluidly, the software stack is almost never discussed. What you can't see from the video: whether the robot is operating autonomously or with human teleoperation, how many attempts were made before the shown take, how narrow the task definition is, what the failure rate is in less controlled conditions, and which layers of the stack are genuinely novel versus using well-established techniques.
None of this means the demonstrations aren't real or the progress isn't genuine. It means that the interesting questions about where a given humanoid system actually is — and how close it is to the general-purpose capability being implied — almost always come down to specifics about the software layers described above. A robot with excellent locomotion but brittle perception is a different product from a robot with capable perception but limited task execution. Both might produce impressive demonstration videos.
The companies that will prove out commercial viability over the next several years are the ones that can build software stacks reliable enough to operate in real environments over extended periods, not just in controlled conditions for a camera. That's a harder problem than the demos suggest — and a more interesting one.