This article was originally published at Analog Devices’ website. It is reprinted here with the permission of Analog Devices.
Training robots in the physical world is slow, expensive, and difficult to scale. Roboticists developing AI policies depend on high quality data—especially for complex tasks like picking up flexible objects or navigating cluttered environments. These tasks rely on data from sensors, motors, and other components used by the robot. Yet generating this data in the real world is time-consuming and requires extensive hardware infrastructure.
Simulation offers a scalable alternative. By running multiple robotic motion scenarios in parallel, teams can significantly reduce the time required for data collection. However, most simulations environments face a trade-off: performance or physical precision.
A model with near-perfect, real-world fidelity often requires vast amounts of computation and time. Such precise but slow simulations produce less data, reducing their usefulness. Instead, many developers choose simplifications that improve speed but result in a disconnect between training and deployment—commonly known as the sim-to-real gap. This means that robots trained solely in simulation will struggle in the real world. Their policies will be confused by actual sensor data that includes noise, interference, and flaws.
To address this challenge and accelerate simulation, Analog Devices developed a machine learning-based surrogate model. In our testing, the model simulated the behavior of an indirect time-of-flight (iToF) sensor with near-real-time performance, while preserving critical characteristics of the real sensor’s output. The model offers a true acceleration breakthrough in scalable, realistic training for robotic policies, and a path forward with complex simulation.
Simulating Sensors with Real-World Accuracy
iToF sensors, such as ADI’s ADTF3175, are common in robotic perception. These sensors emit light in a regular pattern to measure depth by calculating its reflection. In the real world, sensors exhibit readout noise, and accounting for this interference is essential for training reliable robotic policies. However, most simulation environments offer idealized sensor data. For example, NVIDIA’s Isaac Sim™ provides clean depth maps based on geometry, not the noisy output of real-world sensors.
To fill this gap, ADI had previously developed a physics-based simulator that modeled iToF sensor behavior at the pixel level. While accurate, the simulator was too slow for full-frame, real-time use. At just 0.008 frames per second (FPS), it was impractical for training AI policies that require thousands of scenes per second.
Using Machine Learning to Speed Up Simulation
The breakthrough came from using machine learning to emulate the high-fidelity simulator’s output. We trained a multilayer perceptron (MLP) model as a surrogate to approximate the behavior of the precise white-box simulator. Importantly, the team designed this stand-in model to learn not just the average output but also reflect the original’s variability and noise characteristics.
The surrogate model decomposes its task into three sub-tasks:
- Predict the expected depth measurement.
- Estimate the standard deviation, accounting for uncertainty.
- Predict whether a pixel’s depth measurement will be invalid or unresolved.
The surrogate model uses this probabilistic output to capture the essential stochastic behavior of the original simulator while dramatically accelerating inference. The result is a simulation that runs at 17 FPS. That’s fast enough for real-time use while maintaining approximately 1% error from the high-fidelity model.
Real-World Validation in Isaac Sim
After building the surrogate model, the team integrated it into NVIDIA’s Isaac Sim environment. Testing using a digital twin of a robot arm performing peg-insertion tasks showed that the model closely matched the original simulator’s output. The output even included the noise that was absent from standard simulations.
Real-world iToF sensors are sensitive to optical effects in the near-infrared (NIR) range, a property often ignored in standard simulations. Furthermore, iToF performance varies across different surface materials. To ensure the surrogate accounts for both behaviors, the team used fast surrogate inference and adjusted the NIR reflectivity of simulated objects to better match sensor behavior in physical experiments.
This technique helped reduce differences between simulation and real sensor data, particularly on matte surfaces. While imperfect, these adaptations made major strides to minimize the sim-to-real gap. The team is actively exploring additional improvements, including changes to the underlying physics models and
What’s Next: Improving Fidelity and Generalization 
This surrogate model serves as a baseline for enabling fast, realistic simulation of iToF sensors in robotic training workflows. But it’s only the first step. New work involves physics-informed neural operator (PINO) models to improve accuracy, reduce training data needs, and generalize across different scenes and tasks.
In the future, the aim is to eliminate the need for an intermediate white-box simulator. By training models directly on real-world sensor data, simulators could adapt more readily to diverse environments without requiring manual tuning or scene-specific calibration.
These developments could dramatically reduce the time and cost required to deploy robotics systems to real-world environments. Ideally, this work will advance deployments in logistics, manufacturing, product inspection, and beyond.
Philip Sharos, Principal Engineer, Edge AI
