NCore’s interactive 3D viewer showing camera frustums with images, lidar point clouds, 3D cuboid bounding boxes, and rig trajectory from an autonomous vehicle sequence.
This blog post was originally published at NVIDIA’s website. It is reprinted here with the permission of NVIDIA.
Autonomous vehicles, robotics, and broader physical AI research all depend on multi-sensor recordings—cameras, lidars, radars, often capturing at different frame rates and without hardware-synchronized triggers—along with their associated calibrations, poses, and annotations. Yet sensor data today is fragmented across incompatible formats. Every team, every dataset, and every pipeline defines its own conventions for coordinate systems, calibration representations, and storage layouts.
NVIDIA NCore is an open-source Python library that provides a canonical data representation for such multi-sensor recordings. It defines a modular, component-based format with rigorous coordinate conventions, GPU-accelerated sensor models, and a streamable, cluster-friendly storage format optimized for reconstruction use-cases—along with easy-to-use strongly typed data access APIs and associated tools—giving researchers and engineers a single, well-defined foundation for any multi-sensor workflow.
The goal of NCore is to establish a canonical interchange format for multi-sensor recordings: by normalizing diverse sensor configurations, coordinate systems, and calibration representations into a single unambiguous representation, NCore makes it possible to build tools, models, and pipelines that work across datasets and platforms. From this core purpose, two important consequences follow. NCore serves as the data backbone for NVIDIA NuRec—any NuRec user can convert their own proprietary fleet data into NCore as the common ingestion layer for neural reconstruction and simulation workflows—across autonomous vehicles and other physical AI domains. And because the format is open and well-defined, it is equally adopted by independent research projects: NCore dataset loading is already integrated into popular reconstruction engines like 3DGRUT and gsplat, and built-in converters for datasets like Waymo Open, COLMAP/ScanNet++, the NVIDIA Physical AI Dataset (with NCore pre-conversions available), and the PPISP dataset make adoption straightforward.
These goals shape NCore’s layered architecture—from dataset converters at the ingestion layer, through a canonical component-based format with non-redundant storage, to GPU-accelerated sensor models and easy-to-use data access APIs consumed by downstream reconstruction engines:
Component-Based Data Format
NCore organizes datasets as independent components—each containing one aspect of a recording, such as camera images, lidar point clouds, poses, intrinsics, or bounding-box labels. Components are self-contained and can be stored, versioned, and combined independently. This modularity means projects can update calibrations without rewriting image data, add new annotation sets alongside existing ones, or mix components from different processing stages.
The following built-in component types cover the most common multi-sensor recording needs:
| Component | Description |
|---|---|
PosesComponent |
SE(3) pose graph with microsecond timestamps—world poses, sensor-to-rig extrinsics, and optional global ECEF anchor |
IntrinsicsComponent |
Intrinsic calibration parameters for all sensor types—cameras, lidars, and windshield distortion models |
CameraSensorComponent |
Compressed image frames with per-frame timestamps and optional per-frame masks and annotations |
LidarSensorComponent |
Structured ray bundles with per-ray timestamps, distances, intensities, and model elements—native multi-return support |
RadarSensorComponent |
Radar detections with per-point direction, distance, radial velocity, RCS, and SNR |
PointCloudsComponent |
Pre-computed 3D point clouds (e.g., SfM or dense MVS) with per-point position, color, and arbitrary attributes |
CameraLabelsComponent |
Typed per-camera image labels—semantic segmentation, depth, normals, and custom label types with structured schema and quantization |
CuboidLabelComponent |
3D oriented bounding boxes with class labels, track IDs, and attributes |
MaskComponent |
Per-camera static binary masks (e.g., ego-vehicle occlusion regions) |
Custom component types can be defined easily by implementing corresponding reader/writer pairs—allowing projects to extend NCore with application-specific data without modifying the core library.
All component stores are accessed through the same API regardless of where they live. Via UPath and fsspec, NCore reads and writes data transparently from local disk, Amazon S3, Google Cloud Storage, or Azure Blob Storage—with no code changes. The single-file format is particularly well-suited to cluster and distributed training environments, where data can be streamed efficiently from cloud storage without requiring local extraction.
Design Principles
All NCore components follow a core design principle: fully non-redundant data storage. No signal is stored that can be computed from other available signals. This eliminates consistency issues caused by pre-baked computations, simplifies the contract for data producers, and ensures that components remain interchangeable and independent of each other.
This principle shapes several concrete design decisions:
- Raw sensor measurements. Lidar and radar sensors store raw ray bundles—per-ray direction vectors, timestamps, and scalar return distances—rather than pre-computed motion-compensated point clouds. Motion compensation is performed on-demand by the consumer using the pose graph and sensor models. This means egomotion estimates or extrinsic calibrations can be swapped freely without rewriting any sensor data.
- Pose graph. All spatial relationships are represented in a unified pose graph—a tree of named coordinate frames connected by static (e.g., sensor extrinsic calibrations) or dynamic (e.g., ego-vehicle trajectory) SE(3) edges. This makes NCore equally suited to rigidly mounted sensor rigs with known extrinsics and to free-posed datasets like COLMAP/ScanNet++ where each frame has an independent pose—the same API and tools work transparently for both.
- Sensor-independent labels. Cuboid annotations are stored in their own component with explicit reference frame declarations, not coupled to any particular sensor type. This allows labels to exist for camera-only, lidar-only, or any sensor configuration without workarounds.
- Component independence. Each component can be loaded, updated, or replaced without touching any other component. Multiple instances of the same component type—such as different calibration sources or label sets—can coexist as separate component groups, selectable at runtime.
High-Performance Storage
NCore defines a custom (indexed tar) container format as a single-file representation of component stores, tailored to multi-sensor dataset workloads. The format packages zarr chunks as sequential tar members in a single file and appends a compressed index at the end, combining the streaming efficiency of tar with random-access capability.
Regular tar files (top) support linear streaming but no random access. NCore’s indexed tar format (bottom) appends a compressed index, enabling O(1) key lookups and direct seeks to any chunk.
On a synthetic benchmark of 1,000 JPEG images at 2K and 4K resolution (~4.5 GB), significantly outperforms HDF5, Parquet, and WebDataset—both on local SSD (~9.8 GB/s sequential, ~9.5 GB/s random access) and when streaming from S3 (53 MB/s sequential, 16 MB/s random access with a 240 ms time-to-first-read):
 |
 |
|---|
Read throughput on local SSD (left) and streaming from S3 (right). Full bar = sequential; inner bar = random access. See local and S3 benchmark details.
The format is especially effective for cloud-stored data: its compressed trailer index enables a 240 ms time-to-first-read over S3—compared to 119 seconds for tarfile (which must scan all headers over the network) and 8.7 seconds for Parquet (which reads the footer). A single file with an appended index avoids the large number of small HTTP requests that directory-based stores would incur, while still supporting efficient random access via byte-range requests.
GPU-Accelerated Sensor Models
NCore includes PyTorch-based camera and lidar models that run on CUDA, enabling high-throughput batch projection and unprojection directly on the GPU. All sensor models support rolling-shutter-aware projection—interpolating between start-of-frame and end-of-frame poses on a per-scanline basis—which is critical for accurate sensor-to-sensor alignment in moving vehicles.
| Type | Sensor | Description |
|---|---|---|
ftheta |
Camera | NVIDIA polynomial model covering perspective through ultra-wide FOV |
opencv-pinhole |
Camera | Standard pinhole with rational, tangential, and thin-prism distortion |
opencv-fisheye |
Camera | Kannala–Brandt equidistant model (4 coefficients) |
bivariate-windshield |
Camera (external) | Bivariate polynomial windshield refraction distortion |
row-offset-spinning |
Lidar | Structured spinning lidar model (e.g., Hesai Pandar P128) |
The following collage shows rolling-shutter-compensated lidar-to-camera projections across ten non-synchronized rolling-shutter cameras of an autonomous vehicle, computed using NCore’s GPU-accelerated sensor models:
Rolling-shutter-aware lidar-to-camera projections across 10 non-synchronized rolling-shutter cameras of an autonomous vehicle, computed on GPU using NCore sensor models. Point color encodes range.
Dataset Converters and Tools
NCore currently includes built-in converters for Waymo Open, COLMAP/ScanNet++, and NVIDIA Physical AI datasets, with additional converters actively being developed for release in subsequent updates. Each converter transforms source data into NCore’s canonical component format. An extensible converter base class makes it straightforward to add support for any new data source—including proprietary fleet recordings or custom robotics data.
 |
 |
|---|
Point-cloud-to-camera projections from two different dataset sources: Waymo Open (left) and ScanNet++ (right)—both loaded through NCore’s unified API and rendered with NCore’s visualization tools.
An interactive 3D viewer built on viser provides browser-based visualization of camera frustums, lidar and radar point clouds, cuboid bounding boxes, and rig trajectories. Additional CLI tools export point clouds to PLY, camera frames to images or video, and colored point clouds with lidar-to-camera coloring.
 |
 |
|---|
Rolling-shutter-aware point-cloud-to-camera projections on Physical-AI-AV data: lidar projection (left) and radar detections colored by range distance (right).
From raw recordings to reconstruction and beyond. Because NCore is open and format-agnostic, it serves as the data layer for workflows spanning the full spectrum—from proprietary fleet data and robotics recordings ingested through NVIDIA NuRec to independent research projects training NeRF and 3DGS models. The existing converters also serve as examples for writing custom conversions—the extensible converter framework makes it straightforward to add support for any new data source.
Quickstart
$ pip install nvidia-ncore
from ncore.data.v4 import SequenceComponentGroupsReader, SequenceLoaderV4
# Load a multi-sensor sequence from a component group
reader = SequenceComponentGroupsReader(["sequence.json"])
loader = SequenceLoaderV4(reader)
# Discover available sensors
print(loader.camera_ids) # ['camera_front_wide_120fov', ...]
print(loader.lidar_ids) # ['lidar_gt_top_p128']
print(loader.radar_ids) # ['radar_front_center', ...]
# Retrieve a camera frame as an RGB array
image = loader.get_camera_sensor("camera_front_wide_120fov").get_frame_image_array(0)
See the API reference and tutorials for detailed usage.
Get Started with NCore
NCore is open source under the Apache 2.0 license. Install from PyPI, convert your datasets, and start building.
GitHub · PyPI · Documentation
Also refer to the following resources:
- NCore on GitHub—source code, issues, and contributions
- nvidia-ncore on PyPI—install with
- NCore documentation—data format specification, API reference, tutorials, and tools
- NVIDIA NuRec documentation—neural reconstruction workflows using NCore data
- 3DGRUT—NVIDIA’s 3D Gaussian and ray-tracing reconstruction engine (training on NCore datasets)
- gsplat—high-performance Gaussian splatting library (NCore usage guide)
- PPISP—physically-plausible ISP correction for radiance fields (NCore dataset)
- NVIDIA Physical AI Dataset—multi-sensor AV recordings on HuggingFace (NCore pre-conversions)
NCore Contributors
Sérgio Agostinho, Emanuel Attia, Vincent Caux-Brisebois, Antoine Chauveau, Sanja Fidler, Zan Gojcic, Jiahui Huang, Shengyu Huang, Umer Javed, Joey Kang, Ruilong Li, Nick Lucas, Riccardo de Lutio, Janick Martinez Esturo, Michael Shelley, Jonas Toelke, Michal Tyszkiewicz, Qi Wu, Lex Yang, Kangxue Yin, Sipeng Zhang, Tyler Zhu.
