Inside Robot Data Storage: MCAP, Zarr, and Multimodal Datasets at Scale

Introduction

In this post, I'll gradually introduce all of the core components and architectural patterns that make up a modern, production-grade robot data storage system. In particular, I'll be doing an under-the-hood breakdown of how multimodal robot data actually gets stored, from the moment it's captured on-robot to the moment it's loaded into a training batch: MCAP, Zarr, sharding, featurization, and the layout patterns that keep fleet-scale corpora queryable.

This post is the second in a series, and it picks up exactly where the last one left off. Inside ROS 2 covered the communication fabric—how data moves between nodes while the robot is running. This post covers what happens to that data once it stops moving: how a live stream of joint states, camera frames, and policy outputs becomes a durable, indexed, randomly-accessible dataset that a training job can pull batches from a year later, at a thousand times the scale. Like the last post, this one starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without drowning in boilerplate code.

Structure

1. The Problem: Why robot data breaks conventional storage assumptions—multimodal, variable-rate, and read/write patterns that pull in opposite directions.
2. Log-Time Storage: MCAP internals—how data gets captured durably and sequentially as it's generated.
3. Dataset-Time Storage: Zarr internals—how captured data gets reshaped into a chunked, randomly-accessible array format for training.
4. Featurization: The ETL boundary between logs and datasets—resampling, synchronization, and encoding decisions.
5. Scaling Out: Sharding, cloud object storage, and indexing for fleet-scale corpora.
6. Training-Time Access: Dataloaders, chunk-aware shuffling, and prefetching that keep GPUs fed.
7. Production Patterns & Alternatives: Where production patterns (LeRobotDataset, RLDS) and academic formats (HDF5, WebDataset, TFRecord) diverge, and when to reach for each.

Notes & Assumptions

• Target Audience: ML engineers and infra engineers building training pipelines on top of fleet data, as well as roboticists who own the collection side and want to understand what happens to their logs downstream.
• Language Targets: Code examples are Python throughout—this is offline, training-side infrastructure rather than the real-time C++ paths covered in the previous post.
• Scope Boundary: This post draws a hard line between log capture (rosbag2/MCAP, the write path) and dataset storage (Zarr and friends, the read path). The featurization section is the bridge between the two, and understanding that boundary is the key to the whole post.
• This post assumes familiarity with the communication fabric from Inside ROS 2—in particular, rosbag2's role in recording DDS topics to disk, and the stream-synchronization problem introduced there, which resurfaces here as a batch ETL concern rather than a live callback concern.
• Code Versions: Zarr code examples target the Zarr v3 sharding API. Zarr's storage and codec API surface has moved during v3 standardization, so treat method names as illustrative of the pattern and verify against your pinned zarr version before running.

1. The Problem: Why Robot Data Breaks Conventional Storage

Before diving into formats, it's worth establishing exactly what makes robot data hard to store well. It is not simply "big data" in the generic sense—it is data with two fundamentally incompatible access patterns imposed on it at different points in its life.

At collection time, data arrives as an append-only, sequential stream. A robot in the field is generating RGB frames at 30 Hz, joint encoder readings at 500 Hz, force-torque readings, LIDAR sweeps, and sparse language annotations, all simultaneously, all needing to hit disk before the buffer overflows. The write pattern is strictly sequential and cannot block: if the storage layer stalls, you drop packets or stall the control loop that's producing them.

At training time, the access pattern inverts completely. A dataloader wants to pull a randomly-shuffled batch of (episode, timestep) windows from across the entire corpus—episode 4,201's frames 800–812, episode 19 frames 55–67, episode 300,000's frames 12–24—with no relationship to the order any of it was originally written in. The read pattern is random-access and latency-sensitive: if fetching a single training sample is slow, the GPU sits idle.

Heterogeneous Rates, Heterogeneous Types

Layered on top of that read/write inversion is the multimodal problem itself. A single episode contains:

• Images and point clouds: large, dense, compressible tensors arriving at 15–60 Hz.
• Proprioceptive state: small, high-frequency scalars and vectors arriving at 100–1000 Hz.
• Language and task metadata: sparse, low-frequency, often just one annotation per episode.
• Derived signals: actions, rewards, success labels—computed after the fact, not captured directly.

Each of these has a different natural storage shape, a different compression profile, and a different access granularity. A format tuned for one (say, a message bus optimized for heterogeneous, timestamped records) is a poor fit for the other (a format optimized for homogeneous, chunked numerical arrays sliced along a shared time axis).

Resolving the Tension: Two Formats, One Pipeline

Trying to serve both access patterns out of a single format is where most homegrown robot data stacks go wrong. You end up either with a message-log format that's brutal to randomly slice for training, or a training-optimized array format that can't durably absorb a live, crash-prone, multi-topic write stream.

The dominant production pattern splits the problem in two, at the ETL boundary:

• Log-time storage (MCAP): optimized for sequential, crash-safe, self-describing capture, and for coarse time-range queries during debugging and replay.
• Dataset-time storage (Zarr): optimized for chunked, compressed, random-access reads at training time, sharded and indexed for fleet scale.

Everything in the rest of this post follows that split: first the capture format, then the dataset format, then the ETL layer that turns one into the other, then how that dataset format survives being scaled to a fleet and consumed by a training job.

2. Log-Time Storage: MCAP

The previous post touched on MCAP in passing, as the storage backend rosbag2 writes to. Here we open it up properly, because the design decisions baked into this format are what make fleet-scale capture possible in the first place.

MCAP is a self-describing, message-oriented container format, designed by Foxglove specifically to address the shortcomings of both the earlier rosbag1 .bag format and the SQLite3-backed rosbag2 default—more a clean-slate response to both formats' failure modes than a direct evolution of either. "Self-describing" is the operative property: an MCAP file carries its own schema definitions inline, so you can open a file recorded two years ago on a since-deprecated message type and still decode it without a separately-versioned schema registry. That matters enormously at fleet scale, where you cannot guarantee every consumer of a log has the exact interface definitions the producer had.

File Anatomy

An MCAP file is organized into a small number of record types, laid out to support both fast sequential writes and fast indexed reads of the same file:

• Schema and Channel records: registration records that map a topic name to a message type definition—the same schema/channel model that underlies ROS 2's .msg interfaces, serialized once and referenced by every subsequent message on that channel rather than repeated per-message.
• Chunks: the core storage unit. Messages are buffered and written in batches, then compressed as a unit (zstd or lz4) and flushed to disk. Chunking is what makes MCAP crash-safe under streaming writes—each chunk is a self-contained, independently-decompressible unit, so a crash mid-recording loses at most the chunk still buffered in memory, not the file.
• Chunk Index / Summary section: written at the end of the file (or periodically, for very long recordings), this gives $O(\log n)$ seeking by timestamp or by channel without a full linear scan—critical for pulling a 10-second window out of a multi-hour log during debugging.
• Attachments: a side channel for non-message blobs—calibration files, checkpoint snapshots, config dumps—stored alongside the message data they correspond to.

Why MCAP, Specifically, for This Stage

It's worth being precise about what MCAP is optimized for, because it is not optimized for what comes later in this post. Relative to its predecessors:

• vs. rosbag1 (.bag): no built-in per-chunk compression, weaker indexing, and a format that couples tightly to ROS 1's message definitions.
• vs. rosbag2-over-SQLite3: SQLite3 is a general-purpose transactional database, not a purpose-built log format—write throughput suffers under the row-level overhead of a database engine when you're inserting multi-megabyte image blobs at 60 Hz.
• vs. HDF5: HDF5 supports concurrent writers poorly and has no native crash-safety story for a single writer being killed mid-append, which is precisely the failure mode a robot in the field experiences.

MCAP's design center is streaming, crash-safe, self-describing capture with reasonable range-query performance—not fine-grained random access to individual samples. That distinction is the seam the rest of this post is built around.

from mcap.reader import make_reader
from mcap.writer import Writer

# Writing: chunked, compressed, streaming-safe
with open("episode_0042.mcap", "wb") as f:
    writer = Writer(f, chunk_size=1024 * 1024, compression="zstd")
    writer.start()

    schema_id = writer.register_schema(
        name="sensor_msgs/msg/JointState",
        encoding="ros2msg",
        data=joint_state_schema_bytes,
    )
    channel_id = writer.register_channel(
        schema_id=schema_id, topic="/joint_states", message_encoding="cdr"
    )

    for t, msg_bytes in joint_state_stream:
        writer.add_message(
            channel_id=channel_id, log_time=t, publish_time=t, data=msg_bytes
        )

    writer.finish()

# Reading: indexed time-range seek, no full-file scan
with open("episode_0042.mcap", "rb") as f:
    reader = make_reader(f)
    for schema, channel, message in reader.iter_messages(
        topics=["/joint_states"], start_time=t_start, end_time=t_end
    ):
        decode_and_process(message.data)

3. Dataset-Time Storage: Zarr

If MCAP is optimized around the message—a timestamped, typed record on a named channel—Zarr is optimized around the array. That shift in the fundamental unit is the entire reason a second format exists: training doesn't want to replay a channel of discrete messages, it wants to slice a tensor.

Zarr is a chunked, compressed, N-dimensional array storage format with a hierarchical, JSON-described group structure. Where MCAP's atomic unit is a chunk of interleaved, heterogeneous messages, Zarr's atomic unit is a chunk of one homogeneous array—a rectangular block of a single dtype and shape, addressable by index. That's what makes dataset["episode_4201"]["image"][800:812] a cheap, targeted read instead of a full-chunk decompress-and-scan.

Core Mechanics

• Chunking: every array is split into fixed-size chunks along each dimension, and each chunk is compressed and stored independently. Chunk shape is a first-class design decision, not an implementation detail—more on this below.
• Codecs: per-chunk compression (blosc, zstd, and others) trades CPU decompression cost against storage size and I/O volume. Because compression is per-chunk rather than per-file, reading one chunk never requires decompressing its neighbors.
• Store abstraction: Zarr separates the logical array structure from the physical storage backend via a pluggable "store" interface. The exact same array can be backed by a local filesystem, an in-memory dict, or an S3/GCS bucket, with no change to how it's read or written. This abstraction is what makes the sharding and cloud-scale section later in this post possible without a format change.
• Groups: arrays are organized into a hierarchical tree of named groups, closely mirroring a directory structure—e.g., a root group containing one subgroup per episode, each holding sibling arrays for image, state, and action.

Chunk Shape Is the Design Decision

Chunk shape is to Zarr what QoS was to the communication fabric in the last post—a single configuration surface that, tuned wrong, silently degrades everything downstream. Two failure modes sit on either side of it:

• Chunks too large relative to your access pattern: a training loop that reads 16-frame windows out of an array chunked in blocks of 1,000 frames pulls and decompresses 1,000 frames to serve 16, wasting I/O and CPU on every batch.
• Chunks too small relative to your access pattern: an array chunked at 1 frame per chunk turns every batch read into hundreds of tiny independent chunk fetches—cheap on a local SSD, expensive once that store is an S3 bucket charging and rate-limiting per request (a point the sharding section develops further).

The rule of thumb is to align chunk boundaries with your training-time observation horizon: if your policy trains on windows of length $H$, chunking the time axis in multiples of $H$ means a typical batch read touches a small, bounded number of chunks instead of scanning across chunk boundaries on every sample.

Worked example: a $224 \times 224 \times 3$ uint8 image frame is ~150KB uncompressed. At $H=16$, a chunk of shape $(16, 224, 224, 3)$ is ~2.4MB before compression—small enough that a handful of chunks comfortably fits in a prefetch buffer, large enough that you're not issuing a separate object-store request per frame. That 2.4MB figure is also the number the shard-sizing worked example later in this post builds on directly.

import zarr
import numpy as np

# Create a per-episode group with modality arrays chunked along the time axis
store = zarr.storage.LocalStore("dataset.zarr")
root = zarr.group(store=store)
episode = root.create_group("episode_0042")

# Chunk shape aligned to a 16-frame training horizon (H=16)
images = episode.create_array(
    "image", shape=(1200, 224, 224, 3), chunks=(16, 224, 224, 3),
    dtype="uint8", compressors=zarr.codecs.BloscCodec(cname="zstd", clevel=3),
)
state = episode.create_array(
    "state", shape=(1200, 14), chunks=(16, 14), dtype="float32",
)
action = episode.create_array(
    "action", shape=(1200, 7), chunks=(16, 7), dtype="float32",
)

# Random-access slice read: touches exactly one chunk per array
window = images[800:816]  # frames 800-815, a single chunk boundary-aligned read

4. From Logs to Datasets: Featurization and the ETL Boundary

Between the message-shaped world of MCAP and the array-shaped world of Zarr sits an ETL job that has to reconcile the two. This is the featurization pipeline: a batch process that reads raw episode logs, resamples and synchronizes their heterogeneous streams onto a common timeline, encodes each modality into its final training representation, and writes the result into a Zarr store.

It's worth naming the callback explicitly: this is the same stream-synchronization problem introduced in Inside ROS 2's discussion of message_filters and approximate-time coalescing—30 Hz camera frames against 500 Hz joint encoders against a 10 Hz policy loop. There, it was a live problem solved with a sliding-window callback on a running node. Here, it's a batch problem, solved once per episode over the full recorded log rather than once per timestep in real time, but it is the identical underlying misalignment.

The Decisions That Get Baked In Here

Featurization is not a mechanical format conversion—it's where a set of modeling decisions get permanently baked into the dataset, and getting them wrong means a full re-ETL of the corpus later. (This is exactly the gap that dataset versioning and lineage tracking exist to close—see the note in the epilogue.)

• Resampling: choosing a common timeline (often the lowest-rate meaningful signal, or a fixed target Hz) and an interpolation/hold policy for every other stream to align to it.
• Image encoding: raw uint8 arrays in-chunk, JPEG/PNG bytes stored as a compressed byte array, or precomputed vision-backbone embeddings—each a different point on the storage-size vs. training-time-compute tradeoff.
• Action/state representation: absolute vs. delta actions, normalization statistics (mean/std or min/max) computed corpus-wide and stored as dataset-level metadata rather than recomputed per-batch at training time.
• Episode boundaries and annotation: attaching language instructions, task IDs, and success/failure labels at the episode or segment level, and deciding how episode boundaries in the log map to episode groups in the Zarr store.

The ETL Boundary in One Sentence

Once this stage completes, every downstream consumer—training jobs, evaluation scripts, dataset visualizers—reads exclusively from the Zarr side of the boundary. Raw MCAP logs remain the source of truth for re-processing if a featurization decision needs to change, but they stop being on the critical path for training the moment a clean, versioned Zarr dataset exists. That handoff is what lets the next two sections—sharding and dataloading—treat the dataset as a stable, array-shaped artifact rather than a moving target.

5. Scaling Out: Sharding, Cloud Object Storage, and Indexing

Everything up to this point has implicitly assumed a single Zarr store on a local disk. That assumption breaks the moment you go from "one episode" to "a fleet": hundreds of robots, millions of episodes, and a corpus that no longer fits on any one machine's disk, let alone in memory. This section covers three things that change together once the store has to live on cloud object storage and be written and read by many processes at once: how chunks get physically packed (sharding), how metadata gets consolidated, and how episodes get found in the first place (indexing).

Why a Single Zarr Store Breaks Down

The chunking model from the last section, taken naively to fleet scale, has a failure mode: a chunk-per-object storage layout means a corpus of millions of episodes translates into tens or hundreds of millions of individual small objects. Three problems compound at that scale:

• Metadata pressure: opening the dataset means resolving the array metadata for every one of those groups—thousands of small GET requests before a single training sample is read.
• Object-store request cost: cloud object storage bills and rate-limits per request, not just per byte. A chunk-per-object layout that was free on local disk becomes a real dollar cost and a real throughput ceiling on S3 or GCS.
• Single-writer bottlenecks: a naive shared store with many ETL workers writing concurrently invites metadata races and partial-write hazards that a single local writer never had to contend with.

Sharding: Consolidating Chunks into Fewer Objects

Zarr's sharding codec addresses the small-file problem directly: instead of one chunk mapping to one physical object, many chunks are packed into a single larger "shard" file on the store, with an internal index locating each chunk's byte offset within it. The logical chunk boundaries used for reading and writing stay exactly what they were in the previous section—chunk shape is still tuned to the training-time access pattern—but the physical object count on the store drops by orders of magnitude, because sharding is a packing decision at the storage layer, not a change to how data is sliced.

This is the same move MCAP made with chunks-of-messages, applied one layer up: batch many small logical units into fewer physically-addressable containers to amortize per-object overhead.

Alongside sharding, consolidated metadata—aggregating every array's and group's metadata into a single retrievable document rather than one object per array or group—solves the metadata-pressure problem: opening a fleet-scale dataset becomes one request instead of thousands.

On write safety: MCAP's crash-safety story doesn't automatically carry over to sharded Zarr—a shard is a single physical object that many logical chunks depend on, so a worker killed mid-write to a shard risks that shard's other, already-valid chunks along with it. Production ETL pipelines handle this the same way most object-store writers do: write the shard to a temporary key, and only publish it (via an atomic rename or a final metadata update) once the write completes and is verified—never write in place. This is a deliberate scope cut for the rest of this post, but worth flagging since MCAP's crash-safety was described earlier and it would be misleading to let that guarantee seem to extend to the sharding layer by omission.

import zarr

# Sharded array: logical chunking unchanged, physical objects consolidated
store = zarr.storage.FsspecStore.from_url("s3://fleet-data/episodes.zarr")
root = zarr.group(store=store)

images = root.create_array(
    "episode_0042/image",
    shape=(1200, 224, 224, 3),
    chunks=(16, 224, 224, 3),      # logical chunk: unchanged from before
    shards=(160, 224, 224, 3),     # 10 chunks packed per physical object on S3
    dtype="uint8",
    compressors=zarr.codecs.BloscCodec(cname="zstd", clevel=3),
)

# Consolidate all group/array metadata into a single retrievable document
zarr.consolidate_metadata(store)

# Opening the dataset now costs one metadata request, not thousands
root = zarr.open_consolidated(store, mode="r")

One caveat worth flagging on the consolidation step: unlike sharding, which is a finalized part of the Zarr v3 core spec, consolidated metadata is still an experimental extension for v3 stores at the time of writing—it isn't yet part of the formal v3 specification, and other Zarr implementations aren't guaranteed to support it. It's safe to rely on within a single, consistent zarr-python deployment, but worth checking before assuming interoperability with other tooling reading the same store.

The Third Pillar: Indexing, for Filtering Without Opening Every Shard

Sharding and consolidated metadata solve how the bytes are laid out on the store, but not which episodes a given training run should even look at—that's a distinct problem, and it's the third pillar of scaling out alongside sharding and metadata consolidation, not an afterthought to either. A fleet corpus needs a separate, lightweight index—typically a Parquet or SQLite table living alongside the Zarr store—mapping episode ID to its shard location plus queryable attributes: robot embodiment, task label, collection date, a quality-filter flag from an offline QA pass.

Training jobs query this index first ("give me every episode where embodiment == 'arm_v3' and qa_pass == True") and only then touch the Zarr store, pulling exactly the shards those episodes live in. Without this layer, filtering a fleet-scale dataset by any attribute means either opening every episode's metadata individually or maintaining that filtering logic inside the training loop itself—both of which stop scaling well past a few thousand episodes.

Worked example: continuing the earlier chunk-shape numbers, a $(16, 224, 224, 3)$ uint8 chunk is ~2.4MB before compression, roughly 0.7–1MB after zstd. Packing 10 chunks per shard (as in the code sample above) gives shards in the ~7–10MB range—on the small side of the "tens to low hundreds of MB" target below, so a real deployment would likely pack 50–100 chunks per shard rather than 10, trading a larger minimum fetch granularity for fewer, cheaper object-store requests.

6. Training-Time Access: Dataloaders, Caching, Prefetch

Every design decision so far—chunk shape, sharding, indexing—exists in service of one final consumer: a dataloader trying to keep a GPU saturated. This section is about what happens on the read side once a training job actually starts pulling batches from a sharded, indexed Zarr corpus.

Random Shuffling vs. Chunk-Local Access

There's a tension here that echoes straight back to the read/write inversion described earlier: training wants fully-random sample order for good gradient statistics, but the storage layer is fastest when reads are chunk-local. A naive dataloader that shuffles at the individual-sample level defeats the entire point of chunking—every sample pulled is likely to be the sole reason a new chunk gets fetched and decompressed, meaning you pay a full chunk's I/O and decompression cost to serve a single training example.

A standard mitigation is chunk-aware shuffling: shuffle at the chunk (or shard) level first, then shuffle within a bounded in-memory buffer of already-loaded chunks. This keeps physical reads chunk-local—each fetched chunk serves many consecutive samples before being discarded—while still producing an effectively-random sample order at the batch level, provided the shuffle buffer is large enough relative to chunk size.

Caching and Prefetch

Two additional layers sit between the store and the training step:

• Local caching: a hot subset of shards mirrored to local SSD (or held in an OS page cache / memory-mapped) for datasets that get iterated over multiple epochs, converting repeat epochs from repeated cloud GETs into local reads after the first pass.
• Async prefetch: a background worker pool that fetches and decodes the next several batches while the GPU is still consuming the current one, so I/O latency is hidden behind compute rather than serialized in front of it.

In nearly every case, neither of these is optional at fleet scale—without them, the I/O path becomes the bottleneck regardless of how well the storage layer itself is tuned.

import torch
import zarr
import numpy as np

class ShardedEpisodeDataset(torch.utils.data.IterableDataset):
    def __init__(self, store_url, index_df, horizon=16, shuffle_buffer_size=64):
        self.root = zarr.open_consolidated(zarr.storage.FsspecStore.from_url(store_url), mode="r")
        self.index = index_df  # episode_id -> shard location, queried ahead of time
        self.horizon = horizon
        self.shuffle_buffer_size = shuffle_buffer_size

    def __iter__(self):
        # Shuffle at the chunk level, not the sample level
        chunk_refs = self._enumerate_chunks(self.index, self.horizon)
        np.random.shuffle(chunk_refs)

        buffer = []
        for ref in chunk_refs:
            chunk = self._fetch_chunk(ref)  # one chunk-local read, serves `horizon` samples
            buffer.extend(chunk)
            if len(buffer) >= self.shuffle_buffer_size:
                np.random.shuffle(buffer)
                # Drain to half rather than fully empty: this keeps the buffer
                # permanently populated with a mix of just-arrived and
                # longer-resident samples, so the next chunk that streams in
                # gets shuffled against a buffer that's never sample-stale or
                # sample-empty. Draining to zero would briefly force every
                # subsequent sample to come from a single freshly-fetched
                # chunk until the buffer refills — reintroducing exactly the
                # chunk-locality-as-non-randomness problem this buffer exists
                # to avoid.
                while len(buffer) >= self.shuffle_buffer_size // 2:
                    yield buffer.pop()

        np.random.shuffle(buffer)
        while buffer:
            yield buffer.pop()

loader = torch.utils.data.DataLoader(
    ShardedEpisodeDataset("s3://fleet-data/episodes.zarr", index_df),
    batch_size=256,
    num_workers=8,       # async prefetch across worker processes
    prefetch_factor=4,   # batches queued ahead per worker
)

The I/O-Bound / Compute-Bound Model

To formalize when any of this actually matters, it's worth adapting the same lens the last post used for the communication fabric: model steady-state training throughput as bound by the slower of two competing terms, since data fetching and GPU compute happen concurrently once prefetching is in place:

Where $T_{\mathrm{compute}}$ is the GPU forward/backward pass time for a batch, and $T_{\mathrm{fetch}}$ is the time to fetch, decompress, and collate that batch from storage—amortized across prefetch workers, but bounded below by chunk-fetch latency and shuffle-buffer refill rate.

This is a steady-state approximation, not a literal per-step formula: with prefetch depth greater than one, the fetch for batch $N+1$ overlaps the compute for batch $N$ rather than the two being strictly sequential. Once the pipeline is full, throughput converges to whichever term is larger—but a single step's wall-clock time isn't exactly $\max(T_{\mathrm{compute}}, T_{\mathrm{fetch}})$ if the prefetch queue is still filling or draining.

• I/O-Bound Regime ($T_{\mathrm{fetch}} > T_{\mathrm{compute}}$): common with small or lightweight models, high-resolution image data, or under-provisioned shuffle buffers and worker counts. Here, every optimization in this post—chunk alignment, sharding, prefetch depth—translates directly into wall-clock training speedup.
• Compute-Bound Regime ($T_{\mathrm{compute}} > T_{\mathrm{fetch}}$): common with large models (e.g., large VLA backbones) where a single forward/backward pass dwarfs a well-prefetched batch fetch. Here, storage-layer tuning has already done its job by getting T_fetch under the compute floor, and further optimization yields no measurable speedup.

7. Production Patterns vs. Academic Alternatives

Everything so far has described one coherent, production-grade pipeline: MCAP for capture, Zarr for training-time storage, sharded and indexed at fleet scale. But it's worth being honest about where this pattern sits relative to the rest of the ecosystem—what production fleets have converged on, what research datasets still commonly use, and why those choices diverge.

What Production Fleets Converge On

The MCAP-to-sharded-Zarr pipeline described in this post isn't a hypothetical—it's a direct reflection of where the field has landed, under slightly different branding depending on the organization:

• LeRobotDataset (Hugging Face): a standardized dataset schema built on top of chunked, columnar storage with a manifest layer, explicitly designed around the same episode-indexed, random-access, fleet-scale access pattern this post has been building toward.
• Open X-Embodiment / RLDS: Google DeepMind's cross-embodiment dataset format, built on TFRecord rather than Zarr, but converging on the same underlying principles—episode-structured, sharded, indexed for filtering by embodiment and task across a large heterogeneous corpus.
• Internal fleet formats at major robotics labs: rarely published in detail, but consistently reported to follow the same shape—a durable capture format feeding an ETL pipeline into a chunked, sharded, cloud-object-store-backed training format.

The convergence itself is the signal: independent organizations solving the same read/write-pattern-inversion problem keep arriving at structurally similar answers.

Academic and Legacy Alternatives

A few older or narrower formats are still common enough in research code that it's worth knowing what they trade away:

• HDF5: the long-standing default for research-scale robot learning datasets, and still perfectly reasonable at that scale. Its weaknesses are exactly the ones MCAP was designed around at the capture stage—poor concurrent-writer support and no graceful degradation on cloud object storage, since HDF5 assumes something closer to POSIX filesystem semantics than an object store provides.
• WebDataset: a tar-shard-based format built for sequential, streaming reads at very large pretraining scale. It's an excellent fit for single-pass, large-batch pretraining where shuffling is done at the shard level and random access to an arbitrary sample is never required—a weaker fit once you need the kind of fine-grained, attribute-filtered access the indexing layer described earlier provides.
• TFRecord: protobuf-based, sequential-read-oriented, and the backbone of RLDS. Mature and battle-tested within the TensorFlow ecosystem, but it carries that ecosystem's assumptions (schema-as-protobuf, sequential iteration as the primary access pattern) less naturally into a PyTorch/JAX-random-access training loop than Zarr does.

Choosing a Format

The common thread across every row: the right format is a function of access pattern, not scale alone. A research dataset with a thousand episodes and no random-access requirement gains little from sharded Zarr's complexity; a fleet corpus with millions of episodes and an attribute-filtered, randomly-shuffled training loop gains little from a sequential-only format like WebDataset. The read/write tension from the start of this post is the lens to hold onto—every format in this table is a different answer to the same underlying question.

Epilogue

This post stayed on the storage side of the pipeline: MCAP as the durable, self-describing capture format; Zarr as the chunked, random-access dataset format; featurization as the ETL boundary that reconciles the two; sharding, consolidated metadata, and indexing as the three things it takes to keep that pipeline coherent at fleet scale; and dataloader-side chunk-aware shuffling and prefetch as the final mile that keeps a GPU fed. Together with the communication fabric from the last post, that's now the full path a single sensor reading takes—from a live DDS topic, through a chunked MCAP log, through featurization, into a sharded Zarr array, and finally into a training batch.

But notice what that path quietly assumed at every step: that the messages hitting an MCAP chunk were worth recording, that an episode's frames were usable rather than corrupted or redundant, and that a raw log was ready for featurization the moment it landed. None of that is guaranteed in practice, and none of it was addressed here. Deciding what's actually in a corpus—filtering out the episode where the robot sat idle for twenty minutes, catching a camera that silently dropped to 5 Hz for a sensor-driver-restart window, attaching the language and quality labels a raw stream doesn't come with—is a separate system from the one this post described, and it's substantial enough to be its own post. That's next.

Beyond that, plenty else was intentionally left out of scope here, including:

• Dataset Versioning & Lineage: reproducibly tracking which raw MCAP episodes, ETL code version, and featurization decisions produced a given Zarr dataset snapshot—and what breaks when any of those changes underneath a training run already in flight. This post already noted that featurization decisions get permanently baked in and are expensive to unwind; lineage tracking is the system that makes "which decisions, exactly" answerable after the fact rather than reconstructed from memory.
• Schema Evolution Across a Fleet's Lifetime: MCAP's self-describing schemas mean a two-year-old log stays readable even as message definitions change. Zarr's array metadata carries no equivalent story—a shape or dtype baked into a corpus at ETL time is static, and a sensor upgrade, added modality, or changed image resolution years into collection has nowhere obvious to go without either a corpus-wide re-chunk or a second, incompatible Zarr generation living alongside the first.
• Transactional Consistency Across a Multi-Array Episode Write: this post covered atomicity at the shard level—a worker crashing mid-write doesn't corrupt neighboring chunks in the same shard. That's a narrower guarantee than atomicity across an episode: nothing described here stops an ETL crash between writing an episode's image array and its action array from leaving a half-written episode indistinguishable from a complete one to a downstream reader, absent an explicit episode-level commit marker.
• Storage Lifecycle, Tiering & Retention Economics: this post treats raw MCAP logs as the permanent source of truth for re-processing. Nothing here addressed when that stops being true in practice—whether cold raw logs move to archival object storage tiers, at what point a validated Zarr snapshot is trusted enough to let its source logs be pruned, and how that retention decision constrains the lineage problem above (you can't re-run an ETL job against logs that were already deleted).
• Cross-Embodiment Schema Harmonization: the featurization decisions described here were implicitly single-embodiment. A fleet mixing different arm DOF, gripper types, or mobile bases needs an explicit harmonization layer—padding, masking, or a shared action-space convention—before heterogeneous episodes can live in one trainable Zarr corpus rather than several incompatible ones.
• Privacy & PII Redaction: pipelines for detecting and scrubbing identifiable information (faces, license plates, ambient audio) from sensor logs before a dataset is eligible for broader use.
• Simulation Data & Sim/Real Merging: how synthetic data generated at scale in simulation gets stored, labeled, and blended with real fleet data without one silently dominating the other during training.
• Active Learning & Coverage-Driven Collection: deciding what a robot should even upload off-robot in the first place—coverage- or uncertainty-driven selection at the edge, before an episode ever reaches an MCAP log bound for the pipeline described in this post. Not to be confused with filtering an already-collected corpus, which the next post covers; this is the earlier, harder question of what's worth collecting at all.
• Streaming/Online Training: architectures that train directly off live topics, bypassing the storage layer described here entirely, and the different set of tradeoffs that implies.

Each of those is a real system in its own right. This post covered the substrate—if the storage layer can't hold the data or can't serve it fast enough, nothing built on top of it trains at scale.

Next in the series: Inside Robot Data Pipelines—the offline and online systems that decide what a stored dataset is actually made of: filtering out redundant or low-quality episodes and detecting timestamp drift and sensor dropouts, but also dataset mixing and co-training ratios across heterogeneous embodiments, autolabeling and captioning via foundation models, confidence-cascaded labeling, and the online training-time processors—history buffers, action chunking, augmentation, conditioning dropout—that turn a stored corpus into the tensors a training step actually consumes. Policy serving—getting a trained checkpoint back onto the robot—comes later in the series, once the data side of the pipeline is fully covered.