Optimizing Access Patterns

Altay Sansal

Sep 22, 2026

5 min read

Introduction

In this page we will be showing how we can take an existing MDIO and add fast access, lossy, versions of the data in channel and time cross-sections (slices).

We can re-use the MDIO dataset we created in the Quickstart page. Please run it first.

Let’s open the original MDIO first.

from mdio import open_mdio

path = "soda_lake.mdio"

ds = open_mdio(path)
ds

Optimizing Channel and Time

In this cell, we will demonstrate how to create channel and Z (two-way-time) optimized versions.

The API allows us to specify dimensions we want to generate fast access patterns for with desired chunking.

The processing chunks is for efficiency. We want to make sure the processing chunks is at least as large as the largest dimension in every processing chunk dimension. Once these are chunk aligned, performance will be the highest.

By default the optimized access patterns use the ZFP Compressor. However it can be customized if needed.

The default compressor is set to ZFPQuality.LOW. Internally the algorithm dynamically adjusts the ZFP tolerance based on global statistics (standard deviation) of the dataset. This yiels ~1:10 compression out of the box.

from mdio import OptimizedAccessPatternConfig

conf = OptimizedAccessPatternConfig(
    optimize_dimensions={
        "channel": (1, 4, 512),
        "time": (1, 512, 4),
    },
    processing_chunks={"shot_point": 1, "channel": 512, "time": 512},
)

%time optimize_access_patterns(ds, conf, n_workers=4);

We can now open the original MDIO dataset and the fast access patterns.

ds = open_mdio(path)
ds

We can now compare the sizes of the compressed representations to original.

Below commands are for UNIX based operating systems and won’t work on Windows.

!du -hs {path}/amplitude
!du -hs {path}/fast_channel
!du -hs {path}/fast_time

Look at encoding of variable. We show data is rechunked with ZFP compression.

print("Fast channel chunks\t", ds.fast_channel.encoding["chunks"], ds.fast_channel.encoding["serializer"])
print("Fast time chunks\t", ds.fast_time.encoding["chunks"], ds.fast_time.encoding["serializer"])

Comparing local disk read speeds for channels:

%timeit ds.isel(channel=100).amplitude.compute()  # original chunks
%timeit ds.isel(channel=100).fast_channel.compute()  # channel optimized

For time-slices:

%timeit ds.isel(time=500).amplitude.compute()  # original chunks
%timeit ds.isel(time=500).fast_time.compute()  # time-slice optimized

We can check the subjective quality of the compression by visually comparing the original gather to the channel-optimized copy. Similar to the example we had in the Compression page.

import matplotlib.pyplot as plt

from mdio.builder.schemas.v1.stats import SummaryStatistics

stats = SummaryStatistics.model_validate_json(ds.amplitude.attrs["statsV1"])
imshow_kw = {
    "vmin": -3 * stats.std,
    "vmax": 3 * stats.std,
    "cmap": "gray_r",
    "interpolation": "bilinear",
    "yincrease": False,
    "add_colorbar": False,
}

fig, ax = plt.subplots(1, 4, sharex="all", sharey="all", figsize=(8, 5))

ds_gather = ds.sel(shot_point=7733)

ds_gather.amplitude.T.plot.imshow(ax=ax[0], **imshow_kw)
ds_gather.fast_channel.T.plot.imshow(ax=ax[1], **imshow_kw)

diff = ds_gather.amplitude - ds_gather.fast_channel
diff.T.plot.imshow(ax=ax[2], **imshow_kw)
(1000 * diff).T.plot.imshow(ax=ax[3], **imshow_kw)

for axis, title in zip(ax.ravel(), ["original", "lossy", "difference", "1,000xdifference"], strict=False):
    if title != "original":
        axis.set_ylabel("")
    axis.set_title(title)

fig.tight_layout();

Adjusting the Compressor

The compressor can be modified for fast access patterns but the default setting usually works quite well. Given 1:10 compression ratio, the fidelity is quite high with the default ZfpQuality.LOW setting.

If you still want to use the ZFP compression but change the quality settings follow the instructions below. We can also use Blosc compressor available in MDIO as well, but we will not demonstrate that here.

from mdio.optimize import ZfpQuality
from mdio.optimize import get_default_zfp

get_default_zfp(stats, ZfpQuality.HIGH)

Here is a medium example. Note that the tolerance changes because it is based on dataset statistics and compression quality setting.

get_default_zfp(stats, ZfpQuality.MEDIUM)

In conclusion, we show that by generating optimized, lossy compressed copies of the data for certain access patterns yield big performance benefits when reading the data.

The differences are orders of magnitude larger on big datasets and remote stores, given available network bandwidth.