Welcome to Xbitinfo’s documentation!#
xbitinfo: Retrieve bitwise information content and compress accordingly#
Xbitinfo analyses datasets based on their bitwise real information content and applies lossy compression accordingly. Being based on xarray it integrates seamlessly into common research workflows. Additional convienient functions help users to visualize the bitwise information content and to make informed decisions on the real information threshold that is subsequently used as the preserved precision during the compression.
Xbitinfo works in four steps:
Analyse the bitwise information content of a dataset
Decide on a threshold of real information to preserve (e.g. 99%)
Reduce the precision of the dataset accordingly (bitrounding)
Apply lossless compression (e.g. zlib, blosc, zstd) and store the dataset
To fullfill these steps, Xbitinfo relies on:
xarrayfor handling multi-dimensional arrays and file formats (e.g. netcdf, zarr, hdf5, grib)daskfor scaling to large datasetsBitInformation.jl(optional) for computing the bitwise information content based on the original Julia implementation. Continuous integration tests ensure however that the python-implementation shipped with xbitinfo result in identical results.numcodecsfor a wide-range of lossless compression algorithms
Overall, the package presents a pipeline to compress (climate) datasets based on the real information content.
How to install#
Xbitinfo is packaged and distributed both via PyPI and conda-forge and can be installed via pip or conda respectively.
Depending on whether one wants to use the Julia implementation of the bitinformation algorithm (BitInformation.jl) or the native python implementation shipped with xbitinfo,
one might choose one installation option over the other.
Pure-python installation (recommended)#
pip install xbitinfo
We also maintain other dependency sets for different subsets of functionality:
pip install "xbitinfo[viz]" # Optional dependencies for visualisation
pip install "xbitinfo[io]" # Optional dependencies for IO
For conda
conda install -c conda-forge xbitinfo-python
the IO and visualisation dependencies are included by default.
Installation including optional Julia backend#
conda install -c conda-forge xbitinfo
or
pip install "xbitinfo[julia]" # julia needs to be installed manually
How to use#
To install all dependencies needed to run this example,
pip install "xbitinfo[example]"
is recommended.
import xarray as xr
import xbitinfo as xb
# Define output path for compressed dataset
outpath = "example_bitrounded_compressed.nc"
# Load example dataset
# (requires pooch to be installed via e.g. `pip install pooch`)
example_dataset = "eraint_uvz"
ds = xr.tutorial.load_dataset(example_dataset)
# Step 1: analyze bitwise information content
bitinfo = xb.get_bitinformation(ds, dim="longitude", implementation="python")
# Step 2: decide on a threshold of real information to preserve (e.g. 99%)
keepbits = xb.get_keepbits(
bitinfo, inflevel=0.99
) # get number of mantissa bits to keep for 99% real information
# Step 3: reduce the precision of the dataset accordingly (bitrounding)
ds_bitrounded = xb.xr_bitround(
ds, keepbits
) # bitrounding keeping only keepbits mantissa bits
# Step 4: apply lossless compression (e.g. zlib, blosc, zstd) and store the dataset
ds_bitrounded.to_compressed_netcdf(outpath)
How the science works#
Paper#
Klöwer, M., Razinger, M., Dominguez, J. J., Düben, P. D., & Palmer, T. N. (2021). Compressing atmospheric data into its real information content. Nature Computational Science, 1(11), 713–724. doi: 10/gnm4jj
Schulz, H., M. Klöwer, and A. Spring. 2025. “Xbitinfo: Compressing Geospatial Data Based on Information Theory.” Journal of Open Source Software 10 (116): 9178. doi: 10.21105/joss.09178.