Quick start

Bitrounding by itself is straight forward. However, the decision on the number of bits to round to, is much more difficult. Xbitinfo uses information theory to guide a (quantitative) decision.

In order to arrive at an optimal compression strategy that fits the use-case(s) of the compressed dataset, Xbitinfo splits the compression pipeline into four main steps to provide customization options for each step:

1. Analysis of the bitwise information content of a dataset

2. Decision on a threshold of real information to preserve (e.g. 99%)

3. Bitround dataset accordingly (bitrounding)

4. Apply lossless compression of choice (e.g. zlib, blosc, zstd) and store the dataset

To install all dependencies needed to run this example,

pip install "xbitinfo[example]"

is recommended.

Step 0: Open dataset

import xbitinfo as xb

import xarray as xr

ds = xr.tutorial.load_dataset("eraint_uvz").astype("float32")

ds

<xarray.Dataset> Size: 8MB
Dimensions:    (month: 2, level: 3, latitude: 241, longitude: 480)
Coordinates:
  * month      (month) int32 8B 1 7
  * level      (level) int32 12B 200 500 850
  * latitude   (latitude) float32 964B 90.0 89.25 88.5 ... -88.5 -89.25 -90.0
  * longitude  (longitude) float32 2kB -180.0 -179.2 -178.5 ... 178.5 179.2
Data variables:
    z          (month, level, latitude, longitude) float32 3MB 1.068e+05 ... ...
    u          (month, level, latitude, longitude) float32 3MB 1.282 ... 3.539
    v          (month, level, latitude, longitude) float32 3MB -0.04676 ... 3...
Attributes:
    Conventions:  CF-1.0
    Info:         Monthly ERA-Interim data. Downloaded and edited by fabien.m...

xarray.Dataset

• Dimensions:
  • month: 2
  • level: 3
  • latitude: 241
  • longitude: 480
• Coordinates: (4)
  • month
    (month)
    int32
    1 7

    array([1, 7], dtype=int32)

  • level
    (level)
    int32
    200 500 850

    units :

    millibars

    long_name :

    pressure_level

    array([200, 500, 850], dtype=int32)

  • latitude
    (latitude)
    float32
    90.0 89.25 88.5 ... -89.25 -90.0

    units :

    degrees_north

    long_name :

    latitude

    array([ 90.  ,  89.25,  88.5 , ..., -88.5 , -89.25, -90.  ],
          shape=(241,), dtype=float32)

  • longitude
    (longitude)
    float32
    -180.0 -179.2 ... 178.5 179.2

    units :

    degrees_east

    long_name :

    longitude

    array([-180.  , -179.25, -178.5 , ...,  177.75,  178.5 ,  179.25],
          shape=(480,), dtype=float32)

• Data variables: (3)
  • z
    (month, level, latitude, longitude)
    float32
    1.068e+05 1.068e+05 ... 1.178e+04

    number_of_significant_digits :

    5

    units :

    m**2 s**-2

    long_name :

    Geopotential

    standard_name :

    geopotential

    array([[[[106837.516, 106839.234, 106837.516, ..., 106839.234,
              106837.516, 106839.234],
             [106856.484, 106858.21 , 106856.484, ..., 106856.484,
              106856.484, 106856.484],
             [106884.086, 106884.086, 106884.086, ..., 106884.086,
              106884.086, 106884.086],
             ...,
             [109795.94 , 109795.94 , 109795.94 , ..., 109795.94 ,
              109795.94 , 109795.94 ],
             [109802.836, 109804.56 , 109804.56 , ..., 109804.56 ,
              109802.836, 109802.836],
             [109808.01 , 109808.01 , 109808.01 , ..., 109808.01 ,
              109808.01 , 109808.01 ]],
    
            [[ 49723.58 ,  49723.58 ,  49723.58 , ...,  49723.58 ,
               49723.58 ,  49723.58 ],
             [ 49744.277,  49744.277,  49744.277, ...,  49746.004,
               49746.004,  49746.004],
             [ 49771.88 ,  49771.88 ,  49771.88 , ...,  49771.88 ,
               49771.88 ,  49771.88 ],
    ...
               47891.598,  47891.598],
             [ 47941.625,  47941.625,  47941.625, ...,  47943.348,
               47941.625,  47941.625],
             [ 47974.4  ,  47974.4  ,  47974.4  , ...,  47974.4  ,
               47974.4  ,  47974.4  ]],
    
            [[ 13485.926,  13485.926,  13485.926, ...,  13485.926,
               13485.926,  13485.926],
             [ 13484.2  ,  13484.2  ,  13484.2  , ...,  13484.2  ,
               13484.2  ,  13484.2  ],
             [ 13484.2  ,  13484.2  ,  13484.2  , ...,  13482.476,
               13482.476,  13484.2  ],
             ...,
             [ 11360.691,  11358.967,  11357.242, ...,  11364.142,
               11362.417,  11362.417],
             [ 11612.546,  11612.546,  11612.546, ...,  11612.546,
               11612.546,  11612.546],
             [ 11776.424,  11776.424,  11776.424, ...,  11776.424,
               11776.424,  11776.424]]]],
          shape=(2, 3, 241, 480), dtype=float32)

  • u
    (month, level, latitude, longitude)
    float32
    1.282 1.282 1.282 ... 3.578 3.539

    number_of_significant_digits :

    2

    units :

    m s**-1

    long_name :

    U component of wind

    standard_name :

    eastward_wind

    array([[[[ 1.2817602e+00,  1.2817602e+00,  1.2817602e+00, ...,
               1.2817602e+00,  1.2817602e+00,  1.2817602e+00],
             [ 2.0162134e+00,  2.0319405e+00,  2.0240769e+00, ...,
               2.0083499e+00,  2.0083499e+00,  2.0240769e+00],
             [ 2.8198657e+00,  2.8355927e+00,  2.8355927e+00, ...,
               2.7962751e+00,  2.7962751e+00,  2.8198657e+00],
             ...,
             [-5.0797796e-01, -4.9225092e-01, -4.8438740e-01, ...,
              -5.4729557e-01, -5.3156853e-01, -5.2370501e-01],
             [-5.7088619e-01, -5.6302267e-01, -5.4729557e-01, ...,
              -5.9447676e-01, -5.8661324e-01, -5.7874972e-01],
             [-5.3943205e-01, -5.3943205e-01, -5.3156853e-01, ...,
              -5.5515909e-01, -5.4729557e-01, -5.4729557e-01]],
    
            [[ 1.9218512e+00,  1.9061241e+00,  1.8903971e+00, ...,
               1.9611688e+00,  1.9533052e+00,  1.9375782e+00],
             [ 2.3040185e+00,  2.3040185e+00,  2.2820005e+00, ...,
               2.3354726e+00,  2.3197455e+00,  2.3197455e+00],
             [ 2.7428031e+00,  2.7428031e+00,  2.7349396e+00, ...,
               2.7663937e+00,  2.7506666e+00,  2.7585301e+00],
    ...
              -4.4680490e+00, -4.4680490e+00, -4.4995031e+00],
             [-3.5543075e+00, -3.5857615e+00, -3.5936251e+00, ...,
              -3.4929719e+00, -3.5071263e+00, -3.5385804e+00],
             [-2.3905058e+00, -2.4219599e+00, -2.4376869e+00, ...,
              -2.3197341e+00, -2.3354611e+00, -2.3669152e+00]],
    
            [[-1.6355559e-01, -1.5569207e-01, -1.5569207e-01, ...,
              -1.8714617e-01, -1.7928264e-01, -1.7141911e-01],
             [ 7.8692473e-03,  1.5732773e-02,  2.3596296e-02, ...,
              -1.5721327e-02, -7.8578023e-03,  5.7223951e-06],
             [ 2.7365637e-01,  2.8151992e-01,  2.8938344e-01, ...,
               2.4220228e-01,  2.5006580e-01,  2.6579285e-01],
             ...,
             [-1.6875067e+00, -1.6875067e+00, -1.6953702e+00, ...,
              -1.6639161e+00, -1.6717796e+00, -1.6796432e+00],
             [ 1.4531851e+00,  1.4295945e+00,  1.3902769e+00, ...,
               1.5554109e+00,  1.5239568e+00,  1.4925027e+00],
             [ 3.4851198e+00,  3.4536657e+00,  3.3986211e+00, ...,
               3.6250906e+00,  3.5779095e+00,  3.5385919e+00]]]],
          shape=(2, 3, 241, 480), dtype=float32)

  • v
    (month, level, latitude, longitude)
    float32
    -0.04676 -0.06253 ... 3.328 3.383

    number_of_significant_digits :

    2

    units :

    m s**-1

    long_name :

    V component of wind

    standard_name :

    northward_wind

    array([[[[-0.04675769, -0.06252575, -0.07829381, ...,  0.00771379,
              -0.00757645, -0.02334451],
             [ 0.14054775,  0.10948945,  0.08607627, ...,  0.22655535,
               0.19549705,  0.17208387],
             [ 0.33597612,  0.2967949 ,  0.25761366, ...,  0.45304203,
               0.4138608 ,  0.37515736],
             ...,
             [-0.6875143 , -0.70328236, -0.7185726 , ..., -0.656456  ,
              -0.6641011 , -0.6798692 ],
             [-0.5154991 , -0.53126717, -0.5389123 , ..., -0.49208593,
              -0.50020885, -0.507854  ],
             [-0.37502003, -0.38266516, -0.39078808, ..., -0.35160685,
              -0.35925198, -0.3673749 ]],
    
            [[-1.0778933 , -1.1094294 , -1.1328425 , ..., -1.0076537 ,
              -1.0310669 , -1.062603  ],
             [-0.92976904, -0.96082735, -0.99236345, ..., -0.8279934 ,
              -0.8595295 , -0.89823294],
             [-0.765399  , -0.81270313, -0.8514066 , ..., -0.6483331 ,
              -0.6875143 , -0.72669554],
    ...
               1.7187872 ,  1.6485476 ],
             [ 1.8358531 ,  1.7890267 ,  1.7345552 , ...,  1.9844551 ,
               1.9376287 ,  1.8908024 ],
             [ 1.9218607 ,  1.8908024 ,  1.8592663 , ...,  2.0155134 ,
               1.9844551 ,  1.952919  ]],
    
            [[ 0.51563644,  0.51563644,  0.51563644, ...,  0.5079913 ,
               0.5079913 ,  0.5079913 ],
             [ 0.54669476,  0.54669476,  0.54669476, ...,  0.5390496 ,
               0.5390496 ,  0.54669476],
             [ 0.585876  ,  0.585876  ,  0.585876  , ...,  0.585876  ,
               0.57823086,  0.585876  ],
             ...,
             [ 4.734309  ,  4.7500772 ,  4.7500772 , ...,  4.734309  ,
               4.718541  ,  4.734309  ],
             [ 3.195251  ,  3.226787  ,  3.2420774 , ...,  3.1250114 ,
               3.1484246 ,  3.179483  ],
             [ 3.4217377 ,  3.468564  ,  3.5077453 , ...,  3.2889037 ,
               3.328085  ,  3.3830342 ]]]],
          shape=(2, 3, 241, 480), dtype=float32)

• Attributes: (2)

  Conventions :

  CF-1.0

  Info :

  Monthly ERA-Interim data. Downloaded and edited by fabien.maussion@uibk.ac.at

xb.plot_distribution(ds);

NOTE: If you plan to use the example datasets provided by xarray, you will need to install the pooch package separately using the following command:

pip install pooch

Without installing pooch, you will not be able to download and load the example datasets, which may result in errors or unexpected behavior. Alternatively, follow the example installation instructions above.

Step 1: Get information content per bit

using xbitinfo.xbitinfo.get_bitinformation()

info_per_bit = xb.get_bitinformation(ds, dim="longitude", implementation="python")

info_per_bit

<xarray.Dataset> Size: 1kB
Dimensions:     (bitfloat32: 32)
Coordinates:
  * bitfloat32  (bitfloat32) <U3 384B '±' 'e1' 'e2' 'e3' ... 'm21' 'm22' 'm23'
    dim         <U9 36B 'longitude'
Data variables:
    z           (bitfloat32) float64 256B 0.0 0.0 0.0 ... 0.005199 0.007699
    u           (bitfloat32) float64 256B 0.7816 0.4274 0.0 ... 0.01148 0.1475
    v           (bitfloat32) float64 256B 0.8752 0.7756 0.0 ... 0.06165 0.05304
Attributes:
    xbitinfo_description:       bitinformation calculated by xbitinfo.get_bit...
    python_repository:          https://github.com/observingClouds/xbitinfo
    julia_repository:           https://github.com/milankl/BitInformation.jl
    reference_paper:            http://www.nature.com/articles/s43588-021-001...
    xbitinfo_version:           0.0.7
    BitInformation.jl_version:  0.6.3

xarray.Dataset

• Dimensions:
  • bitfloat32: 32
• Coordinates: (2)
  • bitfloat32
    (bitfloat32)
    <U3
    '±' 'e1' 'e2' ... 'm21' 'm22' 'm23'

    description :

    name of the bits: '±' refers to the sign bit, 'e' to the exponents bits and 'm' to the mantissa bits.

    array(['±', 'e1', 'e2', 'e3', 'e4', 'e5', 'e6', 'e7', 'e8', 'm1', 'm2', 'm3',
           'm4', 'm5', 'm6', 'm7', 'm8', 'm9', 'm10', 'm11', 'm12', 'm13', 'm14',
           'm15', 'm16', 'm17', 'm18', 'm19', 'm20', 'm21', 'm22', 'm23'],
          dtype='<U3')

  • dim
    ()
    <U9
    'longitude'

    description :

    dimension of the source dataset along which the bitwise information has been analysed.

    array('longitude', dtype='<U9')

• Data variables: (3)
  • z
    (bitfloat32)
    float64
    0.0 0.0 0.0 ... 0.005199 0.007699

    long_name :

    z bitwise information

    units :

    1

    source_number_of_significant_digits :

    5

    source_units :

    m**2 s**-2

    source_long_name :

    Geopotential

    source_standard_name :

    geopotential

    array([0.        , 0.        , 0.        , 0.        , 0.91829583,
           0.91829583, 0.91829583, 0.91829583, 0.91829583, 0.40596261,
           0.97813287, 0.92341985, 0.94109422, 0.90208436, 0.82484959,
           0.71155894, 0.54404085, 0.34755517, 0.18092331, 0.07436806,
           0.02217038, 0.01245641, 0.00377219, 0.00628822, 0.01224325,
           0.00389673, 0.01004858, 0.0131198 , 0.00564089, 0.00887582,
           0.0051986 , 0.00769888])

  • u
    (bitfloat32)
    float64
    0.7816 0.4274 ... 0.01148 0.1475

    long_name :

    u bitwise information

    units :

    1

    source_number_of_significant_digits :

    2

    source_units :

    m s**-1

    source_long_name :

    U component of wind

    source_standard_name :

    eastward_wind

    array([7.81646180e-01, 4.27436423e-01, 0.00000000e+00, 0.00000000e+00,
           1.71227095e-04, 0.00000000e+00, 6.35104053e-01, 8.05002303e-01,
           6.78224433e-01, 5.35060838e-01, 3.99149342e-01, 2.46776345e-01,
           1.16042153e-01, 3.33513654e-02, 5.73779013e-03, 3.48614977e-03,
           5.59982312e-03, 9.19817640e-03, 7.71894512e-03, 5.38088606e-03,
           2.65337337e-03, 6.27602119e-04, 3.53690832e-04, 4.95650149e-04,
           3.34290451e-03, 2.51371722e-03, 2.11808385e-03, 1.17905219e-03,
           1.13182091e-03, 3.28875231e-03, 1.14760822e-02, 1.47515177e-01])

  • v
    (bitfloat32)
    float64
    0.8752 0.7756 ... 0.06165 0.05304

    long_name :

    v bitwise information

    units :

    1

    source_number_of_significant_digits :

    2

    source_units :

    m s**-1

    source_long_name :

    V component of wind

    source_standard_name :

    northward_wind

    array([8.75249762e-01, 7.75560757e-01, 0.00000000e+00, 0.00000000e+00,
           0.00000000e+00, 2.74549688e-03, 1.17783327e-01, 5.01420134e-01,
           4.06812530e-01, 2.29018531e-01, 1.01658673e-01, 2.73218922e-02,
           3.76688041e-03, 4.52212736e-04, 1.53181859e-04, 1.30129511e-04,
           5.09324245e-05, 4.18079116e-05, 3.43536304e-05, 2.87844951e-05,
           3.84562184e-05, 1.71014710e-04, 7.26738394e-04, 4.54372431e-04,
           6.11464268e-04, 5.30345968e-04, 2.06533122e-04, 7.77135473e-04,
           6.67379672e-03, 1.99001671e-02, 6.16532925e-02, 5.30433184e-02])

• Attributes: (6)

  xbitinfo_description :

  bitinformation calculated by xbitinfo.get_bitinformation wrapping bitinformation.jl

  python_repository :

  https://github.com/observingClouds/xbitinfo

  julia_repository :

  https://github.com/milankl/BitInformation.jl

  reference_paper :

  http://www.nature.com/articles/s43588-021-00156-2

  xbitinfo_version :

  0.0.7

  BitInformation.jl_version :

  0.6.3

Visualize information content

using xbitinfo.graphics.plot_bitinformation()

fig = xb.plot_bitinformation(info_per_bit)

Step 2: Set keepbits

Based on the visualization of the bitinformation plotted above, the number of keepbits can be directly obtained or calculated by setting a threshold of real information content to preserve (e.g. 99%) by using xbitinfo.xbitinfo.get_keepbits():

keepbits = xb.get_keepbits(info_per_bit, 0.99)
keepbits

<xarray.Dataset> Size: 68B
Dimensions:   (inflevel: 1)
Coordinates:
  * inflevel  (inflevel) float64 8B 0.99
    dim       <U9 36B 'longitude'
Data variables:
    z         (inflevel) int64 8B 10
    u         (inflevel) int64 8B 3
    v         (inflevel) int64 8B 2
Attributes:
    xbitinfo_description:       bitinformation calculated by xbitinfo.get_bit...
    python_repository:          https://github.com/observingClouds/xbitinfo
    julia_repository:           https://github.com/milankl/BitInformation.jl
    reference_paper:            http://www.nature.com/articles/s43588-021-001...
    xbitinfo_version:           0.0.7
    BitInformation.jl_version:  0.6.3

xarray.Dataset

• Dimensions:
  • inflevel: 1
• Coordinates: (2)
  • inflevel
    (inflevel)
    float64
    0.99

    array([0.99])

  • dim
    ()
    <U9
    'longitude'

    description :

    dimension of the source dataset along which the bitwise information has been analysed.

    array('longitude', dtype='<U9')

• Data variables: (3)
  • z
    (inflevel)
    int64
    10

    long_name :

    z bitwise information

    units :

    1

    source_number_of_significant_digits :

    5

    source_units :

    m**2 s**-2

    source_long_name :

    Geopotential

    source_standard_name :

    geopotential

    array([10])

  • u
    (inflevel)
    int64
    3

    long_name :

    u bitwise information

    units :

    1

    source_number_of_significant_digits :

    2

    source_units :

    m s**-1

    source_long_name :

    U component of wind

    source_standard_name :

    eastward_wind

    array([3])

  • v
    (inflevel)
    int64
    2

    long_name :

    v bitwise information

    units :

    1

    source_number_of_significant_digits :

    2

    source_units :

    m s**-1

    source_long_name :

    V component of wind

    source_standard_name :

    northward_wind

    array([2])

• Attributes: (6)

  xbitinfo_description :

  bitinformation calculated by xbitinfo.get_bitinformation wrapping bitinformation.jl

  python_repository :

  https://github.com/observingClouds/xbitinfo

  julia_repository :

  https://github.com/milankl/BitInformation.jl

  reference_paper :

  http://www.nature.com/articles/s43588-021-00156-2

  xbitinfo_version :

  0.0.7

  BitInformation.jl_version :

  0.6.3

Step 3: Apply bitrounding

using xbitinfo.bitround.xr_bitround() or xbitinfo.bitround.jl_bitround(). The later does not work with chunked datasets and requires a working installation of Julia.

ds_bitrounded = xb.xr_bitround(ds, keepbits)

xr.concat([ds, ds_bitrounded], "bitround").isel(level=0)["v"].plot(
    col="bitround", row="month"
);

Step 4: Apply compression and save dataset

To leverage the results of bitrounding, the dataset needs to be stored with a (lossless) compression algorithm. Xbitinfo provides two convienience functions that can be used to store the bitrounded dataset into commonly used file formats with default compression settings.

These functions are xbitinfo.save_compressed.ToCompressed_Netcdf and xbitinfo.save_compressed.ToCompressed_Zarr.

NetCDF

ds_bitrounded.to_compressed_netcdf("bitrounded_compressed.nc")

ds.to_compressed_netcdf("compressed.nc")

ds.to_netcdf("original.nc")

!du -hs *.nc

7.5M	0.air_original.nc
532K	bitrounded_compressed.nc
4.1M	compressed.nc
8.0M	original.nc

!rm *.nc

Zarr

ds_bitrounded.to_compressed_zarr("bitrounded_compressed.zarr", mode="w")
ds.to_compressed_zarr("compressed.zarr", mode="w")
ds.to_zarr(
    "original.zarr", mode="w", encoding={v: {"compressors": None} for v in ds.data_vars}
);

/home/docs/checkouts/readthedocs.org/user_builds/xbitinfo/envs/v0.0.7/lib/python3.12/site-packages/zarr/core/group.py:2662: ZarrUserWarning: The `compressor` argument is deprecated. Use `compressors` instead.
  compressors = _parse_deprecated_compressor(
/home/docs/checkouts/readthedocs.org/user_builds/xbitinfo/envs/v0.0.7/lib/python3.12/site-packages/zarr/api/asynchronous.py:231: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.
  warnings.warn(

!du -hs *.zarr

836K	air_bitrounded.zarr
1.1M	air_bitrounded_by_chunks.zarr
7.9M	air_compressed.zarr
1.1M	bitrounded_compressed.zarr
5.0M	compressed.zarr
11M	original.zarr

!rm -r *.zarr