ZarrDatasets

See the documentation of JuliaGeo/CommonDataModel.jl for the full documentation of the API. As a quick reference, here is an example how to create and read a Zarr file store as a quick reference.

Create a Zarr file store

The following example create a Zarr file store in the directory "/tmp/test-zarr":

using ZarrDatasets

# sample data
data = [i+j for i = 1:3, j = 1:5]

directoryname = "/tmp/test-zarr"
mkdir(directoryname)

ds = ZarrDataset(directoryname,"c")
defDim(ds,"lon",size(data,1))
defDim(ds,"lat",size(data,2))
zv = defVar(ds,"varname",Int64,("lon","lat"))
zv[:,:] = data
zv.attrib["units"] = "m"
close(ds)

Loading a Zarr file store

The data and units can be loaded by indexing the data set structure ds.

using ZarrDatasets
directoryname = "/tmp/test-zarr"
ds = ZarrDataset(directoryname)
data = ds["varname"][:,:]
data_units = ds["varname"].attrib["units"]
ZarrDatasets.ZarrDataset — Type
ds = ZarrDataset(url::AbstractString,mode = "r";
                 _omitcode = [404,403],
                 maskingvalue = missing)
ZarrDataset(zg::ZarrCore.ZGroup; _omitcode, maskingvalue)
ZarrDataset(f::Function,url::AbstractString,mode = "r";
                 maskingvalue = missing)

Open the zarr dataset at the url or path url. The URL must be a zgroup (not a zarray). The mode can be "r" (read-only), "w" (write), or "c" (create). ds supports the API of the JuliaGeo/CommonDataModel.jl.

Keywords

  • _omitcode: experimental, allows to define which HTTP error code should be used for missing chunks. For compatibility with python's Zarr, the HTTP error 403 (permission denied) is also used to missing chunks in addition to 404 (not found). Default [404, 403].
  • maskingvalue: The parameter maskingvalue allows to define which special value should be used as replacement for fill values. The default is missing. Defaults to missing`.
  • attrib: Attributes, defualts to Dict()
  • format: Zarr format used when creating a dataset with mode "c" (2 or 3). Defaults to 2.
  • consolidated: if true, open the store from its consolidated metadata (the .zmetadata key written by python's zarr.consolidate_metadata) instead of listing the store. This is required for stores whose backend listing is truncated or unavailable (e.g. paginated cloud buckets), where listing would otherwise miss most arrays. Defaults to false.

Example:

using ZarrDatasets
url = "https://s3.waw3-1.cloudferro.com/mdl-arco-time-035/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-ssh-rean-d_202012/timeChunked.zarr"
ds = ZarrDataset(url);
# see the metadata
display(ds)
# load the variable time
time = ds["time"][:]
# load the the attribute long_name for the variable zos
zos_long_name = ds["zos"].attrib["long_name"]
# load the global attribute
comment = ds.attrib["comment"]
# query the dimension of the variable zos
size(ds["zos"])
close(ds)

Example with a do-block:

using ZarrDatasets
url = "https://s3.waw3-1.cloudferro.com/mdl-arco-time-035/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-ssh-rean-d_202012/timeChunked.zarr"

zos1 = ZarrDataset(url) do ds
  ds["zos"][:,:,end,1]
end # implicit call to close(ds)
source
ZarrDatasets.ZarrVariable — Type
ZarrVariable <: CDM.AbstractVariable

ZarrVariable(zarray, parentdataset)

A CommonDataModel.jl compatible zarr variable.

Usually generated with CommonDataModel.defvar(::ZarrDataset, args...; kw...)

source
CommonDataModel.defVar — Method
defVar(ds::ZarrDataset, name::SymbolOrString, vtype::DataType, dimensionnames; 
    chunksizes=nothing, attrib=Dict(), fillvalue=nothing)

Create a variable name in the dataset ds with the type vtype and the dimension dimensionnames.

For coordinate variables, fill values will be used a background value of undefined chunks and not as missing value as coordinate variables cannot have the _FillValues in the CF convension as in Zarr v2 format a fill_value does not necessarily indicate a missing value.

See also CommonDataModel.defVar for more information.

source

Interoperability with Zarr.jl

Here is a example of how to create a dataset with Zarr.jl that can be read in ZarrDatasets.jl. As in python-xarray, ZaraDatasets.jl assume that there is a _ARRAY_DIMENSIONS attribute containing a list of the dimension names. Note that Zarr.jl uses the C-ordering per default (for compatability with python). Therefore if an array has e.g. the dimensions lon x lat, the corresponding _ARRAY_DIMENSIONS attribute is ("lat","lon").

using Zarr
using ZarrDatasets
using Test

# your file name
fname = tempname()

# sample data (lon x lat)
data = rand(100,101)

store = Zarr.DirectoryStore(fname)
zg = zgroup(store)

# create a variable with the name "temp" where the
# first dimension in lon and the second dimension is lat
z = zcreate(Float64,zg,"temp",size(data)...;
            fill_value = 9999, # optional
            attrs = Dict("_ARRAY_DIMENSIONS" => ("lat","lon"))) # important
z .= data

# read data
ds = ZarrDataset(fname)

# output
#  temp   (100 × 101)
#    Datatype:    Union{Missing, Float64} (Float64)
#    Dimensions:  lon × lat
#    Attributes:
#     _FillValue           = 9999.0


data2 = ds["temp"][:,:]

@test data == data2

Using a pre-existing Zarr group or store

It's also simple to wrap an existing Zarr group, or a manually constructed Zarr store, in a ZarrDataset:

using Zarr, ZarrDatasets
zg = zopen("/path/to/zarr")
zd = ZarrDataset(zg)

and you can pass a constructed Zarr.AbstractStore similarly.

Differences between Zarr and NetCDF files

  • All metadata (in particular attributes) is stored in JSON files for the Zarr format with the following implications:
    • JSON does not distinguish between integers and real numbers. They are all considered as generic numbers. Whole numbers are loaded as Int64 and real numbers Float64. It is not possible to store the number 1.0 as a real number.
    • The order of keys in a JSON document is undefined. It is therefore not possible to have a consistent ordering of the attributes or variables.
    • The JSON standard does not allow the values NaN, +Inf, -Inf which is problematic for attributes (zarr-python #412, zarr-specs #81). However, there is a special case for the fill-value to handle NaN, +Inf and -Inf.
  • All dimensions must be associated to Zarr variables.