Experimental features

Experimental features are feature than not have been thoroughly tested and these features are not considered by the semantic version.

Experimental functions

ncvar = CommonDataModel.ancillaryvariables(ncv::CFVariable,modifier)

data = CommonDataModel.filter(ncv, indices...; accepted_status_flags = nothing)

# da["data"] is 2D matrix
good_data = NCDatasets.filter(ds["data"],:,:, accepted_status_flags = ["good_data","probably_good_data"])

Experimental MPI support

Experimental MPI support is available as a package extension. It is important to load MPI in addition to NCDatasets to enable this package extension. All metadata operations (creating dimensions, variables, attributes, groups or types) must be done collectively. Reading and writing data of netCDF variables can be done independently (default) or collectively. If a variable (or whole dataset) is marked for collective data access, the underlying HDF5 library can enable additional optimization. More information is available in the NetCDF documentation. For the MPI IO standard, collective IO means that all MPI processes execute all the same IO functions (calling for example MPI_File_write_at_all). If this is not the case, then the access is independent (calling for example MPI_File_write_at).

Only the NetCDF 4 format can be currently used for parallel access. On Windows, the MPI interface is currently unsupported. Help from developers with access to Windows would be appreciated.

using MPI
using NCDatasets

MPI.Init()

mpi_comm = MPI.COMM_WORLD
mpi_comm_size = MPI.Comm_size(mpi_comm)
mpi_rank = MPI.Comm_rank(mpi_comm)

# The file needs to be the same for all processes
filename = "file.nc"

# index based on MPI rank
i = mpi_rank + 1

# create the netCDF file
ds = NCDataset(mpi_comm,filename,"c")

# define the dimensions
defDim(ds,"lon",10)
defDim(ds,"lat",mpi_comm_size)
ncv = defVar(ds,"temp",Int32,("lon","lat"))

# enable collective access (:independent is the default)
NCDatasets.paraccess(ncv.var,:collective)

ncv[:,i] .= mpi_rank

ncv.attrib["units"] = "degree Celsius"
ds.attrib["comment"] = "MPI test"
close(ds)

ds = NCDataset(comm::MPI.Comm,filename::AbstractString,
               mode::AbstractString = "r";
               info = MPI.INFO_NULL,
               maskingvalue = missing,
               attrib = [])

NCDatasets.paraccess(ncv::Variable,par_access::Symbol)
NCDatasets.paraccess(ds::NCDataset,par_access::Symbol)

NetCDF 4 user-defined types

NCDatasets.jl supports variable-length arrays, compound types (struct in julia), enums and opaque types. Variable-length arrays can be composed of primitive types, compound types or enums. Variable-length arrays is not considered as an experimental feature.

NetCDF compound types

NetCDF 4 allows the users to define their own type, in particular, compound types which correspond to Julia structures. Compound types can be composed of primitive types, other compound types or enums or vectors of fixed sizes of these types. An array of such structures can be written to and loaded from a NetCDF file. For example:

fname = download("https://raw.githubusercontent.com/Unidata/netcdf-c/refs/tags/v4.8.1/dap4_test/nctestfiles/test_struct_array.nc")
ds = NCDataset(fname)

array = ds["s"][:,:]

propertynames(array[1,1])
# output
#
# (:x, :y)


# access individual elements
array[1,1].x
# output
#
# 1


# access all fields x
getproperty.(array,:x)


struct MyCompoundType
    x::Int32
    y::Int32
end

ds = NCDataset(fname, typemap = Dict("c_t" => MyCompoundType))
array = ds["s"][:,:]
typeof(array)
# output
#
# Matrix{MyCompoundType} (alias for Array{MyCompoundType, 2})

It is preferable in fact that the user defines the compound type as a julia struct and register it using the typemap argument. Users should not rely on the type name generated internally by NCDatasets. Note also that Julia treats two types as different even if they have the same memory layout. When defining these structures, avoid using the type Int as its size is platform-dependent. Vectors of fixed length can also be used in struct fields. They should be declared as NTuples (see Calling C and Fortran Code for the manual). The corresponding julia type definitions of NetCDF compound types and enums can be automatically generated using the ncgen function.

Here is an example to write such a dataset:

n = 5
array2 = MyCompoundType.(1:n,n:-1:1)
fname = tempname()
ds = NCDataset(fname,"c")
defDim(ds,"dim",n)
ncv = defVar(ds,"data",MyCompoundType,("dim",); typename = "my_nc_compound_type")
ncv[:] = array2
close(ds)

# or more compactly:

NCDataset(fname,"c") do ds
    # the Julia type name is used by default in the netcdf file
    # dimension "dim" is created automatically
    ncv = defVar(ds,"data",array2,("dim",))
end

Nested struct/compound types (possibly containing enums) are supported. An important restriction is that the struct must be immutable and contain only immutable fields. The memory layout of a mutable struct is not compatible with the layout expected by the C library. To update a single field in a struct, the user has to recreate the structure. For example to update the field x of the first element to 10:

array2[1] = MyCompoundType(10,array2[1].y)

For large structures, it might be beneficial to use Accessors.

using Accessors
@set array2[1].x = 10

NetCDF enum type

NetCDF enum types are implemented as Julia enum types. This example shows how to create an enum type and write as an vector of enums to a NetCDF file:

@enum TestEnum::Int8 good=1 bad=2 ugly=3

data = [good, bad, good, ugly]
fname = tempname()
NCDataset(fname,"c") do ds
    # the Julia type name is used by default in the netcdf file
    # dimension "dim" is created automatically
    ncv = defVar(ds,"data",data,("dim",))
end

The julia type TestEnum must be internally reconstructed unless it is provided via the typemap parameter (which is preferred). Loading the data:

data2 = NCDataset(fname,"r", typemap = Dict("TestEnum" => TestEnum)) do ds
    ds["data"][:]
end

The array of enums data can be converted to, for example, a CategoricalArray of strings using:

using CategoricalArrays
enum_dict = Dict(inst => string(inst) for inst in instances(eltype(data)))
ca = CategoricalArray([enum_dict[x] for x in data]; levels=collect(values(enum_dict)))

nt = NCDatasets.enums(v::Variable{T}) where T <: Union{Enum,NCEnum}

NCDatasets.typemap!(ds::Dataset, name1 => julia_type1,...)

using NCDatasets
struct MyComplex
    r::Float32
    i::Float32
end

# create some data
n = 10
data = MyComplex.(rand(Float32,n),rand(Float32,n))

fname = tempname()
NCDataset(fname,"c") do ds
    defVar(ds,"data",data,("x",); typename = "nc_complex_t")
end

data = NCDataset(fname) do ds
    # prevent NCDatasets to dynamically reconstruct the struct and use
    # provided type "MyComplex" instead
    NCDatasets.typemap!(ds,"nc_complex_t" => MyComplex)
    ds["data"][:]
end

eltype(data)
# output
#
# MyComplex

NetCDF opaque type

Opaque types represent raw binary data of a fixed size in bytes. They are represented as the Julia type NTuple{len,UInt8} where len is the size of the opaque object in bytes. This example creates an array of 10 elements of a 2-byte opaque type. The user-defined type is called "my_opaque_test" in the NetCDF file.

data_ref = [(UInt8(i),UInt8(i+1)) for i = 1:10]
fname = tempname()

NCDataset(fname,"c") do ds
    defVar(ds,"data",data_ref,("dim",),typename = "my_opaque_test")
end

The data is read in the usual way:

ds = NCDataset(fname)
data = ds["data"][:]
close(ds)