Experimental features

Experimental features are feature than not have been thoroughly tested and these features are not considered by the semantic version.

Experimental functions

CommonDataModel.ancillaryvariables — Function

ncvar = CommonDataModel.ancillaryvariables(ncv::CFVariable,modifier)

Return the first ancillary variables from the NetCDF (or other format) variable ncv with the standard name modifier modifier. It can be used for example to access related variable like status flags.

source

Base.filter — Function

data = CommonDataModel.filter(ncv, indices...; accepted_status_flags = nothing)

Load and filter observations by replacing all variables without an acepted status flag to missing. It is used the attribute ancillary_variables to identify the status flag.

# da["data"] is 2D matrix
good_data = NCDatasets.filter(ds["data"],:,:, accepted_status_flags = ["good_data","probably_good_data"])

source

Experimental MPI support

Experimental MPI support is available as a package extension. It is important to load MPI in addition to NCDatasets to enable this package extension. All metadata operators (creating dimensions, variables, attributes, groups or types) must be done collectively. Reading and writing data of netCDF variables can be done independently (default) or collectively. If a variable (or whole dataset) is marked for collectively data access, the underlying HDF5 library can enable additional optimization. More information is available in the NetCDF documentation. For the MPI IO standard, collective IO means that all MPI processes execute all the same OI functions (calling for example MPI_File_write_at_all). If this is not the case, then the access is independently (calling for example MPI_File_write_at).

Only the NetCDF 4 format can be currently used for parallel access. On Windows, the MPI interface is currently unsupported. Help from developers with access to Windows would be appreciated.

using MPI
using NCDatasets

MPI.Init()

mpi_comm = MPI.COMM_WORLD
mpi_comm_size = MPI.Comm_size(mpi_comm)
mpi_rank = MPI.Comm_rank(mpi_comm)

# The file needs to be the same for all processes
filename = "file.nc"

# index based on MPI rank
i = mpi_rank + 1

# create the netCDF file
ds = NCDataset(mpi_comm,filename,"c")

# define the dimensions
defDim(ds,"lon",10)
defDim(ds,"lat",mpi_comm_size)
ncv = defVar(ds,"temp",Int32,("lon","lat"))

# enable collective access (:independent is the default)
NCDatasets.paraccess(ncv.var,:collective)

ncv[:,i] .= mpi_rank

ncv.attrib["units"] = "degree Celsius"
ds.attrib["comment"] = "MPI test"
close(ds)

NCDatasets.NCDataset — Method

ds = NCDataset(comm::MPI.Comm,filename::AbstractString,
               mode::AbstractString = "r";
               info = MPI.INFO_NULL,
               maskingvalue = missing,
               attrib = [])

Open or create a netCDF file filename for parallel IO using the MPI communicator comm. info is a MPI info object containing IO hints or MPI.INFO_NULL (default). The mode is either "r" (default) to open an existing netCDF file in read-only mode, "c" to create a new netCDF file (an existing file with the same name will be overwritten) or "a" to append to an existing file.

source

NCDatasets.paraccess — Function

NCDatasets.paraccess(ncv::Variable,par_access::Symbol)
NCDatasets.paraccess(ds::NCDataset,par_access::Symbol)

Change the parallel access mode of the variable ncv or all variables of the dataset ds for writing or reading data. par_access is either :collective or :independent. NCDatasets.paraccess will raise an error if MPI is not loaded.

More information is available in the NetCDF documentation.

source

NetCDF compound types

NetCDF 4 allows the users to define their own type, in particular, compound types which correspond to Julia structures. An array of such structures can be written to and loaded from a NetCDF file. For example:

fname = download("https://raw.githubusercontent.com/Unidata/netcdf-c/refs/tags/v4.8.1/dap4_test/nctestfiles/test_struct_array.nc")
ds = NCDataset(fname)

array = ds["s"][:,:]
typeof(array)
# output
# Matrix{c_t} (alias for Array{NCDatasets.ReconstructedTypes_123.c_t, 2})

struct MyCompoundType
    x::Int32
    y::Int32
end

NCDatasets.usertype!(ds,"c_t",MyCompoundType)
array = ds["s"][:,:]
typeof(array)
# output
# Matrix{MyCompoundType} (alias for Array{MyCompoundType, 2})

It is preferable in fact that the user defines the compound type as a julia struct and register it using the NCDatasets.usertype!. Users should not rely on the type name generated internally by NCDatasets. Note also that Julia treats two types as different even if they have the same memory layout. When defining these structures, avoid using the type Int as its size is platform-dependent. Vectors of fixed length can also be used in struct fields. They should be declared as NTuples (see Calling C and Fortran Code for the manual).

Here is an example to write such a dataset:

n = 5
array2 = MyCompoundType.(1:n,n:-1:1)
fname = tempname()
ds = NCDataset(fname,"c")
defDim(ds,"dim",n)
ncv = defVar(ds,"data",MyCompoundType,("dim",); typename = "my_nc_compound_type")
ncv[:] = array2
close(ds)

# or more compactly:

NCDataset(fname,"c") do ds
    # the Julia type name is used by default in the netcdf file
    # dimension "dim" is created automatically
    ncv = defVar(ds,"data",array2,("dim",))
end

An important restriction is that the struct must be immutable and contain only immutable fields. The memory layout of a mutable struct is not compatible with the layout expected by the C library. To update a single field in a struct, the user has to recreate the structure. For example to update the field x of the the first element to 10:

array2[1] = MyCompoundType(10,array2[1].y)

For large structures, it might be beneficial to use Accessors.

using Accessors
@set array2[1].x = 10