Expose dispatch layer so that users can add their own local binary format

[edhartnett]

Introduction

In order to enable the HYCOM model to be updated to the netCDF API, and to provide a general capability to users, I propose to expose the netCDF-C dispatch layer so that users can add their own local format. This feature, already available in the netCDF-C library internals, is not currently exposed to users.

Background

HYCOM Model and Data Format

The HYCOM modeling team would like to upgrade their model to netCDF, in part to take advantage of the ParallelIO libary, which provides a standard way to ensure good performance in the I/O layer, specifically in the case of asynchronous I/O to a subset of processors using parallel I/O to write to disk.

Many legacy tools have been written based on the existing HYCOM native binary format ("AB Format") Converting the model to netCDF would break all existing tools, leading to loss of significant investment of time and effort, as well as disrupting work with a need to convert all necessary tools at the same time.

Providing and Upgrade Path for HYCOM

With the NetCDF-C dispatch layer exposed, an external plug-in can be provided for the netCDF-C library. Once this plug-in is in place, HYCOM programmers can convert model code to the netCDF API, while still reading and writing their local binary format.

All tools based on the netCDF-C library will transparently read the AB Format (including all tools in Fortran 77, Fortran 90, C++, Python, Perl, Mathmatica, etc.) These tools will also be able to write the AB Format with the netCDF API, allowing them to be converted one at a time.

This will give HYCOM modelers an upgrade path for their existing large code base of code which works with the local binary format. One by one, as time permits, these tools can also be upgraded to the netCDF API.

Eventually, when all tools are converted, the arguments of the nc_create commands in the HYCOM model can be changed to use one of the supported netCDF formats. All tools will transparently read netCDF files, and files in their local format. Historical data (in AB Format) will still be transparently available to netCDF programs. When a file is opened, the netCDF-C library will recognize it as an AB file, and use the plug-in code to read the file. User applications will be able to access the file through the netCDF API.

NettCDF-Java Plug-Ins

The netCDF Java library allows users to write a plug-in which allows them to support read/writes to their own binary format, through the netCDF Java API. The plug-in enables the netCDF library to read/write an arbitrary local format. This feature is widely used in the netCDF-Java community.

The proposed changes will bring this capability to the netCDF-C user community.

General Applicability for the NetCDF Community

Although this work will directly benefit the HYCOM mode, the need for an upgrade path from a local format to netCDF is general. I propose that this become part of the netCDF release and become available to all netCDF-C library users.

Technical Approach

The solution consists of two parts: a plug-in to read and write AB Format, and the changes to the C library necessary to support use of the plug-in, and others like it.

Changes to NetCDF-C to Expose the Dispatch Layer

The C library contains a "dispatch" layer which allows for other formats to be added to the C library. This is how netCDF-4/HDF5, OPENDAP, HDF4, and the parallel-netcdf library are currently supported.

But the C library does not expose this plug-in capability. Changing the C library source code is required in order to add a new version.

For the purposes of this discussion, AB Format, and other such formats, will be called "user-defined formats".

Dispatch Layer Library

The user is responsible for building a dispatch layer library.

This library, when linked with netCDF and a user application, allows the netCDF library to understand the user-defined format. The user-defined format dispatch library implements all the necessary functions from the netCDF dispatch table. (This is a subset of the entire netCDF API).

The user-defined format is registered with a new function, nc_def_user_format().

The AB Format

The local format for the HYCOM model is called "AB Format." It consists of two files, one with metadata, and one with the data stored in binary form.

An AB "file" consists of two files on disk, a binary file with extension "a", and an ASCII text file with metadata with a "b" extension. (The names of the file are part of the format definition.)

The A file is IEEE big-endian, direct access, with a fixed record length.

Converting the AB Format to the NetCDF Internal Model

For the netCDF library to read and write files in AB format, the plug-in must read the metadata file, and provide functions in accordance with the internal needs of the netCDF library. Reading and writing are handled separately.

Reading the AB Format

When reading the AB format the netCDF library needs functions which will:

• identify the file as an AB format file
• read metadata and provide it to netCDF internals
• read the data in subsetted arrays.

Identifying AB Files

AB Files can be identified by name. When an nc_open call is made, and the file name is not found, but two files "FILENAME.a" and FILENAME.b" are found, then the library knows it will be opening an AB file.

Reading AB File Metadata

Once a file has been identified as AB Format, the netCDF library will need to read and understand all the metadata pertaining to that file. This includes the names, types, and sizes of all variables, attributes, and dimensions in the file.

With AB format, code will be written to read the .a file and provide metadata information to the netCDF library.

Reading Subsetted Arrays of AB File Data

The core data read operation of netCDF is nc_get_vara_TYPE (where TYPE varies). This call allows the user to read a subset of a data variable into an n-dimensional array. With this operation, the netCDF-C library implements most of the other read operations.

In the case of AB Format the subsetted array read will be straightforward. Based on the information in the metadata file, the exact offsets to any element of the data can be computed.

Writing the AB Format

In writing the AB Foramt, the netCDF library must:

• Create the AB files.
• Fill the metadata file.
• Implement the subset array write operation.

Creating an AB File

The config file which is read at library load will contain the mode flag which indicates the AB format.

Writing the AB Metadata File

Once the nc_create call returns, no further disk access takes place until the nc_enddef() call. During the time between nc_create() and nc_enddef() the user can define variables, attributes, and dimensions for the file.

When nc_enddef() is called, the AB Plug-in code muse write the B file, which contains the metadata for the file.

Only the netCDF-classic data model will be supported by the AB Format Plug-in. Other restrictions on types and objects may be necessary.

Subset Write Operation

The netCDF subset write operation nc_put_vara() is the key write operation from which other write operations are built. Once implement for the AB format, all netCDF write operations will work for the files.

Conclusion

Exposing the netCDF-C library dispatch layer will benefit the HYCOM modeling group, and also other netCDF-C users (including all netCDF fortran, python, perl, etc.)

Comments and feedback welcome. Please add to this issue so that all can participate in discussion.

[edhartnett]

OK, I now have this working on HPC NetCDF. It's really neat. (@wkliao this is one of the interesting new features I mentioned).

To see it in use, take a look at this github project:
https://github.com/HPC-NetCDF/ab-dispatch

In this project, HPC NetCDF is used, and a new netcdf function is called: nc_def_user_format().

After that function is called, the AB Dispatch code is used to read the file. At the moment I've just stuck the HDF4 code in there, so instead of reading AB format, the AB Dispatch code is reading HDF4. ;-) However, that code just has to be swapped out for AB format code that does the same thing (i.e. read the file and present it in the netCDF data model).

Here's how it looks in code:

   printf("\nTesting AB format dispatch layer...");
   if ((ret = nc_def_user_format(NC_UF0, &AB_dispatcher, NULL)))
      return ret;

   if ((ret = nc_open(TEST_FILE, NC_UF0, &ncid)))
      return ret;
   if ((ret = nc_inq(ncid, &ndims, &nvars, &natts, &unlimdimid)))
      return ret;
   /* printf("ndims %d nvars %d natts %d unlimdimid %d\n", ndims, nvars, natts, unlimdimid); */
   if (ndims != 4 || nvars != 6 ||natts !=10 || unlimdimid != -1)
       return 111;
   if ((ret = nc_close(ncid)))
      return ret;

I have added 2 user formats, NC_UF0 and NC_UF1, so the user can define two different formats at the same time, and use the netCDF API to interchangeably read/write them.

It's a neat feature. I will announce it on the netCDF mailing list sometime this week.

It's going to take time to get this into Unidata netCDF, due to the backlog of my PRs. To prevent out of order merging I'll hold off on putting this up until all or most of my currently-pending PRs are merged.

[edhartnett]

Here's a slide showing how user formats will work.

[DennisHeimbigner]

Duplicate of comment at: #814 (comment)

In restrospect, I made a mistake in making the # of entries in the dispatch
table (include/ncdispatch.h) differ depending on whether or not netcdf4
was enabled. The IMO proper way to do this is to make all dispatch table
entries be included and implement them to return NC_ENOTNC4 in e.g.
libsrc and libdap2 and pnetcdf.
P.S Making the dispatch table fixed size will also help Ed attempt at dynamic loading.

[DennisHeimbigner]

One critical issue is telling the netcdf library where and how to load the
dynamic format .so (or .dll) file. It is instructive to see how HDF5 does this for
its filter plugins. I have a short description in docs/filters.md#Process.

[edhartnett]

But this turns out not to be necessary with this solution.

The user-format dispatch layer is a library, which uses the netCDF library. (And the netCDF library does not have to be re-compiled.)

Then the user writes a program, which uses the dispatch layer library he/she has just written. And that library uses netCDF.

So our code code does not have to deal with .so or .dll files. All functions are available at program compile time, either in the user-dispatch library or the netCDF library.

[DennisHeimbigner]

From my point of view, this is a bad and complicated solution.
This no different from our current situation of compiling library usage
into the netcdf library. To call it dynamic seems incorrect.

[DennisHeimbigner]

Let me paraphrase your proposal and tell me where I have it wrong.

1. You have wrapper library on top of netcdf-c
2. In order to add a new format, you implement the netcdf API
   for that format
3. You then add the format implementation code to your wrapper
   source code repo, establish some connections (e.g. pointer to the api/dispatch table
   plus a mode flag + whatever).
4. You then recompile the wrapper (with the format code).
5. The user then uses this newly compiled wrapper in place of the netcdf
   library.

[edhartnett]

First there are some changes in netCDF:

1 - I added function nc_def_user_format() which takes a pointer to a dispatch table, a mode flag (NC_UF0 or NC_UF1), and an optional magic number (not yet used).

2 - I modified the netCDF dispatch code in dfile.c so that when it gets NC_UF0 or NC_UF1 it uses the provided user-format dispatch table. (Magic number modifications still to come.)

Now for a user to implement their own binary format:

1 - User writes a library that implements the required netCDF dispatch functions. It is not a wrapper for netCDF. (Example here: https://github.com/HPC-NetCDF/ab-dispatch). It's the same code as lives in the existing dispatch directories (like HDF4), but it lives in its own repo and library. It is not mixed with netCDF code (but does use netCDF headers). The library contains a dispatch table, with pointers to all the functions needed.

2 - Only the functions needed in a dispatch layer are required. It can easily reuse standard functions that will work (as the HDF4 layer reuses all the existing nc_inq_* functions). So not every function is implemented.

3 - The user writes a program that links to both netCDF and their own, separately compiled and installed, dispatch library. (It actually does not have to be separate from the user application, but it is separate from netCDF.)

4 - The user code calls nc_def_user_format(), with a pointer to the dispatch table in the user-format dispatch library. In this call, user also associates one of two pre-defined mode flags (NC_UF0 or NC_UF1) with the format.

5 - From then on, NC_UF0 in the mode flag means use this custom dispatch table. The netCDF library is not recompiled or wrapped.

6 - Other modeling groups, working on the same HPC with the same netCDF installation, can define their own NC_UF0, by writing and linking to their own dispatch layer. Each program can have up to two user-defined formats in use at the same time (in addition to all the usual netCDF formats).

This way the code for the user-defined dispatch layer is completely and always outside the netCDF code. Users can develop dispatch layers, exchange them, or whatever. They can write a custom dispatch layer for a proprietary format, and not release it. It's a separate library, which plugs in to netCDF. Dynamic loading is taken care of when a pointer to the dispatch table is passed to nc_def_user_format().

So this is quite different from the current situation, where each dispatch layer has to be built into the library, and then the library must be rebuilt.

[edhartnett]

Here's a little sneak preview of some code. This is all working code, BTW.

/* User-defined formats. */
NC_Dispatch* UF0_dispatch_table = NULL;
NC_Dispatch* UF1_dispatch_table = NULL;

/**
 * Add handling of user-defined format.
 *
 * @param mode_flag NC_UF0 or NC_UF1
 * @param dispatch_table Pointer to dispatch table to use for this user format.
 * @param magic_numer Magic number used to identify file. Ignored if NULL.
 *
 * @return ::NC_NOERR No error.
 * @return ::NC_EINVAL Invalid input.
 * @author Ed Hartnett
 */
int
nc_def_user_format(int mode_flag, NC_Dispatch *dispatch_table, char *magic_number)
{
   if (mode_flag != NC_UF0 && mode_flag != NC_UF1)
      return NC_EINVAL;
   if (!dispatch_table)
      return NC_EINVAL;

   switch(mode_flag)
   {
   case NC_UF0:
      UF0_dispatch_table = dispatch_table;
      break;
   case NC_UF1:
      UF1_dispatch_table = dispatch_table;
      break;
   }
   
   return NC_NOERR;
}

Later, in NC_open():

   /* Check for use of user-defined format 0. */
   if (cmode & NC_UF0)
   {
      if (!UF0_dispatch_table)
         return NC_EINVAL;
      model = NC_FORMATX_UF0;
      dispatcher = UF0_dispatch_table;
   }

   /* Check for use of user-defined format 1. */
   if (cmode & NC_UF1)
   {
      if (!UF1_dispatch_table)
         return NC_EINVAL;
      model = NC_FORMATX_UF1;
      dispatcher = UF1_dispatch_table;
   }

Then, in my sample user-defined format library, I have:

NC_Dispatch AB_dispatcher = {

NC_FORMATX_UF0,

AB_create,
AB_open,

AB_redef,
AB__enddef,
AB_sync,
AB_abort,
AB_close,
AB_set_fill,
AB_inq_base_pe,
AB_set_base_pe,
AB_inq_format,
AB_inq_format_extended,

NC4_inq,
NC4_inq_type,

AB_def_dim,
NC4_inq_dimid,
NC4_inq_dim,
NC4_inq_unlimdim,
AB_rename_dim,

NC4_inq_att,
NC4_inq_attid,
NC4_inq_attname,
AB_rename_att,
AB_del_att,
NC4_get_att,
AB_put_att,

AB_def_var,
NC4_inq_varid,
AB_rename_var,
AB_get_vara,
AB_put_vara,
NCDEFAULT_get_vars,
NCDEFAULT_put_vars,
NCDEFAULT_get_varm,
NCDEFAULT_put_varm,

NC4_inq_var_all,

AB_var_par_access,
AB_def_var_fill,

NC4_show_metadata,
NC4_inq_unlimdims,

NC4_inq_ncid,
NC4_inq_grps,
NC4_inq_grpname,
NC4_inq_grpname_full,
NC4_inq_grp_parent,
NC4_inq_grp_full_ncid,
NC4_inq_varids,
NC4_inq_dimids,
NC4_inq_typeids,
NC4_inq_type_equal,
AB_def_grp,
AB_rename_grp,
NC4_inq_user_type,
NC4_inq_typeid,

AB_def_compound,
AB_insert_compound,
AB_insert_array_compound,
AB_inq_compound_field,
AB_inq_compound_fieldindex,
AB_def_vlen,
AB_put_vlen_element,
AB_get_vlen_element,
AB_def_enum,
AB_insert_enum,
AB_inq_enum_member,
AB_inq_enum_ident,
AB_def_opaque,
AB_def_var_deflate,
AB_def_var_fletcher32,
AB_def_var_chunking,
AB_def_var_endian,
AB_def_var_filter,
AB_set_var_chunk_cache,
AB_get_var_chunk_cache,

};

Note the used of NC4_ and NCDEFAULT functions to lighten the load. I only implemented the functions that start AB_, and many of those just return NC_EPERM since this is read only.

To use the AB dispatch layer, I do this:

   printf("\nTesting AB format dispatch layer...");
   if ((ret = nc_def_user_format(NC_UF0, &AB_dispatcher, NULL)))
      return ret;

   if ((ret = nc_open(TEST_FILE, NC_UF0, &ncid)))
      return ret;
   if ((ret = nc_inq(ncid, &ndims, &nvars, &natts, &unlimdimid)))
      return ret;
   /* printf("ndims %d nvars %d natts %d unlimdimid %d\n", ndims, nvars, natts, unlimdimid); */
   if (ndims != 4 || nvars != 6 ||natts !=10 || unlimdimid != -1)
       return 111;
   if ((ret = nc_close(ncid)))
      return ret;

   printf("SUCCESS!\n");
   return 0;

[DennisHeimbigner]

Ok, I see. It is actually a pretty good solution. We ought to propose
a variant to the HDF5 group as a means for defining a filter plugin.

I might suggest we provide an extra function for the dispatch table
that return true/false if an existing file path or URL can be read
by that format. Netcdf-java has such, and it would simplify
the current NC_open code in libdispatch/dfile.c.

However, this solution would not work for existing systems that
use netcdf-c: systems like R and IDL. It would also be
difficult for python and Java users.

One problem I see is mode flag conflicts. We have a limited set of
available mode flags (about 16 bits). We may need to consider extending
the arguments to nc_open and nc_create. We might also consider changing
the mode to be of type long so that is can be 64 bits on 64bit machines.

[edhartnett]

This system will work for R, IDL, Fortran, and any other layer that sits on top of the C library. BUT the user-format dispatch layer must be written in C. And there must be one C call to set the user format (although we can probably do this in most languages with a void *.)

Once that is done, it works completely within the C layer. If the Fortran program passes NC_UF0 in to nf_open() it will all work. Same with python, as it uses the C library.

It does nothing for Java but they already have this feature. We are just trying to catch up. ;-)

For most HPC users and modeling groups, this will be useful. There will be one or two non-netCDF formats, and some C programmer will write the dispatch layer, and all the fortran programmers will just link to the user-format dispatch library, just as they do to netCDF. Useful formats can even be distributed by helpful folk like @opoplawski if they become popular, and then users can just install them with some package management tool.

The user does not add mode flags. I provide two: NC_UF0 and NC_UF1. This allows two different user-defined formats to be in use in the same program. There are still some mode flags available after that. I agree about the eventual mode flag scarcity problem, but we have not run out yet.

So I would propose to separate any discussion of mode flag scarcity from the issue of user-defined formats.

[edhartnett]

Some other programming notes:

• Agree with your proposed extra function.
• Some functions now called NC4_inq_* will be renamed after the HDF5 code is separated from the internal metadata code, functions that use only the internal metadata will get some name like NCDEFAULT_ (to match existing usage).
• Some easier provision can be made for functions that return NC_EPERM. I think read-only user-defined formats will be popular. So we can make it easier for programmers by defining NCDEFAULT style versions of those functions, which just return NC_EPERM.
• When all this happens it will be useful to remove the distinction between classic and netCDF-4 dispatch tables.

[DennisHeimbigner]

This does not sound right. R and IDL are interpreters and have AFAIK no way
to access your format code when they are run.

There are a different set of problems for systems like fortran, java, and python.
writing the c accessing code in these languages is not trivial and generally
requires expert knowledge. So for every format handler, someone will also
have to write the fortran/java/python wrapper for it.

[DennisHeimbigner]

In thinking about this, I do not see why you need to us NC_UF0 and NC_UF1.
There is no obvious difficulty for the user to define multiple formats.

[DennisHeimbigner]

Ok, I take back part of what I said.
It see that the key is that you need to pass a reference to some
structure in the format implementation to the netcdf library so it can set up
the dispatch. So you still need to define a wrapper function in fortran/java/python
that wraps a single C function in the format code.
This function returns the pointer to that structure.
And there needs to be such a function for every format to be used.
However for the languages like python and Java, the format code
needs to be encoded as a shared library. For fortran/c/c++, a shared libary
can be avoided by including the .o files when linking the user's program.
Seems to me that with a little care in defining the passed structure an hdf5
plugin approach can be provided as well.

[edhartnett]

I believe that python, IDL, etc. already have a mechanism for dynamically loading C libraries. So they would use that method to load the dispatch library.

All of these languages will let you wrap a C function that actually calls nc_def_user_format(), so it is not even necessary to wrap that in Fortran, or any other language. As part of their dispatch layer use in a python program, they must load their dispatch library, then they must call a C function that calls nc_def_user_format(). From that point on, they are good to go.

Yes, for fortran and C programs the dispatch layer can just be built right into the user code. However, I am providing a sample library dispatch layer, so probably it will be easiest for users to just copy that and make it a separate library.

I will also say that the primary requirement here is C and Fortran. This is for models and they are not written in Python. However, as I say, I believe it will work just fine for python too.

[edhartnett]

To answer your questions about NC_UF0 and NC_UF1:

• If we let users define as many modes as are available, then when we later want to add a new mode, we can't guarantee is is not already in use by a user. With UF0/UF1 we know that all the others are still free.
• User code will be more clear if everyone uses the same mode flag for their user-defined format. If everyone start making up mode flags, then code will be confusing. But if you see UF0 you know that it is a user defined format.
• If we let the user try to pick an unused mode flag, that will be a rich source of error. They will reuse one, and uproarious hilarity will ensue.

So those are my reasons. Not married to anything about this implementation, it's just my first pass.

[DennisHeimbigner]

Effectively you are just reserving some # of bits (2 in your case) in the mode flag
for user formats. The name is irrelevant.

[DennisHeimbigner]

One thing to remember is that there is no defined mapping between those mode flags
and the format because two different users can attach the same flag to
different formats.

[edhartnett]

Yes and yes.

The mode flags mean nothing until nc_def_user_format() is called to associate them with a dispactch table.

If the users uses a different dispatch table and then tries to read NC_UF0, then obviously it will not work.

[edhartnett]

@DennisHeimbigner I've been thinking about your remarks on ncdump on another thread. Now I understand. Without recompiling ncdump, it cannot understand user defined formats.

I think here is the case where libraries must be dynamically loaded by our code.

If a new command-line argument is added to ncdump which is the user-defined library name, then that library can be dynamically loaded in ncdump, and then ncdump can call its dispatch functions.

[DennisHeimbigner]

Before this goes much further, we need to have
a software architecture and design document to
codfy the API and the critical implementation
details. Something in markdown like a cross between
this https://github.com/Unidata/netcdf-c/blob/master/docs/filters.md
and this https://support.hdfgroup.org/HDF5/doc/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

[DennisHeimbigner]

Some other issues that need to be addressed:

1. format specific error codes
2. NC_FORMAT_XXX, NC_FORMATX_XXX
3. nc_inq_format() and nc_inq_format_extended()

[DennisHeimbigner]

Another issue to consider:
we need to version the data structures we will be using.
Also, we need to consider at some point trying to
simplify the dispatch table. E.g. combine all the filter definition
related stuff into a single dispatch entry..

[edhartnett]

OK, I will assemble some documentation. Also, in light of these conversations I will do some of the dispatch work we have discussed first, to clear away the underbrush and make this feature's implementation more clear. So let me return to this after #856. That will make the necessary interfaces much more clear.

[edhartnett]

@DennisHeimbigner just joking about documentation! No resistance to documenting at all. I love documenting. The hard thing is to get me to shut up.

After the current HDF4 changes are merged I have another HDF4-only change that incrementally moves towards some of these changes. Basically I take the HDF4 only fields out of the netCDF-4 info structs and use a void pointer, as we discussed. In the case of HDF4, the void pointer points to the HDF4 info struct.
(In a future evolution, it will point to the HDF5 info. for HDF5 files.)

This cleans the HDF4 stuff out of the header files, and will allow me to start to isolate the internal API that is needed by the dispatch layer code.

As part of that I will start to add to the internal dispatch documentation, as needed, to explain the changes.

[edhartnett]

There has been a lot of development of the user-defined format feature in HPC NetCDF. And it's a very powerful feature.

I have been writing the ab-dispatcher, a dispatch layer for the AB format used in the HYCOM model.

Recently I have gotten ncdump working with user-defined formats in a general way. Now I can do an ncdump on an AB format file! It's neat!

In order for this to work, the user must build the user-defined format library, and then provide information about it at netCDF compile time. Then, with the use of the new -u option, ncdump can read files of user-defined format.

 ./ncdump -h -u 0 /home/ed/tmp/ab-dispatch/test/surtmp_100l.b
netcdf surtmp_100l {
dimensions:
	day = 123 ;
	j = 175 ;
	i = 258 ;
variables:
	float day(day) ;
	float surtmp(day, j, i) ;
		surtmp:day = 36495.5f, 36495.75f, 36496.f, 36496.25f, 36496.5f, 36496.75f, 36497.f, 36497.25f, 36497.5f, 36497.75f, 36498.f, 36498.25f, 36498.5f, 36498.75f, 36499.f, 36499.25f, 36499.5f, 36499.75f, 36500.f, 36500.25f, 36500.5f, 36500.75f, 36501.f, 36501.25f, 36501.5f, 36501.75f, 36502.f, 36502.25f, 36502.5f, 36502.75f, 36503.f, 36503.25f, 36503.5f, 36503.75f, 36504.f, 36504.25f, 36504.5f, 36504.75f, 36505.f, 36505.25f, 36505.5f, 36505.75f, 36506.f, 36506.25f, 36506.5f, 36506.75f, 36507.f, 36507.25f, 36507.5f, 36507.75f, 36508.f, 36508.25f, 36508.5f, 36508.75f, 36509.f, 36509.25f, 36509.5f, 36509.75f, 36510.f, 36510.25f, 36510.5f, 36510.75f, 36511.f, 36511.25f, 36511.5f, 36511.75f, 36512.f, 36512.25f, 36512.5f, 36512.75f, 36513.f, 36513.25f, 36513.5f, 36513.75f, 36514.f, 36514.25f, 36514.5f, 36514.75f, 36515.f, 36515.25f, 36515.5f, 36515.75f, 36516.f, 36516.25f, 36516.5f, 36516.75f, 36517.f, 36517.25f, 36517.5f, 36517.75f, 36518.f, 36518.25f, 36518.5f, 36518.75f, 36519.f, 36519.25f, 36519.5f, 36519.75f, 36520.f, 36520.25f, 36520.5f, 36520.75f, 36521.f, 36521.25f, 36521.5f, 36521.75f, 36522.f, 36522.25f, 36522.5f, 36522.75f, 36523.f, 36523.25f, 36523.5f, 36523.75f, 36524.f, 36524.25f, 36524.5f, 36524.75f, 36525.f, 36525.25f, 36525.5f, 36525.75f, 36526.f ;
		surtmp:span = 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f, 0.25f ;
		surtmp:min = 9.771892f, 10.78499f, 9.401492f, 6.62843f, 4.607868f, 3.897291f, 4.213979f, 4.424856f, 4.226626f, 4.022602f, 4.136856f, 4.622542f, 4.554771f, 4.664321f, 4.907145f, 5.749945f, 6.398822f, 6.757607f, 7.123997f, 7.393907f, 7.355556f, 7.199125f, 5.509292f, 4.769361f, 5.141631f, 5.896506f, 6.674411f, 7.204914f, 7.653955f, 8.095844f, 8.628254f, 9.426715f, 10.19681f, 10.8054f, 11.26849f, 11.75135f, 11.90478f, 12.05591f, 12.18285f, 12.18917f, 12.19639f, 12.15982f, 9.251261f, 4.093727f, 0.1810672f, -1.352509f, -0.3644944f, 2.234805f, 4.680813f, 5.819631f, 6.325951f, 6.385474f, 6.446225f, 7.241121f, 8.016541f, 9.304774f, 10.86362f, 11.90644f, 12.25582f, 12.66792f, 10.84867f, 7.711861f, 4.648007f, 3.113999f, 2.058515f, -0.4776078f, -1.811485f, -1.072346f, 1.302688f, 2.925264f, 4.287227f, 2.943427f, 2.069484f, -0.971903f, -2.7068f, -2.914489f, -3.389548f, -3.525549f, -2.487972f, -0.9476987f, 0.4470849f, 1.769978f, 2.568594f, 2.248194f, 1.012455f, 0.4613987f, 0.4744164f, 1.511067f, 3.196703f, 4.542253f, 5.570292f, 6.480618f, 7.03563f, 7.337607f, 7.424665f, 7.305072f, 6.841867f, 6.141395f, 5.669068f, 5.445779f, 5.857729f, 6.901529f, 7.535413f, 5.567883f, 4.132844f, 3.37798f, 3.25649f, 4.144887f, 4.801534f, 5.31437f, 6.164811f, 6.354491f, 5.865624f, 4.060117f, 2.045688f, 0.7715751f, -0.4165433f, -1.899319f, -3.70032f, -4.506866f, -3.937608f, -2.206423f, 0.7947709f ;
		surtmp:max = 28.59057f, 28.55973f, 28.51728f, 28.46502f, 28.42407f, 28.39497f, 28.36654f, 28.33718f, 28.35678f, 28.42611f, 28.45828f, 28.4539f, 28.40721f, 28.31663f, 28.22207f, 28.1251f, 28.05723f, 28.01744f, 27.99325f, 27.98465f, 28.02594f, 28.0898f, 28.17212f, 28.27229f, 28.32424f, 28.32764f, 28.31756f, 28.29471f, 28.29529f, 28.31963f, 28.32314f, 28.30661f, 28.28154f, 28.24656f, 28.22466f, 28.21485f, 28.23409f, 28.28334f, 28.33323f, 28.38336f, 28.41042f, 28.41478f, 28.42897f, 28.45376f, 28.46047f, 28.44769f, 28.43469f, 28.42205f, 28.41031f, 28.40067f, 28.36763f, 28.31886f, 28.28504f, 28.2104f, 28.14836f, 28.09814f, 28.03523f, 27.9609f, 27.90944f, 27.92048f, 27.92048f, 27.89946f, 27.87928f, 27.85905f, 27.87305f, 27.95421f, 28.03169f, 28.09938f, 28.16357f, 28.22428f, 28.19538f, 28.07429f, 28.00822f, 27.996f, 28.06264f, 28.20688f, 28.25077f, 28.19443f, 28.13398f, 28.06973f, 28.0083f, 27.95032f, 27.87952f, 27.79702f, 27.74094f, 27.71023f, 27.64829f, 27.55387f, 27.46205f, 27.3736f, 27.27755f, 27.17374f, 27.08916f, 27.0234f, 26.99387f, 27.00098f, 27.0039f, 27.00413f, 27.01702f, 27.04178f, 27.09779f, 27.1843f, 27.27128f, 27.35929f, 27.42496f, 27.46794f, 27.50336f, 27.53091f, 27.5569f, 27.58121f, 28.06085f, 27.65506f, 27.70081f, 27.75047f, 27.78866f, 27.81437f, 27.80693f, 27.76656f, 27.68389f, 27.55849f, 27.43142f, 27.30333f, 27.25132f ;
		surtmp:long_name = " sea surf. temp.  " ;
		surtmp:standard_name = "sea_surface_temperature" ;
		surtmp:units = "degC" ;

// global attributes:
		:att_0 = "FNMOC, 6hrly, degC" ;
		:Conventions = "CF-1.0" ;
}

[WardF]

Great job getting it to work with ncdump, this looks really interesting, congrats!

[DennisHeimbigner]

Agreed. But (there is always a but...:-)
This assumes (as we have discussed)
some way to probe the added dispatch layer to
get information about the format. We need to make sure
that the probe information is sufficiently general.

[edhartnett]

There are two user-defined formats that can be used UF0 and UF1. When you build HPC netCDF you can provide an option like this:

--with-uf0=AB_dispatcher

(If this is provided, you must also provide LDFLAG and LADD options to point to the library that implements a dispatch table called AB_dispatcher, or the build will fail.)

This will tell ncdump that user-format 0 is associated with the AB_dispatcher dispatch table. ncdump registers the new dispatch table before calling nc_open().

When running ncdump, the user must provide the -u0 option, to indicate that user-format 0 should be used for this file. This causes ncdump to use the NC_UF0 for the nc_open() mode flag.

A different user format could also be used as user format 1. It's up to the user to keep straight which is which for their build. (But if they get it wrong, ncdump returns an unknown format error, so that's OK.)

Note sure what you mean by probing. Does the above answer your questions?

[DennisHeimbigner]

No. Put another way, how does the library locate the relevant dispatch table?

[edhartnett]

The library locates the relevant dispatch table using the symbol "AB_dispatcher" (provided to netCDF configure) which is the name of the dispatch table in my dispatch library. The library is specified by adding it to the LIBS flag when building netCDF.

So I can name my dispatch table whatever I like. When I build netCDF, I say --with-uf0=my_dispatch_name, and make sure that the linker can find my dispatch library. The configure defines macro UF0_DISPATCH in config.h. When ncdump is built, the symbol is substituted like this:

#ifdef USE_UF0
extern NC_Dispatch UF0_DISPATCH;
#endif /* USE_UF0 */

...

    /* If user-defined formats are in used, register the dispatch
     * table. */
#ifdef USE_UF0
   if ((ret = nc_def_user_format(NC_UF0, &UF0_DISPATCH, NULL)))
      return ret;
#endif /* USE_UF0 */

This will of course all be fully-documented in a new section of the docs for user-defined formats.

To address some of your other recent comments:

• NC_FORMAT_XXX, NC_FORMATX_XXX - users must pick an unused (by netCDF) value for the format, and it's actually in the dispatch table. I am planning an NCDEFAULT_inq_format() and NCDEFAULT_inq_format_extended() which just return whatever the value in the dispatch table is.

• User-defined error codes are, I think, an unneeded complexity. We can provide a general error for each user-defined format (NC_EUF0/NC_EUF1) which can be used like NC_EHDFERR is used in the HDF5 code. Other than that, they can use one of the already-defined errors, which provide a rich collection appropriate to reading and writing data in the netCDF model. Recall that the vast majority of users are not doing sophisticated return code handling.

• I agree that a refactor of the dispatch table may be advisable before this feature goes public. Once it does the dispatch table becomes a little harder to change. (You will break all user-defined formats when you do.) I suggest that we ponder potential changes, and take a look after I present my PR for Separate internal netCDF-4 data model from HDF5 code #856.

• Another idea is an additional alternative, much slimmer (pseudo-)dispatch table which can be used for read-only formats, which only need 3 or 4 functions. The pseudo-table could be put into a full dispatch table by nc_def_user_format(), and would allow the user to ignore all the other functions, and just take the NCDEFAULT_ versions.

• It turns out that some helper functions in libsrc4 (non-HDF5) code can help all dispatch layers make good use of the netCDF internal data model without exposing its internal elements. We already have many such functions, and with a little refactoring, they can do the job nicely. For example, we have various find, add, and delete functions. We just need to clean up their parameters a bit to hide most of the internals.

[DennisHeimbigner]

I have added to my todo list an item to extend this
to do dynamic loading using code from here:
https://github.com/DennisHeimbigner/cpoco