RELEASE_NOTES

=====================================
PnetCDF Release Notes
=====================================

-------------------------------------
Version _PNETCDF_VERSION_ (_PNETCDF_RELEASE_DATE_)
-------------------------------------

* Bug fixes
  + Fix a bug causing programs to hang when calling a blocking collective put
    API and one of the process has a zero-length request. The bug was reported
    in [Issue #238](https://github.com/Parallel-NetCDF/PnetCDF/issues/238).
    See fix in [PR #239](https://github.com/Parallel-NetCDF/PnetCDF/pull/239).
    Note this process hanging bug only happens when using collective put APIs,
    but not other APIs (i.e. nonblocking and independent APIs).
  + Fix a divide-by-zero error when opening a file that is using the Lustre
    Progressive File Layout (PFL). The bug was reported in
    [Issue #234](https://github.com/Parallel-NetCDF/PnetCDF/issues/234).
    See fix in [PR #236](https://github.com/Parallel-NetCDF/PnetCDF/pull/236).
  + Fix the detection of MPI base compiler names when PrgEnv-cray is loaded,
    where cc, CC, and ftn are wrappers of Cray clang and Cray Fortran
    compilers.
    See [PR #233](https://github.com/Parallel-NetCDF/PnetCDF/pull/233).
  + Fix utility program `pnetcdf_version` for printing the version string of
    base compiler of MPICC wrapper when the base is a classic Intel compiler.
    See [PR #232](https://github.com/Parallel-NetCDF/PnetCDF/pull/232).

* Build recipes
  + When using Intel oneAPI compilers to build PnetCDF, users may encounter
    errors at the link time. An ad hoc solution provided by NERSC Consulting
    and Support has been added to `doc/README.CRAY.md`.

* Other updates:
  + All man pages have been updated with string "Print date" being replaced
    with "Last modified date".
    See [PR #230](https://github.com/Parallel-NetCDF/PnetCDF/pull/230).

* New test programs
  + tst_zero_len_recvar.c and tst_zero_len_recvar2.c in folder test/testcases
    were added to test the bug reported in
    [Issue #238](https://github.com/Parallel-NetCDF/PnetCDF/issues/238).
  + test/Lustre contains programs for testing the bug found in
    [Issue #234](https://github.com/Parallel-NetCDF/PnetCDF/issues/234) and
    resolved in [PR #236](https://github.com/Parallel-NetCDF/PnetCDF/pull/236).

-------------------------------------
Version 1.15.0 (July 1, 2026)
-------------------------------------

* New feature
  + Intra-node aggregation for read requests is added. This is the complement
    to the write counterpart first added to version 1.14.0. Now the intra-node
    aggregation feature supports both write and read operations. This feature
    can be enabled by setting hint `nc_num_aggrs_per_node` to the desired
    number of aggregators per compute node.

* New optimization
  + When creating a new file on the Lustre file system, PnetCDF will try to set
    the Lustre file striping count to the number of compute nodes (NUMA nodes)
    allocated to the MPI communicator passed to the call of `ncmpi_create()`.
    See the section of "New PnetCDF Hints" below for more information.

* New I/O driver
  + A submodule named "GIO" has been added as the PnetCDF's default I/O driver.
    PnetCDF has been using MPI-IO as its only I/O driver since its first
    release. With the GIO driver, PnetCDF users now have an additional option
    to tune the performance. GIO is a parallel I/O library, sharing the design
    concept of the MPI-IO. GIO implements several performance improvement, such
    as:
    * automatically sets `cb_nodes` based on the number of compute nodes and
      number of MPI processes running per node.
    * supports hint `cb_buffer_size` to be a multiple of file striping unit
      size for Lustre.
    * supports Lustre overstriping through hint `overstriping_ratio`. See the
      "New PnetCDF hints" section below.

* New configure options
  + `--disable-gio` disables the GIO driver and uses MPI-IO. The default is to
    enable using the GIO driver.

* APIs deprecated
  + The "vard" family APIs introduced in version 1.6.0 are now deprecated.
    These are the APIs that take an argument of MPI derived data type
    describing the file access layout, which is used to set the fileview
    through a call to `MPI_File_set_view()`. The deprecation is because direct
    file access can be a security risk and error prone.
    * ncmpi_get_vard()
    * ncmpi_get_vard_all()
    * ncmpi_put_vard()
    * ncmpi_put_vard_all()

* API semantics updates
  + API `ncmpi_inq_header_size()` now can be called in the define mode. This
    API returns the file header size with metadata defined by the time of the
    call. The inquired file header size can be helpful to pick proper values
    for arguments `h_minfree`, `v_align`, `v_minfree`, `r_align` when calling
    API `ncmpi__enddef()` which can allow to preserve a sufficiently large free
    space for file header extent and variable data sections to grow later
    without moving data already stored in the file, i.e. when adding new
    variables, dimensions, or attributes.
    See [PR #201](https://github.com/Parallel-NetCDF/PnetCDF/pull/201).

* New error code
  + `NC_EDRIVER` indicates an invalid PnetCDF I/O driver is set in I/O hint,
    `nc_driver`. The current valid drivers are "gio" and "mpiio".
  + `NC_EFSTYPE` indicates a PnetCDF internal error when an invalid file system
    type is detected.
  + `NC_EFILEVIEW` indicates a PnetCDF internal error when creating an MPI
    fileview whose offsets violate the MPI standard requirement of being in a
    monotonically non-decreasing order.

* New PnetCDF hints
  + `nc_data_move_chunk_size` -- When adding new data objects into an existing
    file, the data sections may need to be moved to a higher file offset. The
    movement is performed in chunks. This hint allows users to customized the
    chunk size. The default is 1048576 bytes, i.e. 1 MiB.
    See [PR #203](https://github.com/Parallel-NetCDF/PnetCDF/pull/203).
  + `file_striping` -- When creating a new file on the Lustre file system, this
    hint advises PnetCDF to set the new file's striping configuration. The hint
    value is either "auto" or "inherit". The default is "auto". The former sets
    the new file's striping unit size to 1 MiB and striping count to the number
    of compute nodes found in the MPI communicator passed to the call of
    `ncmpi_create()`.  The latter sets the striping to inherit the new file's
    parent folder's settings, if the folder's striping has been set. Note that
    if users also set the MPI-IO hint `striping_factor` or `striping_unit`,
    then these MPI-IO hints will take a higher precedence.
    See [PR #222](https://github.com/Parallel-NetCDF/PnetCDF/pull/222).
  + `nc_driver` -- To select an internal I/O driver. The value is either string
    "gio" or "mpiio". The default is "gio", which advises PnetCDF to use the
    GIO driver, an external library built as a sub-module of PnetCDF. The other
    is "mpiio", advising PnetCDF to use the MPI-IO driver.
  + `overstriping_ratio` -- This hint is passed to the GIO driver and instructs
    it to make use of Lustre's overstriping feature which allows to create a
    new file to have more than one stripe per OST. For instance, when hints
    `striping_factor` and `overstriping_ratio` are set to 16 and 2,
    respectively, the newly created file will be striped across 16/2=8 OSTs.
    The advantage of Lustre overstriping is to allow a file's striping count to
    be larger than the total number of OTS available on the system.

* New run-time environment variables
  + `GIO_ONLY` - when testing PnetCDF by running command `make check` and `make
    ptest`, setting this environment variable will run all the tests using the
    default GIO driver and skip MPI-IO driver. Without setting this environment
    variable, the default is to test both I/O drivers. Note this environment
    variable only affect when testing PnetCDF and takes no effect on PnetCDF
    library itself.

* Build recipes
  + `doc/NERSC.md` - note for building PnetCDF on machines at NERSC
  + `doc/ALCF.md` - note for building PnetCDF on machines at ALCF

* Updated utility programs
  + `pnetcdf_version` - add information about MPI compiler vendors, supported
    MPI standard version, names of base compilers, and LMOD PrgEnv module
    loaded if available.
  + `ncmpidiff` now will error out when the two input file names are identical.

* Other updates:
  + Running `make check` and `make ptest` takes much longer time to complete.
    This is because the testing mechanism has been revised to become more
    thorough than before.

* Bug fixes
  + Fix problem of building PnetCDF shared library when using NVIDIA HPC
    compilers, nvc, nvc++, and nvfortran.
  + Fix data movement when new record variables are added to an existing file
    that does not change the starting offset of record variable section.
    See [PR #199](https://github.com/Parallel-NetCDF/PnetCDF/pull/199).

* New I/O benchmarks
  + benchmarks/C/indep_data_obj_create.c - It evaluates the performance of
    creating a large number of data objects (dimensions and variables) in an
    output file. This benchmark is used in a paper published in the PDSW
    Workshop of SC 2025. DOI: 10.1145/3731599.3767512

* New test programs
  + test/testcases/tst_grow_data.c -- It tests the case when adding new
    variables by re-entering the define mode multiple time without causing the
    file header extent to grow. It also tests a case when adding a new record
    variable that does not change the starting offset of the record variable
    section in the file.

-------------------------------------
Version 1.14.1 (July 31, 2025)
-------------------------------------

* New optimization
  + When file header extent size grows, moving the data section to a higher
    file offset has changed to be done in chunks of 16 MB per process.
    See [PR #174](https://github.com/Parallel-NetCDF/PnetCDF/pull/174),

* Configure options
  + For PnetCDF developers, the requirement for libtool version has been
    changed to 2.5.4, due to an issue on Mac OS when using OpenMPI. See
    [Issue #155](https://github.com/Parallel-NetCDF/PnetCDF/issues/155),
    [Issue #163](https://github.com/Parallel-NetCDF/PnetCDF/issues/163),
    and [PR #164](https://github.com/Parallel-NetCDF/PnetCDF/pull/164).

* New APIs
  + A set of APIs that read the header of a CDL file header and allow users to
    query the metadata defined in the CDL file. These APIs can be useful for
    creating a program (such as an I/O benchmark) based on an output netCDF
    file from another application (a CDL file can be generated by running the
    utility `ncmpidump/ncdump`). See
    [PR #177](https://github.com/Parallel-NetCDF/PnetCDF/pull/177).
    * `cdl_hdr_open()` opens and parses the CDL file's header
    * `cdl_hdr_inq_format()` returns file format version
    * `cdl_hdr_inq_ndims()` returns number of dimensions defined in CDL file
    * `cdl_hdr_inq_dim()` returns metadata of a dimension
    * `cdl_hdr_inq_nvars()` returns number of variables
    * `cdl_hdr_inq_var()` returns metadata of a variable defined in CDL file
    * `cdl_hdr_inq_nattrs()` returns number of attributes of a given variable
    * `cdl_hdr_inq_attr()` returns metadata of an attribute
    * `cdl_hdr_close()` closes the CDL file
  + The CDL Header C Reference Manual is available in
    [cdl_api_guide.md](https://github.com/Parallel-NetCDF/PnetCDF/blob/master/doc/cdl_api_guide.md).
    See [PR #184](https://github.com/Parallel-NetCDF/PnetCDF/pull/184).

* Bug fixes
  + Fix a GCC 15 compilation error. Thanks to Adrian Reber.
    See [PR #179](https://github.com/Parallel-NetCDF/PnetCDF/pull/179).
  + Fix errors when building an RPM. Thanks to Adrian Reber.
    See [PR #178](https://github.com/Parallel-NetCDF/PnetCDF/pull/178).
  + Fix setting of user hint nc_ibuf_size.
    See [PR #161](https://github.com/Parallel-NetCDF/PnetCDF/pull/161).

* New example programs
  + examples/C/create_from_cdl.c shows how to call the new CDL header APIs to
    create a netCDF file with the same metadata.

* New programs for I/O benchmarks
  + WRF-IO contains an extraction of the I/O kernel of WRF (Weather Research
    and Forecast Model, a weather prediction computer simulation program
    developed at NCAR) that can be used to evaluate the file I/O performance
    of WRF. It's data partitioning pattern is a 2D block-block checkerboard
    pattern, along the longitude and latitude.
    See [PR #165](https://github.com/Parallel-NetCDF/PnetCDF/pull/165)
    and [PR #181](https://github.com/Parallel-NetCDF/PnetCDF/pull/181).

* New test program
  + test/cdf/tst_cdl_hdr_parser.c tests the new CDL header APIs.
  + test/testcases/tst_grow_header.c tests header extent growth by re-entering
    the define mode multiple times and add more fix-sized and record variables.

* Clarifications about of PnetCDF hints
  + There are three ways in PnetCDF for user to set hints to align the starting
    file offset for the data section (header extent) and record variable
    section.
    1. through a call to API `ncmpi__enddef` by setting arguments `h_minfree`,
       `v_align`, `v_minfree`, and `r_align`.
    2. through passing an MPI info object to a call of `ncmpi_create()` and
       `ncmpi_open()`. Hints include `nc_header_align_size`,
       `nc_var_align_size`, and `nc_record_align_size`.
    3. through setting hints in an environment variable `PNETCDF_HINTS` at the
       run time. Hints include `nc_header_align_size`, `nc_var_align_size`, and
       `nc_record_align_size`.
  + As the same hints may be set by one or more of the above methods, PnetCDF
    implements the following hint precedence.
    * `PNETCDF_HINTS` > `ncmpi__enddef()` > `MPI info`.
    * 1st priority: hints set in the run-time environment variable
      `PNETCDF_HINTS`, e.g. `PNETCDF_HINTS="nc_var_align_size=1048576"`. Making
      this the first priority is because it allows the same application
      executable without source code modification to run using different
      alignment settings in the run-time environment variable.
    * 2nd priority: hints set in the MPI info object, e.g.
      `MPI_Info_set("nc_var_align_size", "1048576");`, passed to calls of
      `ncmpi_create()` and `ncmpi_open()`. The reasoning is when a 3rd-party
      library built on top of PnetCDF may call 'ncmpi__enddef' with its setting
      of alignment values. An application that uses such 3rd-party library can
      pass an MPI info object to it, which is then passes to PnetCDF. This
      precedence allows that application to exercise different hints without
      changing the 3rd-party library's source codes.
    * 3rd priority: hints used in the arguments of `ncmpi__enddef()`, e.g.
      `ncmpi__enddef(..., v_align=1048576,...)`.
  + PnetCDF I/O hint `nc_header_align_size` is essentially the same as hint
    `nc_var_align_size`, but its name appears to be closer to the hint's
    intent, i.e. to reserve some space for the header growth in the future when
    new data objects are added. Please note when both hints are set, only hint
    `nc_var_align_size` will take effect and `nc_header_align_size` ignored.
  + When there is no fix-sized variable (i.e. non-record variable) defined,
    argument `v_minfree` passed to `ncmpi__enddef()` is ignored. In this
    case, users should set `h_minfree`, if an extra header space is desired.
  + When there is no fix-sized variables defined and none of hints
    `nc_header_align_size`, `nc_var_align_size`, or argument `v_align` is set,
    `nc_record_align_size` or `r_align`, if set, will be used to align the
    header extent.
  + For the above update to the hint precedence, see
    PnetCDF See [PR #173](https://github.com/Parallel-NetCDF/PnetCDF/pull/173).


-------------------------------------
Version 1.14.0 (November 11, 2024)
-------------------------------------

* New features
  + Intra-node aggregation for write requests -- When the number of MPI
    processes allocated to a compute node is large, this feature can
    effectively reduce the communication congestion caused by an overwhelming
    large number of asynchronous messages posted during the collective write
    of MPI-IO. This new feature can be enabled by setting the PnetCDF I/O hint
    'nc_num_aggrs_per_node' to the desired number of aggregators per compute
    node. The non-aggregators send their requests to the assigned aggregators,
    and then the aggregators make aggregated requests to the file.
    [PR #156](https://github.com/Parallel-NetCDF/PnetCDF/pull/156).
  + Support MPI derived data types that are constructed from the large-count
    derived datatype constructors introduced in MPI 4.0.
    [PR #145](https://github.com/Parallel-NetCDF/PnetCDF/pull/145).

* New optimization
  + When running sequentially (i.e. number of processes is 1), PnetCDF calls
    the MPI independent I/O functions and avoids calls to MPI_Barrier,
    MPI_Bcast, and MPI_Allreduce.
    [PR #149](https://github.com/Parallel-NetCDF/PnetCDF/pull/149).

* Configure options changed
  + The default has been changed to build both shared and static libraries.
    [PR #143](https://github.com/Parallel-NetCDF/PnetCDF/pull/143).

* Configure updates:
  + Fix `pnetcdf-config` of reflecting the installation path when installation
    is done by running command `make install DESTDIR=/alternate/directory`
    which prepends '/alternate/directory' before all installation names.
    [PR #154](https://github.com/Parallel-NetCDF/PnetCDF/pull/154).

* New constants
  + A new C macro `NC_FillValue` replaces `_FillValue` and thus `_FillValue` is
    now deprecated This conforms with NetCDF4's change in its version 4.9.3
    release. [PR #153](https://github.com/Parallel-NetCDF/PnetCDF/pull/153).

* New PnetCDF hints
  + 'nc_num_aggrs_per_node' -- To enable the intra-node aggregation, this I/O
    hint can set to a positive integral value, which indicates the desired
    number of processes per compute node to be selected as the aggregators.
    Setting it to 0 disables the aggregation, which is also the default mode.
    [PR #156](https://github.com/Parallel-NetCDF/PnetCDF/pull/156).

* Build recipes
  + When using OpenMPI on Mac OSX, a link error may appear. The work around is
    to add `LDFLAGS=-ld_classic` into the configure command line. Thanks to
    Rui Chen for reporting and provide the solution.
    [Issue #139](https://github.com/Parallel-NetCDF/PnetCDF/issues/139).

* Updated utility programs
  + none

* Other updates:
  + More document for comparing PnetCDF and NetCDF4 has been added to file
    doc/netcdf4_vs_pnetcdf.md
    [PR #152](https://github.com/Parallel-NetCDF/PnetCDF/pull/152) and
    [PR #140](https://github.com/Parallel-NetCDF/PnetCDF/pull/140).

* New example programs
  + C/flexible_bottom.c and C/vard_bottom.c - These two examples construct MPI
    derived data types using absolute memory addresses first and use MPI_BOTTOM
    when calling the PnetCDF flexible APIs.

* New programs for I/O benchmarks
  + C/pnetcdf_put_vara.c --
    * This program writes a series of 3D variables with 2D block-block
      partitioning pattern. Each variable is a record variable.
      [PR #150](https://github.com/Parallel-NetCDF/PnetCDF/pull/150).
  + C/netcdf_put_vara.c --
    * This sequential NetCDF-C program writes a series of 3D variables. Each
      variable is a record variable.
    * This program and `C/pnetcdf_put_vara.c` can be used to compare the
      performance of NetCDF and PnetCDF when running sequentially, i.e. one
      process.
      [PR #150](https://github.com/Parallel-NetCDF/PnetCDF/pull/150).

* New test program
  + test/testcases/flexible_large_count.c - tests flexible APIs that use MPI
    derived datatypes created by MPI large-count datatype constructors.
    [PR #145](https://github.com/Parallel-NetCDF/PnetCDF/pull/145).


-------------------------------------
Version 1.13.0 (March 29, 2024)
-------------------------------------

* New features
  + A single read/write request made by an MPI process is now allowed to be of
    size larger than 2 GiB. Such large requests will be passed to the MP-IO
    library. This feature makes use of "the large count feature" introduced in
    MPI standard 4.0, which includes `MPI_XXX_c` APIs whose arguments are of
    type `MPI_Count`. `MPI_Count` can be an 8-byte integer type, enabling large
    MPI operations. As some MPI libraries today have begun implementing MPI
    4.0, PnetCDF now can rely on the MPI libraries to support large single
    requests. When the MPI library used to build PnetCDF does not support large
    requests, the MPI errors are returned. Because of this change, the PnetCDF
    configure option `--enable-large-single-req` is thus deprecated.
    See [PR #131](https://github.com/Parallel-NetCDF/PnetCDF/pull/131)
  + Flexible APIs now can operate as high-level APIs, when argument `bufcount`
    is set to `NC_COUNT_IGNORE` and `buftype` is set to an MPI predefined data
    type. See [PR #82](https://github.com/Parallel-NetCDF/PnetCDF/pull/82).
    Below is a write example from a buffer of type float.
    ```
    ncmpi_put_vara_all(ncid, varid, start, count, buf, NC_COUNT_IGNORE, MPI_FLOAT);
    ```
    is equivalent to
    ```
    ncmpi_put_vara_float_all(ncid, varid, start, count, buf);
    ```

* New Limitations
  + Hint `nc_header_read_chunk_size`, introduced in version 1.4.0, is now
    limited to `NC_MAX_INT`. As PnetCDF reads file header in chunks, this hint
    can be used to customize the chunk size. The default is 256 KB.
    See [4209056](https://github.com/Parallel-NetCDF/PnetCDF/commit/4209056e9a66465421f7ce9f1b44518923638b04)

* Configure options
  + `--enable-large-single-req` is deprecated and removed, as PnetCDF now
    allows a single reqd/write request of size larger than 2 GiB.
  + `--disable-file-sync` is deprecated and removed. This configure option
    alone was not able to provide a sufficient data consistency. Users are
    suggested to call `ncmpi_sync` and `MPI_Barrier` to achieve a desired
    consistency, as suggested by MPI standard.
  + A new option `--enable-install-examples` installs the example programs
    under folder `${prefix}/pnetcdf_examples` along with run script files. An
    example is `${prefix}/pnetcdf_examples/C/run_c_examples.sh`. The default of
    this option is `disabled`.
    See [PR #91](https://github.com/Parallel-NetCDF/PnetCDF/pull/91)
  + Add three new environment variables `SEQ_CFLAGS`, `SEQ_LDFLAGS` and
    `SEQ_LIBS` for setting the compile, link, and library flags, respectively
    to be used to build the sequential utility programs, i.e. `cdfdiff`,
    `ncoffsets`, `ncvalidator`, and `pnetcdf_version`.
    See [PR #122](https://github.com/Parallel-NetCDF/PnetCDF/pull/122)

* Configure updates:
  + Upgrade `config.guess` and `config.sub` to 2024-01-01.
    See [PR #116](https://github.com/Parallel-NetCDF/PnetCDF/pull/116)
  + FLASH-IO benchmark - add compile flag "-fallow-argument-mismatch" for GNU
    Fortran 10 and later.
    See [PR #114](https://github.com/Parallel-NetCDF/PnetCDF/pull/114)
  + Handle the case when MPICC environment variable is not set and `--with-mpi`
    is not used.  See commit
    [6142135](https://github.com/Parallel-NetCDF/PnetCDF/commit/61421356ecd38878a4ef46771ed6520d4257251f)
  + Upgrade Autotools version requirement to autoconf 2.71, automake 1.16.5, and
    libtool 2.4.6. (Note this change affects PnetCDF developers only.)
    See [PR #95](https://github.com/Parallel-NetCDF/PnetCDF/pull/95)
    Thanks to Blaise Bourdin for pointing out in
    [Issue #94](https://github.com/Parallel-NetCDF/PnetCDF/issues/94)
    that configure failed when using Intel OneAPI 2022.2.0 compilers. The fix
    is to use autoconf 2.70 and newer.

* New constants
  + `NC_COUNT_IGNORE` - This is used in flexible APIs. When argument `bufcount`
    is `NC_COUNT_IGNORE`, `buftype` must be a predefine MPI datatype and the
    APIs operate as the high-level APIs. Fortran equivalents are
    `NF_COUNT_IGNORE` and `NF90_COUNT_IGNORE`.
    See [PR #92](https://github.com/Parallel-NetCDF/PnetCDF/pull/92)

* API semantics updates
  + File open flag `NC_SHARE` is now deprecated. It is still defined, but takes
    no effect.
    See [PR #119](https://github.com/Parallel-NetCDF/PnetCDF/pull/119)
  + `NC_SHARE` alone is not sufficient to provide data consistency for accessing
    a shared file in parallel and thus is now deprecated. PnetCDF follows the
    file consistency defined in MPI standard, which only addresses the case
    when all file accesses are relative to a specific file handle created from
    a collective open, and thus `NC_SHARE` becomes invalid. See
    doc/README.consistency.md for more information.

* New PnetCDF hints
  + `nc_hash_size_dim` sets the hash table size for dimension names.
    Default: 256
  + `nc_hash_size_var` sets the hash table size for variable names.
    Default: 256
  + `nc_hash_size_gattr` sets the hash table size for global attribute names.
    Default: 64
  + `nc_hash_size_vattr` sets the hash table size for variable attribute names.
    Default: 8
  + The above 4 new hints can be used to set different hash table sizes for
    dimensions, variables, and attributes. Hashing tables are used for quick
    data object name lookup. It can be useful for files containing a large
    number of dimensions, variables, and attributes. For instance, when the
    number of variables to be defined is large and the number of attributes per
    variable is small, increasing the value of `nc_hash_size_var` can speed up
    the variable definition and inquiring time. On the other hand, setting a
    smaller value for hint `nc_hash_size_vattr` can reduce memory footprint.
    See [PR #132](https://github.com/Parallel-NetCDF/PnetCDF/pull/132).

* Updated utility programs
  + `ncvalidator` - When the file size is larger or smaller than what is
    calculated based on the metadata stored in the file header, the file may
    still be a valid netCDF file.
    * The larger-than-expected case can happen if opening an existing file that
      contains no variable. Deleting a global attribute already defined in the
      file will reduce the file header size. In this case, the file is still a
      valid netCDF file. Such mismatch will be detected and the file size
      truncated to the header size.
      See [PR #99](https://github.com/Parallel-NetCDF/PnetCDF/pull/99)
    * The smaller-than-expected case can happen if the last variable is
      partially written. The expected file size is calculated based on the full
      sizes of all variables. In this case, the file is still a valid netCDF
      file, and `ncvalidator` will report a warning, rather than an error.
    * Print the dimension size of a variable on stdout when its size is larger
      than the limitation allowed by the file format. See commit
      [5584d44](https://github.com/Parallel-NetCDF/PnetCDF/commit/5584d44a433a68966b0be601e7a73e939c695dbf)
  + Add file src/utils/README.md which gives short descriptions of the utility
    programs and collapsible bullets to display their manual pages.

* Other updates:
  + When file header extent size grows, PnetCDF now uses a movement unit per
    process of size up to 64 MiB.
    See [PR #137](https://github.com/Parallel-NetCDF/PnetCDF/pull/137)
  + Since version 1.1.0, PnetCDF has been using file striping size, if
    obtainable from the MPI-IO hint `striping_unit`, to align the starting file
    offset of the data section. This offset is also referred to as the file
    header extent, which can be larger than the header size to allow header to
    grow when new data objects are added. Starting from this release, file
    stripe size is no longer used for setting the starting offset of the data
    section. This is because automatically setting file header extent using the
    file striping size may grow the file header unexpectedly when adding new
    objects to an existing file.
    See [PR #124](https://github.com/Parallel-NetCDF/PnetCDF/pull/124) and
    [PR #125](https://github.com/Parallel-NetCDF/PnetCDF/pull/125).
  + Use unsigned int to perform byte swap.
    See [PR #113](https://github.com/Parallel-NetCDF/PnetCDF/pull/113).
  + Silence Intel icc compilation warnings: when CFLAGS contains
    "-Wimplicit-const-int-float-conversion" and "-Wstringop-overread".
    See [PR #110](https://github.com/Parallel-NetCDF/PnetCDF/pull/110).
  + In all previous PnetCDF's implementations, file header is always written/
    read by rank 0 using MPI independent APIs. This can nullify ROMIO hint
    `romio_no_indep_rw` if set by the user. To warrant no independent read/
    write user hint, PnetCDF now checks hint `romio_no_indep_rw` and if set to
    `true`, then all file header I/Os are made through MPI collective I/O
    calls, where only rank 0 makes non-zero length requests while all others
    zero length (in order to participate the collective calls). See
    [PR #104](https://github.com/Parallel-NetCDF/PnetCDF/pull/104) and
    [PR #138](https://github.com/Parallel-NetCDF/PnetCDF/pull/138).
  + In all prior versions, the file name was checked whether it contains
    character ':'. The prefix name ending with ':' is considered by ROMIO as
    the file system type name. The prefix name, if found, is then stripped, so
    the file name can be used in the POSIX function calls internally. However,
    the prefix was not checked against the file system type names recognized
    by ROMIO. Starting from this release, the prefix is checked against the
    known file system type names to ROMIO. If the prefix is not one of the
    recognized types, e.g.  "ufs", "nfs", "xfs", "pvfs2", "gpfs", "panfs",
    "lustre", "daos", "testfs", "ime", or "quobyte", then the prefix name is
    not stripped. This change is for the case when the file name contains ':',
    but it is not for specifying the file system type. See
    [PR #79](https://github.com/Parallel-NetCDF/PnetCDF/pull/79) and
    [MPICH PR 5951](https://github.com/pmodels/mpich/pull/5951).

* Bug fixes
  + Return I/O hints that are actually used. See commit
    [0dcf628](https://github.com/Parallel-NetCDF/PnetCDF/commit/0dcf6284106e42faa5e8fb9ab1aa5d52917ff892).
  + Fix residual values of `v_align` and `r_align` when re-entering the define
    mode multiple times.
    See [PR #126](https://github.com/Parallel-NetCDF/PnetCDF/pull/126).
  + Fix Fortran API `nf90mpi_Inq_buffer_size` which should call
    `nfmpi_inq_buffer_size` internally.
    See [PR #111](https://github.com/Parallel-NetCDF/PnetCDF/pull/111).
  + Fix `ncmpi_inq_num_rec_vars()` and `ncmpi_inq_num_fix_vars()` when opening
    an existing file. See
    [PR #103](https://github.com/Parallel-NetCDF/PnetCDF/pull/103).
  + `ncmpidiff` -  when checking the dimensions defined in the second files
    whether are also defined in the first file. See commit
    [88cd9c1](https://github.com/Parallel-NetCDF/PnetCDF/commit/88cd9c187b9b3dc9018b066e905a10d0c74488f8).
  + Use Fortran subroutine `Get_Environment_Variable` instead of `getenv` if it
    is available.  See commits
    [a0b8aca](https://github.com/Parallel-NetCDF/PnetCDF/commit/a0b8acabc3bf7a7a35d878c9db2afb71942bb7a9), and
    [b796759](https://github.com/Parallel-NetCDF/PnetCDF/commit/b796759f5a8e1749e7c168e05fe1665a35e2a2a1).

* New test program
  + test/largefile/large_attr.c - tests attributes of size > 2 GiB.
  + test/largefile/tst_hash_large_ndims.c - tests hashing performance when
    the number of dimensions is large.
  + test/largefile/tst_hash_large_nvars.c - tests hashing performance when
    the number of variables is large.
  + test/largefile/tst_hash_large_ngattr.c - tests hashing performance when
    the number of global attributes is large.
  + test/largefile/large_header.c - tests file header size larger than 2 GiB.
  + test/largefile/large_reqs.c - tests a single read/write request of size
    larger than 2 GiB.
  + test/testcases/tst_redefine.c - tests multiple entries of `ncmpi__enddef`
    [PR #126](https://github.com/Parallel-NetCDF/PnetCDF/pull/126).
  + test/testcases/tst_symlink.c - tests `NC_CLOBBER` on a symbolic link.
  + test/testcases/tst_del_attr.c - tests delete attributes. See
    [PR #99](https://github.com/Parallel-NetCDF/PnetCDF/pull/99).
  + test/testcases/test_get_varn.c - tests `get_varn` API. See
    [PR #90](https://github.com/Parallel-NetCDF/PnetCDF/pull/90).
  + test/testcases/flexible_var.c - tests flexible var API
  + test/testcases/flexible_api.f - tests flexible API when `bufcount == -1`
  + test/testcases/scalar.c - adds tests for scalar variables using nonblocking
    APIs. See commit
    [07ff7b1](https://github.com/search?q=repo%3AParallel-NetCDF%2FPnetCDF+07ff7b1&type=commits)
  + test/nonblocking/test_bputf.f90, test/nonblocking/test_bputf77.f -
    add tests of APIs `inq_buffer_usage` and `inq_buffer_size`. See commit
    [94ce438](https://github.com/Parallel-NetCDF/PnetCDF/commit/94ce438262fe7fcade031dcae1a677a827549bb3)

* Clarifications
  + Hints `nc_header_align_size` and `nc_record_align_size` are to align the
    file header extent and starting file offset of the record variable section,
    respectively.
  + Hint `nc_record_align_size` is not to align the offsets of individual
    record variables.
  + Prior to version 1.13.0, hint `nc_var_align_size` is used to align the
    starting file offsets of individual fixed-size variables. This design was
    to reduce file lock contention in MPI collective I/O operations. Beginning
    from this release (version 1.13.0), this hint is used to align only the
    starting file offset of the entire data section, not individual variables.
    The reason of such change is because PnetCDF nonblocking APIs can aggregate
    multiple I/O requests and MPI-IO today has been optimized to align I/O to
    file striping boundaries, which makes aligning starting offsets of
    individual variables less effective and may create large empty space in the
    file.
  + Using `NC_CLOBBER` in `ncmpi_create()` can be expensive if the file already
    exists. When the existing file is a regular file, PnetCDF will delete it
    with a call to `unlink()` first and re-created it. Calling `unlink()` may
    be expensive for some parallel file systems. When the existing file is a
    symbolic link, PnetCDF will call `truncate()` or `MPI_File_set_size()` to
    truncate the file size to zero. Calling `truncate()` may also be very
    expensive on some file systems, e.g. Lustre. Sporadically a long time spent
    on `unlink()` and `truncate()` was observed on Perlmutter at NERSC.

-------------------------------------
Version 1.12.3 (February 21, 2022)
-------------------------------------

* New optimization
  + Improve the performance when nonblocking requests contain more than one
    record.
    See [a00f5d2](https://github.com/Parallel-NetCDF/PnetCDF/commit/a00f5d2).

* Update configure options
  + Retire configure options `--enable-netcdf4` and `--enable-adios`, as there
    are already options `--with-netcdf4` and `--with-adios`. According to the
    autoconf manual, `--enable-feature` is for internal packages and
    `--with-feature` is for external.  Adding `--with-feature` is equivalent to
    enabling the feature.

* Updated utility program
  + Replace the definition of "difference ratio" used in utility programs
    `cdfdiff` and `ncmpidiff` with formula
    ```
    |x - y| / max(|x|, |y|)
    ```
    where |x| means the absolute value of x.
    See [issue #78](https://github.com/Parallel-NetCDF/PnetCDF/issues/78) and
    [9b165ceb](https://github.com/Parallel-NetCDF/PnetCDF/commit/9b165ceb).
  + Utility programs `cdfdiff` and `ncmpidiff` now report only the first
    variable element that are different or fail to meet the tolerances. Their
    values and differences are now printed on the standard output.
    See [75d20e6](https://github.com/Parallel-NetCDF/PnetCDF/commit/75d20e6)
    and [58d2d17](https://github.com/Parallel-NetCDF/PnetCDF/commit/58d2d17).
  + Use the same user-provided tolerant difference and tolerant difference
    ratio (through command-line option '-t') to check all variables.
    See [issue #78](https://github.com/Parallel-NetCDF/PnetCDF/issues/78) and
    [75d20e6](https://github.com/Parallel-NetCDF/PnetCDF/commit/75d20e6).

* Other updates:
  + IBM XLF compiler on Summit at OLCF requires compile flag "-qfixed" when
    compiling Fortran programs written in fixed form. Thus, the fixed form flag
    detected at configure time has been added to `AM_FFLAGS` when compiling
    fixed form programs. Similarly, the free form flag has been added to
    `AM_FCFLAGS` when compiling free form programs. This issue affects only the
    test and example Fortran programs. The PnetCDF library is intact.
    See [PR #73](https://github.com/Parallel-NetCDF/PnetCDF/pull/73)
  + Add all PnetCDF I/O hints to the inquired MPI info object returned by API
    `ncmpi_inq_file_info()`.
    See [f0e65cf](https://github.com/Parallel-NetCDF/PnetCDF/commit/f0e65cf).

* Bug fixes
  + Fix a bug in utility program `ncvalidator` for the case when there is no
    record written, but both fixed-size and record variables are defined.
    See [b34bfcd](https://github.com/Parallel-NetCDF/PnetCDF/commit/b34bfcd).
  + Fix configure bug of setting environment variable `SEQ_CC` to the
    sequential `CC` extracted from `MPICC`. Also add configure help message for
    environment variable `SEQ_CC`. Thanks to Carl Ponder for reporting.
    See [4978f6d](https://github.com/Parallel-NetCDF/PnetCDF/commit/4978f6d).
  + Fix a bug when C function `truncate` is not available. Argument `fh` of
    `MPI_File_close` is a pointer.
    See [3e331a6](https://github.com/Parallel-NetCDF/PnetCDF/commit/3e331a6).
  + When using an MPI compiler whose Fortran feature was disabled, the MPI
    Fortran constants and datatypes may not be defined in the header file
    `mpi.h`. This is the case for Open MPI (tested with 4.0.2). PnetCDF used
    some Fortran datatypes without checking whether they are defined, which can
    fail when running 'make'. A fix has been added to check whether the Fortran
    feature is disabled and wraps around the Fortran datatypes with C directive
    'ifdef ENABLE_FORTRAN'. Thanks Bert Wesarg for reporting this bug.
    See [issue #68](https://github.com/Parallel-NetCDF/PnetCDF/issues/68) and
    [PR #69](https://github.com/Parallel-NetCDF/PnetCDF/pull/69).

* Issues with NetCDF library
  + Test program [test/nc4/tst_rec_vars.c](test/nc4/tst_rec_vars.c) fails to
    run when using NetCDF 4.8.0 and 4.8.1. Thanks Bruno Pagani for reporting.
    See [issue #72](https://github.com/Parallel-NetCDF/PnetCDF/issues/72).

* Clarifications
  + Nonblocking APIs have yet to support subfiling feature.

-------------------------------------
Version 1.12.2 (January 15, 2021)
-------------------------------------

* Build recipes
  + Starting from GNU Fortran 10.0.0, function/subroutine argument type
    mismatch becomes a compile error. A new compile command-line option
    "-fallow-argument-mismatch" can turn these errors into warnings. This
    command-line option is added automatically in PnetCDF version 1.12.2 and
    later. When building PnetCDF of version 1.12.1 and earlier versions using
    GNU Fortran 10.0.0 and later, please add "-fallow-argument-mismatch" to
    environment variables FFLAGS and FCFLAGS.
    See [issue #61](https://github.com/Parallel-NetCDF/PnetCDF/issues/61)
    and [GCC 10 Release note](https://gcc.gnu.org/gcc-10/changes.html).
  + README.CRAY has been revised to reflect the recent changes of default
    compiling environment on Cori at NERSC.

* Updated utility program
  + ncvalidator now reports the name of variable that violates the NetCDF
    limitation on large variables for CDF-2 files
  + add corrupted file bad_large_fixed_var.nc2 that contains one large
    fixed-size variables that is not defined last
  + add corrupted file bad_large_rec_2_vars.nc2 that contains 2 large record
    variables
  + add corrupted file bad_large_rec_var.nc2 that contains 1 large record
    variable that is not defined last
  + add URLs of NetCDF limitations on large variables for CDF-1 and 2 file
    formats

* Other updates:
  + When calling ncmpi_create() with NC_CLOBBER flag, PnetCDF now calls
    access() to check whether file exists first. If the file does not exist,
    successive calls to truncate() or unlink() can be skipped.
  + Improve detection of HDF5 signature. The HDF5 signature is located at the
    beginning of the HDF5 superblock, but the location of HDF5 superblock may
    not be at the beginning of the file. It is located at byte offset 0, byte
    offset 512, and at successive locations in the file, each a multiple of two
    of the previous location; in other words, at these byte offsets: 0, 512,
    1024, 2048, and so on.

* Bug fixes
  + Fix more strict aliasing bug when building PnetCDF with -O3 flag. Thanks to
    Gianfranco Costamagna for reporting
    [issue #55](https://github.com/Parallel-NetCDF/PnetCDF/issues/55). See
    [pull request r65](https://github.com/Parallel-NetCDF/PnetCDF/pull/65) for
    the fix.
  + Fix NC_CLOBBER mode for ncmpi_create() when files are existing symbolic
    links. Prior to this release, symbolic links, like other regular files, was
    first deleted and then created. This can result in an unexpected outcome,
    i.e. the deletion of symbolic link. NetCDF-4 library implements this
    differently, by adding O_TRUNC flag when calling open() to truncate the
    file to length 0. Historically, PnetCDF did not adopt the same approach
    because MPI does not define a similar flag to O_TRUNC and the only way to
    achieve the file clobber effect is to through MPI_File_set_size(), which
    can be expensive as the function takes an MPI file handler argument, which
    requires to open the file first with a call to MPI_File_open().
  + Fix various compile and link bugs when NAG Fortran is used. Bugs include
    flag needed to verbose linking output, unrecognized link option -pthread,
    unmatched C compiler underneath. Thanks Sergey Kosukhin for providing the
    fix in [PR #59](https://github.com/Parallel-NetCDF/PnetCDF/pull/59)
    and [PR #60](https://github.com/Parallel-NetCDF/PnetCDF/pull/60)
  + Fix a bug of calling Fortran getarg() with the first argument k with a
    value > 0 when there is no command-line argument is used. NAG Fortran may
    crash the program. See
    [f16bd3c](https://github.com/Parallel-NetCDF/PnetCDF/commit/f16bd3c1ba1b08eade2384f094c519f3f2dc114e)
  + Fix a bug that limits FLASH-IO to run on less than 16K MPI processes. See
    [1d84fa5](https://github.com/Parallel-NetCDF/PnetCDF/commit/1d84fa5d54ca9179da4a5b1a4ee3b92cc92287ed)

-------------------------------------
Version 1.12.1 (December 9, 2019)
-------------------------------------

* New/updated utility program
  + A new command-line option `-t` is added to utility program `cdfdiff` to
    compare variable differences within a tolerance. See the man page of
    `cdfdiff` for usage.

* Issues related to MPI library vendors:
  + When using OpenMPI version 4.0.2 to build PnetCDF 1.12.0 and prior
    versions, running 'make' may encounter a problem related to MPI constants
    that have been deprecated in MPI standard 3.0. Error messages similar to
    below may appear.
    ```
    In file included from src/drivers/common/dtype_decode.c:16:
    src/drivers/common/dtype_decode.c: In function 'ncmpii_dtype_decode':
    /OpenMPI/4.0.2/include/mpi.h:322:57: error: expected expression before '_Static_assert'
     #define THIS_SYMBOL_WAS_REMOVED_IN_MPI30(func, newfunc) _Static_assert(0, #func " was removed in MPI-3.0.  Use " #newfunc " instead.")
                                                             ^~~~~~~~~~~~~~
    /OpenMPI/4.0.2/include/mpi.h:743:46: note: in expansion of macro 'THIS_SYMBOL_WAS_REMOVED_IN_MPI30'
     #        define MPI_COMBINER_HVECTOR_INTEGER THIS_SYMBOL_WAS_REMOVED_IN_MPI30(MPI_COMBINER_HVECTOR_INTEGER, MPI_COMBINER_HVECTOR);
                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    src/drivers/common/dtype_decode.c:390:14: note: in expansion of macro 'MPI_COMBINER_HVECTOR_INTEGER'
             case MPI_COMBINER_HVECTOR_INTEGER:
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ```
    In OpenMPI 4.0.2, the default configuration is not to support MPI constants
    that have been deprecated in MPI 3.0. However, instead of ignoring those
    constants, OpenMPI 4.0.2 still defines them as assertion macros, which can
    cleverly trigger a compile-time error if user programs try to use them.
    This behavior appears when the underlying C compilers supporting 2011
    revision of the C standard (with `__STDC_VERSION__ >= 201112L`) such as GCC
    version 4.6 and later are used to build OpenMPI. However, when using
    earlier versions of C compilers to build OpenMPI 4.0.2, those deprecated
    MPI constants are not defined at all in mpi.h. See discussion
    [issue 7099](https://github.com/open-mpi/ompi/issues/7099).

    Thanks to Carl Ponder for reporting the error and Nick Papior for providing
    a workaround solution that is to rebuild OpenMPI 4.0.2 with configure
    option "--enable-mpi1-compatibility" and use it to build PnetCDF 1.12.0 and
    earlier versions. Note the latest MPICH 3.3.1 does not have such an issue,
    as all deprecated constants are still defined. This issue is now fixed in
    PnetCDF of release 1.12.1 which no longer requires the workaround build of
    OpenMPI.

* Bug fixes
  + Fix strict aliasing bug when building PnetCDF with -O3 flag. See
    [a40aa5f](https://github.com/Parallel-NetCDF/PnetCDF/commit/a40aa5f73938ba1298f92ad471b3e3578ef8dbda)

* New Limitations
  + Configure command now checks whether the supplied MPI C compiler is a
    wrapper of a C++ compiler. If this is detected, the PnetCDF configuration
    will be aborted. This check is enforced because using such an MPI C
    compiler will cause problem for linking Fortran, C and C++ programs, with
    an error message similar to this:
    ```
    conftestf.o: In function `MAIN_': conftestf.f:4: undefined reference to `sub_'
    configure:33318: error: Could not link conftestf.o and conftest.o
    ```

* Build recipes
  + On Theta @ ALCF, when compiling utility programs 'ncoffsets' and 'cdfdiff',
    using Intel-based compilers such as module PrgEnv-intel, one may encounter
    error messages below.
    ```
    In file included from /usr/include/inttypes.h:27:0,
    /theta-archive/intel/compilers_and_libraries_2019.5.281/linux/compiler/include/stdint.h:43:54: error: missing binary operator before token "("
       defined(__has_include_next) && __has_include_next(<stdint.h>)
                                                        ^
    ```
    This issue is due to the environment variable 'CPATH' is set for Intel C
    compilers which is not compatible with GCC. However, this can be simply
    resolved by adding "SEQ_CC=icc" to your make command line, i.e.
    ```
    make SEQ_CC=icc
    ```

* Other updates:
  + The string length of I/O hint `nc_burst_buf_dirname`, the name of burst
    buffer directory must be less than the value of MPI_MAX_INFO_VAL. This is
    because all PnetCDF I/O hints were handled through MPI info mechanism and
    MPI requires the maximum string length of the value of an MPI info object
    to be MPI_MAX_INFO_VAL. If violated, error MPI_ERR_INFO_VALUE will be
    returned.

-------------------------------------
Version 1.12.0 (September 30, 2019)
-------------------------------------

* New features
  + BP file read capability is supported -- By being built on top of ADIOS
    library PnetCDF can now read files in BP format. Note write capability is
    not supported. This feature is added for convenience purpose, i.e. the
    existing PnetCDF programs can read BP files without change. The read
    performance of BP files is expected no difference from using ADIOS library
    directly. More information can be found in doc/README.ADIOS.md.

* New Limitations
  + Writing to BP files is not supported.

* Update configure options
  + To enable ADIOS BP file support, two new options can be used.
    - `--enable-adios`: enables read capability for BP files
    - `--with-adios=/path/to/adios`: can be used to specify the path to ADIOS
      library installation
    By default, this option is disabled.
  + Option `--with-netcdf4` now allows a form of `--with-netcdf4=INC,LIB` in
    addition to `--with-netcdf4=DIR`. This is in case the include and lib
    folders of NetCDF-4 installation are in different locations.

* New C and Fortran constants
  + NC_BP, NF_BP, and NF90_BP are the flags indicating BP file access mode.
  + NC_FORMAT_BP, NF_FORMAT_BP, and NF90_FORMAT_BP indicate BP file format.

* New error codes
  + NC_EADIOS, NF_EADIOS, NF90_EADIOS - indicate ADIOS library internal
    errors.

* New/updated utility program
  + `cdfdiff` is a new utility program. It is a serial version of `ncmpidiff`
    that is compiled with gcc without PnetCDF library. It can run on the login
    node in a cross-compile environment. `cdfdiff` only compares files in the
    classic NetCDF formats, i.e. CDF-1, CDF-2, and CDF-5.
  + `ncmpidiff` now checks file format versions of two files, and reports
    difference if not the same. Even if formats are different, it continues to
    compare the contents of file headers and values in variables.
  + A new command-line option `-t` is added to utility program `ncmpidiff` to
    compare variable differences within a tolerance. See the man page of
    `ncmpidiff` for usage. Thanks to Carl Ponder for contributing the source
    codes. See [PR #53](https://github.com/Parallel-NetCDF/PnetCDF/pull/53)
  + `ncmpidump` can now dump contents of BP files.

* Other updates:
  + The MPI info object now includes all PnetCDF hints once obtained from a
    call to ncmpi_inq_file_info(). If any hint is not set by the users, its
    default value is set in the info object.

* Bug fixes
  + When calling a nonblocking API with a zero-length request and argument
    request ID being NULL, segmentation fault may occur. See
    [PR #51](https://github.com/Parallel-NetCDF/PnetCDF/pull/51)

* New example programs
  + examples/adios/read_metadata.c - dumps metadata of a BP file.
  + examples/adios/read_var.c - reads a variable from a BP file generated by
    the ADIOS example program examples/C/arrays/arrays_write.
  + examples/adios/read_var_nb.c - reads a variable from a BP file generated
    by the ADIOS example program examples/C/arrays/arrays_write using PnetCDF
    non-blocking APIs.

* New test program
  + test/adios/open.c - tests if PnetCDF recognize BP files.
  + test/adios/header.c - tests if PnetCDF can parse BP header.
  + test/adios/var.c - tests if PnetCDF can read variables from a BP file
  + test/adios/varm.c - tests varm APIS for reading a BP file
  + test/adios/vars.c - tests vars APIs for reading a BP file
  + test/adios/ivar.c - tests nonblocking API for reading a BP file
  + test/adios/ivars.c - tests nonblocking vars APIs for reading a BP file
  + test/adios/ivarm.c - tests nonblocking varm APIs for reading a BP file
  + test/adios/att.c - tests reading attributes from a BP file
  + test/adios/indep.c - tests reading a BP file in independent data mode
  + test/burst_buffer/varn.c -- tests varn APIs when burst buffer driver is
    used. The test includes cases when argument `counts` contains some of the
    elements being NULL.
  + test/nc4/notsupport.c - test if error code NC_ENOTSUPPORT is properly
    returned when calling APIs for unsupported NetCDF-4 feature.
  + test/nc4/tst_rec_vars.c - tests writing and reading record variables from
    NetCDF-4 files
  + test/nc4/rec2 - tests reading a record variables with 2 unlimited
    dimensions from NetCDF-4 files.

* Issues related to MPI library vendors:
  + Per-file thread-safe capability is not supported when using OpenMPI with
    ROMIO backend, i.e. when `--mca io romio321` is used at `mpirun` command
    line. See OpenMPI github issue
    [6951](https://github.com/open-mpi/ompi/issues/6951)

-------------------------------------
Version 1.11.2 (May 13, 2019)
-------------------------------------

* Bug fixes
  + When configure command-line option `--enable-netcdf4` is used, errors may
    occur during the configure or make time. Thanks Bruno Pagani for reporting.
    See bug fix in
    [pull request #46](https://github.com/Parallel-NetCDF/PnetCDF/pull/46).
  + Fix configure date setting when environment variable `SOURCE_DATE_EPOCH`
    is set.

* Clarifications
  + Limitation of supporting NetCDF-4 driver -- a list of APIs that are not
    supported for NetCDF-4 files is given in `doc/README.NetCDF4.md`. Error
    code `NC_ENOTSUPPORT` is returned when those APIs are called.

* Other updates:
  + Utility program `ncoffsets` is updated to skip printing the dimension names
    if the number of dimensions of a variable is larger than 64.This predefined
    constant can be changed a different number `N` by adding
    `-DMAX_PRINT_NDIMS=N` to the compile command line.
  + Utility program `ncvalidator` is updated to be able to repair the file
    header to fill in null-byte paddings in the space between the header size
    and header extent.
  + Add a note to `doc/README.CRAY` about a compilation issue encountered on
    Cori @ NERSC when module gcc/7.3.0 is loaded.
  + Change release date format from `Day Month, Year` to `Month Day, Year`.

* New test program
  + test/nc4/notsupport.c - Test if error code NC_ENOTSUPPORT is properly
    returned when calling APIs for unsupported NetCDF-4 feature.
  + test/nc4/rtst_rec_vars.c - Test record variables for NetCDF-4 files.
  + test/nc4/tst_2_rec_dims.c - Test opening a NetCDF-4 file containing more
    than one unlimited dimension on whether correct error can be returned.
  + test/nc4/tst_get_put_size.c - Test the amount of I/O so far for NetCDF-4
    files.
  + test/testcases/tst_version.c - Check whether PnetCDF version string
    returned from ncmpi_inq_libvers() matches the constant PNETCDF_VERSION
    defined in header file pnetcdf.h.

-------------------------------------
Version 1.11.1 (April 12, 2019)
-------------------------------------

* New optimization
  + When inserting nonblocking requests into the PnetCDF internal pending
    queues, the queues are now kept sorted (using an insert sort) into an
    increasing order of variable starting file offsets. This can avoid the
    quick sort when flushing the pending requests. See
    [pull request #37](https://github.com/Parallel-NetCDF/PnetCDF/pull/37).
    To avoid internal sorts completely, users are recommended to post
    nonblocking requests in the increasing order of variable IDs and fixed-size
    variables first followed by record variables.

* New Limitations
  + When building with NetCDF-4 feature enabled, using a NetCDF-4 library that
    has already been built with PnetCDF enabled, i.e. `--enable-pnetcdf`, is not
    supported. See [Issue #33](https://github.com/Parallel-NetCDF/PnetCDF/issues/33).

* Other updates:
  + Add a check whether the MPI library is built with shared-library support.
    If not and `--enable-shared` is used, the configure process of PnetCDF will
    fail.
  + In the NetCDF-4 driver, `nc4io_inq_var()` adds a NULL-argument check for
    `no_fill` and `fill_value`. If both arguments are NULL, it skips the call
    to `nc_inq_var_fill`.
  + File header extent area between end of header and first variable will be
    padded with null bytes if PnetCDF is configured with option
    `--enable-null-byte-header-padding`.
  + Add AC_PROG_CPP in configure.ac to check whether `-E` option is required by
    the C preprocessor.
  + Add a check for whether `m4` utility is available, and abort the configure
    if it is not.

* Bug fixes
  + Fix ncmpidiff when comparing dimension names of 2 variables between files
    whose dimension define orders are different. See
    [Issue #42](https://github.com/Parallel-NetCDF/PnetCDF/pull/42).
  + Fix error checking for programs in examples/C to ignore NC_ENOTENABLED
    if PnetCDF was not built with `--enable-profiling`. Thanks to Bruno Pagani
    and see [Issue #34](https://github.com/Parallel-NetCDF/PnetCDF/issues/34).

* New test program
  + test/burst_buffer/varn.c -- to test varn API when burst buffer driver is
    used. The test includes cases when argument counts is NULL or some elements
    in counts are NULL.

* Clarifications
  + Padding -- NetCDF classic file format specification states "Header padding
    uses null (\x00) bytes. In data, padding uses variable's fill value."
    PnetCDF implements the header padding specification but only enforces it
    when the configure option `--enable-null-byte-header-padding` is set. Note
    PnetCDF has not yet implemented the padding for data section.


-------------------------------------
Version 1.11.0 (December 19, 2018)
-------------------------------------

* New features
  + NetCDF-4 driver -- Accessing HDF5-based NetCDF-4 files is now supported.
    PnetCDF can be built on top of NetCDF-4 library to let users to use PnetCDF
    APIs to read and write a NetCDF-4 file. Users now can add NC_NETCDF4 flag
    when calling ncmpi_create() to create NetCDF-4 files. For opening NetCDF-4
    files, no additional flag is needed, as PnetCDF automatically detects the
    file format and uses the HDF5 I/O driver underneath. This feature is
    provided for convenience purpose. The parallel I/O performance to NetCDF-4
    files is expected no difference from using NetCDF-4 library directly.
  + Per-file thread-safe capability is added. This feature can be enabled at
    configure time by adding command-line option `--enable-thread-safe`. In
    addition, option `--with-pthread` can be used to specify the install path
    to the pthreads library. This feature currently only supports
    one-thread-per-file I/O operations and the classic CDF-1, 2, and 5 files.

* New optimization
  + On some systems, e.g. Cori @NERSC, collective MPI-IO may perform poorly
    when the I/O buffer is noncontiguous, compared to a contiguous one. To
    avoid this, `ncmpi_wait()` and `ncmpi_wait_all()` now check whether the
    buffer is noncontiguous and size is less than 16 MiB. If both are true, a
    temporary contiguous buffer is allocated to copy the data over and used in
    the MPI read or write calls. The size of the buffer can be adjusted through
    a new hint `nc_ibuf_size`. See `New PnetCDF hint` below and
    [PR #26](https://github.com/Parallel-NetCDF/PnetCDF/pull/26). Programs
    developed to test this issue is available in
    https://github.com/Parallel-NetCDF/E3SM-IO/tree/master/mpi_io_test
  + Burst buffer driver is updated to run varn APIs more efficiently. Previous
    implementation breaks a single varn request into multiple vara requests,
    which can be slow and require a large amount of meta data. It has changed
    to consider each varn request a single entity. See
    [PR #30](https://github.com/Parallel-NetCDF/PnetCDF/pull/30) and
    [PR #31](https://github.com/Parallel-NetCDF/PnetCDF/pull/31).

* New Limitations
  + For creating new files, the NetCDF-4 driver in PnetCDF supports only the
    classic model I/O operations. Advanced NetCDF-4 features, such as chunking,
    compression, etc. are not supported in PnetCDF. This is due to the
    unavailability of PnetCDF APIs for those operations.
  + The burst buffering driver does not support NetCDF-4 file formats.
  + Due to a bug in HDF5 1.10.2 that fails zero-length write requests to record
    variables in the collective mode, PnetCDF is not able to support such
    requests when NetCDF-4 feature is enabled. See discussion in
    https://github.com/NCAR/ParallelIO/pull/1304
    The bug fix has appeared in HDF5 1.10.4 release.

* Update configure options
  + Enable NetCDF-4 support.
    - `--enable-netcdf4`: enables NetCDF4 format classic mode support
    - `--with-netcdf4=/path/to/netcdf-4`: can be used to specify the path to
      NetCDF-4 library installation
  + Enable multi-threading support.
    - `--enable-thread-safe`: enable per-file thread-safe support
    - `--with-pthread`: path to the pthread library installation

* New constants
  + none

* New APIs
  + C++ API `NcmpiFile::set_fill()` is added for setting and inquiring the
    fill mode of an opened NetCDF file.

* API syntax changes
  + none

* API semantics updates
  + none

* New error code precedence
  + none

* Updated error strings
  + none

* New error code
  + none

* New PnetCDF hint
  + `nc_ibuf_size` -- to set the size of a temporal buffer to be allocated by
    PnetCDF internally to pack noncontiguous user write buffers supplied to the
    nonblocking requests into a contiguous space. Similarly for read case to
    unpack the temporal buffer to user read buffers, if they are noncontiguous.
    This affects both blocking and nonblocking APIs. On some systems, using
    noncontiguous user buffers in MPI collective read/write functions performs
    significantly worse than using contiguous buffers. Note if the size of
    aggregated user buffers is larger than `nc_ibuf_size`, packing/unpacking
    will be disabled to save memory footprint. The default value is 16 MiB.

* New run-time environment variables
  + none

* Build recipes
  + doc/README.NetCDF4.md is added to describe the usage of the new feature of
    NetCDF-4 support.

* New/updated utility program
  + none

* Other updates:
  + The automatic file layout alignment for fixed-size variables is disabled.
    This is because modern MPI-IO implementations have already aligned the file
    access with the file lock boundaries and the automatic alignment can create
    a file view with "holes" in between variables, which can adversely degrade
    I/O performance. Users can still set hints `nc_header_align_size`,
    `nc_var_align_size`, and `nc_record_align_size` to use customized alignment
    sizes.
  + The internal data buffering mechanism used in the burst buffer driver is
    removed. This mechanism caches the request data in memory until the
    accumulated size is more than 8 MiB, so the write requests to burst buffers
    can be aligned with 8 MiB boundaries. However, experiments on Cray DataWarp
    show a negligible performance improvement unless the I/O request is small
    and fragment. On the other hand, it can degrade performance for mid- and
    large-sized requests. The burst buffer driver now writes directly to the
    burst buffers for each user write request.

* Bug fixes
  + Fix bug of checking interleaved requests for scalar variables. See
    [PR #27](https://github.com/Parallel-NetCDF/PnetCDF/pull/27).
  + When building PnetCDF using the IBM xlc compiler with -O optimization
    option on Little Endian platforms, users may encounter errors related to
    strict ANSI C aliasing rules. Thanks to Jim Edwards for reporting and Rafik
    Zurob for providing the fix. See
    [Issue #23](https://github.com/Parallel-NetCDF/PnetCDF/issues/23) and
    [Pull Request #24](https://github.com/Parallel-NetCDF/PnetCDF/issues/24).
  + Shell ksh has a different way to redirect stdout and stderr from bash.
    PnetCDF configure.ac and acinclude.m4 have been developed mainly on bash.
    This bug can cause configure command to fail when using ksh. Thanks to
    @poohRui for reporting the bug. See
    [Issue #21](https://github.com/Parallel-NetCDF/PnetCDF/issues/21) and
    [PR #22](https://github.com/Parallel-NetCDF/PnetCDF/pull/22).
    However, running configure under ksh is still buggy. A GNU automake bug
    report of hanging problem can be found in
    https://lists.gnu.org/archive/html/bug-automake/2015-04/msg00000.html
    PnetCDF users are recommended to run configure under other shells.
  + For put and get APIs when buftype is MPI_DATATYPE_NULL, bufcount is
    ignored. This is not implemented correctly in blocking put and get APIs.
    See bug fix committed on Aug. 25, 2018.
  + ncmpidiff -- when comparing two files that contain record variables but
    no record has been written. See bug fix committed on Aug. 25, 2018.
  + ncmpidiff -- when comparing two scalar variables, error NC_EBADDIM may
    mistakenly reported. See bug fix committed on Aug. 12, 2018.
  + When the MPI communicator used in ncmpi_create or ncmpi_open is freed by
    the user after the call and before file is closed, programs would crash at
    ncmpi_close with MPI error of "Invalid communicator". The fix moves the
    duplication of MPI communicator to the place before calling driver create
    and open subroutines. See bug fix committed on Jul 21, 2018.

* New example programs
  + examples/C/time_var.c and examples/F77/time_var.f - show how to define,
    write, and read record variables.
  + examples/C/pthread.c - demonstrates the one-file-per-thread I/O example.
    When running on some parallel machines, users may need to set certain
    environment variable to enable MPI multi-threading support, for example on
    Cori @NERSC with command
    ```
    export MPICH_MAX_THREAD_SAFETY=multiple
    ```
  + examples/C/transpose2D.c - a 2D version of examples/C/transpose.c

* New programs for I/O benchmarks
  + none

* New test program
  + test/F90/test_fill.f90 - another test for bug fix r3730.
  + test/testcases/error_precedence.m4 - tests the error code reporting
    precedence
  + test/nc4/tst_zero_req.c - tests a HDF5 1.10.2 bug that causes test program
    to hang when writing to and reading back a 2D record variable in collective
    mode with some of the processes making zero-length requests.
  + test/nc4/put_get_all_kinds.m4 - tests all supported variable read/write
    API. Make sure they are properly wired up
  + test/nc4/interoperability_rd.m4 - tests whether NetCDF-4 file written using
    NetCDF can be read by PnetCDF
  + test/nc4/interoperability_wr.m4 - tests whether NetCDF-4 file written using
    PnetCDF can be read by NetCDF
  + test/nc4/simple_xy.c - tests reading NetCDF-4 files, borrowed the test
    program simple_xy.c from NetCDF
  + test/testcases/tst_pthread.c - tests thread-safe capability for scenario of
    each thread operating on a unique file.
  + test/testcases/tst_free_comm.c - free MPI communicator right after calling
    ncmpi_create to see if PnetCDF duplicates the communicator correctly.

* Conformity with NetCDF library
  + none

* Discrepancy from NetCDF library
  + In contract to NetCDF-4 which allows to read/write variables in define mode
    when the file format is in NetCDF-4 format, PnetCDF still requires reading
    and writing variables in data mode.
  + In contrast to the semantics of nc_set_fill() defined in NetCDF-4,
    ncmpi_set_fill() changes the fill mode of all variables newly defined in
    the current scope of defined mode. Variables affected include the ones
    defined before and after the call to ncmpi_set_fill(). Note this API has no
    effect on the already existing variables created in the previous define
    mode. This behavior follows the convention adopted by NetCDF-3. To change
    fill mode for individual variables after the call to ncmpi_set_fill(), API
    ncmpi_def_var_fill() can be used for this purpose. Refer NetCDF 4.1.3 user
    guide for semantics of
    [nc_set_fill()](https://www.unidata.ucar.edu/software/netcdf/documentation/historic/netcdf-c/nc_005fset_005ffill.html).
    A discussion with NetCDF developers regarding this issue can be found in
    [1114](https://github.com/Unidata/netcdf-c/pull/1114).
  + The error code return precedence can be different between NetCDF and
    PnetCDF in some cases. A test program for error code return precedence is
    available in test/testcases/error_precedence.m4. This program can be used
    to test both PnetCDF and NetCDF libraries. Note when testing NetCDF
    programs, because NetCDF does not follow the same precedence, failures are
    expected. A discussion with NetCDF developers regarding this issue can be
    found in [334](https://github.com/Unidata/netcdf-c/issues/334).

* Issues related to MPI library vendors:
  + none

* Issues related to Darshan library:
  + none

* Clarifications
  + PnetCDF currently does not support Fortran default integer type set to 8
    bytes (for GNU Fortran compiler, this change of default setting is done by
    using compile option -fdefault-integer-8). Checking this has been added
    and configure command will fail, once default 8-byte integer is detected.


-------------------------------------
Version 1.10.0 (July  2, 2018)
-------------------------------------

  o New features
    * A new I/O layer that makes use of burst buffers is added. Burst buffer is
      an I/O driver that implements a log-based I/O aggregation for write
      requests with the present of burst buffer storage devices. The design is
      to save first write requests in burst buffers and later flush the saved
      requests to the destination file system. See doc/README.burst_buffering
      for more detailed description.

  o New optimization
    * Internal memory management for nonblocking APIs has been improved further
      to reduce memory footprint. See r3763.
    * Both blocking and nonblocking varn APIs have been improved for better
      performance and less memory footprint. See r3747-r3750.
    * Prior to this release, when buftype in flexible APIs indicates a
      noncontiguous layout in user buffer, PnetCDF packs the user buffer into
      a contiguous buffer allocated internally, which will later be used in
      MPI-IO calls. To reduce memory footprint, this release avoids the
      additional memory allocation and directly uses the user buffer in MPI-IO
      calls when neither byte-swap nor type-conversion is required. See r3722
      and r3723.

  o New Limitations
    * none

  o Update configure options
    * The default setting for relax-coord-bound option is changed to "enable",
      in accord with NetCDF 4.6.2. Users may use --disable-relax-coord-bound to
      disable this setting at configure time or at run time by setting the
      environment variable PNETCDF_RELAX_COORD_BOUND. See section "New run-time
      environment variables" below for more information.
    * Option in-place byte-swap is expanded into the following three settings.
      --enable-in-place-swap : to perform byte swap on user I/O buffers
        whenever possible. This option results in the least amount of internal
        memory usage. However, if an immutable user buffer is used,
        segmentation fault may occur when byte swap is performed on user
        buffer in place.
      --disable-in-place-swap : when byte swap is required, PnetCDF allocates
        an internal memory to perform byte swap to avoid altering the contents
        of user buffer. This option will increase internal memory footprint.
      The default (neither the above option is used at the command line): an
        internal buffer is allocated for byte swap only when the I/O request
        size is less than 4 KiB. For larger requests, user buffers are used
        for in-place byte swap.
      The mode can also be changed at the run time by setting the PnetCDF hint
      nc_in_place_swap in the run-time environment variable PNETCDF_HINTS. See
      New hints below for more info.  Note -in-place-swap option only affect
      applications running on Little Endian machines, as no byte swap is
      necessary on Big Endian machines.

  o New constants
    * none

  o New APIs
    * ncmpi_flush() flushes data buffered by PnetCDF to the destination file
      system. When using the burst buffers, the data cached in the burst
      buffers will be flushed.

  o API syntax changes
    * none

  o API semantics updates
    * The default setting for checking the coordinate bound is changed to the
      more relaxed check. Users can still change this checking at run time by
      setting the environment variable PNETCDF_RELAX_COORD_BOUND. See section
      of "New run-time environment variables" below for more information.
    * ncmpi_put_vard and ncmpi_get_vard APIs
      1. Now allow to write and read two or more variables in a single call to
         vard APIs. In this case, argument varid will be the ID of first
         variable. Argument filetype should be constructed by using the
         relative distance between any two successive variables. See example
         program examples/C/vard_mvars.c.
      2. When argument filetype is MPI_DATATYPE_NULL, the request is considered
         a zero-length request.
      3. Limitation of vard APIs: The element data type of filetype must
         conform with the variable external NC data type. The element data
         types of all variables accessed in filetype must be the same.
         Otherwise, NC_EMULTITYPES will be returned.
      4. buftype must contain only one same element data type if it is used to
         read or write multiple variables from file. Otherwise NC_EMULTITYPES
         will be returned.

  o New error code precedence
    * none

  o Updated error strings
    * none

  o New error code
    * NC_EFLUSHED when calling ncmpi_cancel() to cancel non-blocking requests
      but the requests have already flushed.
    * NC_EBADLOG when the log files stored in the burst buffers are found
      corrupted.

  o New PnetCDF hints
    * nc_in_place_swap -- to enable or disable in-place byte swap on Little
      Endian architectures. The default mode is auto. See in-place byte-swap
      option above for descriptions of the three modes. Below is an example
      that changes the mode to "enable" regardless the setting used at the
      configure time, by setting the environment variable PNETCDF_HINTS with
      command:
          export PNETCDF_HINTS="nc_in_place_swap=enable"
    * nc_burst_buf -- to use the burst buffering feature. The default setting
      is disable. To enable at the run time, one can set the hint to
      environment variable PNETCDF_HINTS, for example,
          export PNETCDF_HINTS="nc_burst_buf=enable"
    * nc_burst_buf_dirname -- Name of directory on the burst buffer where the
      internal log files will be stored. This is usually set to the path where
      burst buffer is mounted. The default path is the same directory of the
      output file.
    * nc_burst_buf_del_on_close -- Whether the burst buffer log files should
      be deleted when closing the NetCDF file. The default is enable. To
      disable at the run time, one can set the hint to environment variable
      PNETCDF_HINTS, for example,
          export PNETCDF_HINTS="nc_burst_buf_del_on_close=disable"
      Note that on some systems, e.g. DataWarp, all files stored in burst
      buffers may be purged when the user job is completed. In this case, users
      who wish to keep the log files for other jobs must refer to the burst
      buffer user guide for such setting. For DataWarp on Cori @NERSC, the
      settings are described in
      http://www.nersc.gov/users/computational-systems/cori/burst-buffer/example-batch-scripts/
    * nc_burst_buf_flush_buffer_size -- Amount of memory allowed by users for
      PnetCDF to allocate when flushing the burst buffer log files. The unit
      is in bytes. Value 0 means unlimited. Users are suggested to use a
      number that is at least the size of largest individual requests. The
      default is unlimited.

  o New run-time environment variables
    * PNETCDF_RELAX_COORD_BOUND environment variable can be set to 0 to disable
      or 1 to enable the more relaxed boundary check for argument start. Prior
      to version 1.10.0, the default behavior is to use a strict boundary
      check, i.e. error code NC_EINVALCOORDS will be thrown when the value of
      start is larger than or equal to the defined dimension size.

  o Build recipes
    * doc/README.burst_buffering is added to describe the usage of the new
      feature of burst buffering.
    * doc/README.IBM is revised as the compiler behavior, particularly when
      using GNU-based MPI compilers to build static-only libraries, has
      changed on BGQ since the release of 1.9.0.
    * doc/README.CRAY is revised as the compiler behavior, particularly when
      using GNU-based MPI compilers to build static-only libraries, has
      changed on CRAY since the release of 1.9.0.
    * doc/README.Ubuntu is revised and the fix of adding LDFLAGS is no longer
      necessary when building shared libraries.

  o New/updated utility program
    * ncvalidator adds a new option -t to turn on tracing mode which prints all
      successfully validated metadata till the first error encountered.
    * ncvalidator adds a check to detect whether there are two or more
      unlimited dimensions defined in the file and, if yes, reports error code
      NC_EUNLIMIT.

  o Other updates:
    * Inter-library dependencies among C, C++ and Fortran have been updated to
      follow automake user guide: adding FLIBS and FCLIBS to _LIBADD. This
      update resolves problems of building shared libraries for some systems.
    * Move the memory allocation trace feature out of --enable-debug into the
      new configure option --enable-profiling. By default, this is disabled.
    * Add a check for NC_EUNLIMIT in API ncmpi_open to detect whether two or
      more unlimited dimensions are defined to detect a corrupted file.

  o Bug fixes
    * Fix type_create_subarray64() for case when some dimensions of the global
      array is larger than 4G and oldtype is not MPI_BYTE. See r3779. Without
      this fix, test/largefile/large_var.c will fail. Note prior to 1.9.1,
      type_create_subarray64() is always called with oldtype being MPI_BYTE.
    * Fix configure-time bug that configure fails to recognize the compilers
      with full-path names passed from the environment variable MPICC, MPICXX,
      MPIF77, or MPIF90. See Ticket #35 for bug report and r3740 for fix.
    * Fix the bug that writing scalar attributes of type NF_INT64 was
      mistakenly casted to NF_INT. See r3730.
    * Fix test/testcases/erange_fill.m4 for the case of NC_LONG data type.
      NC_LONG is synonym of NC_INT. When setting fill value for variables of
      type NC_LONG, the user buffer must be of type int (internal). See r3726.
    * Fix the calculation of new record number in put_vard API. Thanks to
      Jim Edwards. See r3675.
    * Fix the calculation of growing size of nonblocking request queues to
      include the number of records of a record variable, as such request is
      split into multiple additional requests, one for each record. See r3651.
    * Remove freeing name object because NC_ENULLPAD is not a fatal error. See
      r3644. This bug only appears when "--enable-null-byte-header-padding" is
      set at configure time and opening files whose headers are not padded with
      null bytes.
    * Member "value" of NC_dimarray, NC_attrarray, and NC_vararray can still be
      NULL when a corrupted tag of NC_DIMENSION, NC_ATTRIBUTE, or NC_VARIABLE
      is read from files. Add a check against NULL before freeing it. This bug
      only appears when reading files with corrupted NC tags. See r3645.

  o New example programs
    * example/C/vard_mvars.c shows an example of using a single vard API call
      to write or read two variables.
    * example/burst_buffer/create_open.c shows an example of creating and
      opening a NetCDF file with the burst buffering feature enabled.
    * example/burst_buffer/nonblocking.c shows an example of using non-blocking
      put APIs with the burst buffering feature enabled.

  o New programs for I/O benchmarks
    * none

  o New test program
    * test/burst_buffer/bb_bsize.c - test whether the burst buffer driver works
      under limited flush buffer size.
    * test/burst_buffer/bb_hints.c - test file hints related to the burst
      buffer driver
    * test/burst_buffer/bb_many_reqs.c - stress test the nonblocking request
      pool in the burst buffer driver
    * test/burst_buffer/bb_nonblocking.c - test the behavior of burst buffer
      driver when using nonblocking I/O, particularly for case that tries to
      cancel an already flushed request.
    * test/burst_buffer/highdim.c - test whether the burst buffer driver can
      handle very high dimension variables.
    * test/testcases/test_fillvalue.c - tests PnetCDF allows to put attribute
      _FillValue for global variable and for non-global variable, requires the
      data type of attribute _FillValue be the same as the variable's.
    * test/F90/test_attr_int64.f90 - tests the bug fixed in r3730 that
      nf90mpi_put_att API writes a scalar attribute of type NF90_INT64 that
      was mistakenly casted to NF90_INT.
    * test/testcase/test_vard_rec.c - tests ncmpi_put_vard APIs for writing a
      record variable with one record at a time. This is to test the fix to
      bug reported by Jim Edwards in r3675.
    * test/nonblocking/large_num_reqs.c - tests large number of nonblocking put
      and get requests (larger than NC_REQUEST_CHUNK, the constant used to grow
      the internal queues that store the nonblocking put and get requests. This
      is to test bug fix in r3651.
    * test/testcases/tst_def_var_fill.c - tests API ncmpi_def_var_fill and
      verifies fill values when fill mode is turned on and off.
    * src/utils/ncvalidator/tst_open.c - tests API ncmpi_open against corrupted
      files and checks expected error codes.

  o Conformity with NetCDF library
    * NetCDF allows to put attribute _FillValue of any NC data type to the
      global variable, NC_GLOBAL. Note this is allowed for classic files, but
      not for the HDF5-based NetCDF-4 files. PnetCDF conforms with this
      behavior from this release on. Reference of NetCDF 4.1.3 user guide for
      nc_put_att_type() at https://www.unidata.ucar.edu/software/netcdf/documentation/historic/netcdf-c/nc_005fput_005fatt_005f-type.html
    * For non-global variables, NetCDF treats the classic and HDF5 files
      differently. For classic files, it allows to create attribute _FillValue
      of a data type that is different from the variable's. For HDF5 files, the
      data type of attribute _FillValue must be the same as the variable. Note
      this behavior for classic files is against the convention specified in
      the NetCDF user guide and NetCDF plans to revise the user guide to
      describe the new convention. As for PnetCDF, we decided to enforce the
      rule that requires the data type of attribute _FillValue be the same as
      the variable's.


  o New conflict with NetCDF library
    * none

  o New issues related to MPI library vendors:
    * none

  o New issues related to Darshan library:
    * none

  o New clarifications
    * none


-------------------------------------
Version 1.9.0 (December 19, 2017)
-------------------------------------

  o New features
    * The configure environment has been migrated to use GNU automake and
      libtool.
    * Building shared libraries is now supported with configure option
      --enable-shared. However, the default configuration is to build static
      library only. Note linking shared libraries is not recommended due to
      the likely performance degradation.

  o New optimization
    * Use hash function for faster attribute name lookup. See r3335.

  o New Limitations
    * PnetCDF now requires the underlying MPI library define data type
      MPI_Offset an 8-byte integer. Configuration process will be aborted if
      MPI_Offset is detected to be of size 4 bytes. See r2944.
    * The maximum number of files that can be simultaneously opened by an MPI
      process is now limited to 1024 (a new constant NC_MAX_NFILES defined in
      pnetcdf.h). When this limit is reached, error code NC_ENFILE will be
      thrown.
    * NC_MAX_VAR_DIMS, the maximum number of dimensions per variable is raised
      to 2147483647, i.e. NC_MAX_INT
    * NC_MAX_DIMS, the maximum number of dimensions per file is raised to
      2147483647, i.e. NC_MAX_INT
    * NC_MAX_ATTRS, the maximum number of attributes per file is raised to
      2147483647, i.e. NC_MAX_INT
    * NC_MAX_VARS, the maximum number of variables per file is raised to
      2147483647, i.e. NC_MAX_INT
    * NC_EMAX_REQ error code will be thrown when a single request of a get/put
      API call or the aggregated size of nonblocking requests in a wait API
      call exceeds INT_MAX (max of 4-byte integer).  Note the size limit is per
      MPI process, not across all processes. This is due to MPI-IO is yet to
      support a single MPI-IO request, i.e. MPI_File_read or MPI_File_write, of
      size larger then INT_MAX. A new configure command-line option
      "--enable-large-single-req" has been added to disable the checking. Use
      this option is recommended only when the underneath MPI-IO library
      supports large single requests made by each MPI process.

  o New constants
    * NC_FORMAT_UNKNOWN, NF_FORMAT_UNKNOWN, and NF90_FORMAT_UNKNOWN are new
      constants that are one of the returned values of argument formatp in the
      API ncmpi_inq_file_format when the file under inquiry is unrecognizable
      by PnetCDF library.
    * NC_MAX_NFILES, NF_MAX_NFILES, and NF90_MAX_NFILES define the maximum
      number of files that can be kept opened by an MPI process.

  o New APIs
    * none

  o API syntax changes
    * Add C type qualifier "const" to argument fill_value of ncmpi_def_var_fill
      to indicate the contents pointed by fill_value is immutable.

  o Semantics updates
    * Calling ncmpi_end_indep_data() in collective data mode is no longer
      considered illegal, starting from this release 1.9.0. In this case, the
      API simply returns NC_NOERR. Note calling ncmpi_begin_indep_data() in
      independent data mode has been allowed since PnetCDF 1.2.0. Thanks to
      Gregory Sjaardema for pointing out the asymmetry between the two APIs.
      See r3517.
    * For CDF-5 files, the maximum size of a variable or a variable's record
      is NC_MAX_INT64-3 (i.e. 2^64-4).
    * When the file create/open modes are not consistent among processes, the
      modes on all processes will be replaced by rank 0's mode and PnetCDF
      library will still try to create/open the file. Error code
      NC_EMULTIDEFINE_CMODE or NC_EMULTIDEFINE_OMODE will be returned on
      processes whose modes disagree with rank 0's. If safe mode is on, then
      all processes will also get the same error code.
    * API ncmpi_inq_nreqs() - allows argument nreqs to be NULL. In this case,
      the API is equivalent to checking the validity of argument ncid. In
      previous releases, NC_EINVAL is returned when argument nreqs is NULL.
    * API ncmpi_inq_buffer_usage() - allows argument usage to be NULL. In this
      case, the API is equivalent to checking the validity of argument ncid. In
      previous releases, NC_EINVAL is returned when argument usage is NULL.
    * API ncmpi_inq_buffer_size() - allows argument usage to be NULL. In this
      case, the API is equivalent to checking the validity of argument ncid. In
      previous releases, NC_EINVAL is returned when argument buf_size is NULL.

  o New error code precedence
    * none

  o Updated error strings
    * The error string corresponding to error code NC_EMAXDIMS is changed to
      cover the case when NC_MAX_DIMS or NC_MAX_VAR_DIMS is exceeded.

  o New error code
    * NC_EMAX_REQ - indicates a single request of a get/put API call or the
      aggregated size of nonblocking requests in a wait API call exceeds
      INT_MAX (max of 4-byte integer). The corresponding F77 and F90 error
      codes are NF_EMAX_REQ and NF90_EMAX_REQ, respectively.
    * NC_ENULLPAD - indicates the file fails strict null-byte header check. The
      corresponding F77 and F90 error codes are NF_ENULPAD and NF90_ENULPAD,
      respectively. The same error code has also been defined in NetCDF since
      4.5.1. Note this error is not considered fatal. PnetCDF will continue
      the file open procedure if there is no other error found. Therefore, when
      encountering this error code returned from the file open API, ncmpi_open,
      user can safely ignore it. See section "Conflict with NetCDF library"
      below for further discussion.

  o New PnetCDF hint
    * none

  o New run-time environment variables
    * none

  o New build recipe
    * doc/README.Ubuntu provides build instructions for Ubuntu Linux OS.

  o New/updated utility program
    * Utility program ncmpivalid is renamed to ncvalidator to reflect its
      purpose. It is now running in serial and thus no longer requires an MPI
      compiler to build it. In fact, its source codes are all in a single file,
      ncvalidator.c, which can be compiled using gcc, i.e. command "gcc -o
      ncvalidator ncvalidator.c".  This utility program can be used to validate
      a classic CDF file, and report the locations that fail to comply with CDF
      formats. It can also repair a file whose header fails to use null bytes
      for paddings, when the command-line option "-x" is used. See its man page
      for further description of usage.

  o Other updates:
    * Update UTF8 encoding utility to use utf8proc library version 2.1.0 from
      the Julia Language project, https://github.com/JuliaLang/utf8proc
      License https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md
    * Add quiet mode command-line option "-q" to utility program ncvalidator.
    * Add an internal check for whether the file starting offsets of all
      variables are in an increasing order as the variables defines. See r3369.
    * All build recipes (README files) have been moved to folder doc. Most of
      the recipes have been updated with instructions for building shared
      libraries.
    * Rename configure-time variables TEST_SEQRUN, TEST_MPIRUN, and TEST_OUTDIR
      to TESTSEQRUN, TESTMPIRUN, and TESTOUTDIR, respectively. This change
      avoids conflict to the name convention reserved by automake.
    * Fix TESTMPIRUN default value to use full path specified in --with-mpi at
      configure command line

  o Bug fixes
    * Fix a bug in nonblocking APIs when two separate nonblocking requests that
      can be coalesced into a contiguous MPI fileview or buffer type but the
      coalesced length becomes too big and overflows a 4-byte integer, data
      type of argument blocklengths of MPI_Type_create_hindexed, coalescing
      should be skipped. I.e. such two requests should be treated as two
      separate blocks in MPI_Type_create_hindexed.  Thanks Jim Edwards for
      reporting. See r3565.
    * Fix a bug in nonblocking APIs when one request's access region covers
      more than one other non-interleaved requests. PnetCDF fails to identify
      them as a single group of interleaved requests. Test program
      test/nonblocking/interleaved.c has been revised to test the fix.
      See r3389.
    * Fix a bug in string parsing the environment variable PNETCDF_HINTS and
      print warning messages when hint strings are ill-formed. See r3284.
    * Fix a bug in F90 APIs for reading and writing scalar variables, i.e.
      number of dimensions is zero. The bugs appear only when using Intel
      Fortran compiler which does not allow checking presence of an argument
      and its contents in the same if condition. See r3009, r3012, r3013.
    * Conform with CDF-2 file format specification on using 2^32-1 for vsize
      when the variable size is larger than 2^32-4 bytes. See r2957.
    * Fix a bug in nc_test when running on an ARMv7 architecture, a raspberry
      pi, where char is unsigned and which is little Endian, 32-bit machine.
      See r2922.
    * Fix a bug that fails to do byte-swap when retrieving user-defined fill
      values. See r2864. This fix is critical when PnetCDF is configured with
      option erange-fill enabled (default mode) and the target platform is a
      Little Endian.

  o New example programs
    * none

  o New programs for I/O benchmarks
    * none

  o New test program
    * test/testcases/tst_vars_fill.m4 - tests strided put (vars) APIs with
      fill mode on.
    * test/largefile/large_coalesce.c - tests a bug fixed in r3565 when two
      separate nonblocking requests that can be coalesced into a contiguous MPI
      fileview or buffer type but the coalesced length becomes too big and
      overflows a 4-byte integer.
    * test/testcases/tst_info.c - tests an internal bug in heap memory
      allocation trace mechanism. See r3514.
    * test/cdf_format/tst_corrupt.c
      1. tests whether NC_EMAXDIMS can be thrown when opening a corrupted file
         header with number of dimensions being larger than NC_MAX_DIMS.
      2. tests whether NC_EMAXATTS can be thrown when opening a corrupted file
         header with number of attributes being larger than NC_MAX_ATTRS.
      3. tests whether NC_EBADDIM can be thrown when opening a corrupted file
         header with dimension ID of a variable is beyond the number of
         dimensions defined in the file.
      4. tests whether NC_EBADTYPE can be thrown when opening a corrupted file
         header with nc_type of a variable is not within the legal NC atomic
         data type allowed by the file format.
    * test/testcases/tst_max_var_dims.c - tests the number of dimensions
      against NC_MAX_VAR_DIMS when calling ncmpi_def_var API.
    * test/largefile/tst_cdf5_begin.c - tests CDF-5 file header corruption for
      the case when the starting file offsets of variables are not in an
      increasing order as they defined.
    * test/cdf_format/tst_open_cdf5.c - tests whether NC_ENOTNC can be
      correctly thrown when opening a corrupted CDF-5 (bad_begin.nc5) which
      contains two variables and the second variable's starting file offset is
      less than the first variable's end offset.
    * test/testcases/large_var_cdf5.c - tests whether NC_EVARSIZE can be
      correctly thrown when a variable or a variable record is larger than
      NC_MAX_INT64-3.
    * test/testcases/mix_collectives.c - tests collective get/put APIs that
      allow different processes to call different kinds of APIs (i.e. var1,
      vara, vars, varm, etc.) and use different var IDs. In other words, the
      API kinds can be different among processes in a collective call.
    * test/largefile/high_dim_var.c - tests blocking vars APIs for high-
      dimensional variables. See r3164.
    * test/largefile/large_dims_vars_attrs.c - tests large number of
      dimensions, large number of variables, and large number of attributes.
    * test/testcases/put_all_kinds.m4 - tests various blocking put APIs.
    * test/testcases/tst_dimsizes.c - borrowed from NetCDF library to test
      define/inquire APIs for the maximum allowable dimension sizes. This test
      program is similar to (or a subset of) test/cdf_format/dim_cdf12.c
    * test/testcases/null_args.c - tests whether the correct error codes can
      be returned when using NULL arguments for start, count, stride, or imap

  o Conformity with NetCDF library
    * API ncmpi_open now checks the number of large variables that is legal
      for CDF-1 and CDF-2 files. See r2961. For the information about the
      number of large variables allowed in the classic formats can be found in
      http://www.unidata.ucar.edu/software/netcdf/docs/file_structure_and_performance.html#classic_format_limitations
    * Add checking for error code NC_ELATEFILL. Setting a variable's attribute
      _FillValue must be done in the variable's initial define mode.
    * NetCDF-4 documents in 4.4.1 and prior state that setting attribute
      _FillValue for global variable NC_GLOBAL is not allowed and error code
      NC_EINVAL will be thrown. Revisions r2954, r2955, and r2956 enforce this
      rule. However, because the rule has never been implemented in any NetCDF
      official release, NetCDF developers have decided to abandon this rule
      by revising the NetCDF document starting from v4.5. See discussion in
      NetCDF GitHub pull request 458. To be consistent with NetCDF, PnetCDF
      follows the same decision by wrapping the checking with preprocessor
      conditional #ifdef NO_NC_GLOBAL_FILLVALUE (see r3403). Note NC_EGLOBAL
      will be thrown when using NC_GLOBAL in ncmpi_def_var_fill(). This is
      due to no data type information for the fill value can be passed.
    * From the releases of 1.6.1 through 1.8.1, putting attribute _FillValue
      automatically turns on the fill mode for the variable. Similarly,
      deleting attribute _FillValue turns off the fill mode. However, this
      behavior does not conform with NetCDF library. This release fixes this
      inconsistency, so putting or deleting attribute _FillValue does not
      affect a variable's fill mode. See r2951.

  o Conflict with NetCDF library
    * Null-byte padding for attribute values in file header -- The classic CDF
      specifications require header padding uses null (\x00) bytes. However,
      prior to version 4.5.0, NetCDF did not implement this specification
      entirely. In particular, it has never enforced the null-byte padding for
      attribute values (it has for others, such as names of dimension,
      variables, and attributes.) It also appears that files created by SciPy
      NetCDF module or NetCDF Java module, both developed independent from
      NetCDF-C, also fail to respect this padding specification.  This becomes
      a problem for PnetCDF to read such netCDF files, because PnetCDF enforces
      the header padding from its very first release.  The files violating the
      padding specification will not be readable by PnetCDF of all releases
      prior to 1.9.0 and error code NC_EINVAL or NC_ENOTNC will be thrown when
      opening such files.  Note if the sizes of all attribute values of your
      files are aligned with 4-byte boundaries, then the files are readable by
      PnetCDF.  In order to keep the files in question readable by PnetCDF,
      checking for null-byte padding has been disabled in 1.9.0 by default. A
      new configure command-line option "--enable-null-byte-header-padding" has
      been made available to enable the checking and error code NC_ENULLPAD
      will be thrown when opening files. Note NC_ENULLPAD is not considered a
      fatal error. Users can safely ignore it. However, we keep this checking
      in ncvalidator no matter the new configure option is set or not.
      ncvalidator is a utility program reports whether a CDF file violates the
      file format specification, including this null-byte padding. See r3528
      for source codes changed and discussion in NetCDF Github issue
      https://github.com/Unidata/netcdf-c/issues/657.

  o Issues related to MPI library vendors:
    * When using OpenMPI 2.0.0 through the latest 2.1.2 to build PnetCDF 1.8.1
      and prior, the following message may appear when running make check:
          "MPI error (MPI_File_open) : MPI_ERR_FILE: invalid file"
      but the test programs report "pass". This message can be safely ignored.
      It is due to OpenMPI failing to translate system error code ENOENT into
      MPI_ERR_NO_SUCH_FILE, but returns MPI_ERR_FILE instead. Thanks to Edgar
      Gabriel, a trick to silence the messages is to add the following to the
      command line of make check:
        TEST_SEQRUN="/path/to/OpenMPI/2.x.x/bin/mpirun --mca io romio314 -n 1"
      See discussion thread in OpenMPI:
      https://github.com/open-mpi/ompi/issues/4412
      Note this error message does not appear in PnetCDF 1.9.0 and after.
    * When using OpenMPI 2.1.1 and priors built with Intel compilers 17.0.0,
      the test program nonblocking/mcoll_testf.f90 fails with segmentation
      fault. This is due to a bug in OpenMPI. See discussion thread in OpenMPI:
      https://github.com/open-mpi/ompi/issues/3695
    * PnetCDF can no longer be built on 32-bit machines if using OpenMPI
      versions 2.0.2 and 1.10.6 and earlier versions. This is because OpenMPI
      configures type MPI_Offset as a 4-byte integer on all 32-bit machines,
      even if the machines allow 64-bit file offsets, known as large-file
      support. We suggest to use other MPI libraries to build PnetCDF on 32-bit
      machines. See the discussion thread in OpenMPI:
      https://github.com/open-mpi/ompi/issues/3195
    * Fix test/nc_test/test_read.m4 and test/testcases/modes.c which test
      whether the correct error codes can be returned when opening a
      non-existing file. Thanks to Mark Dixon for testing various MPI
      libraries, C compilers, and file systems, we found that when using
      MVAPICH2 2.2, its Lustre driver always adds O_CREAT flag to the open()
      calls. Because of this behavior, the non-existing file will be created
      with zero length, which causes PnetCDF to spew NC_ENOTNC instead of
      NC_ENOENT. A bug report has been filed to MVAPICH. The fix shall come in
      the next release of MVAPICH. The bug report can be found in
      http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2017-February/006300.html

  o Issues related to Darshan library:
    * When using Darshan in conjunction with PnetCDF, if you encounter
      link-time or run-time errors with messages shown below, then we recommend
      to use Darshan library version 3.1.5 and later.

      During compile/link time:
      /soft/perftools/darshan/lib/libdarshan-mpi-io.a(darshan-pnetcdf.o):
      In function `__wrap_ncmpi_create': darshan-pnetcdf.c:62: undefined
      reference to `ncmpi_create'

      During run time:
      WARNING: Darshan ncmpi_create() stub called; this is probably the result
      of a link-time problem.

  o Clarifications
    * The NetCDF CDF formats define a set of external NC data types and
      describe their intents of use, shown below.

      External type  No. Bits  Intent of use
      -------------  --------  ---------------------------------
      NC_CHAR         8        text data (the only non-numerical type in NetCDF)
      NC_BYTE         8        1-byte integer
      NC_SHORT       16        2-byte signed integer
      NC_INT         32        4-byte signed integer
      NC_FLOAT       32        4-byte floating point number
      NC_DOUBLE      64        8-byte real number in double precision
      NC_UBYTE        8        unsigned 1-byte integer
      NC_USHORT      16        unsigned 2-byte integer
      NC_UINT        32        unsigned 4-byte integer
      NC_INT64       64        signed 8-byte integer
      NC_UINT64      64        unsigned 8-byte integer

      Note NC_CHAR is the only non-numerical data type available in NetCDF
      realm.  All other external types are considered numerical, which are
      illegal to be converted (type-casted) to and from a NetCDF variable
      defined in NC_CHAR type (error code NC_ECHAR will be thrown). The only
      legal APIs to read/write a variable of type NC_CHAR are the "_text" APIs.

    * Starting from 1.7.0, PnetCDF translates internal data types (i.e. data
      types of the I/O buffer and also used in the API name such as text,
      schar, uchar, short, int, etc.) to MPI data types based on the table
      below. Note MPI_BYTE does no correspond to any internal data type used
      in NetCDF/PnetCDF APIs. Thus, when MPI_BYTE is used to construct an MPI
      derived data type which is later used as argument buftype in a flexible
      API, the error code NC_EBADTYPE will be thrown (Not a valid data type).

      Data type of                                  Corresponding
      internal I/O buffer   Example API             MPI datatype
      -------------         ----------------------  -----------------
      text                  ncmpi_put_var_text      MPI_CHAR
      schar                 ncmpi_put_var_schar     MPI_SIGNED_CHAR
      uchar                 ncmpi_put_var_uchar     MPI_UNSIGNED_CHAR
      short                 ncmpi_put_var_short     MPI_SHORT
      ushort                ncmpi_put_var_ushort    MPI_UNSIGNED_SHORT
      int                   ncmpi_put_var_int       MPI_INT
      uint                  ncmpi_put_var_uint      MPI_UNSIGNED
      long                  ncmpi_put_var_long      MPI_LONG
      float                 ncmpi_put_var_float     MPI_FLOAT
      double                ncmpi_put_var_double    MPI_DOUBLE
      longlong              ncmpi_put_var_longlong  MPI_LONG_LONG_INT
      ulonglong             ncmpi_put_var_ulonglong MPI_UNSIGNED_LONG_LONG

-------------------------------------
Version 1.8.1 (January 28, 2017)
-------------------------------------

  This release is mainly for fixing the following bug.

  o Bug fixes:
    * Fortran parameters NF_MAX_DIMS, NF_MAX_ATTRS, NF_MAX_VARS, and NF_MAX_NAME
      defined in the Fortran header file, pnetcdf.inc, were set to obsolete
      values and have been fixed in this release to conform with their
      corresponding C constants. Because these four parameters were not used
      by PnetCDF internally, the end users can safely skip this release and
      1.8.1 should perform the same as 1.8.0. If your programs explicitly use
      these four parameters, then updating to 1.8.1 is recommended.

  o Other updates:
    * To conform with GNU Coding Standards, the installation location for man
      pages are changed to ${prefix}/share/man.
    * Support of DESTDIR for Building Binary Packages has been added, as
      suggested by sunpoet@FreeBSD.org. Ticket #25

-------------------------------------
Version 1.8.0 (December 19, 2016)
-------------------------------------

  o New features
    * Copy fill values to data elements that cause out-of-bound error (i.e.
      NC_ERANGE.) In netCDF v4.4.1 and priors, type conversion for out-of-
      bound elements proceeds even if the out-of-bound error is detected, and
      the contents (in both files and read buffers) are left up to the user
      to handle. See the URL below for details.
      http://www.unidata.ucar.edu/software/netcdf/docs/group__error.html
      Instead of carrying out the type conversion, PnetCDF fills the out-of-
      bound data elements with their "fill values", as specified by the
      variable's attribute "_FillValue" or default CDF fill values if the
      attribute is not defined. Specifically, for PnetCDF put APIs, the value
      of attribute _FillValue, if defined, will be used when writing to files.
      For get APIs, the default CDF _FillValue corresponding to the internal
      data type will be used to fill the read buffer. This feature can be
      disabled by adding option "--disable-erange-fill" to the configure
      command line. In this case, the conventional NetCDF method described
      above is used.
    * A new configure option --enable-relax-coord-bound is added to provide a
      more relaxed boundary check for argument start. Traditionally, both
      NetCDF and PnetCDF detect the error code NC_EINVALCOORDS when the value
      of start is larger than or equal to the defined dimension size. This can
      be inconvenient for some applications that make a zero-length request
      with start being the dimension size, usually the last process. The new
      configure option relaxes the boundary check by allowing the above
      exception, i.e. start[i] can be the size of dimension i only when
      count[i] is zero. The default of this feature is disabled. For more
      detailed information, see discussion in:
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2016-March/001839.html
    * New macros have been added to PnetCDF header file, pnetcdf.h, to indicate
      the configurable options that were enabled/disabled. These macros are
      PNETCDF_ERANGE_FILL, PNETCDF_SUBFILING, PNETCDF_RELAX_COORD_BOUND, and
      PNETCDF_DEBUG_MODE. Their values are set to 1 when enabled, 0 otherwise.
    * API ncmpi_inq_file_format can now detect HDF5 file signature and report
      NC_FORMAT_NETCDF4.
    * Enable ncmpidump to check whether input file is in HDF5 format when -k
      option is used at the command line.
    * In ncmpi_open, add a checking for HDF5 file signature and return error
      code NC_ENOTNC3 if detected.

  o New APIs
    * ncmpi_strerrno, nfmpi_strerrno, nf90mpi_strerrno return the string name
      of a NC error code.
    * ncmpi_inq_path, nfmpi_inq_path, nf90mpi_inq_path for inquiring the file
      pathname which was used to open/create the CDF file.

  o New optimization
    * Use Bernstein hash function for checking whether the name of a variable
      or dimension has already been used.
    * Replace the use of linked lists with arrays for storing the pending
      nonblocking requests. This could save time for traversing the linked
      list.

  o Syntax changes
    * ncmpi_cancel can now be called in define mode.

  o Semantics updates
    * For collective APIs, the following errors are considered fatal:
      NC_EBADID, NC_EPERM, NC_EINDEFINE, NC_EINDEP, and NC_ENOTINDEP. Once
      detects these errors the calling APIs return immediately. If the fatal
      errors happen only to a subset of processes, the running program may
      hang, due to other processes being waiting at an MPI collective call.
      (Note hanging will never occur to independent APIs.) To debug a hanging
      problem, users can run the program in safe mode by setting the
      environment variable PNETCDF_SAFE_MODE to 1. In safe mode, error codes
      are synchronized among all processes. Error codes other than the above,
      if detected, are not as fatal. The processes encountering the non-fatal
      errors will continue to the collective call but with zero-length
      requests. Thus, the collective calls can nicely complete without hanging.
    * Consistency check for header data (defining dimensions, variables, and
      attributes) is moved to individual APIs and only enabled when safe mode
      is enabled. Header data consistency check is no longer performed in
      ncmpi_enddef. As a reminder, PnetCDF APIs that change header data are
      all collective and their arguments are required to be consistent among
      the calling processes.
    * In netCDF-3, whether the contents of an NC_BYTE variable in CDF-1 or
      CDF-2 files are signed or unsigned is determined by the calling APIs. See
      http://www.unidata.ucar.edu/software/netcdf/docs/data_type.html#type_conversion
      "In netcdf-3, we treat NC_BYTE as signed for the purposes of conversion
      to short, int, long, float, or double. (Of course, no conversion takes
      place when the internal type is signed char.) In the _uchar functions, we
      treat NC_BYTE as if it were unsigned. Thus, no NC_ERANGE error can occur
      converting between NC_BYTE and unsigned char." In other words, if called
      from signed APIs, NC_BYTE variables are treated as signed. If called from
      unsigned APIs (i.e. _uchar APIs, the only unsigned APIs in netCDF-3) they
      are unsigned. NetCDF-3 specifically makes an exception to skip NC_ERANGE
      check when calling _uchar APIs on NC_BYTE variables. However, in netCDF-4
      and CDF-5, because of the introduction of the new data type NC_UBYTE, an
      unsigned 8-bit integer, which makes NC_BYTE a signed 8-bit integer and
      thus renders the above exception less sense. Starting from this release,
      for CDF-5 files, regular NC_ERANGE check is performed in PnetCDF for all
      APIs that access NC_BYTE variables. For CDF-1 and 2 files, PnetCDF still
      honors that exception.
    * Prior to version 1.7.0, any outstanding nonblocking requests detected at
      file close are automatically cancelled without reporting an error.
      Starting from this 1.8.0 release, error code NC_EPENDING will be returned
      when pending requests are detected at the file close.
    * API ncmpi_set_default_format is changed to an independent API. However,
      it is expected to be called by all processes that intend to create a file
      later. Inconsistency in default format setting will be detected at file
      create time.

  o New utility program
    * Added pnetcdf-config shell script to display the options used in building
      PnetCDF.

  o New error code precedence
    * When two or more errors are detected by the PnetCDF library (e.g. due to
      the use of more than one bad API argument), the error code returned from
      an API will follow the precedence below. In principle, the errors related
      to ncid is the most serious, followed by varid.
      For put attribute APIs:
          NC_EBADID, NC_EPERM, NC_ENOTVAR, NC_EBADNAME, NC_EBADTYPE, NC_ECHAR,
          NC_EINVAL, NC_ENOTINDEFINE, NC_ERANGE
      For get attribute APIs:
          NC_EBADID, NC_ENOTVAR, NC_EBADNAME, NC_ENOTATT, NC_ECHAR, NC_EINVAL,
          NC_ERANGE
      For put/get variable APIs:
          NC_EBADID, NC_EPERM, NC_EINDEFINE, NC_ENOTVAR, NC_ECHAR,
          NC_EINVALCOORDS, NC_EEDGE, NC_ESTRIDE, NC_EINVAL, NC_ERANGE

  o New error code
    * NC_EPENDING: Pending nonblocking request is found at file close.
    * NC_EINVAL_OMODE: invalid file open mode. This separates the case from the
      file create API which uses NC_EINVAL_CMODE mode. The error messages for
      both error codes are revised accordingly.
    * NC_EMULTIDEFINE_CMODE: inconsistent file create mode. This separates the
      case from NC_EMULTIDEFINE_OMODE. The error messages for both error codes
      are revised accordingly.
    * The Fortran error codes corresponding to the above are also added.

  o New test program
    * test/testcases/scalar.c to test whether arguments start, count, stride,
      and imap are properly ignored when getting/putting a scalar variable
    * test/testcases/erange_fill.m4 to test the new configure-time feature of
      --enable-erange-fill

  o Other updates:
    * Check file open/create mode for NC_DISKLESS and NC_MMAP and return error
      code NC_EINVAL_OMODE/NC_EINVAL_CMODE if either of these two modes is
      used. These modes are currently not supported.
    * Clarify support for CDF-2 and CDF-5 formats. When the size of MPI_Offset
      is less than 8 bytes, support for CDF-2 and CDF-5 is disabled.
    * When using OpenMPI 2.0.1 to build PnetCDF, users may encounter an error
      message of "test_write.c: file ./scratch.nc should not exist" or "FAILURE
      at line 429 of test_ncmpi_abort in test_write.c: expecting NC_ENOENT or
      NC_EFILE but got NC_NOERR" while running "make check". The cause is due
      to a bug in OpenMPI 2.0.1 that fails to return the correct MPI error
      class MPI_ERR_NO_SUCH_FILE when deleting a non-existing file. The bug fix
      will appear in OpenMPI 2.0.2. See the OpenMPI bug report in
      https://github.com/open-mpi/ompi/issues/2232
    * When using OpenMPI 1.10.2 and 1.8.5 to build PnetCDF, users may encounter
      an error message of "An error occurred in MPI_Type_create_hindexed" when
      running "make check". This is due to a bug in OpenMPI that fails to
      create a zero-size datatype in MPI_Type_create_hindexed. The bug fix has
      appeared in the releases of 1.10.3, 1.8.6, and 2.0.1. See the bug report
      in https://github.com/open-mpi/ompi/issues/1611
    * When using SGI MPI with MPT 2.14 to build PnetCDF, users may encounter an
      error message of "function:MPI_TYPE_CREATE_HVECTOR, Invalid argument".
      The bug fix will appear in 2.15 to be released in November 2016.
    * The quiet mode of utility program ncmpidiff, command-line option -q, now
      print no message when two files or two variables in comparison are the
      same.

  o Interoperability with netCDF-4
    * When using 1.8.0 release to build netCDF version 4.4.1.1 and earlier,
      users may encounter errors with message like "test_put.c: NetCDF: Numeric
      conversion not representable" from running command "make test".  This is
      due to an incorrect implementation of uchar APIs for accessing variables
      of type NC_BYTE for CDF-2 files in both PnetCDF and NetCDF. If not using
      such a special case, users can ignore this error. See more information
      above under "Semantics updates".

  o Bug fixes
    * Fix to allow argument no_fill in API ncmpi_inq_var_fill to be NULL. See
      r2609
    * Fix to allow putting global attribute _FillValue. See r2560
    * Fix some missing character string normalizations for names of attributes,
      dimensions, and variables.
    * Fix the return unlimited dimension ID of value -1, not 0, by Fortran API
      nfmpi_inq_unlimdim when no unlimited dimension is defined. This conforms
      with netCDF convention. See r2396 and r1372
    * Fix string processing of the environment variables for PnetCDF I/O hints
      See r2385
    * Fix utility ncmpidiff to use collective read APIs in collective data mode
      See r2382


-------------------------------------
Version 1.7.0 (March 3, 2016)
-------------------------------------

  o New features
    * Nonblocking requests now can be posted while the program is in the define
      mode. This feature enables applications to add new variables and post the
      nonblocking write requests without separating the codes by
      ncmpi_enddef(). Note ncmpi_wait_all and ncmpi_wait must still be called
      in the data mode.
    * When using immutable write buffer in put APIs on a Little Endian machine,
      the in-place byte swap operation performed internally in PnetCDF can
      cause a fatal error (by trying to change the contents of an immutable
      memory space). The solution of copying user's write request to a
      temporary buffer and perform byte swap there conflicts with PnetCDF's
      design principle of avoiding internal memory allocation as much as
      possible. Since the size of immutable buffer in most cases is small,
      PnetCDF now compromise the two by making a copy of write requests that is
      less than 4KB. Users are warned that using immutable write buffer larger
      than 4KB will still cause the fatal error in the PnetCDF default
      configuration. The way to completely disable in-place byte swap is to
      build PnetCDF with --disable-in-place-swap option at the configure time.
      In this case, the internal memory allocation in PnetCDF will increase.

  o Syntax changes
    * In ncmpi_wait_all(), ncmpi_wait(), and ncmpi_cancel(), the API's third
      argument, array of requests IDs, is changed to INOUT. Upon successful
      completion or cancellation of individual nonblocking requests, the
      corresponding request IDs are set to NC_REQ_NULL.

  o New run-time environment variables
    * PNETCDF_VERBOSE_DEBUG_MODE environment variable can be used to print the
      location in the source code where the error code is originated, no matter
      the error is intended or not. This run-time environment variable only
      takes effect when PnetCDF is configure with debug mode, i.e.
      --enable-debug is used at the configure command line.  Set this variable
      to 1 to enable. Set it to 0 or keep it unset disables this mode. Default
      is 0, i.e. disabled.  Users are warned that enabling this mode may result
      in a lot of debugging messages printed in stderr.

  o New example programs
    * examples/C/nonblocking_write_in_def.c shows an example of posting
      nonblocking write requests in the define mode.
    * examples/C/req_all.c shows an example of using NC_REQ_ALL to flush all
      pending nonblocking requests without providing the requests IDs.

  o New test program
    * test/cdf_format/dim_cdf12.c tests defining maximal dimension size for
      CDF-1 and CDF-2 file formats
    * test/testcases/test_erange.c tests if the error code NC_ERANGE can be
      correctly returned for two cases 1) get a NC_UBYTE value of 255 from a
      netCDF file to a memory buffer of type signed char and 2) put a value
      of -1 of signed char to a NC_UBYTE variable in a netCDF file
    * test/testcases/check_type.c tests if the error codes can be correctly
      returned when conflicted in-memory and external data types are used.
    * test/testcases/put_parameter.f tests the use of immutable write buffer
      (e.g. a buffer declared as PARAMETER). Note the buffer size must be
      smaller than 4KB.
    * test/nonblocking/i_varn_indef.c tests posting nonblocking requests in
      define mode.
    * test/nonblocking/req_all.c tests the use of NC_REQ_ALL for flushing
      all pending nonblocking requests without providing the requests IDs.
    * test/last_large_var.c tests the special case when there is no record
      variable, the last fixed-size variable can be larger than 2GiB in size
      if its starting file offset is less than 2GiB.
    * test/testcases/buftype_free.c and test/testcases/buftype_freef.f test
      the bug in r2160.
    * testcases/add_var.c checks the starting file offsets of newly added
      variables from re-entering the define mode.
    * testcases/attrf.f checks NF_ERANGE is returned instead of coredump. This
      is particularly for NAG Fortran compiler that may report "Arithmetics
       exception".
    * testcases/check_striping.c checks if the file striping unit and factor
      returned from MPI-IO hints are consistent among processes.
    * test/nonblocking/column_wise.c checks if PnetCDF detects interleaved
      fileviews from multiple nonblocking requests and correctly breaks and
      reconstructs the fileviews so the combined fileview is monotonic
      non-decreasing in file offsets.

  o New optimization
    * Filling variables at ncmpi_enddef() is now done by aggregating all write
      requests into one MPI collective write call. In v 1.6.1, this is done by
      filling one variable at a time.

  o New utility program
    * ncoffsets reports the file offset information, including the starting and
      ending file offsets, of variables stored in a netCDF file. ncoffsets is
      compiled with gcc if gcc is presented on the build system. Additional
      command-line options are: (-v) reports only for a selected list of
      variables in interest, (-s) prints the variable sizes, (-g) outputs the
      file space gap size from the end of previous variable, (-x) reports
      whether there is a gap between any two adjacent fixed-size variables. See
      the man page for descriptions of all command-line options and examples.

  o Semantics updates
    * All nonblocking APIs now take a NULL pointer for the request ID argument,
      meaning users do not wish to keep track of the request ID. If NULL
      request IDs are used, NC_REQ_ALL should be used when calling
      ncmpi_wait_all/ncmpi_wait to commit all the pending nonblocking requests.
      This feature relinquishes users from the responsibility of tracking the
      IDs of pending requests.
    * Using NC_REQ_ALL as the 2nd argument "num" in ncmpi_wait_all/ncmpi_wait
      APIs will flush all the pending nonblocking requests. In this case, the
      3rd and 4th arguments "array_of_requests" and "array_of_statuses" will be
      ignored and thus these two arguments can be NULLs.
    * Using NC_REQ_ALL in ncmpi_cancel() will cancel all the pending
      nonblocking requests.
    * Using NC_GET_REQ_ALL or NC_PUT_REQ_ALL in ncmpi_wait_all(), ncmpi_wait(),
      and ncmpi_cancel() for all the pending get-only or put-only requests,
      respectively.

  o Other updates:
    * Conform with netCDF on the maximal dimension size for CDF-2 file format
      to be (2^32 - 4)
    * NC_ERANGE checks have been removed from nc_test for text APIs and
      variables that are defined as NC_CHAR type
    * Add README.K-Computer build recipe for using Fujitsu MPI compilers on the
      K computer at RIKEN in Japan
    * Add README.INTEL build recipe for using Intel MPI compilers 4.x
    * Build dependency rule is added for files configure and configure.in
    * PnetCDF checks MPICC/MPICXX/MPIF77/MPIF90 instead of CC/CXX/F77/F90/FC.
      If MPICC/MPICXX/MPIF77/MPIF90 are set, PnetCDF will ignore
      CC/CXX/F77/F90/FC. If CC/CXX/F77/F90/FC is set instead of
      MPICC/MPICXX/MPIF77/MPIF90, PnetCDF will now copy them to
      MPICC/MPICXX/MPIF77/MPIF90.
    * Enforce netCDF convention on error code priority: NC_ECHAR trumps
      NC_EINVALCOORDS, NC_EEDGE, and NC_ESTRIDE.
    * Return error code NC_EGLOBAL instead of NC_ENOTVAR for APIs where using
      NC_GLOBAL as the variable ID argument is prohibited.
    * All Fortran 77 test and example programs (files with .f and .F
      extensions) have been revised to conform with 77 standard, and was tested
      using pgf77.
    * Now provides a pkg-config file
      (http://www.freedesktop.org/wiki/Software/pkg-config/), making it
      slightly easier to set the correct pnetcdf include and library paths.

  o Bug fixes
    * Fix the bus error of invalid address alignment when build with Fujitsu
      compiler. See r2171 and r2180.
    * Fix the bug for the special case when there is no record variable, the
      last fixed-size variable can be larger than 2GiB in size if its starting
      file offset is less than 2GiB. See r2166.
    * Fix the nonblocking flexible APIs that fail to save (duplicate) the user
      MPI derived data type that later is needed to unpack read data to the
      user buffer (a call to MPI_Unpack). See r2160.
    * Fix Fortran 77 constants nf_fill_uint and nf_fill_int64 (thanks Jim
      Edwards) that pgf77-based MPI compiler does not like "_8" modifier.
      See r2045 and r2051. The same issue for NAG Fortran compiler is also
      resolved. See r2089 and r2093.
    * In the example program examples/tutorial/pnetcdf-write-nb.c, the write
      buffers used in two iput API calls should be different. See r2095.
    * Fix the error reporting mechanism for NC_ERANGE, for when an arithmetic
      overflow happens. Overflow checking is now performed before I/O.

-------------------------------------
Version 1.6.1 (June 1, 2015)
-------------------------------------

  o New features
    * PnetCDF now supports fill mode. ncmpi_set_fill() can be used to set the
      fill mode for the entire data set. Note the differences from netCDF:
      1. The default mode in PnetCDF is NC_NOFILL.
      2. Setting fill mode for the entire file or individual variables must be
         done in define mode.
      3. For non-record variables, they are filled at the time ncmpi_enddef()
         is called.
      4. For record variables, users must explicitly call ncmpi_fill_var_rec()
         to fill one record at a time before writing to the record of that
         variable.

  o New APIs
    * ncmpi_def_var_fill() sets fill mode for an individual variable. This API
      must be called in define mode.
    * ncmpi_inq_var_fill() inquires fill mode of a variable.
    * ncmpi_fill_var_rec() is a collective API that fills a record of a record
      variable. This API must be called at data mode.
    * ncmpi_inq_default_format() for inquiring the default file format for
      new file creation. Note the default format can be changed by
      ncmpi_set_default_format().
    * The above new API are also available in Fortran and C++ versions.

  o New error code
    * NC_ENOTRECVAR when attempting operation only for record variables
    * NC_ENOTFILL when attempting to fill a variable when its fill mode is off
    * NC_EMULTIDEFINE_FILL_MODE when inconsistent dataset fill mode is detected
    * NC_EMULTIDEFINE_VAR_FILL_MODE when inconsistent variable fill mode is
      detected
    * NC_EMULTIDEFINE_VAR_FILL_VALUE when inconsistent variable fill value is
      detected
    * Fortran versions of the above error codes are also added.

  o New example programs
    * C/fill_mode.c shows the use of ncmpi_set_fill(), ncmpi_def_var_fill(),
      ncmpi_inq_var_fill() and ncmpi_fill_var_rec()
      F77/fill_mode.f is the Fortran version.
      F90/fill_mode.f90 is the Fortran 90 version.
      CXX/fill_mode.cpp is the C++ version.
    * C/ghost_cell.c shows how to use varm API for writing from a user buffer
      as a 2D array with ghost cells on both ends of every dimension.

  o New test programs
    * nc_test/tst_nofill.c borrowed from netCDF
    * testcases/ivarn.c tests bug fix r2023 when the request IDs stored in
      argument array_of_requests[] of ncmpi_wait_all() are not in an
      increasing order.

  o Other updates:
    * Change the chunk size used for moving variable data when the file header
      extent expands. The default is now 1MB. If the file's striping unit
      size is known (from MPI-IO hint striping_unit) then the chunk size is
      set to the striping unit size.

  o Bug fixes
    * Add missing F90 function overloading for f90mpi_put_var_all,
      f90mpi_get_var_all, f90mpi_put_vard_all, and f90mpi_get_vard_all,
      when the user buffer is a scalar.
    * Fix when the request IDs passed in argument array_of_requests[] of
      ncmpi_wait_all() are not in an increasing order. See r2023.
    * Fix C++ compile error for converting NULL to string. See r2039.

-------------------------------------
Version 1.6.0 (February 2, 2015)
-------------------------------------

  o Format conformation updates:
    * Conform with netCDF4 on CDF-1 and CDF-2 formats. The only difference now
      between the two formats are the OFFSET item in the format spec (32 vs.
      64 bit integers.) All names (variable, dimension, attribute) now allow
      extended characters (e.g. special2 and MUTF8).

  o New APIs
    * Nonblocking buffered varn API family.
      For C,   ncmpi_bput_varn_<type>()
      For F77, nfmpi_bput_varn_<type>()
      For F90, nf90mpi_bput_varn()
      For C++, NcmpiVar::bputVarn()
    * Nonblocking varn API family.
      For C,   ncmpi_iput_varn_<type>() and ncmpi_iget_varn_<type>()
      For F77, nfmpi_iput_varn_<type>() and nfmpi_iget_varn_<type>()
      For F90, nf90mpi_iput_varn()      and nf90mpi_iget_varn()
      For C++, NcmpiVar::iputVarn()     and NcmpiVar::igetVarn()
    * Blocking vard API family takes an argument of MPI derived data type that
      describes the file access layout, as opposed to vara and vars APIs that
      use start[] and count[].
      For C,   ncmpi_put_vard()    and ncmpi_get_vard()
      For F77, nfmpi_put_vard()    and nfmpi_get_vard()
      For F90, nf90mpi_put_vard()  and nf90mpi_get_vard()
      For C++, NcmpiVar::putVard() and NcmpiVar::getVard()
    * Collective var1 API family
      For C,   ncmpi_put_var1_all()        ncmpi_get_var1_all()
               ncmpi_put_var1_<type>_all() ncmpi_get_var1_<type>_all()
      For F77, nfmpi_put_var1_all()        nfmpi_get_var1_all()
               nfmpi_put_var1_<type>_all() nfmpi_get_var1_<type>_all()
      For F90, nf90mpi_put_var_all()       nf90mpi_get_var_all()
      For C++, NcmpiVar::putVar_all()      NcmpiVar::getVar_all()
    * ncmpi_inq_buffer_size() returns the size of buffer previously attached
      for use of bput APIs. With ncmpi_inq_buffer_usage() one can calculate
      the space remaining for additional bput requests.
      For F77, nfmpi_inq_buffer_size()
      For F90, nf90mpi_inq_buffer_size()
      For C++, NcmpiFile::Inq_buffer_size()
    * ncmpi_inq_recsize() returns the size of record block, i.e. the sum of
      single records of all record variables.
      For F77, nfmpi_inq_recsize()
      For F90, nf90mpi_inq_recsize()
      For C++, NcmpiGroup::getRecSize()
    * ncmpi_inq_num_rec_vars() and ncmpi_inq_num_fix_vars() report the number
      of record and fixed-size variables, respectively.
      For F77, nfmpi_inq_num_rec_vars()     and nfmpi_inq_num_fix_vars()
      For F90, nf90mpi_inq_num_rec_vars()   and nf90mpi_inq_num_fix_vars()
      For C++, NcmpiGroup::getRecVarCount() and NcmpiGroup::getFixVarCount()

  o New PnetCDF hint
    * pnetcdf_subfiling -- it can be set in an MPI info object or in the
      environment variable PNETCDF_HINTS to enable/disable subfiling.
      The value is either "enable" or "disable".

  o PnetCDF hint priority
    * The alignment hints set in the environment variable "PNETCDF_HINTS" have
      the highest priority, which overwrite the alignment hints set in
      `ncmpi__enddef()`, which overwrite the alignment hints set in the MPI_Info
      object used in the call of ncmpi_create() and ncmpi_open().

  o New error code
    * NC_ESTRICTCDF2 for attempting CDF-5 operation on CDF-2 file. For
      example, define a variable of type NC_INT64 in a CDF-2 file.
    * NC_ETYPESIZE when filetype size is bigger than the variable size
    * NC_ETYPE_MISMATCH when the element type of filetype mismatches the
      variable type
    * NC_ETYPESIZE_MISMATCH when filetype size mismatches buffer type size
    * NC_ENULLSTART when argument start is a NULL pointer
    * NC_ENULLCOUNT when argument count is a NULL pointer
    * NC_EINVAL_CMODE when invalid file create mode is set, (e.g. cannot have
      both NC_64BIT_OFFSET & NC_64BIT_DATA. In PnetCDF 1.5.0 and earlier
      versions, if both flags were set, then NC_64BIT_DATA triumphs
      NC_64BIT_OFFSET.)

  o New example programs
    * C/bput_varn_uint.c and F77/bput_varn_int8.f show the use of
      nonblocking bput_varn APIs
    * C/i_varn_int64.c and F77/i_varn_real.f show the use of nonblocking
      iput_varn and iget_varn APIs
    * C/vard_int.c F77/vard_int.f F90/vard_int.f90 CXX/vard_int.cpp show the
      use of vard API to write/read record and fixed-size variables.
    * C/transpose.c shows the use of ncmpi_put_vara_int_all to write a 3D array
      that is dimensional-transposed from the one stored in memory. Six
      transposed 3D arrays are saved whose dimensions are organized as ZYX,
      ZXY, YZX, YXZ, XZY, and XYZ. The C++, Fortran 77, and Fortran 90
      versions are also provided.

  o New test program
    * nonblocking/wait_after_indep.c tests if ncmpi_end_indep_data() returns
      properly when nonblocking APIs are called in independent data mode and
      the wait call is made after exiting the independent mode.
    * nonblocking/flexible_bput.c tests flexible bput APIs that use
      noncontiguous buffer type, noncontiguous imap and require type conversion
    * testcases/flexible2.c tests flexible APIs that perform type conversion
    * testcases/flexible_varm.c tests flexible varm APIs
    * testcases/varn_contig.c tests the case when the fileview is actually a
      contiguous chunk. PnetCDF should be able to merge all.
    * nonblocking/bput_varn_uint.c tests nonblocking bput_varn APIs
    * nonblocking/i_varn_int64.c tests nonblocking iput_varn and iget_varn APIs
    * test/testcases/test_vard.c test/testcases/test_vardf.f
      test/testcases/test_vardf90.f90 test the new vard APIs.
    * test/testcases/inq_recsize.c tests if one can correctly inquire the
      size of record block from in a netCDF file. A similar program in F90,
      named inq_recsizef.f90, is also added.
    * In test/nc_test, the test programs below are borrowed from netCDF test
      programs: t_nc.c tst_misc.c tst_norm.c tst_small.c tst_names.c
      tst_atts3.c tst_atts.c
    * test/testcases/one_record.c tests the special case defined in CDF-1 and
      CDF-2 specification that "A special case: Where there is exactly one
      record variable, we drop the requirement that each record be four-byte
      aligned, so in this case there is no record padding."
    * test/testcases/modes.c tests if correct error codes are returned when
      various file create/open modes are used.
    * Under test/testcases, varn_int.c varn_intf.f varn_real.f90 test varn APIs
    * test/testcases/inq_num_vars.c test if one can correctly inquire the
      numbers of record and fixed-size variables defined in a netCDF file.
      A similar program in F90, named inq_num_varsf.f90, is also added.
    * test/nonblocking/interleaved.c tests a combination of interleaved
      file types. This is to test the bug fix in r1758.

  o New optimization
    * Prior to this release, PnetCDF wraps each MPI read/write call in put/get
      APIs with two MPI_File_set_view(). One is before the MPI read/write call
      to take advantage of MPI's fileview feature for accessing non-contiguous
      file locations. The other is after the MPI read/write call to make the
      whole file visible, as the root process may write to file header later
      in the data mode and it alone cannot make a call to MPI_File_set_view()
      because the function is collective.

      In this release, the second MPI_File_set_view() has been removed. The
      root process's fileview is changed to always keep the whole file header
      visible. Saving a collective call to MPI_File_set_view() is expected to
      improve some performance.

  o Semantics updates
    * Header consistency mechanism has been updated. See README.consistency for
      details.
    * The use of NC_SHARE is also revised. See README.consistency for details.

  o Other updates:
    * The subfiling feature is now disabled in ncmpidump and ncmpidiff until
      a bug fix to allow reading the master file with the number of processes
      that is smaller than the number of subfiles.
    * The attribute names reserved for subfiling feature are now changed to
      use a prefix of "_PnetCDF_SubFiling". The leading "_" underscore is a CDL
      convention.
    * The flexible APIs now allow argument buftype to be MPI_DATATYPE_NULL.
      In this case, argument bufcount is ignored and argument buf's data type
      must match the data type of variable defined in the file - no data
      conversion will be done. This extension makes the flexible APIs be
      able to correspond to the netCDF APIs whose names do not contain a data
      type, e.g. nc_put_vara().
    * Type conversion between NC_BYTE and unsigned char no longer checks for
      out of range error (NC_ERANGE). CDF file format specification make a
      special case for this kind of data type conversion. See: "Note on byte
      data" in the format specification.
    * Conform with the CDF-2 and CDF-5 formats that names are normalized
      according to Unicode NFC normalization rules during encoding as UTF-8
      for storing in the file header.
    * A new configure option, --enable-large-file-test, to enable testing
      I/O on large files and large variables. Note the testing programs will
      run very slowly.

  o Bug fixes
    * fix the bug for flexible get_varn API. When buftype is noncontiguous, the
      bug forgot to "unpacks" the temporary buffer that reads data from file to
      the user buffer using buftype.
    * fix the bug in blocking flexible get APIs when buftype is noncontiguous,
      swap is needed, type conversion is not. The bug sets a NULL pointer to
      the read buffer and passes it to MPI_File_read functions. See r1815.
    * fix the bug for the NetCDF special case when there is only one record
      variable and the record size is not four-byte aligned. In this case,
      NetCDF spec says the alignment must dropped (no padding). See r1791.
    * fix the bug in nonblocking APIs when requests are resorted into
      nonoverlapping groups and the first group of file types are interleaved,
      it was mistakenly identified as non-interleaved. See r1758.
    * fix the setting for PNETCDF_RELEASE_DATE in configure.in. Using read
      command together with IFS does not work as expected in bash 4.3.11. See
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2014-July/001586.html
      for further detailed discussion.

-------------------------------------
Version 1.5.0 (July 8, 2014)
-------------------------------------

  o New features
    * A new configure option "--disable-in-place-swap" is added. It disables
      the byte-swap operations running in-place on the user's write buffers.
      The purpose of providing this option is to deal with the problem when a
      Fortran program uses a immutable buffer for put APIs, e.g. the buffer is
      declared as a PARAMETER, and in-place byte swap on this buffer causes
      segmentation fault. See discussion threads of
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2013-July/001498.html

      Impacts:
      1. It takes effect only on Little Endian machines.
      2. It only affects put/iput data APIs, but not attribute APIs.
      3. The INTENT of buffer arguments in all Fortran 90 put/iput APIs will be
         declared as "IN". Without this setting, the default is "INOUT".
      4. It has an impact on performance, as an extra internal temporary buffer
         will be allocated to copy data over from user's put buffer, so byte
         swap can be run on the temporary buffer.

      The default setting is to enable in-place byte swap. PnetCDF tries not to
      allocate additional memory space, due to performance concern. Users are
      discouraged to use Fortran PARAMETER buffers in put APIs.

    * A new configure option "--enable-debug" is added. It enables a memory
      allocation tracing mechanism internal in PnetCDF. In addition, it enables
      the PnetCDF safe mode. (Note that setting the environment variable
      PNETCDF_SAFE_MODE at the run time can still overwrite the safe mode.)
      Default debug mode is disabled. When enabled, a user program can call
      three new APIs below: ncmpi_inq_malloc_size, ncmpi_inq_malloc_max_size,
      and ncmpi_inq_malloc_list to obtain the size in bytes of current memory
      allocated internally. This feature uses a binary tree to manage all
      malloc buffers, e.g. tsearch() and tdelete().

    * Add three Fortran parameters for PnetCDF library version numbers:
      PNETCDF_VERSION_MAJOR, PNETCDF_VERSION_MINOR, and PNETCDF_VERSION_SUB.
      Similarly in C programs, these are defined in pnetcdf.h as constants.

  o New APIs
    * C++ APIs are now available. They are developed based on netCDF-4 C++
      library. However, users are encouraged to use C APIs, instead C++.

    * ncmpi_put_att, ncmpi_get_att, nfmpi_put_att, and nfmpi_get_att - these
      APIs correspond to nc_put_att, nc_get_att, nf_put_att, and nf_get_att.
      Note they are not the "flexible" APIs. Flexible APIs have an MPI derived
      datatype argument.

    * `ncmpi__enddef`, `nfmpi__enddef`, and `nf90mpi_enddef` - these APIs
      correspond to netCDF `nc__enddef`, `nf__enddef`, and `nf90_enddef` (with
      additional optional arguments).

    * ncmpi_inq_file_info - the naming for ncmpi_get_file_info may cause
      confusion, as "get" has a different meaning on PnetCDF. The correct
      name should use "inq". However, ncmpi_get_file_info is kept for backward
      compatibility.

    * ncmpi_inq_striping, nfmpi_inq_striping, nf90mpi_inq_striping report the
      file system striping settings of the opened file: striping size and
      striping count, if the underneath MPI-IO can find their values from the
      file system in use.

    * ncmpi_inq_malloc_size, ncmpi_inq_malloc_max_size, ncmpi_inq_malloc_list
      report the size in bytes of current memory allocated internally by
      PnetCDF. Similar APIs for Fortran 77 and 90 are also available. These
      APIs are enabled only when PnetCDF is configured with option
      --enable-debug. When this option is not enabled, calling these APIs will
      return the error code NC_ENOTENABLED. These APIs are usefully for
      debugging.

    * ncmpi_inq_files_opened reports the number of files that are currently
      opened. Similar API for Fortran 77 and 90 are also available. The API
      takes 2 arguments: number of files and array of file IDs. If the second
      argument, array of IDs, is not NULL, then it will filled with the netCDF
      dataset IDs. This API is useful for debugging.

  o Syntax changes
    * For all Fortran put/iput APIs, the INTENT of write buffer arguments is
      changed to "INOUT" on Little Endian machines, if option
      "--disable-in-place-swap" is not used at configuration. Otherwise, i.e.
      on Big Endian machines or PnetCDF is configured with
      "--disable-in-place-swap" on Little Endian machines, the INTENT is "IN".

  o New PnetCDF hint
    * nc_record_align_size - aligns the starting file offset of the record
      variable section. Note this is for the entire section, not individual
      records.

  o New error code
    * NC_ENOTENABLED indicates the API is available only when the corresponding
      feature is enabled. For example, nfmpi_inq_malloc_size() returns this
      error code when "--enable-debug" is not used at configure.

    * NC_EBAD_FILE corresponds to MPI error code MPI_ERR_BAD_FILE, meaning
      "Invalid file name (e.g., path name too long)."

    * NC_ENO_SPACE corresponds to MPI error code MPI_ERR_NO_SPACE, meaning
      "Not enough space."

    * NC_EQUOTA corresponds to MPI error code MPI_ERR_QUOTA, meaning
      "Quota exceeded."

    * NC_EMULTIDEFINE_FNC_ARGS corresponds to MPI error code MPI_ERR_NOT_SAME,
      meaning "inconsistent function arguments used in collective API."

  o New run-time environment variables
    * none

  o New example programs
    * Example programs now report if there is any PnetCDF internal malloc
      residues yet to be freed, if --enable-debug option is used at
      configure.

    * Under examples/C, three examples are added: create_open.c, get_vara.c,
      and global_attributes.c. File examples/README contains their
      descriptions.

    * Under examples/CXX, several example programs corresponding to those in
      examples/C are added.

  o New programs for I/O benchmarks
    * none

  o New test program
    * Many test programs now report if there is any PnetCDF internal malloc
      residues yet to be freed, if --enable-debug option is used at configure.

    * add tests for flexible APIs. The tests borrow from nc_test/test_write.c
      that tests nc_put_var1, nc_put_vara, nc_put_vars, and nc_put_varm.
      Similarly for get APIs.

    * testcases/record.c checks if the number of records is updated correctly.
      It writes to a variable's 2nd record followed by the 1st record. A call
      to ncmpi_inq_dim() or ncmpi_inq_dimlen() should report 2 records after
      the writes complete.

    * testcases/noclobber.c checks if error code NC_EEXIST can be returned
      correctly when NC_NOCLOBBER modes is used in ncmpi_create and in the
      meantime the file exists.

    * Some test programs are developed to run in parallel. The test mechanism
      for parallel runs is command "make ptest". Two environment variables
      can be used to set the MPI run command and output file directory:
      TEST_MPIRUN and TEST_OUTDIR. Their defaults are mpiexec and "." (current
      directory), respectively. For example,
          make ptest TEST_MPIRUN="aprun -n NP" TEST_OUTDIR=/scratch
      Note the keyword "NP" will be replaced by the different numbers of
      processes used to run the tests. The testing uses up to 8 MPI processes.

    * A sample PBS script file is provided to test "make ptest" on machines
      with a batch queue system: doc/pbs.script. This example script can be
      submitted from the build root directory (where you run "make" command to
      build PnetCDF library).

    * For cross compile environment (and batch queue system), the environment
      variable TEST_SEQRUN can be used to set the MPI run command. For example,
      the command for testing sequential programs:
          make check TEST_SEQRUN="aprun -n 1" TEST_OUTDIR=/scratch

      For non-cross compile environment, there is no need to set the environment
      variables, as long as one can run the MPI executable without mpirun or
      mpiexec.

  o New optimization
    * none

  o New utility program
    * pnetcdf_version prints the version information of the PnetCDF library and
      command-line arguments used at configure

  o Other updates:
    * Revise FLASH-IO benchmark to use nonblocking APIs for both checkpoint and
      plot writes. The control variable to switch between nonblocking and
      blocking API is "use_nonblocking_io". Set it to .FALSE. in
      flash_benchmark_io.F90 to switch to blocking APIs. Using nonblocking APIs
      is now the default.

    * To match ncdump, ncmpidump now only supports one input file.

    * Makefiles are revised for better recursive make and fixed some build
      target dependency for parallel make.

    * File creation was revised for handling NC_CLOBBER and NC_NOCLOBBER modes.
      On systems where Unix calls access() and unlink() are available, they
      are used to check if a file exits and to delete an existing file.

    * subfiling is now enabled by default. Users can use --disable-subfiling to
      disable it

    * man page of ncmpigen is updated to add the description for option "-v"
      which lets users to specify the desired output file formats, e.g.
      CDF-1, CDF-2, or CDF-5.

    * flex, lex, bison, or yacc are no longer needed for building ncmpigen.

  o Bug fixes
    * fix ncmpigen.y on parsing CDL file to get the number of records. The bug
      failed the command "make b-test", due to getting a wrong number of
      records (current value for the unlimited dimension).

    * fix the update for number of records when writing to a record that is
      not the last record.

-------------------------------------
Version 1.4.1 (December 23, 2013)
-------------------------------------

  o Bug fix:
    * Improve pnetcdf.inc portability for fixed/free-form Fortran programs

  o Fortran API syntax changes
    * For nfmpi_put_att and nf90mpi_put_att family, the intent modifier for
      the put buffer arguments are now declared as INTENT(IN).
    * For nfmpi_put_var* and nf90mpi_put_var family
      + On Big Endian machines, the intent modifier for the put buffer
        arguments are now declared as INTENT(IN).
      + On Little Endian machines, the intent modifier for the put buffer
        arguments are still declared as INTENT(INOUT). This is because PnetCDF
        does in-place byte-swap on user's put buffer. If user's buffer is
        declared as parameter, then segment fault can happen when PnetCDF
        tries to byte-swap a read-only memory.

  o Subfiling
    * Subfiling is a new PnetCDF feature that divides a file transparently
      into several smaller subfiles, each of which stores subarrays
      in CDF file formats. The file name supplied by the users serves as a
      master file that contains all metadata about array partitioning
      information among the subfiles. Because data partitioning is made
      transparently from users, data accessing is kept the same through
      the conventional PnetCDF APIs and the master file.
    * To enable this feature at configure time, add configure option
      "--enable-subfiling".


-------------------------------------
Version 1.4.0 (November 17, 2013)
-------------------------------------

  o New APIs
    * Fortran 90 APIs (adopted from netcdf-fortran-4.2). All F90 APIs have
      prefix name "nf90mpi_". The APIs support function overloading.
    * get/put_varn_<type> for reading/writing a list of subrequests (each is
      specified by starts[i][ndims] and counts[i][ndims] for subrequest i.
      ndims is the number of dimension of the variable) to a single variable.
    * multiple put/get requests with explicit buffer type names:
      ncmpi_mput_var_type(), ncmpi_mput_var1_type(), ncmpi_mput_vara_type(),
      ncmpi_mput_vars_type(), ncmpi_mput_varm_type(). Similar for get APIs.
      "type" is one of the followings: text, schar, uchar, short, ushort, int,
      uint, long, float, double, longlong, or ulonglong.
    * ncmpi_inq_nreqs() reports the number of pending nonblocking requests
    * ncmpi_inq_header_size() reports the size of the file header
    * ncmpi_inq_header_extent() reports the space currently allocated for the
      file header, (also the file offset of the first variable)
    * ncmpi_inq_put_size() reports the write amount committed by far
    * ncmpi_inq_get_size() reports the read amount committed by far
    * ncmpi_sync_numrecs() a collective API that can be called in independent
      data mode to synchronize the number of records in memory across all
      processes, and update to the file if NC_SHARE is set.

  o Syntax change for Fortran put APIs
    * intent of buffer argument in all Fortran APIs is changed to inout, as
      byte-swap might be performed directly on the buffer. This change is
      for performance consideration. For example, if the buffer is declared
      as Fortran parameter, then compile will fail.

  o New PnetCDF hint
    * nc_header_read_chunk_size: PnetCDF reads the file headers in chunks. This
      hint indicates the chunk size (in bytes). The default is 256 KB.

  o New error code
    * NC_EINTOVERFLOW reports the error of 4-byte integer overflow. This
      usually happens due to MPI-IO data type constructor APIs' arguments using
      4-byte integers.
    * Error codes to report metadata defined inconsistently across processes.
        NC_EMULTIDEFINE_OMODE      - create/open mode
        NC_EMULTIDEFINE_DIM_NUM    - number of dimensions
        NC_EMULTIDEFINE_DIM_SIZE   - size of dimension
        NC_EMULTIDEFINE_DIM_NAME   - dimension names
        NC_EMULTIDEFINE_VAR_NUM    - number of variables
        NC_EMULTIDEFINE_VAR_NAME   - variable name
        NC_EMULTIDEFINE_VAR_NDIMS  - variable's number of dimensions
        NC_EMULTIDEFINE_VAR_DIMIDS - variable's dimid
        NC_EMULTIDEFINE_VAR_TYPE   - variable's data type
        NC_EMULTIDEFINE_VAR_LEN    - variable's size
        NC_EMULTIDEFINE_NUMRECS    - number of records
        NC_EMULTIDEFINE_VAR_BEGIN  - variable file begin offset
        NC_EMULTIDEFINE_ATTR_NUM   - number of attributes
        NC_EMULTIDEFINE_ATTR_SIZE  - memory space used by attribute
        NC_EMULTIDEFINE_ATTR_NAME  - attribute name
        NC_EMULTIDEFINE_ATTR_TYPE  - attribute type
        NC_EMULTIDEFINE_ATTR_LEN   - attribute length
        NC_EMULTIDEFINE_ATTR_VAL   - attribute value

  o New run-time environment variables
    * PNETCDF_SAFE_MODE environment variable can be used to enable/disable the
      internal checking for data/argument consistency across all processes (by
      calling collective MPI_Allreduce). Set it to 1 to enable the checking.
      Default is 0, i.e. disabled.
    * PNETCDF_HINTS environment variable can be used to pass the I/O hints to
      PnetCDF library. Hints include both PnetCDF and MPI-IO hints.
      PNETCDF_HINTS is a string of hints separated by ";" and each hint is in
      the form of hint=value. E.g.
          romio_ds_write=disable;nc_header_align_size=1048576
      If this environment variable is set, it overrides any values that
      were set by using calls to MPI_Info_set in the application code.

  o New example programs
    * example programs are now categorized into C, F77, and F90 directories
    * nonblocking_write.f and nonblocking_write.f90 are the Fortran version of
      nonblocking_write.c
    * put_varn_float.c for using the new APIs ncmpi_put_varn_float_all()
      put_varn_real.f and put_varn_real.f90 are the Fortran versions
    * put_varn_int.c, put_varn_int.f, and put_varn_int.f90, for using the new
      APIs ncmpi_put_varn_int_all() and nfmpi_put_varn_int_all()
    * hints.c, hints.f, and hints.f90 for using PnetCDF hints
    * flexible_api.c, flexible_api.f, and flexible_api.f90, for using blocking
      and nonblocking flexible APIs
    * mput.c for using ncmpi_mput_vara_all() to write a series of arbitrary
      start[] and count[]
    * block_cyclic.c, block_cyclic.f, and block_cyclic.f90 are for a
      *-(block-cyclic) 2D partitioning pattern
    * column_wise.c, for a *-cyclic 2D partitioning pattern
    * put_vara.c, put_vara.f, and put_var.f90 for using for
      nfmpi_put_vara_int_all()

  o New programs for I/O benchmarks
    * benchmarks/C/aggregation.c -- evaluate PnetCDF's performance on I/O
      aggregation across multiple requests with different data access patterns.
    * benchmarks/C/write_block_read_column.c -- writes variables and reads back
      using different data partitioning patterns
    * benchmarks/FLASH-IO -- I/O kernel of FLASH, a reacting hydrodynamics code
      developed at University of Chicago. This benchmark can be built
      independently from PnetCDF release.

  o New test program
    * test/F90 contains test programs adopted from netcdf-fortran-4.2
    * test/nf90_test contains test programs adopted from test/nf_test
    * testcases/alignment_test.c -- test for header and fixed variable file
      offset alignments when entering redef mode
    * testcases/nonblocking.c -- test nonblocking APIs ncmpi_iput_vara_int()
      and ncmpi_iget_vara_int()
    * testcases/flexible.c -- test flexible API ncmpi_get_vara_int_all() using
      an MPI derived data type created by MPI_Type_create_hindex()
    * test/header/header_consistency.c -- test header inconsistency and see if
      inconsistent metadata is overwritten by root's

  o New semantics for attribute APIs in data mode
    The following APIs can modify file header while being called in data mode.
    Note that these APIs can only modify existing attributes, rename variables
    or dimensions, given that the new attributes and names do not take more
    file space than the old ones. Otherwise, these APIs are prohibited in data
    mode.
      ncmpi_rename_dim(),
      ncmpi_rename_var(),
      ncmpi_copy_att(),
      ncmpi_rename_att(), and
      ncmpi_put_att_<type>()
    Starting from this release of PnetCDF, these APIs must be called
    collectively when in data mode. This new requirement is to ensure the
    file header cached in memory is consistent across all processes.

  o New synchronization for number of records
    In collective data mode, the number of records cached in memory is
    always synchronized across all processes. In independent mode, the value
    can be inconsistent, unless ncmpi_sync_numrecs() (a collective API) is
    called explicitly. Otherwise, the synchronization will have to wait until
    the call to ncmpi_end_indep_data().

    Flushing the number of records to file when it is changed used to be
    delayed until ncmpi_close() or ncmpi_sync() is called. If a strong file
    consistency is desired, users must enable NC_SHARE mode when opening the
    file. However, note that in this mode any header change will cause file
    I/O to flush  the dirty header data (not just number of records). For
    programs that do not change header frequently, enabling NC_SHARE should
    have no significant performance impact.

    This flushing behavior has been changed in 1.4.0 to the following. At the
    end of collective calls, if the number of records increases, the changed
    value will be written to the file, no matter if NC_SHARE mode is used or
    not. This change is to ensure the number of records is up-to-dated in
    file, in case the application program does not close file (due to crash
    or programming error). Note this change applies to collective APIs only.

  o New optimization: I/O request aggregation
    The original design of nonblocking I/O is to concatenate the fileviews of
    individual nonblocking requests and serve them with a single MPI-IO call,
    if possible. However, because MPI-IO requires the file displacements of
    the flattened fileview be in a monotonically nondecreasing order, the
    original approach (in v1.3.1 and prior) divides the nonblocking requests
    into groups such that each group abides by this MPI-IO fileview
    requirement. Each group is then carried out by a separate MPI-IO call.
    Performance can be poor if there are multiple groups and each group's
    aggregate access region is non-contiguous in the file.

    This revision fixes this problem by 1) sorting the starting offset of all
    nonblocking requests into a non-decreasing order; 2) dividing the requests
    into groups (two types of groups are identified: interleaving and
    non-interleaving); 3) for each non-interleaving group, concatenating
    fileviews of all requests in the group; 4) for each interleaving group,
    flattening fileviews of all requests in the group, merging the
    offset-length pairs, and concatenating them into a new integrated fileview;
    5) concatenating the fileviews of all groups into a single one; 6) the
    final combined fileview is used by a single MPI-IO call to carry out the
    requests.  Performance is expected to be improved as the number of MPI-IO
    calls is reduced to one.

    However, be warned about the additional memory requirement. The additional
    memory needed for flattening the fileviews might be more than the I/O data
    itself. For example, a request to accessing a single column of a 2D integer
    array will result in offset-length pairs, each representing only a 4-byte
    integer where the C struct in PnetCDF for storing an offset-length pair
    takes 3*sizeof(MPI_Offset)=24 bytes (offset, length, and I/O buffer
    pointer).

  o Other updates:
    * configure.in and Makefile.in have been revised to detect MPI compilers
      and other compile options automatically.
    * A new configure option "--disable-file-sync" to disable calling file
      sync. This is to be used when the underlying file system provides data
      consistency control.
    * add build recipe for IBM BGQ (e.g. Vesta/Mira/Cetus @ANL) in README.IBM
    * add build recipe for CRAY-XE6, Hopper @NERSC in README.CRAY
    * add build recipe for CRAY-XC30, Edison @NERSC in README.CRAY
    * add build recipe for Endeavour @ NASA in README.SGI
    * add declaration of flexible APIs for Fortran 90
    * "make testing" now hides most of the stdout. Use "make verbose_testing"
      for verbose output.
    * ncmpidump: add the command-line option "-k" to report the format of
      netCDF file.
    * ncvalid is renamed to ncmpivalid, a tool to validate the structure of
      netCDF files for conforming with CDF formats.
    * Fortran type NFMPI_OFFSET is removed. It was merely a shortcut to
      integer(KIND=MPI_OFFSET_KIND). Some Fortran 77 compiler does not
      recognize keyword KIND. In that case, please set MPIF77 to the MPI
      Fortran 90 compiler, e.g. ./configure MPIF77=mpif90
    * configure now automatically checks Fortran module compile flags
    * Support additional Fortran netCDF data types: nf_ubyte, nf_ushort,
      nf_uint, nf_uint64, nf90_ubyte, nf90_ushort, nf90_uint, nf90_uint64.
    * Error codes and messages conform with netCDF's

  o Bug fixes
    * Argument unlimdimidp of nfmpi_inq() returns -1 when no unlimited length
      dimension has been defined (to conform nf_inq()).
    * Argument varid of nfmpi_inq_varoffset() is fixed to be the C's varid
      plus one.
    * For collective APIs, many places have been changed to prevent program
      from hanging if a subset of the processes got errors. The fix will allow
      all processes participating the MPI collective calls in the PnetCDF, even
      if errors are detected on a subset of processes.
    * set the nonblocking request ID to NULL when the request length is zero
    * report error when bogus request IDs are passed in ncmpi_wait
    * when entering redef with different alignment hints, fixed-size
      variables' file starting offsets will only be changed when it is bigger
      than the old ones
    * Fix some Fortran API intent in/out argument declarations
    * ncmpi_def_var is fixed to detect if CDF-5 data types is used on CDF-1 or
      CDF-2 files. Error code NC_ESTRICTNC3 will return.



-------------------------------------
Version 1.3.1 (September 24, 2012)
-------------------------------------

    This release is primarily a bug-fix release, tidying up a few issues and
    incorporating some early feedback on our "buffered put" interface (see
    http://trac.mcs.anl.gov/projects/parallel-netcdf/wiki/BufferedInterface for
    more information)

    - add a new API ncmpi_inq_buffer_usage/nfmpi_inq_buffer_usage for inquiring
      the current usage of the internal buffer allocated by the "buffered"-put
      APIs.
    - bug fix to make bput APIs properly return error code NC_EINSUFFBUF.
    - bug fixes for ncmpidump to avoid residue contents from a previous read
      when it read beyond EOF.
    - bug fixes in the tutorial example codes.
    - add more in-line comments for the tutorial example codes.
    - add the error string for error code NC_ENOENT.

-------------------------------------
Version 1.3.0 (June 26, 2012)
-------------------------------------

    - Bug fixes in new ncmpidiff tool.
    - Small optimizations to reduce communication in library.
    - Improved documentation, including more test programs and a QuickTutorial.
    - Bug fixes in our Fortran 90 support.
    - Better compatibility with NetCDF-4: no need for a modified pnetcdf.h from
      Unidata when building NetCDF-4 with PnetCDF support.
    - PnetCDF now duplicates the MPI communicator internally, which fixed at
      least one odd behavior seen in a PnetCDF-using application.
    - Improvements to PnetCDF header and variable alignment (see wiki page
      VariableAlignment).
    - Add a checking for file create mode consistency across all processes and
      an error code for it, NC_ECMODE.
    - Bug fix for updating the number of record variables in nonblocking APIs.
    - Bug fix for variable starting file offsets when defining multiple large
      variables (> 232 elements) in one CDF-5 file.
    - Add a new configure option that takes environment variable $RM to replace
      the default "rm" command in all Makefiles.
    - Bug fix for nonblocking varm APIs.
    - Support for CDF-2's special2 characters in names (variables, attributes,
      dimensions, etc.)
    - Release of the official CDF-5 file format specification (see wiki page
      CD-5).
    - Support for CDF-5 data types: NC_UBYTE, NC_USHORT, NC_UINT, NC_INT64,
      NC_UINT64, and NF_INT64.
    - New C APIs: ncmpi_put_vara_ushort, ncmpi_put_vara_uint,
      ncmpi_put_vara_longlong, and ncmpi_put_vara_ulonglong. Similarly for
      var1, var, vars and varm APIs. Also for get and nonblocking APIs.
    - New Fortran APIs: nfmpi_put_vars_int8 and similarly for var1, var, vars,
      varm, get, and nonblocking APIs.
    - Add a new error code, NC_ENOTSUPPORT, for not-yet-supported features.
    - A new set of buffered put APIs (eg. ncmpi_bput_vara_float) is added (see
      BufferedInterface). They make a copy of the user's buffer internally, so
      the user's buffer can be reused when the call returns. Their usage are
      similar to the iput APIs.
    - Add new error codes for buffered put APIs: NC_ENULLBUF,
      NC_EPREVATTACHBUF, NC_ENULLABUF, NC_EPENDINGBPUT, and NC_EINSUFFBUF.
    - Add new test and example programs for the buffered put APIs.
    - The error string returned by ncmpi_strerror() for an undefined error code
      is updated from "Unknown Error" to "Unknown Error: Unrecognized PnetCDF
      error code ID" to be more descriptive.
    - Remove the use of POSIX error codes: EINVAL, ENOERR, ERANGE, and ENOMEM.
      They are replaced by NC error codes now: NC_EINVAL, NC_NOERR, NC_ERANGE,
      and NC_ENOMEM.
    - A better translation of MPI-IO error classes to netCDF/PnetCDF error
      codes. For example, MPI-IO error class MPI_ERR_NO_SUCH_FILE is translated
      to NC_ENOENT.

    - Compatibility note

    - In testing with version 11.7 of the Portland Group compiler, some of the
      Fortran test programs fail to compile if built with pgf77. The tests work
      if built with the Fortran 90 compiler.

-------------------------------------
Version 1.2.0 (August 19, 2010)
-------------------------------------

    - Several bugs fixed, memory leaks reduced,

    - Internal housekeeping

    - Reduce the places where PnetCDF will call MPI_FILE_SYNC

    - A new 'ncmpidiff' utility to compare two datasets in parallel

    - Addressed a build failure when compiling with some versions of MVAPICH2.
      See
      http://lists.mcs.anl.gov/pipermail/parallel-netcdf/2010-July/001045.html
      for an example of the build failure.

    - If you build this release and still get "undefined reference to 'yyin'",
      then you have some stale object files. "make clean" and re-build.

    - Additional error checking for collective routines

    - We have also made API changes to our non-blocking interface. Our
      non-blocking semantics have been relaxed somewhat, and should be more
      usable in more situations.


-------------------------------------
Version 1.1.0 (November 2, 2009)
-------------------------------------

    - A new file format, CDF-5, is introduced. This format allows defining
      large array variables with more than 2^32 elements.

    - A new optimization in PnetCDF with this release. If the hint
      "striping_unit" is set, then PnetCDF will align the start of non-record
      variables to a multiple of that value.

    - A new set of APIs for reading/writing multiple variables. The existing
      asynchronous APIs have also been improved to enable combination of
      multiple variable access into fewer I/O requests.

    - There is now a simple pnetcdf.F90 module for F90 codes. Please consider
      this as an "early feedback" version and not a hard guarantee of a fixed
      F90 API (though we don't expect it to change drastically if at all).
      This module is generated from the F77 'pnetcdf.inc': thanks to
      Annette Koontz for the idea.

    - The 'ncmpigen' utility can now create "big variable" (CDF-5) files, in
      addition to the older "big file" (CDF-2) files.

-------------------------------------
Version 1.0.0 (July 27, 2005)
-------------------------------------

    - The PnetCDF developers are quite happy to announce our 1.0.0 release.
      This release reflects the culmination of several years of work and lots
      of community feedback.

      A brief list of the major changes since 0.9.4:

    - Has both the high level and flexible data mode interfaces
    - Improved support for many more platforms
    - Support for all serial NetCDF access patterns (var, vara, vars,
      varm) and types (text, char, uchar, schar, short, int, float, long,
      double)
    - Synced up with netcdf-3.6.0
    - The usual array of bug fixes and improvements.