Hyrax-1.17

Submitted by jimg on Thu, 01/25/2024 - 16:01

Hyrax-1.17.0 (25 Jan 2024)

New Features

Added in Hyrax-1.17.0

  • DMR++ support (direct access to data in S3) now supports much more of the HDF5 data model. We still have two gaps: support for certain Compound variables and support for some kinds of string arrays. 
  • Support for direct I/O transfers from HDF5 to NetCDF4 when using DMR++.
  • Two-level caching of remote objects.
  • Much improved support for response size limits and better error messages for users when those (configurable) limits are exceeded.
  • The bundled grid() and geogrid() server side functions now support DAP4.
  • Support for very large data objects.
  • Many bug fixes.

olfs

  • OLFS now uses Java17 (jdk17)
  • Changed maxResponseSize type from int to long.
  • Units scalar set to 2^10 (kilobytes) when the units value is junk.
  • Preformatted text for 400 errors
  • Reduced beslistener pool from 200 to 25 for NGAP deployments
  • Fixes NASA/ESDIS Earthdata Login (EDL) token handling so that redirected are not issued and session (aka cookie) management is not required of the client.

bes

  • Reworked the way we implement the BES's singletons (or more of them at least) so that they use the Meyers Singleton pattern. This includes the EffectiveUrlCache and TheBESKeys classes.
  • We have made significant advances to supporting HDF5 files in the DMR++ builder and interpreter. Our support is close to complete with only a very few features not supported. This support now includes HDF5's Varying-length string arrays. The software now also supports scalar strings in all their forms. Note that we also added support for fixed length string arrays in the DMR++ builder and interpreter.
  • This version of the BES includes a new feature, dependent on the newest version of libdap, that reduces the time needed to build NetCDF4 responses by a factor of 10 or more when the variables in the response are not spatially subset (that is, when all variables in the response are returned in their entirety). It is still possible to subset at the variable level and get this performance boost. This is the 'Direct I/O' feature added ti libdap. NOTE: To use this feature, you must rebuild DMR++ files using the version of the DMR++ builder associated with this version of the BES (or get_dmrpp tool from the hyrax-1.16.8-356 build).
  • We improved the performance of finding the effective URL for a data item when it is accessed via a series of redirect operations, the last of which is a signed AWS URL. This is a common case for data stored in S3.
  • We have added generic Memory and File caching, tailored specifically toward the cases that arise when serving data from S3 using the DMR++ system.
  • The BES now has much better support for response size and time limits. Users are warned about responses that are too large before they are built and the BES now exits gracefully when a response takes too long to build. The limits are configurable.
  • We replaced a home-grown HTTP connection pool with a scheme provided by libcurl. The two have equivalent performance, but the libcurl version is much easier to maintain and might offer room for improvement in the future.
  • C99 compatibility improvements for the GCTP code in/used-by the HDF5 handler. Thanks ti Florian Weimer for those fixes.
  • We have added a BES module that can work with S3 using the DMR++ system. This provides a data flow that is similar to the one we provide for Hyrax in the Cloud as developer for NASA, but this new module does not make use of the NASA/ESDIS CMR system to resolve 'NASA Granules' to URLs. This will enable other groups to use the DMR++ system to serve data from S3.
  • We have adopted C++-14 as aggressively as we can, resulting in fewer lines of code, better memory management and more efficient code.
  • We have moved more of the handlers to build/use DAP4 as the default response format and build DAP2 responses from those. This reduces the amount of code we have to maintain.
  • The BES can sign S3 URLs using the AWS V4 signing scheme. This uses the Credentials Manager system.
  • The bundled grid() and geogrid() server side functions now support DAP4.

libdap4

  • Added new Direct I/O support so that modules written using libdap can pass compressed data buffers read directly from disk to output files without expensive decompression and recompression operations.
  • Merged contributed fixes from Bo Anderson Fix handling of libtirpc pkg-config files with -L flags (#228)
  • Merged contributed fixes from Dan Horák <da at @danny.cz> add missing include (#227) With GCC 13 the header isn't included thru other headers any more, thus include it explicitly. Otherwise uint8_t or uint32_t type remain undefined in Vector.cc. Fixes: #226 add missing big endian baselines (#196)
  • Fixed a bug where the copy ctor for D4Maps failed to correctly set the parent Array.
  • Merged contributed fix from Orion Poplawski Add missing cstdint include for uint32_t (#219)
  • Removed support for RHEL 7 (CentOS 7)
  • Now require C++-14 to build the code.
  • Moved the functionality of is_dap4_projected() into libdap4 (#213). This had a number of consequences and there are new methods to support the feature.
  • Added support for 64-bit sized arrays. This was done by adding a set of 'size methods' that have the suffix '_ll' (for long long). These should be used in place of the old methods, which are still in the code.

Hyrax Software Downloads

Hyrax is open-source and so is available as compiled binaries and source code. We also produce Docker images of Hyrax and it's components.

Binary Packages

    Download Binaries for Enterprise Linux 8

    Install Binaries

Docker Images (About the Docker build process)

 ​Hyrax - The complete Hyrax service in a single Docker image.

 ​ Hyrax with ncWMS2 - The Hyrax service bundled with ncWMS2 in a single Docker image

 ​ besd - The BES daemon in a single Docker image, typically used with Docker compose and the olfs image.

 ​ olfs - The OLFS (and Tomcat) in a single Docker image, typically used with Docker compose and the besd image

Source Code

Continuous Integration

Required External Dependencies

In order to run Hyrax 1.17, you will need:

  • Java 11 or greater
  • Tomcat 9 or greater
  • Linux (We provide RPMs for RHEL-8; install them with dnf/yum)

Or

  • Docker and the Containerized version of the server

Binaries for Hyrax 1.17.0

Software Components

To run the Hyrax server, download and install the following (from source or binary):

  • OLFS (Java 1.11+)
  • libdap
  • BES
  • ncWMS2 (optional)

  OLFS (Java-1.8) - EDIT

  ncWMS2 (Java-1.8) (optional)

 BES

REHL-8

Our Enterprise Linux builds are released on el8 (using rocky-8)

Linux (RHEL-8) x86_64 RPMs  All of the RHEL-8 RPMs we build, including the devel and debuginfo packages

  • libdap-3.21.0-27 (gpg signature) - The libdap library RPM for this release.
     
  • bes-3.21.0-46.static (gpg signature) - This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.).

Installing the binary distribution

BES Installation

  • Download the RPM packages found (see above) for your target operating system.
  • Use yum to install the libdap and bes RPMs:
    sudo yum install libdap-3.21.*.rpm bes-3.21.*.rpm
    (Unless you're going to be building software from source for Hyrax, skip the *-devel and *-debuginfo RPMs.)
  • Look at the /etc/bes/bes.conf.rpmnew file. Localize and merge the new BES.ServerAdministrator information into your bes.conf file. Note the format of the new BES.ServerAdministrator entries as it has changed from the previous version.
  • At this point you can test the BES by typing the following into a terminal:
    • start it:
        sudo service besd start
    • connect using a simple client:
        bescmdln
    • and get version information:
        show version;
    • exit from bescmdln:
        exit

Installing the OLFS and Starting the Server

Enterprise Linux 8

In el8 the Apache Tomcat application has been removed from yum/dnf. Thus one will need to go to the Apache Tomcat site and retrieve that latest release.

Please note that Hyrax-1.17.0 was tested using Tomcat-9.0.64

  • Install Apache Tomcat (for this example it's in /usr/share/tomcat)
  • Make the directory /etc/olfs and ensure tomcat can write to it. (sudo mkdir /etc/olfs; chgrp tomcat /etc/olfs; chmod g+w /etc/olfs)
  • Install the opendap.war file (sudo cp opendap.war /usr/share/tomcat/webaps)
  • Start Tomcat (The bes should be running already)

Test the server:

Notes:

  • If you are installing the OLFS in conjunction with ncWMS2 version 2.0 or higher: Copy both the opendap.war and the ncWMS2.war files into the Tomcat webapps directory. (Re)Start Tomcat. Go read about, and then configure ncWMS2 and the OLFS to work together.
  • From here, or if you are having problems, see our new Hyrax Manual and the older Hyrax documentation page
  • ATTENTION - If you are upgrading Hyrax from any previous installation older than 1.16.5, read this!
    The internal format of the olfs.xml file has been revised. No previous version off this file will work with Hyrax-1.16.5. In order to upgrade your system, move your old configuration directory aside (ex: mv /etc/olfs ~/olfs-OLD) and then follow the instruction to install a new OLFS. Once you have it installed and running you will need to review your old configuration and make the appropriate changes to the new olfs.xml to restore your server's behavior. The other OLFS configuration files have not undergone any structural changes and you may simply replace the new ones that were installed with copies of your previously working ones.
  • To make the server restart when the host boots, use systemctl enable besd and systemctl enable tomcat or chkconfig besd on and chkconfig tomcat on depending on specifics of your Linux distribution

Source code for Hyrax 1.17.0

Source from GitHub

Snapshot builds

Snapshot builds from the Continuous Integration and Delivery (CI/CD) system are available in Docker images.

See our Docker Hub page for the latest "snapshot" CI/CD build of the server.

Tags
Hyrax
Hyrax Release
Data Server