This release, Hyrax-1.16.2 , contains new features and bug fixes
Welcome to the Hyrax-1.16.2 release page (updated 27 April 2020).
- New Features
- Bug Fixes
- Required External Dependencies
- Software Binaries
- Software Source Code
- Server Documentation
Added in Hyrax-1.16.2
Hyrax In The Cloud
Hyrax is currently deployed and running as a fault-tolerant, scalable, highly available service in the NASA NGAP 2.0 production system using AWS CloudFront scripts deployed using Bamboo. The NASA/NGAP system is a subset of Amazon's AWS cloud system and the same architecture could be deployed outside of NGAP with minimal effort. Thanks to Doug Newman and the NGAP team for this work.
- Fault tolerant: The Hyrax deployment uses multiple instances and Amazon's ALB to distribute load across multiple instances. If one instance fails, others can take on the load.
- Scalable:An aAuto-scaling group will adjust to load if necessary.
- Highly available: Failing instances will be detected and replaced. Deployments will be 'blue green' and thus not cause a service outage.
- Hyrax can generate signed S3 requests when processing dmr++ files whose data content live in S3 when the correct credentials are provided (injected) into the server.
- Hyrax can use Earthdata Login to authenticate users for data access.
Hyrax Regression Tests
The Hyrax regression tests have been moved out of the OLFS and into their own project. With this change comes new capabilities:
- The regression tests can be run against any Hyrax server instance with the default (packaged) data available.
- The target Hyrax instance can be running at any endpoint URL, as long as it is specified at runtime.
- The regressiontests can be made to authenticate, if needed, by specifying a netrc file at runtime.
- These regression tests will soon become part of the Hyrax continuous integration process.
Kent Yang of the HDFGroup has been developing code to resolve problems encountered by dmr++ representations when the underlying data do not contain domain coordinate variables. Hyrax can synthesize these variables at runtime, and Kent has been applying these techniques to the dmr++ generation. Stay tuned for more on this front as we integrate the results into the automated dmr++ production.
Added in Hyrax-1.16.1
Added (alpha) support support for S3 authentication credentials
- Hyrax can access data in a protected Amazon Web Services S3 bucket using credentials provided by a pair of environment variables.
- For situations where Hyrax needs to use several sets of credentials with S3, it now supports storing credentials in a configuration file. Credential sets are associated with URL prefixes, making the configuration easy.
Improved Server Logging
- The server can now be set to add the OLFS log content to the BES log file, simplifying configuration and problem diagnosis.
NASA Earthdata Login User Authentication Support
- Hyrax support NASA's Earthdata Login system, which is based largely on the OAuth2 protocol. If you need to stage a server behind OAuth2, you may be able to use/extend/modify the implementation in Hyrax.
Data Request Form and Catalogs
- The Data Request Form now offers a configuration parameter to control if users must choose individual variables before getting data. For some sites, it makes sense to enable users to access all the data in a dataset with one click, while for other sites this is not appropriate. Now you can configure this behavior using <RequireUserSelection /> in the olfs.xml file.
- You can now disable the automatic generation of THREDDS catalog files, and have Hyrax use catalogs you provide, instead. If <NoDynamicNavigation/> is uncommented in the olfs.xml file, then all of the dynamically generated catalog/navigation pages will be disabled. The server admin must either supply and maintain THREDDS catalogs, or provide their own system of navigation and discovery to generate links to the dataset endpoints. Note that this new option does not disable the Data Request Form.
- Reduced the time to first byte for users by eliminating the unnecessary construction of metadata objects for the data response.
Added in Hyrax-1.16.0
Dataset Search Engines
Datasets served by Hyrax now provide information Google and other search engines need to make these data findable. All dataset landing pages and catalog navigation (contents.html) pages now contain embedded json-ld which crawlers such as Google Dataset Search, NSF's GeoCODES, and other data sensitive web crawlers use for indexing datasets. In order to facilitate this, certain steps can be taken by the server administrator to bring the Hyrax service to Google (and other) crawlers attention. Find more about Hyrax and JSON-LD here. Our work on JSON-LD support was funded by NSF Grant #1740704.
Serving Data From S3
Hyrax 1.16 has prototype support for subset-in-place of HDF5 and NetCDF4 data files that are stored on AWS S3. See the preliminary documentation in GitHub.
The new support includes software that can configure data already stored in S3 and still on spinning disk so that it can be served (and subset) in-place from S3 without reformatting the original data files. Support for other web object stores besides S3 has also been demonstrated.
This work on serving data from S3 was supported NASA, Raytheon, and The HDF Group.
Experimental support for STARE Indexing
We have added experimental support for STARE (Spatio Temporal Adaptive-Resolution Encoding). STARE provides a way for locations on the Earth to be denoted using a single integer number instead of the conventional Latitude and Longitude notation and provides rapid intercomparisons for finding co-located data. Our work on STARE indexing was supported by NASA Grant 17-ACCESS17-0039.
Many bugs were fixed, and a lot of effort was put into continuous integration and testing. Rather than itemizing the tickets here, if you want you can see all of the tickets we processed here.
We also worked extensively with NASA on their NGAP project. We processed 71 tickets during the release period. If you have access to NASA's JIRA you can see the details here
For Hyrax-1.16.1: The following issues have been fixed:
- HK-272 - MDS bug - LMT of data not used
- HK-361 - More performant handling of contiguous data for the DMR++ handler
- HK-376 - Have Travis add the source distribution tar balls to the S3 bucket.
- HK-411 - Fix the --baselines feature of the libdap DMRTest
- HK-413 - Persistent leaks in the libxml2-based NCML parser.
- HK-404 - Address operational and efficiency issues in the MDS
- HK-426 - Form interface bug - Structures do not work correctly - two issues
- HK-439 - bes source release
- HK-444 - Build initial version of ncdmr.cc that can read the fnoc1.nc and build a DMR.
- HK-445 - Modify the simple ncdmr.cc code so that it includes the attributes.
- HK-446 - Modify the ncdmr.cc code so that it correctly recognizes shared dimensions in fnoc1.nc and coads_climatology.nc
- HK-447 - Modify the ncdmr.cc code so that it can work with netCDF4 files that use groups.
- HK-448 - Modify the ncdmr.cc code so that it can work with netCDF4 files that contain structures.
- HK-449 - Integrate the ncdmr.cc code into the netCDF handler so that it is used for the DMR response.
- HK-454 - the dmrpp_module is unable to build a dmr++ for the test file data/dmrpp/grid_1_2d.h5
- HK-456 - Install the BES RPM package built from a PR and start the BES from that install. Check for failure.
- HK-457 - The class BESRegex is utilized in a way that is incompatible with the underlying implementation. FIX
- HK-458 - Web interface bug for Structures and Sequences
- HK-459 - When Hyrax 1.16 runs, we see some error messages "leaking out of stderr"
- HK-472 - BESInternalError Exception thrown by the NcML handler not handled properly
- HK-473 - Implement combined olfs/bes log.
- HK-474 - BES 3.20.5 memory errors
- HK-485 - Modify the CI/CD process to make the docker image
- HK-492 - Review the Travis activities for olfs, bes, and libdap
- HK-537 - Reported problem in fileout_netcdf associated with _FillValue in Ocean Color dataset
- HK-574 - Memory leak in AWSV4
For Hyrax-1.16.0: The following issues have been fixed:
- Issues and Improvements with the CovJSON response were contributed by Corey Hemphill.
- NetCDF file responses were not compressed when thy should have been. Fix by Aron Bartle at mechdyne.com.
- HDF5 handler: CF option: Fixed a small memory leak when handling the OCO2 Lite product. Fix by Kent Yang at The HDF Group
- HK-22 - The max_response_size limit is not working. Why? Fix!
- HK-23 - Fileout netCDF cannot generate a valid netCDF file when string datatype has a _FillValue
- HK-128 - FreeForm: Added regex pattern matching for format application.
- HK-311 - When running the httpd_catalog _tests_ I get intermittent errors on the first test.
- HK-327 - Add response size logging to Hyrax.
- HK-338 - In the remote THREDDS catalog presentation pages and dataset detail URL links contain spurious "/"(as "//") characters.
- HK-351 - Gmljp2 output seems empty/broken.
- HK-352 - fileout geotiff doesn't work for NCEP dataset.
- HK-354 - Rewrite the Hyrax-Guide so that the OLFS configuration section reflects current situation.
- HK-357 - Add tests for C++-11 support.
- HK-360 - Improvement of Time Aggregation when using the DMR++ software.
- HK-364 - Adopt New HDF5 library API for chunk info in the DMR++ handler.
- HK-365 - Document how to serve data from S3.
- HK-366 - Reanimate the BesCatalogCache (as BesNodeCache) but without worker threads.
- HK-369 - Fix IFH for variable names containing things like "-" or "." which breaks the java script.
- HK-372 - WCS fails to implement the needful atomic types. FIX.
- HK-375 - Create SiteMap cache file to improve response site map navigation speeds for large holdings.
- HK-387 - The httpd catalog is not showing content from the IRIS data server.
- HK-388 - DMRs built (by libdap) fail to correctly XML encode attribute values and this breaks things.
- HK-389 - fileout_netcdf not making compressed files when it should.
- HK-398 - Error found by the Google JSON-LD checker in JSON-LD added for BALTO.
- HK-403 - Memory leak in ncml_handler when accessing data from aggregated dmr++ files.
- HK-407 - Improve the dmrpp parser.
- HK-409 - Further GDAL tests: local netCDF tests.
- HK-410 - D4ParserSax2 removes newline chars from element content.
- HK-417 - Debug the httpd_catalog for IRIS on balto.o.o.
- HK-421 - Catch up on sonar cloud issues in OLFS now the that the CI scanner is working.
Required External Dependencies
In order to run Hyrax 1.16, you will need:
- Java 1.7 or greater
- Tomcat 7.x or 8.x Note: On CentOS 6.6 use Java 7 + Tomcat 7 if you're using yum to install software.
- Linux (We provide RPMs for CentOS 6.9 and 7.13; install them with yum), Ubuntu, OSX or another suitable Unix OS.
Software downloads for Hyrax 1.16.2
To run the Hyrax server, download and install the following (from source or binary):
- OLFS (Java 1.7+)
- ncWMS2 (optional)
Binaries for Hyrax 1.16.2
Docker Images (About the Docker build process)
- OLFS 1.18.7 Web Archive File (gpg signature) Unpack using 'tar -xvf filename' and follow the instructions in the README file. (Requires Java 1.7 Built using Java 8 Tested against Tomcat 8.5.34).
- OLFS Automatic robots.txt generation for 1.18.5 (gpg_signature) This archive contains a web archive file that runs in the Tomcat server's root context that returns a response for '/robots.txt' so that your site can be crawled using the automatically-built site maps added in 1.15.2. This is beta software; we'd appreciate feedback on it.
- CentOS 6.3 comes with Java 6; Oracle has stopped supporting that version of Java; update your machine if you haven't already. Use yum search java-1.7 to find the correct package. For our servlet you will only need the JRE - the Java Runtime Environment. You will also need to employ the "alternatives" tool to make java 7 JVM the preferred alternative.
- Note: On CentOS 6.6 it's Java 7 + Tomcat 7 if you're using yum to install things. Tomcat 7 + Java 8 will not work!
- ncWMS2 (Java-1.7)
- Use the EDAL web page to locate the latest ncWMS2 "Servlet Container" software bundle as a WAR file. Install it into the same Tomcat instance as the OLFS. The configuration instructions may be found here.
- Linux (CentOS 6.x) x86_64 RPMs - All of the RPMs we build, including the devel and debuginfo packages
- libdap-3.20.6-1 (gpg signature)
- bes-3.20.7-1.static (gpg signature) This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.). Note: the bes.conf file has important changes in support of JSON-LD. Make sure to look at /etc/bes/bes.conf.rpmnew after you insta/upgrade the BES with these RPMs.
- Linux (CentOS 7.x) x86_64 RPMs - The RPM packages for CentOS 7 / RHEL 7, otherwise, the same code as above.
- Snapshot** builds from the Continuous Integration and Delivery (CI/CD) system
- Currently disabled. Contact firstname.lastname@example.org or see our Docker Hub page for the latest CI/CD build of the server.
Installing the binary distribution
- Download the RPM packages found (see above) for your target operating system.
- Use yum to install the libdap and bes RPMs:
sudo yum install libdap-3.20.*.rpm bes-3.20.*.rpm
(Unless you're going to be building software from source for Hyrax, skip the *-devel and *-debuginfo RPMs.)
- Look at the /etc/bes/bes.conf.rpmnew file. Localize and merge the new BES.ServerAdministrator information into your bes.conf file. Note the format of the new BES.ServerAdministrator entries as it has changed from the previous version.
- At this point you can test the BES by typing the following into a terminal:
- start it:
sudo service besd start
- connect using a simple client:
- and get version information:
- exit from bescmdln:
- start it:
BES Notes - If you are upgrading from an existing installation older than 1.13.0
- In the bes.conf file the keys BES.CacheDir, BES.CacheSize, and BES.CachePrefix have been replaced with BES.UncompressCache.dir, BES.UncompressCache.size, and BES.UncompressCache.prefix respectively. Other changes include the gateway cache configuration (gateway.conf) which now uses the keys Gateway.Cache.dir, Gateway.Cache.size, and Gateway.Cache.prefix to configure its cache. Changing the names enabled the BES to use separate parameters for each of its several caches, which fixes the problem of 'cache collisions.'
OLFS and Starting the Server
CentOS 7, modern Ubuntu/Debian systems:
Install tomcat (sudo yum install tomcat)
- Make the directory /etc/olfs and ensure tomcat can write to it. (sudo mkdir /etc/olfs; chgrp tomcat /etc/olfs; chmod g+w /etc/olfs)
- Unpack the opendap.war web archive file from olfs-1.18.1-webapp.tgz (tar -xzf olfs-1.18.1-webapp.tgz)
- Install the opendap.war file (sudo cp opendap.war /usr/share/tomcat/webaps)
NOTE: On the current CentOS-7 default SELinux rules will now prohibit Tomcat from reading the war file :(
This can be remediated by issuing the following two commands as the super user:
sudo semanage fcontext -a -t tomcat_var_lib_t \ /var/lib/tomcat/webapps/opendap.war
sudo restorecon -rv /var/lib/tomcat/webapps/
- Start tomcat:
sudo service tomcat start
CentOS 6, older systems:
- Check the java version on your system: (java -version)
- You need at least java 7 (aka 1.7.0), although Java 8 is better.
- If you need to update java, the easiest way on Linux is using yum or apt-get
- On CentOS Linux you may need to use the alternatives tool to set the Java version: alternatives --config java
- Use yum to install tomcat.noarch. (sudo yum install tomcat)
- On CentOS 6, you will need to first install the EPEL info for yum. To do that...
- yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm (See https://fedoraproject.org/wiki/EPEL for more info)
- On CentOS 6, you may need to open a port in the iptables-managed firewall. To do that...
- Example, open port 8080: sudo iptables -I 1 -i eth0 -p tcp --dport 8080 -j ACCEPT
- Now follow the CentOS 7 steps (but skip the SELinux bits as they probably do not apply).
Test the server:
- Test the server:
- In a web browser, use http://localhost:8080/opendap/
- Look at sample data files shipped with the server
- If you are installing the OLFS in conjunction with ncWMS2 version 2.0 or higher: Copy both the opendap.war and the ncWMS2.war files into the Tomcat webapps directory. (Re)Start Tomcat. Go read about, and then configure ncWMS2 and the OLFS to work together.
- From here, or if you are having problems, see our new Hyrax Manual and the older Hyrax documentation page
- ATTENTION - If you are upgrading Hyrax from any previous installation older than 1.15, read this!
The internal format of the olfs.xml file has been revised. No previous version off this file will work with Hyrax-1.15. In order to upgrade your system, move your old configuration directory aside (ex: mv /etc/olfs ~/olfs-OLD) and then follow the instruction to install a new OLFS. Once you have it installed and running you will need to review your old configuration and make the appropriate changes to the new olfs.xml to restore your server's behavior. The other OLFS configuration files have not undergone any structural changes and you may simply replace the new ones that were installed with copies of your previously working ones.
- To make the server restart when the host boots, use systemctl enable besd and systemctl enable tomcat or chkconfig besd on and chkconfig tomcat on depending on specifics of your Linux distribution
Source code for Hyrax 1.16.2
- libdap4 3.20.6, gpg signature
- BES 3.20.7, gpg signature
- Collected dependencies for Hyrax 1.16.2 (gpg signature) - This bundles the NetCDF, HDF4, HDF5, and other libraries that the Hyrax handlers require.
- OLFS 1.18.6 (requires Java 1.7)
Source from GitHub
- All of our source code is on GitHub. There you will find the hyrax repo, which is a meta-project that contains scripts to clone and build all of Hyrax. You will also see all of the repos that contain the Hyrax source code (libdap4, bes, and all of the handlers).
- Directions on building Hyrax from GitHub are available at our documentation site.
We build RPM and Debian packages whenever new code is pushed to a Hyrax project's master branch. We were making those publicly available, but have stopped for the time being. Stay tuned for updates. In the meantime, you can find the most recent build of the server in a Docker image.