Back to top

Hyrax-1.16

Submitted by ndp on Mon, 07/01/2019 - 15:15

Hyrax by OPeNDAP logo

This release, Hyrax-1.16.0 , contains new features and bug fixes

Welcome to the Hyrax-1.16.0 release page (updated 10 July 2019).

New Features

Added in Hyrax-1.16.0

Dataset Search Engines

Datasets served by Hyrax now provide information Google and other search engines need to make these data findable. All dataset landing pages and catalog navigation (contents.html) pages now contain embedded json-ld which crawlers such as Google Dataset Search, NSF's GeoCODES, and other data sensitive web crawlers use for indexing datasets. In order to facilitate this, certain steps can be taken by the server administrator to bring the Hyrax service to Google (and other) crawlers attention. Find more about Hyrax and JSON-LD here. Our work on JSON-LD support was funded by NSF Grant #1740704.

Serving Data From S3

Hyrax 1.16 has prototype support for subset-in-place of HDF5 and NetCDF4 data files that are stored on AWS S3. See the preliminary documentation in GitHub.

The new support includes software that can configure data already stored in S3 and still on spinning disk so that it can be served (and subset) in-place from S3 without reformatting the original data files. Support for other web object stores besides S3 has also been demonstrated.

This work on serving data from S3 was supported NASA, Raytheon, and The HDF Group.

Experimental support for STARE Indexing

We have added experimental support for STARE (Spatio Temporal Adaptive-Resolution Encoding). STARE provides a way for locations on the Earth to be denoted using a single integer number instead of the conventional Latitude and Longitude notation and provides rapid intercomparisons for finding co-located data. Our work on STAE indexing was supported by NASA Grant 17-ACCESS17-0039.

Bug Fixes

For Hyrax-1.16.0: The following issues have been fixed:

  • Issues and Improvements with the CovJSON response were contributed by Corey Hemphill.
  • NetCDF file responses were not compressed when thy should have been. Fix by Aron Bartle at mechdyne.com.
  • HDF5 handler: CF option: Fixed a small memory leak when handling the OCO2 Lite product. Fix by Kent Yang at The HDF Group
  • HK-22 - The max_response_size limit is not working. Why? Fix! 
  • HK-23 - Fileout netCDF cannot generate a valid netCDF file when string datatype has a _FillValue 
  • HK-128 - FreeForm: Added regex pattern matching for format application. 
  • HK-311 - When running the httpd_catalog _tests_ I get intermittent errors on the first test.
  • HK-327 - Add response size logging to Hyrax.
  • HK-338 - In the remote THREDDS catalog presentation pages and dataset detail URL links contain spurious "/"(as "//") characters.
  • HK-351 - Gmljp2 output seems empty/broken.
  • HK-352 - fileout geotiff doesn't work for NCEP dataset.
  • HK-354 - Rewrite the Hyrax-Guide so that the OLFS configuration section reflects current situation.
  • HK-357 - Add tests for C++-11 support.
  • HK-360 - Improvement of Time Aggregation when using the DMR++ software.
  • HK-364 - Adopt New HDF5 library API for chunk info in the DMR++ handler.
  • HK-365 - Document how to serve data from S3.
  • HK-366 - Reanimate the BesCatalogCache (as BesNodeCache) but without worker threads.
  • HK-369 - Fix IFH for variable names containing things like "-" or "." which breaks the java script.
  • HK-372 - WCS fails to implement the needful atomic types. FIX.
  • HK-375 - Create SiteMap cache file to improve response site map navigation speeds for large holdings.
  • HK-387 - The httpd catalog is not showing content from the IRIS data server.
  • HK-388 - DMRs built (by libdap) fail to correctly XML encode attribute values and this breaks things.
  • HK-389 - fileout_netcdf not making compressed files when it should.
  • HK-398 - Error found by the Google JSON-LD checker in JSON-LD added for BALTO.
  • HK-403 - Memory leak in ncml_handler when accessing data from aggregated dmr++ files.
  • HK-407 - Improve the dmrpp parser.
  • HK-409 - Further GDAL tests: local netCDF tests.
  • HK-410 - D4ParserSax2 removes newline chars from element content.
  • HK-417 - Debug the httpd_catalog for IRIS on balto.o.o.
  • HK-421 - Catch up on sonar cloud issues in OLFS now the that the CI scanner is working.

Required External Dependencies

In order to run Hyrax 1.16, you will need:

  • Java 1.7 or greater
  • Tomcat 7.x or 8.x Note: On CentOS 6.6 use Java 7 + Tomcat 7 if you're using yum to install software.
  • Linux (We provide RPMs for CentOS 6.9 and 7.13; install them with yum), Ubuntu, OSX or another suitable Unix OS.

Software downloads for Hyrax 1.16.0

To run the Hyrax server, download and install the following (from source or binary):

  • OLFS (Java 1.7+)
  • libdap
  • BES
  • ncWMS2 (optional)

Binaries for Hyrax 1.16.0

Docker Docker Images - Hyrax, Hyrax with ncWMS (Comming soon)
 
Java icon OLFS (Java-1.7)
  • OLFS 1.18.5 Web Archive File (gpg signature) Unpack using 'tar -xvf filename' and follow the instructions in the README file. (Requires Java 1.7 Built using Java 8 Tested against Tomcat 8.5.34).
  • OLFS Automatic robots.txt generation for 1.18.5 (gpg_signature) This archive contains a web archive file that runs in the Tomcat server's root context that returns a response for '/robots.txt' so that your site can be crawled using the automatically-built site maps added in 1.15.2. This is beta software; we'd appreciate feedback on it.
  • CentOS 6.3 comes with Java 6; Oracle has stopped supporting that version of Java; update your machine if you haven't already. Use yum search java-1.7 to find the correct package. For our servlet you will only need the JRE - the Java Runtime Environment. You will also need to employ the "alternatives" tool to make java 7 JVM the preferred alternative.
  • Note: On CentOS 6.6 it's Java 7 + Tomcat 7 if you're using yum to install things. Tomcat 7 + Java 8 will not work!
Java iconncWMS2 (Java-1.7)
Use the EDAL web page to locate the latest ncWMS2 "Servlet Container" software bundle as a WAR file. Install it into the same Tomcat instance as the OLFS. The configuration instructions may be found here.
 
Linux Tux LogoBES
Linux (CentOS 6.x) x86_64 RPMs - All of the RPMs we build, including the devel and debuginfo packages
  • libdap-3.20.4-1 (gpg signature)
  • bes-3.20.5-1.static (gpg signature) This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.). Note: the bes.conf file has important changes in support of JSON-LD. Make sure to look at /etc/bes/bes.conf.rpmnew after you insta/upgrade the BES with these RPMs.
Linux (CentOS 7.x) x86_64 RPMs - The RPM packages for CentOS 7 / RHEL 7, otherwise, the same code as above.
Snapshot** builds from the Continuous Integration and Delivery (CI/CD) system
Linux x86_64 RPMs for CentOS6/7 and Debian packages for Ubuntu Trusty

** Snapshot packages are built and uploaded to the S3 bucket whenever the master branch for the BES is built. This generally happens anytime a new feature or fix in made. Caveat Emptor

Installing the binary distribution

BES installation

  • Download the RPM packages found (see above) for your target operating system.
  • Use yum to install the libdap and bes RPMs:
    sudo yum install libdap-3.20.*.rpm bes-3.20.*.rpm
    (Unless you're going to be building software from source for Hyrax, skip the *-devel and *-debuginfo RPMs.)
  • Look at the /etc/bes/bes.conf.rpmnew file. Localize and merge the new BES.ServerAdministrator information into your bes.conf file. Note the format of the new BES.ServerAdministrator entries as it has changed from the previous version.
  • At this point you can test the BES by typing the following into a terminal:
    • start it:
        sudo service besd start
    • connect using a simple client:
        bescmdln
    • and get version information:
        show version;
    • exit from bescmdln:
        exit

BES Notes - If you are upgrading from an existing installation older than 1.13.0

  • In the bes.conf file the keys BES.CacheDir, BES.CacheSize, and BES.CachePrefix have been replaced with BES.UncompressCache.dir, BES.UncompressCache.size, and BES.UncompressCache.prefix respectively. Other changes include the gateway cache configuration (gateway.conf) which now uses the keys Gateway.Cache.dir, Gateway.Cache.size, and Gateway.Cache.prefix to configure its cache. Changing the names enabled the BES to use separate parameters for each of its several caches, which fixes the problem of 'cache collisions.'

OLFS and Starting the Server

CentOS 7, modern Ubuntu/Debian systems:

Install tomcat (sudo yum install tomcat)

  • Make the directory /etc/olfs and ensure tomcat can write to it. (sudo mkdir /etc/olfs; chgrp tomcat /etc/olfs; chmod g+w /etc/olfs)
  • Unpack the opendap.war web archive file from olfs-1.18.1-webapp.tgz (tar -xzf olfs-1.18.1-webapp.tgz)
  • Install the opendap.war file (sudo cp opendap.war /usr/share/tomcat/webaps)
    NOTEOn the current CentOS-7 default SELinux rules will now prohibit Tomcat from reading the war file :(
    This can be remediated by issuing the following two commands as the super user
    :
    • sudo semanage fcontext -a -t tomcat_var_lib_t /var/lib/tomcat/webapps/opendap.war
    • sudo restorecon -rv /var/lib/tomcat/webapps/
  • Start tomcat: 
    • sudo service tomcat start

CentOS 6, older systems:

  • Check the java version on your system: (java -version)
  • You need at least java 7 (aka 1.7.0), although Java 8 is better.
    • If you need to update java, the easiest way on Linux is using yum or apt-get
    • On CentOS Linux you may need to use the alternatives tool to set the Java version: alternatives --config java
  • Use yum to install tomcat.noarch. (sudo yum install tomcat
    • On CentOS 6, you will need to first install the EPEL info for yum. To do that...
    • yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm (See https://fedoraproject.org/wiki/EPEL for more info)
  • On CentOS 6, you may need to open a port in the iptables-managed firewall. To do that...
    • Example, open port 8080: sudo iptables -I 1 -i eth0 -p tcp --dport 8080 -j ACCEPT
  • Now follow the CentOS 7 steps (but skip the SELinux bits as they probably do not apply).

Test the server:

  • Test the server:
    • In a web browser, use http://localhost:8080/opendap/
    • Look at sample data files shipped with the server

Notes:

  • If you are installing the OLFS in conjunction with ncWMS2 version 2.0 or higher: Copy both the opendap.war and the ncWMS2.war files into the Tomcat webapps directory. (Re)Start Tomcat. Go read about, and then configure ncWMS2 and the OLFS to work together.
  • From here, or if you are having problems, see our new Hyrax Manual and the older Hyrax documentation page
  • ATTENTION - If you are upgrading to Hyrax 1.15 from any previous installation read this!
    The internal format of the olfs.xml file has been revised. No previous version off this file will work with Hyrax-1.15. In order to upgrade your system, move your old configuration directory aside (ex: mv /etc/olfs ~/olfs-OLD) and then follow the instruction to install a new OLFS. Once you have it installed and running you will need to review your old configuration and make the appropriate changes to the new olfs.xml to restore your server's behavior. The other OLFS configuration files have not undergone any structural changes and you may simply replace the new ones that were installed with copies of your previously working ones.
  • To make the server restart when the host boots, use systemctl enable besd and systemctl enable tomcat or chkconfig besd on and chkconfig tomcat on depending on specifics of your Linux distribution

Source code for Hyrax 1.16.0

Source from GitHub

  • All of our source code is on GitHub. There you will find the hyrax repo, which is a meta-project that contains scripts to clone and build all of Hyrax. You will also see all of the repos that contain the Hyrax source code (libdap4, bes, and all of the handlers).
  • Directions on building Hyrax from GitHub are available at our documentation site.

Snapshot builds

We build RPM and Debian packages whenever new code is pushed to a Hyrax project's master branch. Those builds available from an Amazon S3 bucket at https://s3.amazonaws.com/opendap.travis.build/. These are automatically built packages; caveat emptor. Email support@opendap.org for help.