Back to top

Hyrax-1.15.0

Submitted by ndp on Wed, 09/19/2018 - 14:08

Hyrax by OPeNDAP logoThis release, Hyrax-1.15, contains new features and bug fixes

Server Documentation

1.15.0 release (24 September 2018)

Welcome to the Hyrax 1.15.0 release

New Features

CoverageJSON - For datasets that contain geo-spatial data, we now provide the option to get those data (and related metadata) encoded using the covjson format. Thanks to Corey Hemphill, Riley Rimer, and Lewis McGibbney for this contribution.

JSON-LD - With this release Hyrax adds JSON-LD content to every browser navigable catalog page ( i.e. "*/contents.html") and to every dataset/granule OPeNDAP Data Access Form. This feature along with the site map generation can be used to assist search engines like Google to catalog/index/find the data you wish the world to access.

New Data Access Form - The OPeNDAP Data Access Form is renewed. A more compact and response UI that we hope to be improving even more in the months to come.

NASA Earthdata Login (URS) Support - In addition to utilizing the mod_auth_urs Apache module to provide Earthdata Login services Hyrax now has it's own standalone implementation of the Earthdata Login client. This means that it can be deployed using only Tomcat (or other servlet Engine) and still work with Earthdata Login with out binding it to an instance of Apache httpd

MetaData Store (MDS) - A new cache has been added to the BES for Metadata Responses (aka, the MDS or MetaData Store). This cache is unlike the other BES caches in that it is intended to be operated as either a 'cache' or a 'store.' In the latter case, items added will never be removed - it is an open-ended place where metadata response objects will be kept indefinitely. The MDS contents (as a cache or a store) will persist through Hyrax restarts.

Prototype support for data stored on Amazon's S3 Web Object Store - Hyrax 1.15 has prototype support for subset-in-place of HDF5 and NetCDF4 data files that are stored on AWS S3. Data files stored in S3 must be configured for use with this feature of Hyrax. To configure files for us with this feature, a special metadata file must be built and installed in the MDS. For this feature, the MDS should be run in 'store' mode. Note: The software has a known problem with requests for very large numbers of variables in a single URL. Contact us for details. We welcome feedback on this new feature.

Improved catalog support - We have generalized the Hyrax catalog system so that it is much easier to extend. As an example, we have implemented a new module that reads information about datasets from NASA's Common Metadata Repository (CMR) and uses that to display virtual directories for NASA data holdings. This software is currently available in source form only - contact us if you would like to extend the Hyrax Catalog system for your own data collections.

Aggregation service web API - Hyrax 1.15 has support for a web API that provides a way for users to apply an operation on a group of data files or data sets. The results can be packages in netcdf3 or 4 files, as ASCII data or in tables as CSV data. The service can form a complex kind of user-specified aggregation. See the aggregation service documentation  for more information.

Additional support for CF for HDF4 and HDF5 files

HDF4

CF option - Enhance the support of handling the scale_factor and add_offset to follow the CF. The scale_factor and
add_offset rule for the MOD16A3 product is different than other MODIS products. We make an exception for
this product only to ensure the scale_factor and add_offset follow the CF conventions.

For other the new features and fixes see the section titled “What’s new for Hyrax 1.15.0” in the HDF4 handler README files: https://github.com/OPENDAP/hdf4_handler/blob/master/README


HDF5

CF option -

  1. Added support of the HDF-EOS5 Polar Stereographic(PS) and Lambert Azimuthal Equal Area(LAMAZ) grid projection files. Both projection files can be found in NASA LANCE products.
  2. Added HDF-EOS5 grid latitude and longitude cache support. This is the same as what we did for the HDF-EOS2 grid.
  3. Added support for TROP-OMI, new OMI level 2 and OMPS-NPP product.
  4. Removed the internal reserved netCDF-4 attributes for DAP output.
  5. Made the behavior of the "drop long string" BES key consistent with the current limitation of netCDF Java.

For other the new features and fixes see the section titled “What’s new for Hyrax 1.15.0” in the HDF5 handler README files:  https://github.com/OPENDAP/hdf5_handler/blob/master/README

Bug Fixes

About 40 bugs have been fixed for this release. 

Specific tickets fixed:

  • HYRAX-10 The fileout netCDF-4 doesn't generate the correct dimensions for aggregated files
  • HYRAX-247 <Dimension> elements in a constrained DMR sometimes have 'random' order
  • HYRAX-248 fileout_gdal seems to build broken JP2k files
  • HYRAX-362 Make the GeoTiff (GDAL) handler work with NCML aggregations
  • HYRAX-554 BES now includes DMR++; hack the configure script WRT libcurl
  • HYRAX-561 The fileout_netcdf, and/or the ncml_handler code does not clean up the temporary netCDF result file if the requesting client is killed during the transaction.
  • HYRAX-588 Gateway HTML Form Rendering Failure
  • HYRAX-591 Tests that create files fail 'make distcheck' (e.g., the tests for HYRAX-561)
  • HYRAX-595 The "SHOW HELP" button in the DAP2 Data Request Form points to a broken link
  • HYRAX-598 NULL pointer dereference in D4ParserSax2
  • HYRAX-599 Symbolic links to data not showing up in Hyrax 1.14.0 RPMs on CentOS 7
  • HYRAX-600 Unable to startup Hyrax installed from RPM on boot on Centos OS 7
  • HYRAX-603 The OLFS authentication code is no longer compatible with the current deployment "bootstrap".
  • HYRAX-612 Renaming the result of an aggregation (only join new?) fails.
  • HYRAX-613 OLFS installation bootstrap is broken on CentOS-7
  • HYRAX-621 Replace logo.gif with a transparent logo.png in Hyrax
  • HYRAX-623 Fix the CI build
  • HYRAX-630 The Keywords feature of libdap is hosed
  • HYRAX-645 Build issue causes make -j check to fail
  • HYRAX-646 Target collect-coverage-data doesn't work
  • HYRAX-647 The DDS print_das() method does not produce the same DAS as the DAS::print() method for Grids sometimes
  • HYRAX-648 MDS tests 61 and 62 fail on the first run of ./testsuite --conf=bes.mds.conf
  • HYRAX-670 Reading values from olfs.xml file is fragile
  • HYRAX-692 cppunit-config is no longer present in cppunit
  • HYRAX-721 The implementation of Sequence in the XSLT based Data Request Form (IFH) is broken
  • HYRAX-723 The DMR++ parser doesn't see a newline (cr?) as whitepspace.
  • HYRAX-745 Broker service needs to make correct links for data access.
  • HYRAX-755 The build_dmrpp code seems to fail on DMRs/Files with several variables.
  • HYRAX-756 get_dmrpp fails on datasets where variables are not in the root group
  • HYRAX-764 Fileout_netcdf returns empty response for dataset when no query is provided.
  • HYRAX-767 Change the Data Request Form code (all 3 versions) so that it URL encodes the query before using it.
  • HYRAX-775 The DMR response from the MDS has the xml:base attribute in a random place.
  • HYRAX-790 geogrid is failing in OLFS regression tests
  • HYRAX-791 w10n syntax collides with URI encoding rules enforced by recent Tomcat
  • HYRAX-794 Some tests regarding the enum type in the netcdf handler  fail randomly
  • HYRAX-801 ASAN Reveals 104 issues in the BES
  • HYRAX-802 Issues remain in the Aggregation rename bug
  • HYRAX-803 NcML Memory errors
  • HYRAX-804 Error in ResponseBuilderTest - a unit test
  • HYRAX-805 fileout_netcdf memory errors
  • HYRAX-818 showNode removes catalog name from the path name of the node - stop it.
  • HYRAX-833 Update baselines for BES cmdlm tests
  • HYRAX-837 libdap seems to return DAP2 and DAP3.2 DDX responses in kind of random way.
  • HYRAX-844 THREDDS catalog produced by Hyrax no longer work with Unidata code because of time zone issues
  • HYRAX-845 BESUtil::get_time() has a pointer problem
  • HYRAX-851 Memory leak in BESCatalog
  • Required External Dependencies

In order to run Hyrax 1.15.0, you will need:

  • Java 1.7 or greater
  • Tomcat 7.x or 8.x Note: On CentOS 6.6 use Java 7 + Tomcat 7 if you're using yum to install software.
  • Linux (We provide RPMs for CentOS 6.9 and 7.13; install them with yum), Ubuntu, OSX or another suitable Unix OS.

Software downloads for Hyrax 1.15.0

To run the Hyrax server, download and install the following (from source or binary):

  • OLFS (Java 1.7)
  • libdap
  • BES
  • ncWMS2 (optional)

Binaries for Hyrax 1.15.0

Docker Offical Hyrax-1.15.0 Docker Images
 
Java icon OLFS (Java-1.7)
  • OLFS 1.18.0 Web Archive File (gpg signature) Unpack using 'tar -xvf filename' and follow the instructions in the README file. (Requires Java 1.7 Built using Java 8 Tested against Tomcat 8.5.34).
  • CentOS 6.3 comes with Java 6; Oracle has stopped supporting that version of Java; update your machine if you haven't already. Use yum search java-1.7 to find the correct package. For our servlet you will only need the JRE - the Java Runtime Environment. You will also need to employ the "alternatives" tool to make java 7 JVM the preferred alternative.
  • Note: On CentOS 6.6 it's Java 7 + Tomcat 7 if you're using yum to install things. Tomcat 7 + Java 8 will not work!
Java iconncWMS2 (Java-1.7)
Use the EDAL web page to locate the latest ncWMS2 "Servlet Container" software bundle as a WAR file. Install it into the same Tomcat instance as the OLFS. The configuration instructions may be found here.
 
Linux Tux LogoBES - These are the same RPM packages as for Hyrax 1.15.0
Linux (CentOS 6.x) x86_64 RPMs - All of the RPMs we build, including the devel and debuginfo packages
  • libdap-3.20.0-1 (gpg signature)
  • bes-3.20.0-2.static (gpg signature) This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.). RPMFind is a good tool to locate those RPMs. (Updated with build 2 of the RPMs on 10/10/18. This fixes an issue where older versions of libcurl did not work with the BES)
Linux (CentOS 7.x) x86_64 RPMs - The RPM packages for CentOS 7 / RHEL 7, otherwise, the same code as above.

Installing the binary distribution

  • Download the RPM packages above
  • Use yum to install the libdap and bes RPMs (sudo yum install libdap-3.20*.rpm bes-3.20.*.rpm). Unless you're going to be building software from source for Hyrax, skip the *-devel and *-debuginfo RPMs.
  • For version 1.15.0, you must change the owner and group of /usr/share/mds and move the mds_ledger.txt file into /usr/share/mds. Do this
    • sudo chown -R bes /usr/share/mds
    • sudo chgrp -R bes /usr/share/mds
    • sudo vi /etc/bes/modules/dap.conf; then edit the line for DAP.GlobalMetadataStore.ledger
      • to read DAP.GlobalMetadataStore.ledger = /usr/share/mds/mds_ledger.txt
  • At this point you can test the BES by typing the following into a terminal:
    • start it:
        sudo service besd start
    • connect using a simple client:
        bescmdln
    • and get version information:
        show version;
    • exit from bescmdln:
        exit

BES Notes - If you are upgrading from an existing installation older than 1.13.0

  • In the bes.conf file the keys BES.CacheDir, BES.CacheSize, and BES.CachePrefix have been replaced with BES.UncompressCache.dir, BES.UncompressCache.size, and BES.UncompressCache.prefix respectively. Other changes include the gateway cache configuration (gateway.conf) which now uses the keys Gateway.Cache.dir, Gateway.Cache.size, and Gateway.Cache.prefix to configure its cache. Changing the names enabled the BES to use separate parameters for each of its several caches, which fixes the problem of 'cache collisions.'

OLFS Notes

  • Check the java version on your system: java -version
  • You need at least java 7 (aka 1.7.0), although Java 8 is better.
    • If you need to update java, the easiest way on Linux is using yum or apt-get
    • On CentOS Linux you may need to use the alternatives tool to set the Java version: alternatives --config java
  • Use yum to install tomcat.noarch. Make the directory /etc/olfs, change its group to tomcat, and set it 'group writable' (mkdir /etc/olfs; chgrp tomcat /etc/olfs; chmod g+w /etc/olfs). Alternatively, get Apache Tomcat-8.x from the Apache Software Foundation and install it wherever you'd like (e.g., /usr/local/).
  • Download the OLFS Web ARchive (war) file (see above, by the Java logo) and put opendap.war in the tomcat webapps directory (cp opendap.war /usr/share/tomcat/webaps or, if you installed tomcat from the ASF distribution, its web application directory, e.g., /usr/local/apache-tomcat-8.5.34/webapps)
  • At this point you should be able to start tomcat (sudo service tomcat start or ./apache-tomcat-8.5.34/bin/startup.sh) and test the server:
    • In a web browser, use http://localhost:8080/opendap/
    • Look at sample data files shipped with the server
  • If you are installing the OLFS in conjunction with ncWMS2 version 2.0 or higher: Copy both the opendap.war and the ncWMS2.war files into the Tomcat webapps directory. (Re)Start Tomcat. Go read about, and then configure ncWMS2 and the OLFS to work together.
  • From here, or if you are having problems, see our new Hyrax Manual and the older Hyrax documentation page
  • ATTENTION - If you are upgrading to Hyrax 1.15.0 from any previous installation read this!
    The internal format of the olfs.xml file has been revised. No previous version off this file will work with Hyrax-1.15.0. In order to upgrade your system, move your old configuration directory aside (ex: mv /etc/olfs ~/olfs-OLD) and then follow the instruction to install a new OLFS. Once you have it installed and running you will need to review your old configuration and make the appropriate changes to the new olfs.xml to restore your server's behavior. The other OLFS configuration files have not undergone any structural changes and you may simply replace the new ones that were installed with copies of your previously working ones.

Source code for Hyrax 1.15.0

Source from GitHub

  • All of our source code is on GitHub. There you will find the hyrax repo, which is a meta-project that contains scripts to clone and build all of Hyrax. You will also see all of the repos that contain the Hyrax source code (libdap4, bes, and all of the handlers).
  • Directions on building Hyrax from GitHub are available at our documentation site.