This incremental release, Hyrax-1.15.4, contains new features and bug fixes
Welcome to the Hyrax-1.15.4 release page (25 March 2019).
- New Features
- Bug Fixes
- Required External Dependencies
- Software Binaries
- Software Source Code
- Server Documentation
Added in Hyrax-1.15.4
- Added in mempory LRU cache for BES showNode responses. (HK-366)
- Added a site map cache file so that site maps for large instances of Hyrax(>500k datasets) can be navigated in a few seconds and not hours. (HK-375)
- Updated Docker containers so that Tomcat is producing compressed responses when the requesting client indicates that this is acceptable. (HK-329)
Response Size Logging
- Added response size logging to all metadata and data responses. Error responses do not have ssociated response size values in the log.
Improved GeoTIFF and Jpeg2000 responses
- The improvement creates byte valued geoTIFF files which can be opened with common tools like Preview (on OS-X) the floating point valued geoTIFF files can still be produced by changing a configuration parameter. (HK-263, HK-351, HK-352)
AIRS version 6 support for HDF4
- Enabling the CF option in the hdf4_handler (H4.EnableCF=true) now creates a usable mapping between AIRS version 6 HDF-EOS Grid/Swath attributes into the DAP2 data model.
Added previously to 1.15.x
THREDDS Catalogs - Added a new feature that allows Hyrax to "broker" remote data accessible via THREDDS catalogs. If the static THREDDS catalogs served by Hyrax (via the OLFS) contain catalogRef values that point to catalogs outside of the server, the server will provide navigational and data access brokering services for them. HK-144 , HK-264
NcML Handler - Added the ability to rename named dimensions. HK-247
Configuration Management - The server now installs a site.conf.proto file alongside the bes.conf file. The site.conf.proto file can be copied to site.conf and edited to create a localized BES configuration that persists across updates. HK-306
Data Request Form interface and Support Links - Modified the Data Request Form interface pages so that the user is now required to make a selection before downloading data. Previously clicking a download button with no selection resulted in the user receiving the entire dataset. Entire dataset retrieval can still can still be accomplished by utilizing a command line tool such as getdap or curl, or by manipulating the request URL in a browser. (HK-337) Decoupled support email configuration from site/server administrator configuration information. Reworked mailto:support links so that the generated emails contain context sensitive service information. Until now acquiring this information required an additional interchange with the user seeking support. The generated email can easily be edited by the user to remove this information if they desire. (HK-149, HK-217)
HDF5 Handler - The handler is updated to successfully compile and test with both HDF5 1.8.21 and HDF5 1.10.4.
Site Maps - Automated generation of site maps for use with Dataset crawlers such as Google Dataset Search and EarthCube's Project 418. When enabled, the site maps are accessible at the URL http://<host>/opendap/siteMap/ (In order to enable this in Hyrax 1.15.2 one must edit the web.xml for the opendap web application. This will be enabled by default in 1.15.3) There is also a new web application that provides an automatically built robots.txt file that will reference the new site maps and is useful if you are running Hyrax behind Tomcat and cannot easily put a document in / (the root document directory). Added in version 1.15.2.
A BES catalog for Apache httpd directory pages - an Alpha version of a module that enables the BES to use data made accessible by Apache httpd's directory pages. This is part of the BALTO broker project. The module is available only to developers (i.e., with the source build) at this point. Added in version 1.15.1.
Prototype support for data stored on Amazon's S3 Web Object Store - Hyrax 1.15 has prototype support for subset-in-place of HDF5 and NetCDF4 data files that are stored on AWS S3. Data files stored in S3 must be configured for use with this feature of Hyrax. To configure files for us with this feature, a special metadata file must be built and installed in the MDS. For this feature, the MDS should be run in 'store' mode. Note: The software has a known problem with requests for very large numbers of variables in a single URL. Contact us for details. We welcome feedback on this new feature.
CoverageJSON - For datasets that contain geo-spatial data, we now provide the option to get those data (and related metadata) encoded using the covjson format. Thanks to Corey Hemphill, Riley Rimer, and Lewis McGibbney for this contribution.
JSON-LD - With this release Hyrax adds JSON-LD content to every browser navigable catalog page ( i.e. "*/contents.html") and to every dataset/granule OPeNDAP Data Access Form. This feature along with the site map generation can be used to assist search engines like Google to catalog/index/find the data you wish the world to access.
New Data Access Form - The OPeNDAP Data Access Form is renewed. A more compact and response UI that we hope to be improving even more in the months to come.
NASA Earthdata Login (URS) Support - In addition to utilizing the mod_auth_urs Apache module to provide Earthdata Login services Hyrax now has it's own standalone implementation of the Earthdata Login client. This means that it can be deployed using only Tomcat (or other servlet Engine) and still work with Earthdata Login with out binding it to an instance of Apache httpd
MetaData Store (MDS) - A new cache has been added to the BES for Metadata Responses (aka, the MDS or MetaData Store). This cache is unlike the other BES caches in that it is intended to be operated as either a 'cache' or a 'store.' In the latter case, items added will never be removed - it is an open-ended place where metadata response objects will be kept indefinitely. The MDS contents (as a cache or a store) will persist through Hyrax restarts.
Improved catalog support - We have generalized the Hyrax catalog system so that it is much easier to extend. As an example, we have implemented a new module that reads information about datasets from NASA's Common Metadata Repository (CMR) and uses that to display virtual directories for NASA data holdings. This software is currently available in source form only - contact us if you would like to extend the Hyrax Catalog system for your own data collections.
Aggregation service web API - Hyrax 1.15 has support for a web API that provides a way for users to apply an operation on a group of data files or data sets. The results can be packages in netcdf3 or 4 files, as ASCII data or in tables as CSV data. The service can form a complex kind of user-specified aggregation. See the aggregation service documentation for more information.
Additional support for CF for HDF4 and HDF5 files
CF option - Enhance the support of handling the scale_factor and add_offset to follow the CF. The scale_factor and
add_offset rule for the MOD16A3 product is different than other MODIS products. We make an exception for
this product only to ensure the scale_factor and add_offset follow the CF conventions.
For other the new features and fixes see the section titled “What’s new for Hyrax 1.15.0” in the HDF4 handler README files: https://github.com/OPENDAP/hdf4_handler/blob/master/README
Graceful error handling for DAP4 DMR response when the dimension name is not in the dimension list.
CF option -
- Added support of the HDF-EOS5 Polar Stereographic(PS) and Lambert Azimuthal Equal Area(LAMAZ) grid projection files. Both projection files can be found in NASA LANCE products.
- Added HDF-EOS5 grid latitude and longitude cache support. This is the same as what we did for the HDF-EOS2 grid.
- Added support for TROP-OMI, new OMI level 2 and OMPS-NPP product.
- Removed the internal reserved netCDF-4 attributes for DAP output.
- Made the behavior of the "drop long string" BES key consistent with the current limitation of netCDF Java.
For other the new features and fixes see the section titled “What’s new for Hyrax 1.15” in the HDF5 handler README files: https://github.com/OPENDAP/hdf5_handler/blob/master/README
For Hyrax-1.15.4: The following issues have been fixed:
- HK-22 The maximum response size limit (set in the olfs.xml file) now works as intended.
- HK-372 Added missing code to support all DAP2 and DAP4 numeric atomic types in WCS-2.0
For Hyrax 1.15.3: The follow issues have been fixed
- Corrected the CovJson response Content-Type headers
- Repaired incompatibility between Data Request Forms and CentOS-6
- HK-14 CMR/Default BES Catalog bug
- HK-34 Memory leaks in the ncml module
- HK-40 Renaming of shared dimensions doesn't work (and should) - Could be duplicate of renaming a variable
- HK-54 Renaming dimensions in NCML handler fails
- HK-71 The HTML form help url is broken
- HK-86 Sometimes in an NcML modified dataset the DDX response contains 2 copies (one modified, one not) of each variable
- HK-88 The asan compiler finds issues in netcdf_handler
- HK-95 Fix the NCML parser error messages to not have awful XML entities in them!
- HK-96 Issues with ICESat-2 data.
- HK-204 NCML handler conceptual mismatch when aggregating only some of the variables in a set of files
- HK-266 Problems with port numbers in Load Balancer configuration.
- HK-270 CentOS 6 yum installs Tomcat 6 by default
- HK-271 libdap Debian package does not install on a plain Ubuntu 14 VM.
- HK-274 MDS does not support the ancillary DAS
- HK-275 bes cmdln tests fail on master for the Travis build
- HK-285 The BES Centos 7 RPM as packaged does not support `systemctl enable bes`
- HK-296 Building the Hyrax + ncWMS Docker container results in a container that doesn't work for 1.15.2
- HK-298 Repair misclassification of error in BESStreamResponseHandler
- HK-299 Unchecked return value from opendir() causes segmentation fault.
- HK-307 Vector attribute values are not separated by a comma (or anything) in the new form interface
- HK-312 The bes.conf.in file in _bes/modules/httpd_catalog/tests_ Should be cleaned up
- HK-313 The virtual catalogs (e.g., CMR or httpd_catalog) crash when a path does not end in slash in one (rare) case.
- HK-314 Sticky state for containers (?) when switching back and forth between the default and virtual file systems
- HK-319 When the caching software is used by code that access remote resources, empty files are left in the file system when exceptions are thrown
- HK-320 libcurl fails when called from the bes daemon on OSX, but not on CentOS 6 or 7
- HK-323 Hyrax siteMap generation adds an extraneous ""opendap"" to the map urls.
- HK-325 I think the httpd_catalog module, or maybe the HttpdCatalogContainer, is not setting the container type field
- HK-340 Viewers servlet fails to correctly handle query string URL decode task.
For Hyrax 1.15.2: The follow issues have been fixed
- HK-39 Ncml_handler fails when ascii data (or a server function result) is requested for renamed variables.
- HK-247 Add a rename dimension feature to the ncml_handler.
- HK-40 Renaming of shared dimensions doesn't work (and should) - Could be duplicate of renaming a variable.
- HK-287 We need to have the BES site map command return the site map as a result.
- HK-146 Make JSON-LD localization configuration for OLFS.
For Hyrax 1.15.1: The follow issues have been fixed
- HK-277 hyrax-dependencies on the web site are out of date
- HK-269 The current BES RPM package installs with MDS unwritable
- HK-258 Modify roi() server-side function so that it is more flexible in the kinds of arguments it can use
- HK-215, HK-246 Support subsetting a 3D+ variable using the roi() function
- HK-268, HK-108, HK-257, HK-136, HK-224, HK-249, HK-152, HK-222 Implement Continuous Delivery for Hyrax binary packages
- HK-214, Add Debian package build to libdap and BES
- HK-103, Build an initial version of the BALTO broker for two different data source types: Hyrax and Files accessible using HTTP
- HK-22 The max_response_size limit is not working.
- HK-1, HK-2, HK-3, HK-10 Data Transfer Errors in DMR++
Additional Specific tickets fixed in Hyrax-1.15.x:
- HYRAX-10 The fileout netCDF-4 doesn't generate the correct dimensions for aggregated files
- HYRAX-247 <Dimension> elements in a constrained DMR sometimes have 'random' order
- HYRAX-248 fileout_gdal seems to build broken JP2k files
- HYRAX-362 Make the GeoTiff (GDAL) handler work with NCML aggregations
- HYRAX-554 BES now includes DMR++; hack the configure script WRT libcurl
- HYRAX-561 The fileout_netcdf, and/or the ncml_handler code does not clean up the temporary netCDF result file if the requesting client is killed during the transaction.
- HYRAX-588 Gateway HTML Form Rendering Failure
- HYRAX-591 Tests that create files fail 'make distcheck' (e.g., the tests for HYRAX-561)
- HYRAX-595 The "SHOW HELP" button in the DAP2 Data Request Form points to a broken link
- HYRAX-598 NULL pointer dereference in D4ParserSax2
- HYRAX-599 Symbolic links to data not showing up in Hyrax 1.14.0 RPMs on CentOS 7
- HYRAX-600 Unable to startup Hyrax installed from RPM on boot on Centos OS 7
- HYRAX-603 The OLFS authentication code is no longer compatible with the current deployment "bootstrap".
- HYRAX-612 Renaming the result of an aggregation (only join new?) fails.
- HYRAX-613 OLFS installation bootstrap is broken on CentOS-7
- HYRAX-621 Replace logo.gif with a transparent logo.png in Hyrax
- HYRAX-623 Fix the CI build
- HYRAX-630 The Keywords feature of libdap is hosed
- HYRAX-645 Build issue causes make -j check to fail
- HYRAX-646 Target collect-coverage-data doesn't work
- HYRAX-647 The DDS print_das() method does not produce the same DAS as the DAS::print() method for Grids sometimes
- HYRAX-648 MDS tests 61 and 62 fail on the first run of ./testsuite --conf=bes.mds.conf
- HYRAX-670 Reading values from olfs.xml file is fragile
- HYRAX-692 cppunit-config is no longer present in cppunit
- HYRAX-721 The implementation of Sequence in the XSLT based Data Request Form (IFH) is broken
- HYRAX-723 The DMR++ parser doesn't see a newline (cr?) as whitepspace.
- HYRAX-745 Broker service needs to make correct links for data access.
- HYRAX-755 The build_dmrpp code seems to fail on DMRs/Files with several variables.
- HYRAX-756 get_dmrpp fails on datasets where variables are not in the root group
- HYRAX-764 Fileout_netcdf returns empty response for dataset when no query is provided.
- HYRAX-767 Change the Data Request Form code (all 3 versions) so that it URL encodes the query before using it.
- HYRAX-775 The DMR response from the MDS has the xml:base attribute in a random place.
- HYRAX-790 geogrid is failing in OLFS regression tests
- HYRAX-791 w10n syntax collides with URI encoding rules enforced by recent Tomcat
- HYRAX-794 Some tests regarding the enum type in the netcdf handler fail randomly
- HYRAX-801 ASAN Reveals 104 issues in the BES
- HYRAX-802 Issues remain in the Aggregation rename bug
- HYRAX-803 NcML Memory errors
- HYRAX-804 Error in ResponseBuilderTest - a unit test
- HYRAX-805 fileout_netcdf memory errors
- HYRAX-818 showNode removes catalog name from the path name of the node - stop it.
- HYRAX-833 Update baselines for BES cmdlm tests
- HYRAX-837 libdap seems to return DAP2 and DAP3.2 DDX responses in kind of random way.
- HYRAX-844 THREDDS catalog produced by Hyrax no longer work with Unidata code because of time zone issues
- HYRAX-845 BESUtil::get_time() has a pointer problem
- HYRAX-851 Memory leak in BESCatalog
Required External Dependencies
In order to run Hyrax 1.15.4, you will need:
- Java 1.7 or greater
- Tomcat 7.x or 8.x Note: On CentOS 6.6 use Java 7 + Tomcat 7 if you're using yum to install software.
- Linux (We provide RPMs for CentOS 6.9 and 7.13; install them with yum), Ubuntu, OSX or another suitable Unix OS.
Software downloads for Hyrax 1.15.4
To run the Hyrax server, download and install the following (from source or binary):
- OLFS (Java 1.7+)
- ncWMS2 (optional)
Binaries for Hyrax 1.15.4
- Docker Images -
Hyrax, Hyrax with ncWMS(Comming soon)
- OLFS (Java-1.7)
- OLFS 1.18.4 Web Archive File (gpg signature) Unpack using 'tar -xvf filename' and follow the instructions in the README file. (Requires Java 1.7 Built using Java 8 Tested against Tomcat 8.5.34).
- OLFS Automatic robots.txt generation for 1.18.4 (gpg_signature) This archive contains a web archive file that runs in the Tomcat server's root context that returns a response for '/robots.txt' so that your site can be crawled using the automatically-built site maps added in 1.15.2. This is beta software; we'd appreciate feedback on it.
- CentOS 6.3 comes with Java 6; Oracle has stopped supporting that version of Java; update your machine if you haven't already. Use yum search java-1.7 to find the correct package. For our servlet you will only need the JRE - the Java Runtime Environment. You will also need to employ the "alternatives" tool to make java 7 JVM the preferred alternative.
- Note: On CentOS 6.6 it's Java 7 + Tomcat 7 if you're using yum to install things. Tomcat 7 + Java 8 will not work!
- ncWMS2 (Java-1.7)
- Use the EDAL web page to locate the latest ncWMS2 "Servlet Container" software bundle as a WAR file. Install it into the same Tomcat instance as the OLFS. The configuration instructions may be found here.
- Linux (CentOS 6.x) x86_64 RPMs - All of the RPMs we build, including the devel and debuginfo packages
- libdap-3.20.3-1 (gpg signature)
- bes-3.20.4-1.static (gpg signature) This RPM includes statically linked copies of all of the modules/handlers we support, including HDF4 & 5 with HDFEOS support. There is no need to install packages from EPEL with this RPM. Other sources of RPM packages will likely provide a bes RPM that uses handlers linked (dynamically) to dependencies from their distributions (CentOS, Fedora, etc.). Note: the bes.conf file has important changes in support of JSON-LD. Make sure to look at /etc/bes/bes.conf.rpmnew after you insta/upgrade the BES with these RPMs.
- Linux (CentOS 7.x) x86_64 RPMs - The RPM packages for CentOS 7 / RHEL 7, otherwise, the same code as above.
- Snapshot** builds from the Continuous Integration and Delivery (CI/CD) system
- Linux x86_64 RPMs for CentOS6/7 and Debian packages for Ubuntu Trusty
** Snapshot packages are built and uploaded to the S3 bucket whenever the master branch for the BES is built. This generally happens anytime a new feature or fix in made. Caveat Emptor
Installing the binary distribution
- Download the RPM packages above
- Use yum to install the libdap and bes RPMs (sudo yum install libdap-3.20.*.rpm bes-3.20.*.rpm). Unless you're going to be building software from source for Hyrax, skip the *-devel and *-debuginfo RPMs.
- Look at the /etc/bes/bes.conf.rpmnew file. Localize and merge the new BES.ServerAdministrator information into your bes.conf file. Note the format of the new BES.ServerAdministrator entries as it has changed from the previous version.
- At this point you can test the BES by typing the following into a terminal:
- start it:
sudo service besd start
- connect using a simple client:
- and get version information:
- exit from bescmdln:
- start it:
BES Notes - If you are upgrading from an existing installation older than 1.13.0
- In the bes.conf file the keys BES.CacheDir, BES.CacheSize, and BES.CachePrefix have been replaced with BES.UncompressCache.dir, BES.UncompressCache.size, and BES.UncompressCache.prefix respectively. Other changes include the gateway cache configuration (gateway.conf) which now uses the keys Gateway.Cache.dir, Gateway.Cache.size, and Gateway.Cache.prefix to configure its cache. Changing the names enabled the BES to use separate parameters for each of its several caches, which fixes the problem of 'cache collisions.'
OLFS and Starting the Server
CentOS 7, modern Ubuntu/Debian systems:
Install tomcat (sudo yum install tomcat)
- Make the directory /etc/olfs and ensure tomcat can write to it. (sudo mkdir /etc/olfs; chgrp tomcat /etc/olfs; chmod g+w /etc/olfs)
- Unpack the opendap.war web archive file from olfs-1.18.1-webapp.tgz (tar -xzf olfs-1.18.1-webapp.tgz)
- Install the opendap.war file (sudo cp opendap.war /usr/share/tomcat/webaps)
NOTE: On the current CentOS-7 default SELinux rules will now prohibit Tomcat from reading the war file :(
This can be remediated by issuing the following two commands as the super user:
sudo semanage fcontext -a -t tomcat_var_lib_t /var/lib/tomcat/webapps/opendap.war
sudo restorecon -rv /var/lib/tomcat/webapps/
- Start tomcat:
sudo service tomcat start
CentOS 6, older systems:
- Check the java version on your system: (java -version)
- You need at least java 7 (aka 1.7.0), although Java 8 is better.
- If you need to update java, the easiest way on Linux is using yum or apt-get
- On CentOS Linux you may need to use the alternatives tool to set the Java version: alternatives --config java
- Use yum to install tomcat.noarch. (sudo yum install tomcat)
- On CentOS 6, you will need to first install the EPEL info for yum. To do that...
- yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm (See https://fedoraproject.org/wiki/EPEL for more info)
- On CentOS 6, you may need to open a port in the iptables-managed firewall. To do that...
- Example, open port 8080: sudo iptables -I 1 -i eth0 -p tcp --dport 8080 -j ACCEPT
- Now follow the CentOS 7 steps (but skip the SELinux bits as they probably do not apply).
Test the server:
- Test the server:
- In a web browser, use http://localhost:8080/opendap/
- Look at sample data files shipped with the server
- If you are installing the OLFS in conjunction with ncWMS2 version 2.0 or higher: Copy both the opendap.war and the ncWMS2.war files into the Tomcat webapps directory. (Re)Start Tomcat. Go read about, and then configure ncWMS2 and the OLFS to work together.
- From here, or if you are having problems, see our new Hyrax Manual and the older Hyrax documentation page
- ATTENTION - If you are upgrading to Hyrax 1.15 from any previous installation read this!
The internal format of the olfs.xml file has been revised. No previous version off this file will work with Hyrax-1.15. In order to upgrade your system, move your old configuration directory aside (ex: mv /etc/olfs ~/olfs-OLD) and then follow the instruction to install a new OLFS. Once you have it installed and running you will need to review your old configuration and make the appropriate changes to the new olfs.xml to restore your server's behavior. The other OLFS configuration files have not undergone any structural changes and you may simply replace the new ones that were installed with copies of your previously working ones.
- To make the server restart when the host boots, use systemctl enable besd and systemctl enable tomcat or chkconfig besd on and chkconfig tomcat on depending on specifics of your Linux distribution
Source code for Hyrax 1.15.4
- libdap 3.20.3, gpg signature
- bes 3.20.4, gpg signature
- Collected dependencies for Hyrax 1.15 (gpg signature) - This bundles the NetCDF, HDF4, HDF5, and other libraries that the Hyrax handlers require.
- OLFS 1.18.4 (requires Java 1.7)
Source from GitHub
- All of our source code is on GitHub. There you will find the hyrax repo, which is a meta-project that contains scripts to clone and build all of Hyrax. You will also see all of the repos that contain the Hyrax source code (libdap4, bes, and all of the handlers).
- Directions on building Hyrax from GitHub are available at our documentation site.
We build RPM and Debian packages whenever new code is pushed to a Hyrax project's master branch. Those builds available from an Amazon S3 bucket at https://s3.amazonaws.com/opendap.travis.build/. These are automatically built packages; caveat emptor. Email firstname.lastname@example.org for help.