Index of /pub/dods/DODS-3.2/3.2.1/

$Id: README,v 1.6.4.4 2001/10/11 19:29:38 edavis Exp $ README DODS, version 3.2 Table of Contents ----------------- 1. Introduction 2. DODS source distributions and their dependencies 3. Compiling and installing DODS 4. Using DODS 5. Support A. Source Code Organization B. About Data Security 1. Introduction --------------- DODS is a software framework for scientific data networking. The DODS framework simplifies all aspects of remote data access. Local data can be made accessible to remote locations regardless of local storage format by using DODS servers. Existing, familiar data analysis and visualization applications can be transformed into DODS clients (i.e., applications able to access remote DODS served data). For more information, visit the DODS web site at http://www.unidata.ucar.edu/packages/dods/ DODS is a freely available open source package. 2. DODS source distributions and their dependencies --------------------------------------------------- The DODS software has numerous components. The ones you will require depends on the data you want to serve and the applications you want to make into DODS clients. Here is a guide on the DODS source distribution components: This information is also available at http://www.unidata.ucar.edu/packages/dods/home/faq/componentsSource.html DODS-packages-<ver>.tar.gz: Third party packages used by DODS. - Results: 3rd-party libraries (i.e., libwww, librx, libz, libtcl, libtk) - Requires: no other DODS software - Required by: DODS-dap-<ver>.tar.gz and thus by all other DODS software. DODS-dap-<ver>.tar.gz: The DODS Data Access Protocol implementation. - Results: DODS core libraries (i.e., libdap++.a and libdap++-gui.a) - Requires: DODS-packages-<ver>.tar.gz - Required by: all DODS software DODS-asciival-<ver>.tar.gz: Tool for displaying DODS data in an ASCII format - Results: asciival - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz - Required by: all DODS servers DODS-www_interface-<ver>.tar.gz: Tool that provides the DODS web interface - Results: www_int - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz - Required by: all DODS servers DODS-dsp-dods-<ver>.tar.gz: The DSP server. - Results: DSP server components (i.e., dsp_dds, dsp_das, dsp_dods) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz builds but server will be incomplete without DODS-asciival-<ver>.tar.gz and DODS-www_interface-<ver>.tar.gz - Required by: none DODS-ff-dods-<ver>.tar.gz: The FreeFormND server and some FF tools. - Results: FF server components (i.e., ff_dds, ff_das, ff_dods) and FF utilities (i.e., bufform, checkvar, checkform, newform) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz builds but server will be incomplete without DODS-asciival-<ver>.tar.gz and DODS-www_interface-<ver>.tar.gz - Required by: none DODS-hdf-dods-<ver>.tar.gz: The HDF 4 and HDF-EOS server. - Results: HDf server components (i.e., hdf_dds, hdf_das, hdf_dods) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz builds but server will be incomplete without DODS-asciival-<ver>.tar.gz and DODS-www_interface-<ver>.tar.gz - Required by: none DODS-jg-dods-<ver>.tar.gz: The JGOFS server, client library, and an example client. - Results: JGOFS server components (i.e., jg_dds, jg_das, jg_dods), JGOFS client library (i.e., libjg-dods.a), and JGOFS example client (i.e., list) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz builds but server will be incomplete without DODS-asciival-<ver>.tar.gz and DODS-www_interface-<ver>.tar.gz - Required by: none DODS-nc3-dods-<ver>.tar.gz: The netCDF 3 server, client library, and an example client. - Results: netCDF server components (i.e., nc_dds, nc_das, nc_dods), netCDF client library (i.e., libnc-dods.a), and netCDF example client (i.e., dncdump) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz builds but server will be incomplete without DODS-asciival-<ver>.tar.gz and DODS-www_interface-<ver>.tar.gz - Required by: DODS-ncview-<ver>.tar.gz DODS-idl-cmdln-<ver>.tar.gz: The IDL command-line client. - Results: IDL client components (i.e., idl_dods.so and numerous .pro files) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz - Required by: none DODS-ml-cmdln-<ver>.tar.gz: The Matlab command-line client. - Results: Matlab client components (i.e., loaddods and writeval) - Requires: DODS-dap-<ver>.tar.gz and DODS-packages-<ver>.tar.gz - Required by: none DODS-ncview-<ver>.tar.gz: The ncview graphical client. - Results: dncview - Requires: DODS-dap-<ver>.tar.gz, DODS-packages-<ver>.tar.gz, and DODS-nc3-dods<ver>.tar.gz - Required by: none 3. Compiling and installing DODS -------------------------------- See the file INSTALL in this directory for directions on building and installing DODS software. 4. Using DODS ------------- The best place to look for guidance on how to use DODS is the DODS User Guide which is online at http://www.unidata.ucar.edu/packages/dods/user/guide-html/ Also, several of the DODS source distributions contain USING files that give information specific to using the software in that distribution. See the next section for how to get support for DODS software. 5. Support ---------- Our goal is to support DODS to the best of our abilities. We think the best way to do this is to encourage community-based support. If DODS users will try to follow the support guidelines below, we can all work to help each other: 1) Consult our online documentation http://www.unidata.ucar.edu/packages/dods/home/docs.html - DODS User Guide http://www.unidata.ucar.edu/packages/dods/user/guide-html/ - DODS Quick Start Guide http://www.unidata.ucar.edu/packages/dods/user/quick-html/ - DODS Server Installation Guide http://www.unidata.ucar.edu/packages/dods/user/install-html/ 2) Consult our Frequently Asked Questions list http://www.unidata.ucar.edu/packages/dods/home/faq/ 3) Consult the searchable DODS support archive http://www.unidata.ucar.edu/glimpsedocs/ghdods.html and the searchable DODS email list archive http://www.unidata.ucar.edu/glimpsedocs/ghdods-list.html 4) Post your question to the dods@unidata.ucar.edu email list for the DODS user community to answer. Often, a knowledgable DODS user will answer your question. OR Send your question to support@unidata.ucar.edu (Unidata support). Your question will be addressed by the appropriate DODS developer. We try to respond to your email within a few hours of recieving it, either with an answer, a request for more information, or a note letting you know we are looking into the issue. With either of these routes, your email (or the response) will be archived and searchable on the web at the URL listed above. Please consider in advance how to make your question intelligible to an audience unfamiliar with your work. A few things that may help: - include information on the operating system that you are running on (e.g., paste the output of the Unix command "uname -a" into your message) - include information on which DODS component (and the version) you are using - If you post a question and recieve an answer in private emails, please post a brief summary of the solution to dods@unidata.ucar.edu (or support@unidata.ucar.edu) so that others can see the solution to the question you raised and so that the email archive contains the solution to your question. Also, we'd like to express our thanks to the DODS community for all the helpful suggestions, bug reports, and patches we have received. With your help, DODS will continue to improve. A. Source Code Organization --------------------------- This README file was found in the top-level of the DODS file hierarchy; this directory contains the following directories: bin: Empty until some of the DODS software is built. etc: Each of the servers is initially installed in the etc/ directory. Use the installServers script (also here) to move the server(s) to your web daemon's cgi-bin directory. This directory also contains various support files used during the build of the DODS source code. It also contains templates for the autoconf configuration and make init files. include: Header files, both for the DODS libraries. lib: Once built, the various DODS libraries are put here. These libraries include both the DODS core software which is common to all the servers and reimplemented APIs and the two sample DODS-IZED APIs (netcdf and jgofs). src: All the DODS source code lives here. Within src there are the following: dap-<ver>: An implementation of the DODS Data Access Protocol. This is used by all the clients and servers. packages-<ver>: This directory holds various third-party software packages used by DODS. At the present time only packages used by the core software are contained in this directory. That is, you must supply a copy of the HDF library to build the HDF server, the Matlab libraries to build the Matlab server, etc. In the future we hope to include with the servers all the software they use as long as that software is freely available. tools/asciival-<ver>: tools/www_interface-<ver>: These programs are used by the server. DODS servers are made up of a collection of programs most of which are specific to a particular data storage technology (e.g., HDF) but these two programs are common to all the servers. clients/idl-cmdln-<ver>: An IDL command line client. This client accepts DODS URLs and loads data into IDL. clients/ml-cmdln-<ver>: Like the IDL command line client, but for Matlab. Both of these clients use the DAP directly. clients/ncview-<ver>: A client made from a popular NetCDF analysis program relinked with our (re)implementation of netCDF. You can used our version of the netCDF library to relink any program which uses the netCDF API. Some examples of other programs that have been relinked are Ferret (available from their web site) and GrADS (still in testing). dsp-dods-<ver>: A server for the U of Miami's DSP image file format. nc3-dods-<ver>: A server netcdf files. This is a rewrite of our older netCDF software, it now uses the new netcdf 3.x library. jg-dods-<ver>: A server for JGOFS datasets. This server provides a way to customize the datatype of each variable in the dataset. JGOFS datasets (which are all on-line) are typically limited to either String or Floating point datatypes. Our JGOFS server provides a way to specify each variable's datatype independently. No change to the dataset is required. It is easy to configure this server for data stored in collections of text files. ff-dods-<ver>: The FreeForm server. Like the JGOFS, this server is easy to customize for `homegrown' data formats. This server also has a rich set of date and time functions which simplifies setting up catalogs of large data collections. hdf-dods-<ver>: An HDF 4 and HDF-EOS server. B. About Data Security ---------------------- There are two levels of security which DODS data servers support: domain restrictions and user restrictions. In conjunction with a World Wide Web server, access to a DODS server can be limited to a specific group of users (authenticated by password), specific machine(s) or a group of machines within a given domain or domains. NOTE DODS versions 3.2 and greater software contains significant improvements in the way password authentication is handled. Older versions of the DODS clients prompted for the password with each and every interaction between client and server. Now credentials may be embedded in URLs and are remembered and reused for the duration of a session. The security features of DODS servers depend heavily on the underlying WWW daemon because we felt this was the best way to solve the thorny problem of ensuring only authorized users accessed data. By using the daemon's authorization software we are ensuring that the security checks used by DODS have been tested by many many sites. In addition, WWW daemons already support a very full set of security features and many system administrators are comfortable and confidant with them. The tradeoff with using the web daemon's security system for our servers is that two security settings must be made for each group of data to be configured and more than one server may be needed even if provides access to the same type of data. Because the security features rely almost entirely on the host machine's WWW server, the steps required to install a secure DODS server will vary depending on the WWW server used. Thus, before installing a secure DODS server, check over your WWW server's documentation to make sure it provides the following security features: access limits to files in the document root on a per user and/or per machine basis, and the ability to place CGI scripts within the document root directory. As an alternative to the second requirement, a server may provide a way to place access limits on a CGI script not within the document root directory hierarchy. IMPORTANT Because security features are used to protect sensitive or otherwise important information, once set-up they should be tested until you are comfortable that they work. You should try accessing from at least one machine that is not allowed to access your data. If you would like, we will try to access your data, assuming that our machines are among those not allowed, to help you evaluate your set-up. Since the security features are provided by a WWW server, it is highly likely that they are functional and extensively tested. While problems with these features have shown up in the past (e.g., the Netscape SSL server bug) they are generally fixed quickly. Thus there is good reason to assume that your data are safe if you choose to set-up your DODS server as a secure one. However, *there is a chance* that a defect in the WWW server software will allow unauthorized people access; how big that chance is depends on the WWW server software you use and how extensively its security features are tested. That level of testing is completely beyond our control. It is important to distinguish securing a DODS server from securing data. If data are served using DODS then those data are also also accessible through a web browser (although it might be hard to figure out the URLs, it is still possible for the data to be accessed). So the data themselves need to be stored in directories that have limited access. If all data access will take place through a DODS server this limitation can exclude all access *except* the local machine. This is the case because some the DODS server's function requires being able to read the data through the local host's web server. For example, if the DODS server cannot read information about the dataset as DODS objects then it cannot build the INFO document. It bears repeating: If you're serving sensitive information with DODS, that information is accessible two ways, one via the DODS server and two through the WWW server. You need to make sure *both* are protected. In the past it was possible to install two or more DODS servers on a computer and assign different protections to each one. However, in practice this has proven to be very hard for to configure correctly. In many cases where this feature was used, a secure server was setup up for one group of data while an open server was set up for another. It was often the case that all the data were accessible using the open server! Thus, if you need to secure data and serve it with DODS, it is best to host all the sensitive information on one machine and put other data on a second machine with an open-access server. If you *must* run two or more servers from the same physical host, we suggest that you configure your web server to see two (or more) virtual hosts. This will provide the needed separation between the groups of data. USING A SECURE DODS SERVER Using a secure DODS sever is transparent if the server is configured to allow access based on hosts or domains. Give the DODS URL to a client; the server will respond by answering the request if allowed or with an error message if access is not allowed. Accessing a server which requires password authentication is a little different and varies depending on the type of client being used. All DODS clients support passing the authentication information along with the URL. To do this add `<username>:<password>@' before the machine name in a URL. For example, suppose I have a secure server set up on `www.dods.org' and the user `guest' is allowed access with the password `demo'. A URL for that server would start out: http://guest:demo@www.dods.org/... For example, http://guest:demo@www.dods.org/secure/nph-dods/sdata/nc/fnoc1.nc.info will return the info on the data set fnoc1.nc from a secure server. You cannot access the data without including the username and password `guest' and `demo'. Some clients will pop up a dialog box and prompt for the username and password. Netscape, and some other web browsers, for example, will do this. Similarly, some DODS clients (e.g., the geturl test client and some clients that use our NetCDF interface) will also popup a dialog. For DODS clients, the general rule is that if the client pops up progress information it will also prompt for authentication credentials. CONFIGURING A SERVER In the following I'll use the Apache 1.3.12 server as an example and describe how to install a server which limits access to a set of users. While this example is limited to the Apache server, it should be simple to perform the equivalent steps for any other WWW server that supports the set of required security features (See ABOUT DATA SECURITY). I. Create a directory for the server. To limit access to a dataset to particular machine, begin by creating a special directory for the server. This maybe either an additional CGI bin directory or a directory within the web server's document root. In this example, I chose the latter. cd /home/httpd/html/ mkdir secure II. Establish access limitations for that directory. Establish the access limitations for this directory. For the Apache server, this is done either by adding lines to the server's httpd.conf file or by using a per-directory file. Note: The use of per-directory access limit files is a configurable feature of the Apache server; look in the server's httpd.conf file for the value of the AccessFileName resource. I modified Apache's httpd.conf file so that it contains the following: # Only valid users can use the server in secure. 7/6/2000 jhrg <Directory /home/httpd/html/secure> Options ExecCGI Indexes FollowSymLinks order deny,allow deny from all allow from dcz dcz.dods.org AuthType Basic AuthUserFile /etc/httpd/conf/htpasswd.users AuthGroupFile /etc/httpd/conf/htpasswd.groups AuthName /secure require valid-user satisfy any </Directory> # Protect the directory used to hold the secure data. <Directory /home/httpd/html/sdata> Options Indexes order deny,allow deny from all allow from dcz dcz.dods.org AuthType Basic AuthUserFile /etc/httpd/conf/htpasswd.users AuthGroupFile /etc/httpd/conf/htpasswd.groups AuthName /sdata require valid-user satisfy any </Directory> and ScriptAlias /secure/ "/home/httpd/html/secure/" The first group of lines establishes the options allowed for the `secure' directory, including that it can contain CGI programs. The lines following that establish that only users in the Apache password file can access the contents of the directory. The second group of lines secure the data itself from accesses which bypass the DODS server. The ScriptAlias line tells Apache that executable files in the directory are CGIs. You can also do this by renaming the nph-dods script to nph-dods.cgi and making sure httpd.conf contains the line: AddHandler cgi-script .cgi III. Copy the server into the new directory. Copy the CGI dispatch program and the server filter programs in to the newly created directory. Use the `installServers' script for this. The script is available in the etc directory of our source distributions and is also bundled with our binary distributions. Note that if you're using the extension `.cgi' to tell Apache that nph-dods is a CGI you must rename nph-dods to nph-dods.cgi. If you forget to do that then you will get a Not Found (404) error from the server and debugging information generated by the DODS server won't appear in Apache's error_log even if it has been turned on. You are done. IV. Tips. Here are some tips on setting up secure servers: Using the per-directory limit files makes changing limits easier since the server reads those every time it accesses the directory, while changes made to the httpd.conf file are not read until the server is restarted or sent the HUP signal. Using httpd.conf for your security configuration seems a bit simpler since all the information is in one place. Using the installServers script it is easy to set up a suite of DODS servers and then use symbolic links (make sure to turn FollowSymLinks on in httpd.conf) to `mount' datasets under those servers. When doing this look for `loops' in Apache's DocumentRoot that will allow users to circumvent your security by accessing data using a different path. If the protections are set up so that it is impossible for the server host to access the data and/or the DODS server itself, then an infinite loop can result. This can be frustrating to debug, but if you see that accesses generate an endless series of entries in the access_log file, it is likely that is the problem. Make sure that you have `allow from <server host name>' set for both the directory that holds the DODS server and that holds the data. Also make sure that the server's name is set to the full name of the host. Configuring a secure DODS server can be frustrating if you're testing the server using a web browser that remembers passwords. You can turn this feature off in some browers. Also, the geturl tool supplied with DODS can be useful to test the server since it will not remember passwords between runs.