$Id: SERVLETS,v 1.8 2002/04/09 17:51:49 ndp Exp $ ---------------------------------------------------------------------------- Release 1.0.6 04/08/2002 (ndp) Requirements: You will need a servlet engine. I developed This release of Java-DODS using "Tomcat". Tomcat: First you will need to install a web server which supports servlets (aka a servlet engine). The current "reference standard" for this release is jakarta-tomcat-3.3a from Apache. Go get it from: http://jakarta.apache.org/tomcat/ Follow the instructions and get it installed. javax.servlet: In order to compile the servlet code you will need the javax.servlet package. You can get this from Sun (http://java.sun.com/products/servlet/2.1/index.html) Or you can get it from inside the Tomcat distribution (TOMCATHOME/lib/common) Either way you will need to get the javax.servlet jar file on your CLASSPATH before you can compile the dods.servlet and dods.servers packages. To build the servlets: Once you've got the javax.servlet stuff and your CLASSPATH straightened out a call to "make" in the top level directory will build everything. To specifically build the server stuff do a "make servers" in the top level directory. Core Servlets: The servlets distributed with the core Java-DODS software are: DODS Test Server (DTS): Class name: dods.servers.test.dts This servlet is primarily used to test the core code (in particular the DAP). This servlet will take any DDS in it's DDScache (see below) and populate it with invented data per client request. This allows the testing of unusual DDS structures. DODS Relational Database Server (DRDS): Class name: dods.servers.sql.drds This DODS server serves data from a relational database to DODS clients. The DRDS uses the JDBC interface to connect to the database. This means that to use this server you will need to locate and install JDBC drivers for your database system. This requirement is for the server only. The DODS clients that use the server need know nothing about JDBC or SQL. In this release the DRDS does not support SQL JOIN operations. Each database table must appear as a DODS Sequence data type in its own DDS file. If your data crosses multiple tables then you will need to make a "view" or a "virtual table" in the database in order to serve the data with the DRDS. This situation will improve in the next major revision. (I have an as yet un-implemented plan to allow the DRDS to support SQL JOIN operations.) Installation: The easiest way to install the distibuted servlets is to get the "dods.war" file from: ftp://ftp.unidata.ucar.edu/pub/dods/DODS-Java-1.1/ Or, you can build it by doing a "make war" in the Java-DODS directory. This will make the "dods.war" file from the source code and place it in the Java-DODS/webapps directory. Put the "dods.war" file in appropriate directory of your servlet engine. For tomacat3.3a you would place it in $TOMCATHOME/webapps. Restart the servlet engine (tomcat), this will unpack the "dods.war" file into the directory $TOMCATHOME/webapps/dods Once it's unpacked you will need to configure the servlets. Go to the directory $TOMCATHOME/webapps/dods/WEB-INF and edit the file "web.xml" Instructions for it's modifcation are in the "Servlet Configuration" section below. Once you have the "web.xml" file whipped into shape, restart the servlet engine (tomcat) again. You should be up and running! If things aren't working for you, try turning on the Debug flags showRequest and showResponse. (See bleow) If you get stuck contact support@unidata.ucar.edu If they can't help you, they will find me (Nathan Potter), and I will help you get it working. Servlet Configuration: The DODSServlet relies on the javax.servlet.ServletConfig interface (in particular the getInitParameter() method) to retrieve configuration information used by the servlet. InitParameters common to all DODSServlets are: DebugOn - This controls ouput to the terminal from which the servlet engine was launched. The value is a list of flags that turn on debugging instrumentation in different parts of the code. Common values are : showRequest, showResponse Other debugging values that are specific to each servlet should be documented in each servlets javadoc documentation. Look in Java-DODS/doc. INFOcache - This is should be set to the directory containing the files used by the ".info" service for the servlet. This directory should contain any dataset specific "over-ride" files (see below), any dataset specific additional information files (see below), and any servlet specific information files(see below). DDScache - This is should be set to the directory containing the DDS files for the datasets used by the servlet. Some servlets have been developed that do not use DDS's that are cached on the disk, however the default behaviour is for the servlet to load DDS images from disk. DAScache - This is should be set to the directory containing the DAS files for the datasets used by the servlet. Some servlets have been developed that do not use DAS's that are cached on the disk, however the default behaviour is for the servlet to load DAS images from disk. Here is an example entry from the web.xml file (for tomcat3.3a) for the DODS Test Server (DTS): dts dods.servers.test.dts DebugOn showRequest showResponse INFOcache /usr/Java-DODS/sdds-testsuite/info/ DDScache /usr/Java-DODS/sdds-testsuite/dds/ DAScache /usr/Java-DODS/sdds-testsuite/das/ Other servlets have other (very important) InitParameters. The DODS Relational Database Server (DRDS) relies on JDBC to make it's connections to DBMS's. Specific InitParameters for the DRDS are: JDBCdriver - This should be set to the java CLASSPATH name of the JDBC drivers that the servlet is to use to make the JDBC connection to the DBMS. For example, in my server that is using the Merant Data Direct drivers the value is set to: "com.merant.sequelink.jdbc.SequeLinkDriver" JDBCconnectionURL - This is the connection URL (aka the connection string) that the DRDS is to use to connecto to the DBMS. This is usually defined by the devlopers of the JDBC driver. It is not always easy to ascertain for a particular installation. For example, in my server that is using the Merant Data Direct drivers the value is set to: "jdbc:sequelink://sugar.oce.orst.edu:19996" JDBCusername - This is the user name for the DBMS that the JDBC connection will be made under. This is often set to "guest". JDBCpassword - The password associated with the above username. This is stored as simple text, so make sure that the JDBC user doesn't have any significant privileges! JDBCMaxResponseLength - This limits the number of lines that the DRDS will for a given client request. For debugging I use 300. For production I use 300000. UseDatasetName - This is a (probably temporary) hack. Some databases (MS-SQL Server 7.0) require that the database name and the owner of the database be specified in every variable and table name. This is akward for the current implmentation of the DRDS. The work around is to name the dataset (in the DDS file) with the database name and owner name of the table being served. For example in one dataset that I server the database name is "EOSDB" and the owner of the database is "DBO". For this database I set the value of UseDatasetName to true and then I define the DDS as follows: Dataset { Sequence { Float64 battery; Float64 checksum; Float64 data_age; } Drifters; } EOSDB.DBO; Thus the hack is invoked. If you don't want to use this hack then DO NOT even included the InitParameter "UseDatasetName" in the web.xml entry for the DRDS. Here is an example entry from the web.xml file (for tomcat3.3a) associated with a DRDS servlet: drds dods.servers.sql.drds JDBCdriver com.merant.sequelink.jdbc.SequeLinkDriver JDBCconnectionURL jdbc:sequelink://sugar:19996 JDBCusername guest JDBCpassword JDBCMaxResponseLength 300 UseDatasetName INFOcache /usr/Java-DODS/sdds-testsuite/info/ DDScache /usr/Java-DODS/sdds-testsuite/dds/ DAScache /usr/Java-DODS/sdds-testsuite/das/ DebugOn showRequest showResponse JDBC Additional Information Files: The current implementaion of the DODSServlet provides a ".info" service in addition to the ".dds", ".das", and ".dods" services. This ".info" service provides an html document with human readable content. This document is a synthesis of the contents of the DDS and the DAS. There are provisions for adding additional information, and for overriding the default document. There are three types of files used by the ".info" service to achieve this. They are: Dataset Over-Ride files: These files live in the INFOcache directory. To override the .info service output for a particular dataset, place a file of html text containing the overide document in the INFOcache directory. The file should be a complete HTML document that can be rendered by most browsers. The file should have the same name as the dataset with an additonal ".ovr" suffix. For example, is my dataset is called "test.22" then the over-ride file is called "test.22.ovr" Dataset Additional Information Files: These files live in the INFOcache directory. If you want to supplement the ".info" service output provided by the servlet, place a file of html text containing the additional information in the INFOcache directory. This html text should be a snippet of html, not a complete document. It will be inserted into the ".info" service response. It should not contain , , or other document definition tags. The file should have the same name as the dataset with an additonal ".html" suffix. For example, is my dataset is called "test.22" then the additional information file is called "test.22.html" Servlet Information Files: These files live in the INFOcache directory. They are used to provide servlet specific information in the ".info" service responce. If you write a servlet and want to add information to the ".info" service response about your servlets idiosyncrasies, place a file of html text containing the additional information in the INFOcache directory. This html text should be a snippet of html, not a complete document. It will be inserted into the ".info" service response. It should not contain , , or other document definition tags. The file should have the same name as the java CLASSPATH of the servlet followed by a ".html" suffix. For example the DODS Test Servlet (DTS) that I developed lives in "Java-DODS/dods/servers/dts". Its class name is "dods.servers.test.dts" Thus, its servlet information file is called "dods.servers.test.dts.html" Examples of all of these files can be found in the directory "Java-DODS/sdds-testsuite/info" in the full distribution of the Java-DODS core. Legacy code: Jake,s original servlet code still exists in dods.servlet.jake but we have basically scrapped Jake's original design in favor of the current one in dods.servlet. 4/8/02 ndp ---------------------------------------------------------------------------- 10/19/98 jhrg This is the text of an e-mail from Jake Hamby which describes how to install the DODS Java servlets. Note that he mentions that the `Server:' header we're using is also used by the Java Servlets. We will change from `Server:' to `XDODS-Server:' around release 2.19-2.20. 10/19/98 jhrg First, you will need to install a web server which supports servlets. Previously, I had used JRun with somewhat mixed success (it worked okay on Solaris, but for some reason, I was unable to install it on IRIX the last time I tried). This time, I thought I'd install JServ, the official servlet module from http://java.apache.org/. Instead of getting the 0.9.11 version, which is many months old, I skipped ahead and downloaded the latest daily build. This requires you to get a copy of Sun's servlet developer kit (JSDK 2.0), and is not the simplest thing in the world to get configured and running (I found a few setup bugs which may be fixed by the time you download it). However, since these servlets are very generic, you should be able to use JRun, Sun's Java Webserver, or anything else you choose, although the installation instructions will be slightly different, of course. Anyway, I haven't added a Makefile target to compile the servlets, because you'll need to make sure the servlet classes are in your CLASSPATH first. In the case of Sun's JSDK, you'll need to add jsdk.jar to your CLASSPATH. Once you've done this, typing "javac dods/server/*.java" should do the trick. I did add the package to "make doc", however. Now, you'll need to install the servlets. Make sure you can run some of the example servlets, like SimpleServlet, before you try this. I just made a symlink from the servlets directory to the dods directory containing the Java classes, so that the classes could be found in /usr/local/apache/servlets/dods/server/*. Now, you'll need to tell the server which servlets are installed and what parameters to use. In the case of JServ (at least the beta version I used), there's a zone.properties file, which you can use to create aliases (which is necessary since the servlets are in a package, not the main directory) and startup parameters (only necessary for the proxy server). Here's what I used: servlet.proxy.code=dods.server.ProxyServlet servlet.proxy.initArgs=manta=http://manta.jpl.nasa.gov/cgi-bin/nph-hdf servlet.dap.code=dods.server.DAPServlet Now, assuming you can run the servlets, what do they do? DAPServlet fakes a real DODS server by reading the DAS, DDS, and data from separate files in the document root. If you access: http://localhost/servlets/dap/foo.hdf it will try to find foo.hdf.das, foo.hdf.dds, and foo.hdf.dods for DAS, DDS, and Data requests, respectively. The .dods file is a binary dataset file (compressed or uncompressed) as created by dods.client.Geturl -p, or by calling an existing DODS server CGI by hand. ProxyServlet looks at the first component of its path, and searches for it in its list of arguments. If it finds a match, it constructs a new URL to retrieve the information from, then sends it back out to the client. For example, http://localhost/servlets/proxy/manta/foo.hdf turns into: http://manta.jpl.nasa.gov/cgi-bin/nph-hdf/foo.hdf When testing this, I discovered one big problem, which only affects sequences: Apache won't normally allow you to override the "Server:" field (I'm guessing that the DODS CGI's can do it only because they construct their own headers). Thus, even though I set the Server header in DispatchServlet (the parent of both servlets), it wasn't working, and so clients assumed the old sequence deserialize protocol instead of the new. I've attached a patch to Apache's http_protocol.c which fixes this. I don't know if you'll have the same problem with JRun (although I'd expect you would, and I'd expect the attached patch to fix it there too), and I don't know if other servlet engines, such as Sun's Java Webserver allow you to override Server or not. I guess that's about it. If you have any problems, let me know. Naturally, these servlets are pretty naive and don't perform any caching (which is one thing you would probably expect a proxy server to do). If you plan to build on this framework, it might be a good idea to build some sort of generic caching mechanism that all DODS servers could use. Of course you have to be careful about threadsafety because doGet() can be called from several threads at once. As for adding the constraint evaluator, I suggest you don't add any state to the regular BaseType classes. My fear is that, if you do start caching DDS's, two simultaneous client requests might want to access different parts of the same dataset, and then you couldn't share a single copy of that dataset in the cache. Since a big benefit of servlets are that they're multithreaded, it would be a good idea to handle this case properly. One approach would be to have some sort of shadow BaseType classes which point to the real data, and contain an extra boolean for whether to send that variable. Then you could call serialize on the shadow class and expect it to do the right thing. The downside of this approach is that you'd need to write a shadow class for each DODS type, and override the serialize methods to work properly. You'd also need a method to set up an entire shadow class hierarchy for each simultaneous client request (of course, when you're finished, you could put them into a pool for the next client to reuse, instead of throwing it away and making a new one each time). The only other possibility I could think of would be to keep all the constraint information in a single object (let's call it CE), which lists all variables to send, then pass that object as an extra argument to the serialize method. The only problem with that is if you stored the variable list by name, how could you easily find the full name of a given variable from inside its serialize method? Since there's no getParent method, you probably couldn't. Perhaps you could just store a reference to the BaseType inside the CE object, instead of the name: then you'd get the added benefit that comparison would be much faster than any String compare. Does this make sense? -Jake --------------474CBB33E22F57292E88B46B Content-Type: text/plain; charset=us-ascii; name="patch1" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch1" *** http_protocol.c.orig Tue Oct 13 17:19:43 1998 --- http_protocol.c Tue Oct 13 17:24:31 1998 *************** *** 1160,1165 **** --- 1160,1166 ---- API_EXPORT(void) ap_basic_http_header(request_rec *r) { char *protocol; + const char *server; #ifdef CHARSET_EBCDIC int convert = ap_bgetflag(r->connection->client, B_EBCDIC2ASCII); #endif /*CHARSET_EBCDIC*/ *************** *** 1193,1199 **** protocol, " ", r->status_line, "\015\012", NULL); ap_send_header_field(r, "Date", ap_gm_timestr_822(r->pool, r->request_time)); ! ap_send_header_field(r, "Server", ap_get_server_version()); ap_table_unset(r->headers_out, "Date"); /* Avoid bogosity */ ap_table_unset(r->headers_out, "Server"); --- 1194,1203 ---- protocol, " ", r->status_line, "\015\012", NULL); ap_send_header_field(r, "Date", ap_gm_timestr_822(r->pool, r->request_time)); ! /* allow servlets to override Server field if desired ! */ ! server = ap_table_get(r->headers_out, "Server"); ! ap_send_header_field(r, "Server", server ? server : ap_get_server_version()); ap_table_unset(r->headers_out, "Date"); /* Avoid bogosity */ ap_table_unset(r->headers_out, "Server"); --------------474CBB33E22F57292E88B46B--