4.1.2 The DAS and DDS filter programs

The simplest way to learn about creating a new filter program to return a dataset's DAS or DDS is to examine the existing filter programs. In this section, we will examine the NetCDF servers.

The source code for the DAS filter program distributed with the NetCDFserver software is shown in figure 4.1.2. The DAS and DDS filters are very similar, so only the DAS filter will be discussed here. The important differences between the two will be pointed out.

The CGI dispatch program makes heavy use of commonly used functions collected in the DODS_Dispatch class. In the same way, the DODSFilter class collects several commonly used functions for the construction of filter programs. The example program uses several methods of that class. Other useful utility functions are in the cgi-util collection.  

The filter program in figure 4.1.2 can be separated into the following steps:

line 16

Step 1: The DODSFilter class provides a constructor that parses the argument list to create the data. You can use the OK method to check that the list was parsed properly. Any errors here indicate a mistake in the dispatch CGI itself. This is why the print_usage function prints its message to the WWW server log file when it returns an error object to the client.

line 21

Step 2: If the user has only requested version information from the server, it is provided here.

line 26

Step 3: The read_variables function performs the real work of this program. This involves scanning the dataset itself for data variable attributes and using the DAS method functions to assemble the corresponding DAS. This operation is specific to the data access API in use, so does not make a good example.

line 29

Step 4: Each of the filter programs must create a Multipurpose Internet Mail Extensions document to hold its return value. The DAS and DDS filters return a text MIME document; they set up the MIME headers using the utility function set_mime_text.

line 34

Step 5: Once the data set has been read and the attribute table built, the DAS ancillary file is loaded. The example filter looks for a file with the same root name as the data set and an extension of .das. If such a file exists, it is read in using the DAS member function DAS::parse and the information it contains is merged with the DAS built from the dataset.

line 37

Step 6: Finally the DAS member function print is used to send the textual representation of the DAS to the client. When it is invoked by the httpd daemon, the dispatch CGI's standard input and output are a socket connected to the remote client process. This means that since the filter is invoked by the dispatch script, its output goes directly to the client. The DODSFilter send_das method looks something like this:

DODSFilter::send_das(DAS &das)
{
    set_mime_text(dods_das);
    das.print();

    return true;
}

#include <iostream.h>

#include "DAS.h"
#include "cgi_util.h"
#include "DODSFilter.h"

extern bool read_variables(DAS &das, 
        const char *filename, String *error);

int 
main(int argc, char *argv[])
{
    DAS das;
    DODSFilter df(argc, argv);

    if (!df.OK()) {
        df.print_usage();
        return 1;
    }

    if (df.version()) {
        df.send_version_info();
        return 0;
    }

    String errMsg;
    if(!read_variables(das, df.get_dataset_name(), &errMsg)){
      Error e(no_such_file, errMsg);
      set_mime_text(dods_error);
      e.print();
      return 1;
    }

    if (!df.read_ancillary_das(das))
        return 1;

    if (!df.send_das(das))
        return 1;

    return 0;
}

Note that the example filter in figure 4.1.2 does not use any caching. It is possible to build a more sophisticated filter program that saves the generated DAS to a text file and then uses that file without first interrogating the data set, thus saving on access. It is also possible to write a DAS by hand and always use that if the data set does not contain any of the type of information that the DAS has.

Caching DAS and DDS Objects

Because the construction of the DAS and DDS objects requires that an entire data set be scanned, it can become very inefficient to continually rebuild these objects. Because the DAS and DDS filter programs use a text representation for transmission from the server to the client, it is simple to store both the DAS and DDS objects once they have been created. Subsequent accesses to these objects can be accomplished by reading and transmitting the textual representation without actually building the binary data object.

When taking advantage of this optimization, it is important that the server check the date stamp of the DAS/DDS text objects and compare it to the latest modification date of the data set. For any dataset to which new data is periodically added, the DAS/DDS text object must clearly be updated so that the cached text object matches exactly the object that would be created if the object were built by querying the data set.

The update of the DAS/DDS text object can itself be optimized significantly. It is not actually necessary to completely re-read the entire data set. Because the software used to build both the DAS and the DDS binary objects work incrementally, it is possible to read text version of the DAS/DDS object, and then read only the new parts of the data set. The binary object will be added to as needed.

NOTE: The DAS/DDS software may not properly update changed data (data that was present in a previous version of the data set, but is now different) nor is it straightforward to remove data which is no longer present in the data set. In these cases it is usually better to regenerate the DAS/DDS from scratch.