This chapter is an introduction to Web programming. We look at how dynamic Web sites can provide a rich experience for Web users. Then we go behind the scenes to look at how Web servers produce these dynamic pages.
We look at the most popular server interface options and detail how CGI and WinCGI work. When you understand these two interface options, you'll be prepared to implement ISAPI. And you'll have a better understanding of how the Web works.
We close this chapter by looking at the programming language options you have. This should help you to choose the right language for your development projects.
One of the exciting things about the Web is that it offers easy and instant access to so much information on so many varied topics. The ease of putting information on the Web, along with the ease of navigating it, has provided a treasure trove of information that at one time was not readily accessible.
In many cases, the information was locked away in the archives of an educational institution or corporation where few had access to it. Because the Web makes it so easy to share information, countless gigabytes of new information are published on the Web every month.
First-generation Web sites consisted of static HTML pages. Static means that the pages were not generated on the fly by software on the server.
Static pages are typically created and uploaded to the host Web server. There, they are served to Web users time after time until the HTML author (called the Webmaster) decides to make changes. The Webmaster makes the changes and reuploads the pages. The pages once again remain static until they are revised again.
Figure 3.1 illustrates this concept.
This Microsoft Internet Information Server FAQ is static HTML.
So just what are dynamic HTML pages? Unlike a static file that resides on the server's disk and goes out to the Web user's browser without change, dynamic pages change from HTTP request to request.
In many cases, there is no static HTML file at all. Instead, the complete HTML file is generated when the server gets the HTTP request from a Web user.
HTML output can change in a number of ways. For example:
Through dynamic HTML, the Web user's experience is enhanced because the dynamic HTML is customized to the needs, tastes, or requests of that particular Web user.
Figure 3.2 is an example of how Microsoft Network's (MSN) MSN.com service is allowing users to customize the information they see.
MSN on the Web offers custom user pages based on a user's selections.
Once users of MSN.com choose the information they want to see, MSN sends a client cookie to the user's Web browser. On future visits, the user is greeted with HTML tailored to that user's selections, as shown in Figure 3.3.
For more information about Cookies, see Chapter 2, "HTTP Cookies."
An example of MSN's customized HTML.
The HTML in Figure 3.3 could be the output from a database the user has searched. Or the page could be tailored to take advantage of the features of the user's Web browser.
And, of course, dynamic HTML allows us to create interactive pages in which Web users choose the information they get. Figure 3.4 is another example of this concept.
Microsoft's Tripoli Search Engine produces dynamic HTML based on search information provided by the user.
A number of interface options are available to deliver dynamic documents via the Web. These include the common gateway interface (CGI), the Windows CGI (WinCGI), Microsoft ActiveX Server (code-named Denali), Internet Server API (ISAPI), Netscape's API (NSAPI), and others such a FastCGI, Spry's BGI, and Spyglass's ADI.
CGI and WinCGI are the most commonly used interfaces to an HTTP server. We'll take a look at each of these in a little more detail.
The CGI specification was created by John Franks (firstname.lastname@example.org), Ari Luotonen (email@example.com), George Phillips (firstname.lastname@example.org), and Tony Sanders (email@example.com), the original lead authors of the earliest HTTP server.
CGI was created because it became obvious that only certain functions were appropriate to build into the HTTP server software. Yet other server functionality was useful and even necessary.
The term gateway was chosen for a software program that handles information requests and returns the appropriate document, or even generates a document on the fly. The idea is that this additional program is a gateway to information that the server could not normally access.
CGI programming needs a moderate degree of technical skill. CGI programs can be written in almost any programming language, including C or C++, Perl, Pascal, or even with a batch file.
WinCGI was created by Robert B. Denny (firstname.lastname@example.org), the author of Win-HTTPD and O'Reilly's HTTP server named WebSite. Bob's Win-HTTPD server runs under Microsoft Windows 3.1 and gave the Windows world its earliest HTTP server.
With a large pool of Windows programmers created by products such as Microsoft Visual Basic, Bob recognized the need to update the CGI specification for the Windows environment. WinCGI and CGI work in much the same way, as we'll see shortly.
WinCGI is like CGI in that you need a moderate degree of technical skill to create a WinCGI application. WinCGI programs can be written in almost any programming language.
But the languages that work in the Windows environment are particularly suited to WinCGI. These include Visual Basic, C and C++, and Delphi.
Microsoft ActiveX Server should be shipping before the end of 1996. ActiveX will revolutionize dynamic HTML generation by scripting together ActiveX components. These components span a array of functionality, accessing data across the Web and providing content directly into the HTML stream.
These include Browser Capability objects that specify client capabilities, ActiveX Data Objects that interface with local and remote data sources, and server objects that provide a power interface to server function. The beauty of the ActiveX is that you don't need a high level of technical skill to build dynamic HTML pages. An ActiveX application is HTML with in-line scripting. ActiveX supports an open scripting standard, interfacing with any scripting engine that meets the ActiveX scripting specification.
ISAPI was created by Microsoft and Process Software as an answer to the deficiencies of CGI and WinCGI. The formal specification was released in March 1996 by both companies. As we saw in Chapter 1, "Introducing ISAPI," ISAPI has been embraced by a number of HTTP server vendors.
You need a relatively high level of technical skill to use ISAPI. Two examples of skills you'll need are the ability to write thread-safe DLLs and to avoid tying up server threads for long periods of time. You can develop ISAPI extensions and filters using most 32-bit C/C++ compilers and Delphi 2.0.
The Netscape API (NSAPI) was developed by Netscape Communications Corporation as an interface to their server software. NSAPI allows developers to extend the function of Netscape server products much as ISAPI does. So far, no other HTTP vendors have embraced the NSAPI.
You need a relatively high level of technical skill to use NSAPI. The only compilers known to work with NSAPI are C compilers outlined in the NSAPI documentation (http://home.netscape.com/newsref/std/server_api.html).
A number of other interfaces have been developed to overcome the inefficiencies of CGI. These include FastCGI, the Spry binary gateway interface (Spry BGI), and the Spyglass application development interface (http://www.spyglass.com/techspec/specs/adi_spec.html). These interfaces have limited use in the Web development community.
Table 3.1 lists the standard interface options.
Table 3.1 Standard Interface Options
|CGI||C, C++, Pascal, Basic, Perl, Tkl|
|WinCGI||C, C++, Pascal, Visual Basic|
|ActiveX Server||VB Script, Jscript, Java, Perl|
|ISAPI||C, C++, Pascal|
CGI, regardless of the operating system, works like this:
Remember that a user agent can be either a Web browser or another type of program such as a robot, spider, or crawler, as discussed in Chapter 2, "The Web and Windows NT.
Figure 3.5 illustrates this sequence of events.
Environment variables are used to pass information from the operating system to application software or from application to application as in the case of CGI. Table 3.2 lists the environment variables available to a CGI application.
Table 3.2 CGI Environment Variables
|AUTH_TYPE||If the server has user authentication and the script is protected, this is the protocol-specific authentication method to validate the user.|
|CONTENT_LENGTH||The length of the content as given by the client.|
|CONTENT_TYPE||For queries with attached information, such as HTTP POST and PUT, this is the content type of the data.|
|GATEWAY_INTERFACE||The revision of the CGI specification to which this server complies. The format is CGI/revision.|
|HTTP_ACCEPT||The MIME types the client accepts, as given by HTTP headers. Other protocols may need to get this information from elsewhere. Separate each item in this list by commas according to the HTTP specifications. The format is type/subtype, type/subtype.|
|HTTP_USER_AGENT||The browser the client is using to send the request. The format is software/version library/version.|
|PATH_INFO||The extra path information, as given by the client. In other words, scripts can be accessed by their virtual path name, followed by extra information at the end of this path.
The extra information is sent as PATH_INFO. This information should be decoded by the server if it comes from a URL before it is passed to the CGI script.
|PATH_TRANSLATED||The server provides a translated version of PATH_INFO, which takes the path and does any virtual-to-physical mapping to it.|
|QUERY_STRING||The information following the "?" in the URL that referenced this script. This is the query information and it should not be decoded in any fashion. Always set this variable when there is query information, regardless of command line decoding.|
|REMOTE_HOST||The name of the host making the request. If the server does not have this information, it should set REMOTE_ADDR and leave this unset.|
|REMOTE_ADDR||The IP address of the remote host making the request.|
|REMOTE_IDENT||If the HTTP server uses request for comments (RFC) 931identification, this variable is set to the remote user name retrieved from the server. Limit use of this variable to logging only.|
|REQUEST_METHOD||The method with which the request was made. For HTTP, this is GET, HEAD, POST, and so forth.|
|REMOTE_USER||If the server has user authentication and the script is protected, this is the user name the Web client was authenticated as.|
|SERVER_NAME||The server's host name, domain name service (DNS) alias, or IP address as it would appear in self-referencing URLs.|
|SERVER_SOFTWARE||The name and version of the server software answering the request (and running the gateway). The format is name/version.|
|SERVER_PORT||The port number to which the request was sent.|
|SERVER_PROTOCOL||The name and revision of the information protocol this request came in with. The format is protocol/revision.|
|SCRIPT_NAME||A virtual path to the script being executed, used for self-referencing URLs.|
For requests with information attached after the header, such as HTTP POST or PUT, the information is sent to the script on stdin.
The server sends CONTENT_LENGTH bytes on this file descriptor. Remember that it also gives the CONTENT_TYPE of the data.
The script sends its output to stdout. This output can be either a document generated by the script or instructions to the server for retrieving the output. Stdout is read by the server and sent back to the client connection.
WinCGI works like CGI. The difference is that Windows INI files pass information to the WinCGI application instead of through stdin and environment variables.
This is how WinCGI works:
Figure 3.6 illustrates the WinCGI process.
As we mentioned earlier, Bob Denny was the creator of the WinCGI specification. We'll now take a look at the WinCGI specification that Bob created.
A key feature of Windows CGI is its spooled exchange of data between the server and the CGI program. It is essential that the server provide efficient transfer of data between the spool files and the network. This means that the server should use memory-mapped techniques, and minimize the number of separate network I/O requests used.
The reasons for using spooled I/O are:
HTML Form Data Decoding
Windows CGI requires that the web server decode HTML form data if present in a POST request. It is not required that the server decode form data if it appears in the "query string" portion of a request URL.
There are two ways in which form data can be sent by a browser to the server, explained in the following sections.
This is the most common form data format. The contents of form fields are "escaped" according to the rules in the HTML 1.0 Specification, then concatenated using unescaped ampersand characters. This URL-encoded data is sent as a stream to the server, with a content type of application/x-www-form-urlencoded.
This format has been introduced to permit efficient file uploading with forms. It may be used without explicitly including a file upload form field, however. The contents of the form fields are sent as a MIME multipart message. Each field is contained within a single part. The content type indicated by the browser is multipart/form-data.
Compliant servers must decode both form data types.
Launching the CGI program
The server uses the CreateProcess() service to launch the CGI program. The server maintains synchronization with the CGI program so it can detect when the CGI program exits. This is done using the Win32 WaitForSingleObject() service, waiting for the CGI process handle to become signalled, indicating program exit. The server must never use a shell to execute the CGI program. This can create serious security risks.
The CGI program's process handle becomes signaled before the process rundown is complete. Reliance on rundown to close files, inherited handles, etc., can cause obscure synchronization problems.
The server must execute a CGI program request by doing a CreateProcess() with a command line in the following form:
The complete path to the CGI program executable. The server does not depend on the "current directory" or the PATH environment variable. Note that the "executable" need not be a .EXE file. It may be a document, provided an "association" with a corresponding executable has been established.
The complete path to the CGI data file.
The server issues the CreateProcess() such that the process being launched has its main window hidden. The launched process itself should not cause the appearance of a window nor a change in the Z-order of the windows on the desktop.
The server supports a CGI program/script debugging mode. If that mode is enabled, the CGI program is launched such that its window shows and is made active. This can help in debugging CGI applications.
The server must honor document associations. If the target of a Windows CGI request is a document (not an executable), the server must attempt to find the associated application for the document and launch the application such that the document is "processed".
The CGI Data File
The server passes data to the CGI program via a Windows "private profile" file, in key-value format. The CGI program may then use the standard Windows API services for enumerating and retrieving the key-value pairs in the data file.
The CGI data file contains the following sections:
This section contains most of the CGI data items (accept types, content, and extra headers are defined in separate sections). Each item is provided as a string value. If the value is an empty string, the keyword is omitted. Table 3.3 lists the variables found in the CGI section of the WinCGI INI file.
Table 3.3 The [CGI] Section
|Request Protocol||The name and revision of the information protocol this request came in with. Format: protocol/revision. Example: "HTTP/1.0".|
|Request Method||The method with which the request was made. For HTTP, this is "GET", "HEAD", "POST", etc.|
|Executable Path||The logical path to the CGI program executable, as needed for self-referencing URLs. This may vary if the server supports multihoming with separate logical path spaces.
The server must provide the physical path equivalent using the logical to physical mapping for the identity on which the current request was received.
|Document Root||The physical path to the logical root "/". This may vary if the server supports multihoming with separate logical path spaces. The server must provide the physical path to the logical root for the identity on which the current request was received.|
|Logical Path||A request can specify a path to a resource needed to complete that request. This path can be in a logical path name space. This item holds the path name exactly as received by the server, without logical-to-physical translation.|
|Physical Path||If the request contained logical path information, the server provides the path in physical form, in the native object (such as file) access syntax of the operating system. This may vary if the server supports multi- homing with separate logical path spaces.
The server must provide the physical path equivalent using the logical to physical mapping for the identity on which the current request was received.
|Query String||The information following the "?" in the URL that generated the request is the "query" information. The server furnishes this to the back end whenever it is present on the request URL, without any decoding or translation.|
|Request Range||Byte-range specification received with request (if any). See the current Internet Draft (or RFC) describing the byte-range extension to HTTP for more information. The server must support CGI program participation in byte- ranging to be compliant with this specification.|
|Referer||The URL of the document that contained the link pointing to this CGI program. Note that in some browsers, the implementation of this is broken and cannot be relied on.|
|From||The e-mail address of the browser user. Note that this is in the HTTP specification but is not implemented in some browsers because of privacy concerns.|
|User Agent||A string description of the client (browser) software. Not generated by all browsers.|
|Content Type||For requests with attached data, this is the MIME content type of that data. Format: type/subtype.|
|Content Length||For requests with attached data, this is the length of the content in bytes.|
|Content File||For requests with attached data, the server makes the data available to the CGI program by putting it into this file. The value of this item is the complete path name of that file.|
|Server Software||The name and version of the information server software answering the request (and running the CGI program). Format: name/version.|
|Server Name||The network host name or alias of the server, as needed for self-referencing URLs. This (in combination with the ServerPort) could be used to manufacture a full URL to the server, for URL fix-ups.
This may vary if the server supports multihoming. The value of this item must be the host name on which the current request was received.
|Server Port||The network port number on which the server is listening. This is also needed for self-referencing URLs.|
|Server Admin||The e-mail address of the server's administrator. This is used in error messages and might be used to send MAPI mail to the administrator, or to form "mailto:" URLs in generated documents.|
|CGI Version||The revision of the CGI specification to which this server complies. Format: CGI/revision. For this version, CGI/1.2 (Win).|
|Remote Host||The network host name of the client (requestor) system, if available. This item is used for logging.|
|Remote Address||The network (IP) address of the client (requestor) system. This item is used for logging if the host name is not available.|
|Authentication Method||The protocol-specific authentication method specified in the request. If present, this is normally Basic. The server must provide this whether or not it was used by the server for authentication.|
|Authentication Realm||The method-specific authentication realm given in the request. If present in the request, the server must provide this whether or not it was used by the server for authentication.|
|Authenticated Username||The user name (in the indicated realm) that the client used to try authentication, as specified in the request. If present in the request, the server must provide this whether or not it was used by the server for authentication.|
|Authenticated Password||The password the client used to attempt authentication, as specified in the request. If present in the request, the server must provide this whether or not it was used by the server for authentication.|
This section contains the client's acceptable data types found in the request header as:
If the parameters (e.g., "q=0.100") are present, they are passed as the value of the item. If there are no parameters, the value is "Yes".
The accept types may easily be enumerated by the CGI program with a call to GetPrivateProfileString() with NULL for the key name. This returns all of the keys in the section as a null-delimited string with a double-null terminator.
This section contains items that are specific to the Windows implementation of CGI. Table 3.4 lists the variables found in the System section of the WinCGI INI file.
Table 3.4 The [System] Section
|GMT Offset||The number of seconds to be added to GMT time to reach local time. For pacific standard time, this number is -28,800. Useful for computing GMT times.|
|Debug Mode||This is No unless the server's CGI/script tracing mode is enabled, then it is Yes. Useful for conditional tracing within the CGI program.|
|Output File||The full path name of the file in which the server expects to receive the CGI program's results.|
|Content File||The full path name of the file that contains the content (if any) that came with the request.|
This section contains the "extra" headers that were included with the request, in key=value form. The server must URL-unescape both the key and the value prior to writing them to the CGI data file.
The extra headers may easily be enumerated by the CGI program with a call to GetPrivateProfileString() with NULL for the key name. This returns all the keys in the section as a null-delimited string with a double-null terminator.
If the request is an HTTP POST from an HTTP form (with content type of application/x-www-form-urlencoded or multipart/form-data), the server decodea the form data and puts it into the [Form Literal] section.
For URL-encoded form data, raw form input is of the form key=value&key=value&..., with the value parts in URL-encoded format. The server splits the key=value pairs at the '&', then splits the key and value at the '=', URL-decodes the value string, and puts the result into key=(decoded)value form in the [Form Literal] section.
For multipart form data, raw form input is in a MIME-style multipart format, with each field in a separate part. The server extracts the field named and value from each part and puts the result into key=value form in the [Form Literal] section.
If the form contains any SELECT MULTIPLE elements, there will be multiple occurrences of the same key. In this case, the server generates a normal key=value pair for the first occurrence and appends a sequence number to subsequent occurrences. It is up to the CGI program to know about this possibility and to recognize the tagged keys.
If the decoded value string is more than 254 characters long, or if the decoded value string contains any control characters or double-quotes, the server puts the decoded value into an external temp file and lists the field into the [Form External] section as:
where path name is the path and name of the temp file containing the decoded value string, and length is the length in bytes of the decoded value string.
Be sure to open this file in binary mode unless you are certain that the form data is text!
If the raw value string is more than 65,535 bytes long, the server does no decoding. But it does get the keyword and mark the location and size of the value in the Content File. The server lists the huge field in the [Form Huge] section as:
where offset is the offset from the beginning of the Content File at which the raw value string for this key is located, and length is the length in bytes of the raw value string. You can use the offset to do a Seek to the start of the raw value string and use the length to know when you have read the entire raw string into your decoder. Note: Be sure to open this file in binary mode unless you are sure the form data is text!
If the request is in the multipart/form-data format, it may contain one or more file uploads. In this case, each file upload is placed into an external temp file similar to the form external data. Each such file upload is listed in the [Form File] section as:
where path name is the path name of the external tempfile containing the uploaded file, length is the length in bytes of the uploaded file, type is the MIME content type of the uploaded file, xfer is the content-transfer encoding of the uploaded file, and filename is the original name of the uploaded file. The square brackets must be included. They are used to delimit the file and path names, which may contain spaces.
In the following example of form decoding, the form contains a small field, a SELECT MULTIPLE with two small selections, a field with 300 characters in it, one with line breaks (a text area), and a 230-kbyte field.
smallfield=123 Main St. #122
Now we'll look at the most common language options for CGI, WinCGI and ISAPI. Each language has its own advantages and its own disadvantages. You should carefully consider:
C and C++ need the most technical skill of the language options we'll look at, but it also gives developers the most advantages. For the sake of this discussion, we'll consider C and C++ to be the same.
Using C, you can develop for any of the Internet server interface options. Developers using Microsoft Visual C++ can create CGI, WinCGI, ActiveX components, ISAPI extensions and filters, NSAPI extensions and FastCGI-compliant applications. C is also a compiled language (versus an interpreted language) for creating small and fast binary executables.
For this reason, if speed is a concern, C should be one of your top options. The debugging facilities allow interactive debugging in some cases, which is a real development advantage not currently offered by the other language options.
Example CGI Application Using C
This C CGI demo sets a client-side cookie and presents the user with a screen that says "Greetings from CGI."
printf("<html><head><title>CGI C Demo</title></head>");
printf("<body><h1>Greetings from CGI</h1>");
Sample ISAPI Extension Using C
The same program written with an ISAPI interface follows.
BOOL WINAPI GetExtensionVersion( HSE_VERSION_INFO *pVer )
pVer->dwExtensionVersion = MAKELONG(HSE_VERSION_MINOR,
lstrcpyn( pVer->lpszExtensionDesc, "ISAPI C Demo",
DWORD WINAPI HttpExtensionProc(EXTENSION_CONTROL_BLOCK *pECB)
"<head><title>ISAPI C Demo</title></head>\n"
"<body><h1>Greetings from ISAPI</h1>\n");
if (!pECB->ServerSupportFunction( pECB->ConnID,
HSE_REQ_SEND_RESPONSE_HEADER, "200 OK",
&dwLen,(LPDWORD) buff ))
Microsoft VB is the premier, GUI-building RAD environment for the Windows platform. Of course, with WinCGI you can't create GUI applications, but the environment is still excellent for creating these Internet and intranet applications.
Microsoft VB 3.0 and 4.0 can produce WinCGI applications or OLEISAPI extensions. Of the products we consider here, VB is one of the easiest to learn.
VB 5.0 supports the creation of ActiveX components that can be placed on the server.
Borland's Delphi is an Object Pascal-based visual development environment for the Windows platform. Delphi 1.0 is a 16-bit compiler and can be used to create 16-bit WinCGI applications. Delphi 2.0 is a 32-bit version which can produce 32-bit WinCGI, Windows console applications for CGI support and can also produce ISAPI DLLs.
Delphi is a slightly more technical product than VB 3.0 and 4.0 but not quite as complicated as the Visual C++ environment. A number of libraries are available for producing Internet applications using Delphi.
Example CGI Application Using Delphi 2.0
This Delphi CGI demo sets a client-side cookie and shows the user a screen that says "Greetings from CGI."
WriteLn('<html><head><title>CGI Delphi Demo</title></head>');
WriteLn('<body><h1>Greetings from CGI</h1>');
Sample ISAPI Extension Using Delphi 2.0
The same program written with an ISAPI interface follows.
CrLF = ^M^J;
Function GetExtensionVersion (var pVer : HSE_VERSION_INFO):BOOL;
pVer.dwExtensionVersion := MAKELONG(HSE_VERSION_MINOR,
StrPCopy(pVer.lpszExtensionDesc, 'Delphi Test ISAPI DLL');
result := True;
Function HttpExtensionProc (var pECB : EXTENSION_CONTROL_BLOCK):
DWORD; export; stdcall;
Buffer : Array[0..2047] of Char;
Response : Array[0..8] of Char;
dwLen : longint;
StrPCopy(Buffer, 'Content-Type: text/html' + CrLf +
'Set-Cookie: Lang=Delphi' + CrLf + CrLf +
'<html><head><title>CGI Delphi Demo</title></head>' +
'<body><h1>Greetings from CGI</h1></body></html>');
StrPCopy(Response, '200 OK');
dwLen := StrLen(Buffer);
HSE_REQ_SEND_RESPONSE_HEADER, @Response, @dwlen, @Buffer);
result := HSE_STATUS_SUCCESS;
Perl was created by Larry Wall in the early 1980s. Perl is an interpreted language that is a cross between awk, C, sed, and UNIX shells. The syntax of Perl is similar to C.
Perl was originally developed to process text files. As it turned out, the Web's popularity created a sudden demand for processing HTML (text) files. Perl has rapidly become the most popular CGI language on UNIX-based systems.
Because Perl is an interpreted language there are no known libraries for WinCGI, developers are currently limited to writing CGI scripts with Perl.
NT developers can pick up the latest copy of Win32 Perl from http://www.perl.hip.com/.
In this chapter we explore the CGI and WinCGI server interfaces. Because ISAPI builds on the basic concepts of these interfaces, new Web developers should familiarize themselves with these interfaces.
We also review the programming language options that are available and list considerations for starting Web development projects.