Hypertext Transfer Protocol (HTTP)
Any communication between two parties requires a clear understanding of how the communication will be conducted. For example, a telephone conversation is initiated by the caller lifting the handset from the cradle, checking for a dial tone, and if the dial tone is present, beginning to dial the desired number. This is usually comprised of a country code prefix, followed by the country code, an area code, and the phone number. If the call is made to another phone within the same country, or within the same area code, the caller might skip these elements.
At the receiving end, the protocol is much simpler. The called party waits until the phone begins to ring, at which point, she picks up the phone and indicates that she is ready to receive the communication by saying 'Hello'.
This telephony protocol has been in place for so many years that it has now become second nature to us, and seems quite unremarkable. Until of course, you move to a different country, and suddenly realize that your international dialing prefix is not 011, but is 00 instead, or that the area code prefix is now a 0 rather than a 1.
HTTP is just another communication protocol, except that it defines how one computer may connect to another to request a particular resource. This particular protocol has been wildly successful since its introduction in 1990 as HTTP/0.9 and is today the lingua franca of the Internet.
There are some characteristics that are common to the two communication models.
First, there is a connection that needs to be established between the two parties. This connection has to be held open until the interaction has been completed and the information has been completely exchanged. In the HTTP/1.1 protocol, this is called a persistent connection.
Second, there are two aspects to the communication, the protocol and the actual payload. In the case of HTTP, the payload is usually text marked up using tags that are defined by the HyperText Markup Language (HTML). However, HTTP may be used to transfer other data, including audio and video content. The protocol aspects are carried in communication headers that indicate various capabilities of the client (such as the client's preferred character encoding), metadata about the resource being retrieved (such as the length of the payload data), the server's success at retrieving this resource, and so on.
Third, there is the question of conversational state. In the typical interaction, a client (usually a web browser) makes a request for a resource, such as a text file or an image, and a server responds with that resource.
In reality, a single interaction may consist of multiple such request/response pairs. This usually happens when a browser requests a page that contains one or more images. In this case, the connection is held open while a request/response pair occurs for each of the images on that page. However, once the interaction has been completed, the connection between the two computers is terminated.
This is why HTTP is considered a stateless protocol. The next time the client connects to the server to make a request, the server has no memory of the prior interaction with this client.
HTTP, therefore, defines a connection oriented but stateless protocol.
Note
The astute reader will have noticed that a telephone conversation is not quite the best analogy for an HTTP interaction. While it is connection oriented, there is an element of state that is retained in the memories of both parties while the communication is in progress. On the other hand, the HTTP protocol is completely stateless.
Note that it is always the client that initiates the connection in this model. The server has no means to push content to the client. In this usage, the term client refers to the computer that initiates the communication. While this is usually a client's browser, it could also be a server host that initiates a web services type of connection. One way of simulating a push model, for a browser client, is to simply have the client reconnect, every so often, to refresh its content (say using the refresh meta tag or with JavaScript).
The underlying protocol used by HTTP is TCP/IP. While this protocol uses the well known TCP port 80, other ports may be used. For example, by default, your Tomcat installation will start up and listen on port 8080 for incoming requests.
Uniform Resource Locator, URL
A Uniform Resource Locator (URL) is described in RFC 3986 (http://www.ietf.org/rfc/rfc3986.txt) as a uniform way of uniquely identifying a resource such as an electronic document or an image. URLs are designed to implicitly provide a means of locating the resource by describing its 'location' on a network.
A generic URL is a hierarchical sequence of components, structured as follows:
scheme://hostName:portNumber/path/to/resource?queryString#fragment
An example of which is:
http://www.swengsol.com:80/file.htm?name=bwayne&city=gotham#profile
Here the scheme is http, the host name is www.swengsol.com, the port number is 80, the path to the resource is /index.html, the query string is name=bwayne&city=gotham, and the fragment is profile.
The combination of host name and port number taken together is termed an authority.
A number of these parts are optional, including the port number (which defaults to the well known ports 80 and 443 for the http and https schemes respectively), the query string, and the fragment.
When present, a query string is a series of name-value pairs that are preceded with a question mark and with an ampersand (&) separating the pairs.
The hierarchical organization of a URL becomes apparent as you travel from left to right, with its components increasing in specificity the farther right you go. You start off with a server domain name, a port on that server, and finally an absolute path on that server. The portion that comes after '?' is additional information that is useful to process the located resource.
A URL is composed of characters that are taken from a very limited set—the letters of the basic Latin alphabet (ISO-8859-1), digits, and a few special characters ('-', '.', '_', '~'). All other characters are deemed to be reserved and must be percent-encoded before use. In other words, you must convert them to their hexadecimal representation and prefix them with a % sign. For example, a space character (32 in ISO-8859-1) is encoded as %20. Other characters that should be encoded include the percent itself (%25) as well as characters that are used as delimiters (such as '&', '$', and ':').
HTTP methods
An HTTP method, or request type, defines the actions that a client may request from a server. HTTP/1.1 specifies seven such methods.
This is the oldest and most commonly used method. It is used to GET a resource (such as a document or an image file) from a server. It can also request dynamic content by passing in additional parameters in a query string. This method is intended to be used only for idempotent requests, that is requests that do not cause state changes on the server. For example, a retrieval of a resource from the server is considered idempotent.
The URL for a GET request is portable as it is a self contained unit that can be bookmarked or emailed. Typing it in over and over into a browser's address bar should return the same content.
There are some downsides to using this method. Though the protocol does not place any limit on the length of a URL, lengths should be limited to 255 bytes because some older browsers or proxies may not properly support greater lengths. As the query string is counted against this limit, this restricts the amount of information that can be passed using this mechanism. A security risk associated with this method is that the parameters that are passed to the web server appear on the address bar in plain text and so can be easily viewed by anyone. In addition, the URL along with its query string is often logged at the server and at other hops along its journey, leading to additional security risks.
Finally, while the specification recommends that a GET request is used only for idempotent requests, nothing prevents an errant application from violating this recommendation. It is therefore not at all uncommon to find, for example, a GET request that causes a shopping cart order to be saved, or a credit card to be charged.
This method was introduced to handle non-idempotent requests, that is, requests that cause a state change on the server, such as inserting a record in the server's database.
Repeated submission of a non idempotent request is unsafe, as it may end up causing undesirable effects such as a shopping cart order being saved to the database multiple times. This method sends its parameters to the server as part of the request body, rather than in a plainly visible query string. This means that a large amount of data (even megabytes) can be sent from the client.
A key downside is that the URL is no longer self-sufficient, as the parameters are now part of the body's payload. As a result, the URL cannot be bookmarked or emailed to others.
The TRACE method simply returns an exact copy of a request to a client. It is intended for debugging use and traces the path taken by a request.
This method allows the client to query the server about the communication options that are available with regards to a particular resource without explicitly requesting that resource. The returned response would include header fields that indicate optional features that are implemented by the server which are applicable to that resource. For example, the Allow header lists the set of methods (GET, POST, and so on) that are supported for a given resource.
HTTP requests
The simplest HTTP request is composed of the following elements:
[request-method] [/path/to/resource] [HTTP protocol version] [request-header=value]+ [blank-line to indicate the end of the request headers] [POST:request-payload]
The first line of the request contains the following:
- The type of the request, which is usually
GET, POST, PUT, DELETE, OPTIONS, HEAD, andTRACE - The name of the resource that is being requested
- The protocol that the browser wishes to use for this communication
Following this first line are the request headers, which in turn are followed by the request's payload (for a POST request).
As shown, the browser is asking the server (specified by the Host: header) for the specified resource and is stating that it would like to use HTTP 1.1 for the communication.
Each request header conveys information either about the client's capabilities or about the request body. Each header is on a separate line, and the end of the headers is indicated by a blank line. The final section is the request's body, which contains form parameters for a POST request.
Request headers are name-value pairs, which take the form Header-Name: value. Headers allow the client to send optional metadata about the request. The information provided in these headers can be used by the server to tailor its response to the specific circumstances and capabilities of the requesting client.
Some commonly seen request headers include the following:
Accept:This header specifies the MIME types that the client can handle. The server can query this request header to determine how to craft the response to the client.Accept-Charset:The character sets that the client can handle.Accept-Encoding:The encodings that the client can handle, such as thegziporcompresscompression formats, which serve to save transmission time. On receiving the response, the browser first reverses this encoding. It then uses theContent-Typeresponse header to determine how to handle the decoded content.Accept-Language:It allows the client to request a preferred language for the response. In the previous image, this request header indicates that the browser would prefer a response in English (United States), which is represented by the language codeen-us.Content-Length:This header is applicable to POST requests and gives the size of the request body in bytes.Content-Type:This header is applicable to POST requests, and is used to communicate the type of the POST data.Cookie:This header is used to return a cookie that has been previously sent by a server.Host:This helps the server to determine the original host name and port for this request, which might otherwise be obscured due to request forwarding.If-Modified-SinceandIf-Unmodified-Since:These are used by the client to support caching of resources.Referer:This header indicates the URL of the referring web page.User-Agent:This header identifies the browser that is making the request. A server can use this to customize the content by the type of browser that is being used.
HTTP responses
The basic response that is returned by a server is comprised of a response line, headers, and a body.
The first line of the response indicates the overall status of the request (for example, success or failure at a very specific level of detail). This is followed by response headers, which indicate control information to the browser (such as the length of the content being sent in the response's body). Finally, you have the response body, which is the response's main content—an HTML document, a graphical image, and so on.
The response line contains the HTTP version of the server, a response code, and a reason phrase that describes the response in a human readable format (such as 'OK' or 'Not Found').
A response code is a 3 digit number, where the first digit indicates a response category. The supported response categories include 1xx for informational messages, 2xx for successful requests, 3xx to indicate files that have moved to a different URL, 4xx for client browser errors, and 5xx for server errors.
Common examples of response codes are as follows:
- 200 (OK) indicates a successful response, and that a document is being returned for GET and POST requests.
- 302 (Found) indicates that the requested document has been moved, and that the new URL for that document is provided in the
Locationresponse header. A browser receiving a 302 response code will automatically request the new document. - 404 indicates that the requested resource could not be found at the given URL, the dreaded 'File Not Found' error.
- 500 indicates that an internal server error has occurred. This is normally caused by a severe program error in your servlet code.
The server sends response headers as name-value pairs to provide information to the client about the server and the content of the response body.
Some commonly seen response headers include the following:
Cache-Control:This header tells the client whether a response may be cached. For example, setting it tono-cacheindicates that the document should never be cached.Content-Encoding:This indicates how the page was encoded, say by compressing it usinggzip, for transmission.Content-Language:It indicates the language in which the document was written (for example,frores).Content-Length:This is the number of bytes in the response.Content-MD5:This header provides an MD5 digest for the subsequent body content to allow clients to verify that the document was not modified in transit.Content-Type:This indicates the Multipurpose Internet Mail Extension (MIME) type of the response document. A MIME type is a universally accepted way of identifying the type of content when a file is transmitted over the Web. It serves as a standard way of associating content with the application that can be used to view/edit it, and is more portable than file extensions. The most common MIME types aretext/plainortext/htmlthat indicates a response payload that consists of ordinary text or HTML content. Other types includeimage/giffor GIF images, orvideo/mpegfor MPEG video clips.Last-Modified:This header indicates when this resource was last changed, enabling a client to cache the document, and to use caching request headers likeIf-Modified-Sinceto request efficient updates in later requests.Location:It provides the location of the new document that is set when the response status code is in the 300s.Pragma:This is used with a value ofno-cacheto prevent the caching of the response, and is recognized by older HTTP/1.0 clients.Refresh:This header indicates the seconds after which the client browser should ask the server for an updated version of this resource. You must continue to supplyRefreshin all subsequent responses until the server returns a 204 response, which should stop the client browser from requesting further refreshes. Note that using theRefreshmeta tag is a bit more powerful, as you can request a page other than the current page.Set-Cookie:It sets a given cookie on the client.WWW-Authenticate:This is used with a 401 (Unauthorized) status code to indicate to the client which authorization type the client should supply in itsAuthorizationheader.
Spying on HTTP
A couple of nifty tools allow you to eavesdrop on the HTTP communication that occurs between your browser and the server.
For Internet Explorer users, there is the amazingly featured Fiddler v2 HTTP Debugging proxy at http://www.fiddlertool.com/fiddler/.
For Firefox users, there's the http://livehttpheaders.mozdev.org/ add-on.
Both these tools give you amazing insights into the request/response process and are an invaluable aid while debugging server-side applications.