Exception handling

There are two main type of exceptions that the user of HttpClient may encounter when executing HTTP methods:

  1. transport exceptions
  2. protocol exceptions
Not all of these exceptions will be propagated to the user in regular HttpClient use. Exceptions handled internally by HttpClient are marked below as INTERNAL.

Transport exceptions

Transport exceptions are those caused by input/output failures such as an unreliable connection or an inability to complete the execution of an HTTP method within the given time constraint (socket timeout). Generally transport exceptions are non-fatal and may be recovered from by retrying the failed method. However, special care must be taken when recovering from exceptions in non-idempotent methods (refer to HTTP transport safety for details).

java.io.IOException

Generic transport exceptions in HttpClient are represented by the standard Java java.io.IOException class or its sub classes such as java.net.SocketException and java.net.InterruptedIOException.

In addition to standard input/output exception classes HttpClient defines several custom transport exceptions that convey HttpClient specific information.

org.apache.commons.httpclient.NoHttpResponseException

java.io.IOException
  +- org.apache.commons.httpclient.NoHttpResponseException

In some circumstances, usually when under heavy load, the web server may be able to receive requests but unable to process them. A lack of sufficient resources like worker threads is a good example. This may cause the server to drop the connection to the client without giving any response. HttpClient throws NoHttpResponseException when it encounters such a condition. In most cases it is safe to retry a method that failed with NoHttpResponseException.

org.apache.commons.httpclient.ConnectTimeoutException

java.io.IOException
  +- java.io.InterruptedIOException
    +- org.apache.commons.httpclient.ConnectTimeoutException

This exception signals that HttpClient is unable to establish a connection with the target server or proxy server within the given period of time.

org.apache.commons.httpclient.ConnectionPoolTimeoutException

java.io.IOException
  +- java.io.InterruptedIOException
    +- org.apache.commons.httpclient.ConnectTimeoutException
      +- org.apache.commons.httpclient.ConnectionPoolTimeoutException

This exception can only occur when using the multithreaded connection manager. The exception signals that the connection manager fails to obtain a free connection from the connection pool within the given period of time.

org.apache.commons.httpclient.HttpRecoverableException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.HttpRecoverableException

Deprecated and no longer thrown any of the standard HttpClient classes.

Protocol exceptions

Protocol exceptions generally indicate logical errors caused by a mismatch between the client and the server (web server or proxy server) in their interpretation of the HTTP specification. Usually protocol exceptions cannot be recovered from without making adjustments to either the client request or the server. Some aspects of the HTTP specification allow for different, at times conflicting, interpretations. HttpClient can be configured to support different degrees of HTTP specification compliance varying from very lenient to very strict.

org.apache.commons.httpclient.HttpException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException

HttpException represents an abstract logical error in HttpClient. Generally this kind of exception cannot be automatically recovered from.

org.apache.commons.httpclient.ProtocolException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException

ProtocolException signals a violation of the HTTP specification. It is important to note that HTTP proxies and HTTP servers can have different level of HTTP specification compliance. It may be possible to recover from some HTTP protocol exceptions by configuring HttpClient to be more lenient about non-fatal protocol violations.

org.apache.commons.httpclient.auth.MalformedChallengeException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.auth.MalformedChallengeException

INTERNAL

MalformedChallengeException signals that an authentication challenge is in some way invalid or illegal in the given authentication context.

org.apache.commons.httpclient.auth.AuthenticationException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.auth.AuthenticationException

INTERNAL

AuthenticationException signals a failure in the authentication process. Usually authentication exceptions are handled internally when executing HTTP methods and are not propagated to the caller.

org.apache.commons.httpclient.auth.AuthChallengeException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.auth.AuthenticationException
        +- org.apache.commons.httpclient.auth.AuthChallengeException

INTERNAL

AuthenticationException is thrown when HttpClient is unable to respond to any of the authentication challenges sent by the server.

org.apache.commons.httpclient.auth.CredentialsNotAvailableException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.auth.AuthenticationException
        +- org.apache.commons.httpclient.auth.CredentialsNotAvailableException

INTERNAL

CredentialsNotAvailableException indicates that credentials required to respond to the authentication challenge are not available.

org.apache.commons.httpclient.auth.InvalidCredentialsException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.auth.AuthenticationException
        +- org.apache.commons.httpclient.auth.InvalidCredentialsException

INTERNAL

InvalidCredentialsException indicates that the credentials used to respond to the authentication challenge have been rejected by the server.

org.apache.commons.httpclient.cookie.MalformedCookieException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.cookie.MalformedCookieException

INTERNAL

MalformedCookieException signals that the cookie is in some way invalid or illegal in the given HTTP session context.

There are several cookie specifications that are often incompatible. Thus the validity of a cookie is established within a context of a specific cookie specification used to parse and validate the cookie header(s) sent by the server. If the application needs to process cookies differently from the commonly used cookie specifications, it may choose to provide a custom cookie policy or extend the existing one. Please see cookies for more details.

org.apache.commons.httpclient.RedirectException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.ProtocolException
      +- org.apache.commons.httpclient.RedirectException

RedirectException signals violation of the HTTP specification caused by an invalid redirect response. If the application that uses HttpClient needs to be more lenient about redirect responses, it may choose to disable automatic redirect processing and implement a custom redirect strategy.

org.apache.commons.httpclient.URIException

java.io.IOException
  +- org.apache.commons.httpclient.HttpException
    +- org.apache.commons.httpclient.URIException

URIException is thrown when the request URI violates the URI specification.

HTTP transport safety

It is important to understand that the HTTP protocol is not well suited for all types of applications. HTTP is a simple request/response oriented protocol which was initially designed to support static or dynamically generated content retrieval. It has never been intended to support transactional operations. For instance, the HTTP server will consider its part of the contract fulfilled if it succeeds in receiving and processing the request, generating a response and sending a status code back to the client. The server will make no attempts to roll back the transaction if the client fails to receive the response in its entirety due to a read timeout, a request cancellation or a system crash. If the client decides to retry the same request, the server will inevitably end up executing the same transaction more than once. In some cases this may lead to application data corruption or inconsistent application state.

Even though HTTP has never been designed to support transactional processing, it can still be used as a transport protocol for mission critical applications provided certain conditions are met. To ensure HTTP transport layer safety the system must ensure the idempotency of HTTP methods on the application layer.

Idempotent methods

HTTP/1.1 specification defines idempotent method as

Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.

In other words the application ought to ensure that it is prepared to deal with the implications of multiple execution of the same method. This can be achieved, for instance, by providing a unique transaction id and by other means of avoiding execution of the same logical operation.

Please note that this problem is not specific to HttpClient. Browser based applications are subject to exactly the same issues related to HTTP methods non-idempotency.

Automatic exception recovery

By default HttpClient attempts to automatically recover from exceptions. The default auto-recovery mechanism is limited to just a few exceptions that are known to be safe.

HttpClient will make no attempt to recover from any logical or HTTP protocol error (those derived from HttpException class).

HttpClient will automatically retry up to 5 times those methods that fail with a transport exception while the HTTP request is still being transmitted to the target server (i.e. the request has not been fully transmitted to the server).

HttpClient will automatically retry up to 5 times those methods that have been fully transmitted to the server, but the server failed to respond with an HTTP status code (the server simply drops the connection without sending anything back). In this case it is assumed that the request has not been processed by the server and the application state has not changed. If this assumption may not hold true for the web server your application is targeting it is highly recommended to provide a custom exception handler.

Custom exception handler

In order to enable a custom exception recovery mechanism one should provide an implementation of the HttpMethodRetryHandler interface. 

HttpClient client = new HttpClient();

HttpMethodRetryHandler myretryhandler = new HttpMethodRetryHandler() {
    public boolean retryMethod(
        final HttpMethod method, 
        final IOException exception, 
        int executionCount) {
        if (executionCount >= 5) {
            // Do not retry if over max retry count
            return false;
        }
        if (exception instanceof NoHttpResponseException) {
            // Retry if the server dropped connection on us
            return true;
        }
        if (!method.isRequestSent()) {
            // Retry if the request has not been sent fully or
            // if it's OK to retry methods that have been sent
            return true;
        }
        // otherwise do not retry
        return false;
    }
};
        
GetMethod httpget = new GetMethod("http://www.whatever.com/");
httpget.getParams().
    setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler);
try {
    client.executeMethod(httpget);
    System.out.println(httpget.getStatusLine().toString());
} finally {
    httpget.releaseConnection();
}