Introduction

This document provides an overview of how HttpClient handles character encodings and how to use HttpClient in an encoding safe way. There are three main sections: HTTP Headers, Request/Response Body and URLs.

HTTP Headers

The headers of a HTTP request or response must be in US-ASCII format. It is not possible to use non US-ASCII characters in the header of a request or response. Generally this is not an issue however, because the HTTP headers are designed to facilite the transfer of data rather than to actually transfer the data itself.

One exception however are cookies. Since cookies are transfered as HTTP Headers they are confined to the US-ASCII character set. See the Cookie Guide for more information.

Request/Response Body

The request or response body can be any encoding, but by default is ISO-8859-1. The encoding may be specified in the Content-Type header, for example:

Content-Type: text/html; charset=UTF-8

In this case the application should be careful to use UTF-8 encoding when converting the body to a String or some characters may be corrupt. You can set the content type header for a request with the addRequestHeader method in each method and retrieve the encoding for the response body with the getResponseCharSet method.

If the response is known to be a String, you can use the getResponseBodyAsString method which will automatically use the encoding specified in the Content-Type header or ISO-8859-1 if no charset is specified.

Note that some document types, such as HTML and XML allow the author to specify the content type of the file. In this case, you should consult the appropriate standards regarding how to resovle any conflicts in the reported charsets.

URLs

The standard for URLs (RFC1738) explictly states that URLs may only contain graphic printable characters of the US-ASCII coded character set and is defined in terms of octets. The octets 80-FF hexadecimal are not used in US-ASCII and the octets OO-1F hexadecimal represent control characters; characters in these ranges must be encoded.

Characters which cannot be represented by an 8-bit ASCII code, can not be used in an URL as there is no way to reliably encode them (the encoding scheme for URLs is based off of octets). Despite this, some servers do support varying means of encoding double byte characters in URLs, the most common technique seems to be to use UTF-8 encoding and encode each octet separately even if a pair of octets represents one character. This however, is not specified by the standard and is highly prone to error, so it is recommended that URLs be restricted to the 8-bit ASCII range.