# LibHTP Manual

Qualys, Inc.
Ivan Ristić

## Introduction

LibHTP is a security-aware parser for the HTTP protocol and the related bits and pieces. The goals of the project, in the order of importance, are as follows:

Completeness of coverage

  LibHTP must be able to parse virtually all traffic that is found in practice.

Permissive parsing

  LibHTP must never fail to parse a stream that would be parsed by some other web server.

Awareness of evasion techniques

  LibHTP must be able to detect and effectively deal with various evasion techniques, producing, where practical, identical or practically identical results as the web server processing the same traffic stream.

Performance

  The performance must be adequate for the desired tasks. Completeness and security are often detrimental to performance. Our idea of handling the conflicting requirements is to put the library user in control, allowing him to choose the most desired library characteristic.

LibHTP is not yet considered stable. Use at your own risk. Do not use in production. Work is currently under way to ensure that the code is secure and that it performs well. The library is still young; it will take some time before it can be considered feature-complete.

### License

LibHTP is licensed under the BSD 3-Clause license (also known as "BSD New" and "BSD Simplified".):

    Copyright (c) 2009-2010 Open Information Security Foundation
    Copyright (c) 2010-2013 Qualys, Inc.
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are
    met:

    - Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.

    - Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.

    - Neither the name of the Qualys, Inc. nor the names of its
      contributors may be used to endorse or promote products derived from
      this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
    HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

### Installation

Assuming you're using an already packaged version of LibHTP, the installation process should be as simple as:

    $ ./configure
    $ make
    $ sudo make install

If you've retrieved your LibHTP directly from the repository, you will need to perform the following steps first:

1. Retrieve the sources of the version you wish to use

2. Update the version number in htp.pc.in and configure.ac

3. Run autogen.sh, which will update the build system

4. Run doxygen to generate the API documentation

5. Continue to install as described above

### Usage

#### Normalization Features

##### Raw NUL terminates request line

##### Lowercase

Convert characters to lowercase. Typically used on paths, with servers with case-insensitive filesystems.

##### Convert backslashes to forwardslashes

Convert all backlash characters to forwardslash characters. Typically used on paths, with servers that support both as path separators.

##### Compress path separator characters

Convert multiple path separator characters into one (e.g., `///` is converted to `/`).

##### RFC URI normalization

Normalize URI according to the RFC. Handles self-references (`./`) and back-references (`../`).

##### URL decoding

Decode URL-encoded characters (`%HH`). Configurable handling of invalid encoding: decode anyway, remove the `%` character, and ignore (continues decoding after the `%` character).

Request query string and request body in `application/x-www-form-urlencoded` are always decoded.

##### %u decoding

Decode `%u`-encoded characters (`%uHHHH`). Configurable handling of invalid encoding.

##### Convert UTF-8 characters

Convert UTF-8 characters into single bytes. Useful to counter evasion with overlong UTF-8 characters (ASCII characters needlessly encoded as UTF-8). Best-fit mapping is used when a characters cannot be cleanly converted to a single byte.

##### URL-decode request line before parsing

##### Best-fit mapping

Used whenever a Unicode character needs to be converted to a single byte. There's a default map, but a custom map can be provided. Also supports the setting of a default replacement character, which is used when a better fit is not available.

##### Transcoding

Convert from one character encoding to another.

#### Dealing with Proxies

Under normal circumstances, when you're monitoring HTTP traffic the TCP addresses and ports will be those of the clients and servers involved in direct communication. If there are proxies on the network, however, server address and port may be those used by proxies, and not of the target HTTP servers.

If you want LibHTP to "see" and propagate actual server address and port, the solution is to wait for LibHTP to parse the request line and and use a `REQUEST_LINE` callback to overwrite the TCP address and port with the information extracted from the request.