:tocdepth: 3

base/protocols/http/main.zeek
=============================
.. zeek:namespace:: HTTP

Implements base functionality for HTTP analysis.  The logging model is
to log request/response pairs and all relevant metadata together in
a single record.

:Namespace: HTTP
:Imports: :doc:`base/frameworks/tunnels </scripts/base/frameworks/tunnels/index>`, :doc:`base/protocols/conn/removal-hooks.zeek </scripts/base/protocols/conn/removal-hooks.zeek>`, :doc:`base/utils/files.zeek </scripts/base/utils/files.zeek>`, :doc:`base/utils/numbers.zeek </scripts/base/utils/numbers.zeek>`

Summary
~~~~~~~
Runtime Options
###############
================================================================================ ====================================================================
:zeek:id:`HTTP::default_capture_password`: :zeek:type:`bool` :zeek:attr:`&redef` This setting changes if passwords used in Basic-Auth are captured or
                                                                                 not.
:zeek:id:`HTTP::http_methods`: :zeek:type:`set` :zeek:attr:`&redef`              A list of HTTP methods.
:zeek:id:`HTTP::max_pending_requests`: :zeek:type:`count` :zeek:attr:`&redef`    Only allow that many pending requests on a single connection.
:zeek:id:`HTTP::proxy_headers`: :zeek:type:`set` :zeek:attr:`&redef`             A list of HTTP headers typically used to indicate proxied requests.
================================================================================ ====================================================================

Types
#####
============================================= ===================================================================
:zeek:type:`HTTP::Info`: :zeek:type:`record`  The record type which contains the fields of the HTTP log.
:zeek:type:`HTTP::State`: :zeek:type:`record` Structure to maintain state for an HTTP connection with multiple
                                              requests and responses.
:zeek:type:`HTTP::Tags`: :zeek:type:`enum`    Indicate a type of attack or compromise in the record to be logged.
============================================= ===================================================================

Redefinitions
#############
==================================================================== =============================================================
:zeek:type:`Log::ID`: :zeek:type:`enum`                              
                                                                     
                                                                     * :zeek:enum:`HTTP::LOG`
:zeek:type:`connection`: :zeek:type:`record`                         
                                                                     
                                                                     :New Fields: :zeek:type:`connection`
                                                                     
                                                                       http: :zeek:type:`HTTP::Info` :zeek:attr:`&optional`
                                                                     
                                                                       http_state: :zeek:type:`HTTP::State` :zeek:attr:`&optional`
:zeek:id:`likely_server_ports`: :zeek:type:`set` :zeek:attr:`&redef` 
==================================================================== =============================================================

Events
######
============================================= ====================================================================
:zeek:id:`HTTP::log_http`: :zeek:type:`event` Event that can be handled to access the HTTP record as it is sent on
                                              to the logging framework.
============================================= ====================================================================

Hooks
#####
============================================================== =======================
:zeek:id:`HTTP::finalize_http`: :zeek:type:`Conn::RemovalHook` HTTP finalization hook.
:zeek:id:`HTTP::log_policy`: :zeek:type:`Log::PolicyHook`      
============================================================== =======================


Detailed Interface
~~~~~~~~~~~~~~~~~~
Runtime Options
###############
.. zeek:id:: HTTP::default_capture_password
   :source-code: base/protocols/http/main.zeek 25 25

   :Type: :zeek:type:`bool`
   :Attributes: :zeek:attr:`&redef`
   :Default: ``F``

   This setting changes if passwords used in Basic-Auth are captured or
   not.

.. zeek:id:: HTTP::http_methods
   :source-code: base/protocols/http/main.zeek 120 120

   :Type: :zeek:type:`set` [:zeek:type:`string`]
   :Attributes: :zeek:attr:`&redef`
   :Default:

      ::

         {
            "POST",
            "PUT",
            "CONNECT",
            "BMOVE",
            "SEARCH",
            "TRACE",
            "LOCK",
            "PROPPATCH",
            "HEAD",
            "OPTIONS",
            "POLL",
            "REPORT",
            "SUBSCRIBE",
            "MOVE",
            "GET",
            "UNLOCK",
            "DELETE",
            "COPY",
            "MKCOL",
            "PROPFIND"
         }


   A list of HTTP methods. Other methods will generate a weird. Note
   that the HTTP analyzer will only accept methods consisting solely
   of letters ``[A-Za-z]``.

.. zeek:id:: HTTP::max_pending_requests
   :source-code: base/protocols/http/main.zeek 141 141

   :Type: :zeek:type:`count`
   :Attributes: :zeek:attr:`&redef`
   :Default: ``100``

   Only allow that many pending requests on a single connection.
   If this number is exceeded, all pending requests are flushed
   out and request/response tracking reset to prevent unbounded
   state growth.

.. zeek:id:: HTTP::proxy_headers
   :source-code: base/protocols/http/main.zeek 107 107

   :Type: :zeek:type:`set` [:zeek:type:`string`]
   :Attributes: :zeek:attr:`&redef`
   :Default:

      ::

         {
            "CLIENT-IP",
            "X-FORWARDED-FROM",
            "VIA",
            "XROXY-CONNECTION",
            "PROXY-CONNECTION",
            "X-FORWARDED-FOR",
            "FORWARDED"
         }


   A list of HTTP headers typically used to indicate proxied requests.

Types
#####
.. zeek:type:: HTTP::Info
   :source-code: base/protocols/http/main.zeek 28 89

   :Type: :zeek:type:`record`


   .. zeek:field:: ts :zeek:type:`time` :zeek:attr:`&log`

      Timestamp for when the request happened.


   .. zeek:field:: uid :zeek:type:`string` :zeek:attr:`&log`

      Unique ID for the connection.


   .. zeek:field:: id :zeek:type:`conn_id` :zeek:attr:`&log`

      The connection's 4-tuple of endpoint addresses/ports.


   .. zeek:field:: trans_depth :zeek:type:`count` :zeek:attr:`&log`

      Represents the pipelined depth into the connection of this
      request/response transaction.


   .. zeek:field:: method :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Verb used in the HTTP request (GET, POST, HEAD, etc.).


   .. zeek:field:: host :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the HOST header.


   .. zeek:field:: uri :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      URI used in the request.


   .. zeek:field:: referrer :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the "referer" header.  The comment is deliberately
      misspelled like the standard declares, but the name used here
      is "referrer", spelled correctly.


   .. zeek:field:: version :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the version portion of the reply. If you require
      message-level detail, consider the :zeek:see:`http_request` and
      :zeek:see:`http_reply` events, which report each message's
      version string.


   .. zeek:field:: user_agent :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the User-Agent header from the client.


   .. zeek:field:: origin :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the Origin header from the client.


   .. zeek:field:: request_body_len :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Actual uncompressed content size of the data transferred from
      the client.


   .. zeek:field:: response_body_len :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Actual uncompressed content size of the data transferred from
      the server.


   .. zeek:field:: status_code :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&optional`

      Status code returned by the server.


   .. zeek:field:: status_msg :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Status message returned by the server.


   .. zeek:field:: info_code :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&optional`

      Last seen 1xx informational reply code returned by the server.


   .. zeek:field:: info_msg :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Last seen 1xx informational reply message returned by the server.


   .. zeek:field:: tags :zeek:type:`set` [:zeek:type:`HTTP::Tags`] :zeek:attr:`&log`

      A set of indicators of various attributes discovered and
      related to a particular request/response pair.


   .. zeek:field:: username :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Username if basic-auth is performed for the request.


   .. zeek:field:: password :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Password if basic-auth is performed for the request.


   .. zeek:field:: capture_password :zeek:type:`bool` :zeek:attr:`&default` = :zeek:see:`HTTP::default_capture_password` :zeek:attr:`&optional`

      Determines if the password will be captured for this request.


   .. zeek:field:: proxied :zeek:type:`set` [:zeek:type:`string`] :zeek:attr:`&log` :zeek:attr:`&optional`

      All of the headers that may indicate if the request was proxied.


   .. zeek:field:: range_request :zeek:type:`bool` :zeek:attr:`&default` = ``F`` :zeek:attr:`&optional`

      Indicates if this request can assume 206 partial content in
      response.


   .. zeek:field:: orig_fuids :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of file unique IDs.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: orig_filenames :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of filenames from the client.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: orig_mime_types :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of mime types.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: resp_fuids :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of file unique IDs.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: resp_filenames :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of filenames from the server.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: resp_mime_types :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of mime types.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: current_entity :zeek:type:`HTTP::Entity` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      The current entity.


   .. zeek:field:: orig_mime_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      Current number of MIME entities in the HTTP request message
      body.


   .. zeek:field:: resp_mime_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      Current number of MIME entities in the HTTP response message
      body.


   .. zeek:field:: client_header_names :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/header-names.zeek` is loaded)

      The vector of HTTP header names sent by the client.  No
      header values are included here, just the header names.


   .. zeek:field:: server_header_names :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/header-names.zeek` is loaded)

      The vector of HTTP header names sent by the server.  No
      header values are included here, just the header names.


   .. zeek:field:: omniture :zeek:type:`bool` :zeek:attr:`&default` = ``F`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/software-browser-plugins.zeek` is loaded)

      Indicates if the server is an omniture advertising server.


   .. zeek:field:: flash_version :zeek:type:`string` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/software-browser-plugins.zeek` is loaded)

      The unparsed Flash version, if detected.


   .. zeek:field:: cookie_vars :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&optional` :zeek:attr:`&log`

      (present if :doc:`/scripts/policy/protocols/http/var-extraction-cookies.zeek` is loaded)

      Variable names extracted from all cookies.


   .. zeek:field:: uri_vars :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&optional` :zeek:attr:`&log`

      (present if :doc:`/scripts/policy/protocols/http/var-extraction-uri.zeek` is loaded)

      Variable names from the URI.


   The record type which contains the fields of the HTTP log.

.. zeek:type:: HTTP::State
   :source-code: base/protocols/http/main.zeek 93 104

   :Type: :zeek:type:`record`


   .. zeek:field:: pending :zeek:type:`table` [:zeek:type:`count`] of :zeek:type:`HTTP::Info`

      Pending requests.


   .. zeek:field:: current_request :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Current request in the pending queue.


   .. zeek:field:: current_response :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Current response in the pending queue.


   .. zeek:field:: trans_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Track the current deepest transaction.
      This is meant to cope with missing requests
      and responses.


   Structure to maintain state for an HTTP connection with multiple
   requests and responses.

.. zeek:type:: HTTP::Tags
   :source-code: base/protocols/http/main.zeek 18 22

   :Type: :zeek:type:`enum`

      .. zeek:enum:: HTTP::EMPTY HTTP::Tags

         Placeholder.

      .. zeek:enum:: HTTP::URI_SQLI HTTP::Tags

         (present if :doc:`/scripts/policy/protocols/http/detect-sql-injection.zeek` is loaded)


         Indicator of a URI based SQL injection attack.

   Indicate a type of attack or compromise in the record to be logged.

Events
######
.. zeek:id:: HTTP::log_http
   :source-code: base/protocols/http/main.zeek 132 132

   :Type: :zeek:type:`event` (rec: :zeek:type:`HTTP::Info`)

   Event that can be handled to access the HTTP record as it is sent on
   to the logging framework.

Hooks
#####
.. zeek:id:: HTTP::finalize_http
   :source-code: base/protocols/http/main.zeek 384 396

   :Type: :zeek:type:`Conn::RemovalHook`

   HTTP finalization hook.  Remaining HTTP info may get logged when it's called.

.. zeek:id:: HTTP::log_policy
   :source-code: base/protocols/http/main.zeek 15 15

   :Type: :zeek:type:`Log::PolicyHook`



