Application Identification (AppId) Service
================================================================================
Dilbagh Chahal

Overview
================================================================================
With increasingly complex networks and growing network traffic, network administrators 
require application awareness in managing networks. An administrator may allow only 
applications that are business relevant, low bandwidth and/or deal with certain
subject matter. 

Past snort releases have supported this requirement by allowing users to
specify service meta data, urls and contents. This new preprocessor adds a
simplified single point application awareness by making a set of application
identifiers (AppId) available to Snort Rule writers. One would just specify an
appId instead of URLs and contents in a rule.

To provide network usage awareness, the new preprocessor outputs statistics to
show network bandwidth used by each application seen on network. Administrators can
monitor bandwidth usage and may decide to block applications that are wasteful.

The preprocessor enables administrators to create their own application detectors
to detect new applications. The detectors are written in Lua and interface with Snort
using a well-defined C-Lua API.

Sections:
    Dependency Requirements
    Portability
    Application Identification Configuration
    Session Application Identifiers
    AppId Rule Detection Option
    AppId Events
    AppId Usage Statistics
    Open Detector Package (ODP) Installation
    User Created Application Detectors

Dependency Requirements
================================================================================
For proper functioning of the file processing:
 
    Stream session tracking must be enabled, i.e. stream5. TCP and UDP must be
    enabled in stream5. The preprocessor requires a session tracker to keep 
    its data. 
    
    IP defragmentation should be enabled, i.e. the frag3 preprocessor should be
    enabled and configured.

    HTTP preprocessor must be enabled and configured. There are no appId specific
    configuration. The preprocessor uses HTTP preprocessor for parsing HTTP headers
    for application determination. Without HTTP preprocessor, appId preprocessor will
    identify only non-HTTP applications.

    LuaJIT version 2.0.2 must be installed on host where snort is being compiled and run. 
    Newer versions of LuaJIT are not tested for compatibility.

Portability
================================================================================
Some Operating Systems require additional steps to compile and run snort. The following
sections identify these steps.

OpenBSD/FreeBSD
    use 'gmake' for LuaJIT compilation and installation:

    Before running './configure' command for snort, set environment with the following commands
        export PKG_CONFIG=/usr/bin/pkg-config
        export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
        sudo ln -f -s  /usr/local/lib/libluajit-5.1.so.2.0.2 /usr/local/lib/libluajit-5.1.so


Application Identification Configuration
================================================================================
AppId dynamic preprocessor can be enabled during build time. The 
following options must be included in ./configure:

--enable-open-appid

The configuration name is "appid": 

 app_stats_filename appstats-unified.log, app_stats_period 60, app_detector_dir /usr/local/cisco/apps



Option                  Argument             Required  Default
app_detector_dir        <directory>          No        /usr/local/etc/appid
app_stats_filename      <filename>           No        NULL
app_stats_period        <time in seconds>    No        300 seconds
app_stats_rollover_size <disk size in bytes> No        20 MB
app_stats_rollover_time <time in seconds>    No        1 day
memcap                  <memory limit bytes> No        256MB       
debug                   <"yes">              No        disabled
dump_ports              No                   No        disabled

preprocessor appid :\
            < app_detector_dir <directory>>, \
            < app_stats_filename <file name>>, \
            < app_stats_period <time in seconds>>, \
            < app_stats_rollover_size <disk size in bytes>>, \
            < app_stats_rollover_time <time in seconds>>, \
            < memcap <memory limit in bytes>>, \
            < debug <"yes">>, \
            < dump_ports >

Options:
  < app_detector_dir >:
      specifies base path where Cisco provided detectors and application
      configuration files are installed by ODP (Open Detector Package) package.
      The package contains Lua detectors and some application metadata. Customer
      written detectors are stored in subdirectory ‘custom’ under the same base path.

  < app_stats_filename >:
      name of file. If this configuration is missing, application stats are disabled.

  < app_stats_period >:
      bucket size in seconds. Default 5 minutes.

  < app_stats_rollover_size >:
      file size which will cause file rollover. Default 20 MB.

  < app_stats_rollover_time >:
      time since file creation which will cause of rollover. Default 1 day.

  < memcap >:
      upper bound for memory used by appId internal structures. Default 32MB.

  < dump_ports >:
      prints port only detectors and information on active detectors. Used for troubleshooting.
 
  < debug >:
      Used in some old detectors for debugging.
 
Example:

   preprocessor appid : app_stats_filename appstats-unified.log, \
                app_stats_period 60, \
                app_detector_dir /usr/local/cisco/apps


Session Application Identifiers
================================================================================
There are up to four appId stored in a session as defined below:
    serviceAppId - appId associated with server side of a session. Example http server.
    clientAppId - appId associated with application on client side of a session. Example 
          Firefox. 
    payloadAppId - For services like http, this is appId associated with webserver host. 
          Example Facebook.
    miscAppId - For some encapsulated protocols, this is the highest encapsulated application.

AppId Rule Detection Option
================================================================================
Snort rules have new detection option "appid" that allows users to customize
rules to specific application in a simple manner. The option can take up to 10
application names separated by spaces, tabs, or commas. Application names in
rules are the names you will see in last column in appMapping.data file.  A
rule is considered a match if one of the appId in a rule match an appId in a
session. 

For client side packets, payloadAppid in a session is matched with all AppIds
in a rule.  Thereafter miscAppId, clientAppId and serviceAppId are matched.
Since Alert Events contain one AppId, only the first match is reported. If rule
without appId option matches, then the most specific appId (in order of
payload, misc, client, server) is reported.

The same logic is followed for server side packets with one exception. Order of
matching is changed to make serviceAppId higher then clientAppId.

Examples:
    alert tcp any any -> any any  (msg:"test for app HTTP";     appid: http;     sid:18759; rev:4; )
    alert tcp any any -> any any  (msg:"test for app FTP";      appid: ftp;      sid:18760; rev:4; )
    alert tcp any any -> any any  (msg:"test for app FTP Data"; appid: ftp-data; sid:18761; rev:4; )
    alert tcp any any -> any any  (msg:"test for app CNN zappos"; appid: cnn.com zappos;  sid:18762; rev:4; )

AppId Events
================================================================================
This feature defined new event types for logging application name in Snort
Alerts in unified2 format only. These events contain only one application name.
The Events can be enabled for unified2 output using 'appid_event_types keyword. 

For example, the following configuration will log alert in my.alert file with
application name.

    output alert_unified2: filename my.alert, appid_event_types

u2spewfoo, u2openappid, u2streamer tools can be used to print alerts in new format. 
Events will contains additional application name at the end of event.

Example: u2spewfoo output the following event
(Event)
        sensor id: 0    event id: 6     event second: 1292962302        event microsecond: 227323
        sig id: 18763   gen id: 1       revision: 4      classification: 0
        priority: 0     ip source: 98.27.88.56  ip destination: 10.4.10.79
        src port: 80    dest port: 54767        protocol: 6     impact_flag: 0  blocked: 0
        mpls label: 0   vland id: 0     policy id: 0    appid: zappos

AppId Usage Statistics
================================================================================
AppId preprocessor prints application network usage periodically in snort
log directory in unified2 format. File name, time interval for statistic and
file rollover are controlled by appId preprocessor configuration. u2spewfoo,
u2openappid, u2streamer tools can be used to print contents of these files. An
example output from u2openappid tools is as follows:

    statTime="1292962290",appName="firefox",txBytes="9395",rxBytes="77021"
    statTime="1292962290",appName="google_analytic",txBytes="2024",rxBytes="928"
    statTime="1292962290",appName="http",txBytes="28954",rxBytes="238000"
    statTime="1292962290",appName="zappos",txBytes="26930",rxBytes="237072"

Open Detector Package (ODP) Installation
================================================================================
Application detectors from Snort team will be delivered in a separate package
called Open Detector Package. ODP is a package that contains the following
artifacts:

a. Application detectors in Lua language. 

b. Port detectors, which are port only application detectors, in  meta-data in YAML format.

c. appMapping.data file containing application metadata. This file should not
   be modified.  The first column contains application identifier and second column 
   contains application name.  Other columns contain internal information.

d. Lua library files DetectorCommon.lua, flowTrackerModule.lua and
   hostServiceTrackerModule.lua 

User can install ODP package in any directory of its choosing and configure
this directory in app_detector_dir option in appId preprocessor configuration.
Installing ODP will not modify any subdirectory named custom, where
user-created detectors are located. 

When installed, ODP will create following sub-directories:
    odp/port    //Cisco port-only detectors
    odp/lua     //Cisco Lua detectors
    odp/libs    //Cisco Lua modules

User Created Application Detectors
================================================================================
Users can create new applications by coding detectors in Lua language. A
document will be posted on Snort Website with details on API. Users can also
copy over Snort team provided detectors and modify it. Users can also use the
detector creation tool described in the next section.

Users must organize their Lua detectors and libraries by creating the following
directory structure, under ODP installation directory. The root path is specified
by the "app_detector_dir" parameter of the "preprocessor appid" line in snort.conf
    custom/port    //port-only detectors
    custom/lua     //Lua detectors
    custom/libs    //Lua modules
    
For example, Snort instructions for building from scratch indicate adding to your
snort.conf a line:
preprocessor appid : app_stats_filename appstats-unified.log, app_stats_period 60, app_detector_dir /usr/local/lib/openappid

So the path to the user-created lua files would be /usr/local/lib/openappid/custom/lua/

None of the directories below /usr/local/lib/openappid/ would be added for you.
    
Application Detector Creation Tool
================================================================================
For rudimentary Lua detectors, there is a tool provided called appid_detector_builder.sh.
This is a simple, menu-driven bash script which creates .lua files in your current
directory, based on your choices and on patterns you supply.

When you launch the script, it will prompt for the Application Id that you are giving
for your detector. This is free-form ASCII with minor restrictions. The Lua detector file
will be named based on your Application Id. If the file name already exists you will be 
prompted to overwrite it.

You will also be prompted for a description of your detector to be placed in the comments
of the Lua source code. This is optional.

You will then be asked a series of questions designed to construct Lua code based
on the kind of pattern data, protocol, port(s), etc.

When complete, the Protocol menu will be changed to include the option, "Save Detector".
Instead of saving the file and exiting the script, you are allowed to give additional
criteria for another pattern which may also be incorporated in the detection scheme. Then
either pattern, when matched, will be considered a valid detection.

For example, your first choices might create an HTTP detection pattern of "example.com",
and the next set of choices would add the HTTP detection pattern of "example.uk.co" (an 
equally fictional British counterpart). They would then co-exist in the Lua detector, and
either would cause a detection with the name you give for your Application Id.

The resulting .lua file will need to be placed in the directory, "custom/lua", described
in the previous section of the README above called "User Created Application Detectors"

"make install" will place the tool, with proper privileges, in /usr/local/bin/

