The Application Identification inspector (AppId) is a network inspector that inspects the packets
of a flow to discover the application (client, service, payload) that is running over
the network connection. Example applications include:

    - service: HTTP
    - client: Chrome
    - payload: Facebook

AppId inspects packets from the client and server endpoints to identify the applications on each
end of the connection.

The AppIdModule and AppIdInspector classes subclass the framework Module and Inspector classes to
provide the logic to plug this inspector into the snort framework.  AppIdModule represents the
configuration for AppId and AppIdInspector provides the packet processing context.  An AppId
inspector is instantiated for each packet thread created by the framework.

AppId registers to receive any IP packet, it does not process rebuilt packets.

AppIdModule contains all the logic to process the AppId inspector Lua configuration which is identified
by the 'appid' keyword.  This configuration includes settings for logging, statistics, etc. and also
the directory path or filename for three additional configuration items:

    - directory for the application id mapping table and Lua detectors
    - path to legacy 'RNA' configuration (appid uses a small part of this to configure monitored networks)
    - path to third party application identification configuration

The configuration syntax for these items has not been converted to use Lua so currently custom
parsing code is used to process these files.  Converting these to use Lua is on the AppId roadmap.

The application id mapping table is a key component of AppId and must be configured in order for detection
to work for rules with the 'appids' option.  The value of the 'appids' rule option is a list of the names of
the applications required for a match to that rule (the match logic is any of the named applications).
AppId uses numbers to identify applications and the mapping table defines the association of each number
to the application name.

The set of Lua detectors that AppId loads are located in the odp/lua subdirectory of the directory that
contains the mapping configuration file.

The legacy 'RNA' configuration is processed by the AppIdContext class.  This is currently not supported so
no additional details provided here at this time.  This section should be updated once this feature is
supported.

<TBD: Provide description for third party configuration>

AppId maintains three categories of statistics counters:

1. Statistics on general events such as packets processed that are known at build time.  These statistics
are defined as part of AppIdModule and use the PegCounts framework and reporting mechanism of the
ModuleManager.  See appid_module.[h|cc] for implementation of these counts.

2. Statistics on detector discovery events.  These are counts for each application that is defined in the
application id mapping table and thus discoverable by AppId.  This is not known until configuration is
processed at runtime so the data structures for the counts are built dynamically during AppId initialization.
These counts also use the PegCounts type and a custom 'print' method is provided to dump the counts when
required.  See appid_peg_counts.[h|cc] for implementation of these counts.

3. AppId periodic 'bucket' statistics.  These statistics count the amount of data processed by AppId and are
periodically dumped to file for post processing.  These statistics are legacy and are a candidate for
review to determine if they should be refactored to use the enhanced snort3 statistics capabilities.  See
appid_stats.[h|cc] for implementation of these counts.

Supporting the 'appids' option on detection rules is one of the primary features of AppId and this
support is implemented in the AppIdIpsOption class.  This is a subclass of the IpsOption with an IpsApi
defined to plug into the detection framework.  The AppIdIpsOption eval method is called for all rules that
have the 'appids' option to determine if any of the applications discovered on a flow match the rule.
The application id mapping table is required to map the numerical ids maintained by AppId to the
corresponding name that is used in IPS rules.  Since this table is loaded after the framework has
initialized all the rule options AppIdIpsOption must generate its lookup table on the first call to check
for a match.  This is not ideal but until the rule option framework supports a 'post-config' callback to
handle situations like this AppId will need this workaround.

The application discovery state for a flow is maintained in the AppIdSession object that AppId instantiates
for each flow.  This is typically done of the first packet of the flow, however for expected flows this
object may have already been allocated.  AppIdSession is a subclass of FlowData and currently is a kitchen
sink of data and methods to support discovery and report results to interested parties. Many of the member
variables are public to support access from legacy code.  Refactoring this class to improve organization
and encapsulation would be a worthy undertaking.

The application discovery process for a flow is managed in the AppIdDiscovery class or the client or
service discovery classes derived from this class.  An instance of the client and service discovery classes
is created during initialization and these classes in turn instantiate each of builtin detectors for its
category (client or service).  Detectors register with discovery the ports and patterns they are interested
in and when a flow matches the port or the packet payload matches a pattern registered then the detector is
added to the list of candidates to do more detailed inspection of the payload for the current packet.
Once the list of candidates is created each detector is dispatched in turn to examine the packet.

External detectors coded in Lua are also loading during the initialization process and these detectors use
AppId's Lua API to register themselves and the ports and patterns to match for selecting them as candidates
to inspect a flow.

<NOTE: add details for how third-party discovery fits into this process>

Application 'detectors' are the workhorses of the AppId inspector.  Detectors inspect packets for either
the server side or the client side (there are a few exceptions where a client detector may look at packets in
both directions in some scenarios).  Common behavior for both detector types is implemented in the
AppIdDetector base class.  ClientDetector and ServiceDetector subclass AppIdDetector to implement behavior
that is common to all detectors with each category.  Finally each application specific detector is a
subclass of either the client or service detector class.

When a detector is instantiated it registers the ports for which it should be dispatched when packets
arrive on that port and it also registers the set of patterns to match against the content of the packet
payloads.  The application ids the detector supports and flags indicating additional meta-data the detector
may discover are also registered.  When a detector is selected based on a port or pattern match the
discovery framework will call the 'validate' function of the detector to initiate detailed inspection of
the payload.  The detector will determine one of the following:

1. The flow is not a match for its applications.  In this case the detector is removed from the list of
candidates and discovery moves on to the next detector.

2. The flow may match but more packets must be inspected to make a final decision.  In this case the
detector remains a candidate and discovery moves on to the next detector.

3. The flow is a match.  In this case the AppIdSession is updated with the detected application id and
any meta-data that has been extracted.  If more meta-data is expected the detector will continue to be
dispatched on subsequent packets until that process completes.  Otherwise the detection process is
finished.  In either case the list of any other candidate detectors is purged.

As mentioned before, Lua detectors are client and service detectors written in Lua.
LuaClientDetector and LuaServiceDetector subclass ClientDetector and ServiceDetector respectively to
represent client and service Lua detectors.

AppId's Lua API relies on Lua States to interact with Lua detectors. Every thread has an independent state
and the detectors need to be initialized once per thread/state, whereas the C detectors, are only initialized
once in the control thread. To mimic this behavior, session information is stored in a LuaStateDescriptor
object and is associated with every detector in a thread/state. The Lua detectors are initialized once in the
control thread and are shared by all the threads/states.

During initialization and discovery there are multiple callbacks between C and Lua functions. To keep track
of the session and detector another layer of abstraction is added. The LuaObject class encapsulates a Lua
detector and a LuaStateDescriptor object, it is stored in the Lua State and is accessed by C functions
during Lua callbacks. LuaServiceObject and LuaClientObject subclass LuaObject to represent stateful (session)
LuaServiceDetectors and LuaClientDetectors.

When a Lua detector is initialized, a unique environment/table is created in the global registry of the
Lua State and a LuaObject object is created in C. And when a detector is activated, a call is made to
it's initialization function in Lua code and the corresponding LuaObject object is stored in it's local stack.
Callbacks to C functions to register ports and patterns are processed only in the control thread and
ignored in the packet processing threads.

During discovery, if a Lua detector is selected based on a port or pattern and "validate" is called,
the table corresponding to that detector is pulled from the Lua State and a call is made to the
corresponding "validate" function in Lua code. The "validate" function in Lua can in turn make callbacks
to C functions and shares its local stack with the C function. These functions make sure that the call
is made only during discovery before executing.

A custom first packet lua detector API which would map IP address, port and protocol on the very first packet to 
application protocol (service appid), client application (client appid) and web application (payload appid). 
This API is only used if a user creates a custom lua detector containing the IP, port, protocol values to be mapped to AppIDs.
The first packet API shall offer performance improvements, reinspection and early detection of the traffic. 

The values are stored in a cache during load time which are parsed in the lua detector API from the values passed
through the custom detector lua file. During runtime, for every session, the first packet API is invoked, to find any cache entries 
for the first packet. If there's an entry in the cache, the found appids are assigned accordingly, if no entries are found in the cache,
further discovery is carried out on the incoming traffic.
Here, there could be two scenarios, if the reinspection flag is enabled, discovery process is further continued and 
appids found on first packet may or may not change, else if it is disabled, the discovery is stopped at the first packet itself 
and appids remains the same for this entire session. 

LUA detectors can log messages at any point within their workflow ( e.g initialisation, deinitialisation and packet processing ) through
snort via the detector_log_snort_message ( LUA cLog ) API. This function requires 2 arguments - log_level and message.
The log_level argument tells snort what information needs to be logged. LUA_LOG_TRACE corresponds to packet-centric logging and
thus instructs snort to attempt to add packet 5-tuple information to the log messages. Control logic should use the LUA_LOG_INFO log_level to
log messages at a higher urgency and without packet information. Packet processing log should use LUA_LOG_TRACE instead, as described above. 
