Flows are preallocated at startup and stored in protocol specific caches.
FlowKey is used for quick look up in the cache hash table.

Each flow may have associated inspectors:

* clouseau is the Wizard bound to the flow to help determine the
  appropriate service inspector

* gadget is the service inspector

* data is a passive service inspector such as a client config.

FlowData is used by various inspectors to store specific data on the flow
for later use.  Any inspector may store data on the flow, not just clouseau
gadget.

FlowData reference counts the associated inspector so that the inspector
can be freed (via garbage collection) after a reload.

There are many flags that may be set on a flow to indicate session tracking
state, disposition, etc.

==== High Availability

HighAvailability (ha.cc, ha.h) serves to synchronize session state between high
availability partners.  The primary purpose is to exchange session state with the
goal of keeping the session caches in sync.  Other clients may also register
with the HA service to exchange additional session data by implementing one or
more FlowHAClient classes.

HA uses Side Channel Connectors and/or DAQ module IOCTLs to transmit and receive
HA state messages.  A full duplex Side Channel is required for Side Channel
communications.  A DAQ module that supports the IOCTLs for setting and getting
HA state is required for a functional DAQ-backed setup.  Modules that do not
support these IOCTLs will silently fail and present no HA state on future
packets in the flow.  HA state will be queried from the DAQ module prior to new
flow creation when the appropriate DAQ packet message flag has been set on the
initiating packet.

The HA subsystem exchanges two high level message types:
  - DELETE: Indicate to the partner that a session has been removed.  No
additional HA client status will be exchanged.  (Not used for DAQ-backed
storage.)
  - UPDATE: Indicate all other state changes.  The message always includes
the session state and optionally may include state from other HA clients.
By default, the update messages are incremental.  In the case of DAQ-backed
storage, the update messages are always fully formed.

The HA subsystem implements these classes:
  - HighAvailabilityManager - A collection of static elements providing the
    top-most interface to HA capabilities.
  - HAMessage - A wrapper around the actual message and includes a cursor and
    convenience functions for producer/consumer activity.  Passed around
    among all message handing classes/methods.
  - HighAvailability - If HA is enabled, instantiated in each packet thread and
    provides all primary HA functionality for the thread.  Referenced via a
    THREAD_LOCAL object pointer.
  - FlowHAState - One per flow and referenced via a pointer in the Flow object.
    Contains all dynamic state information.  A set of get'ers and set'ers are
    implemented to help manage the flags, client pending flags, and timing
    information.
  - FlowHAClient - All HA coordination is managed via a set of FlowHAClient's.
    Every function that desires to use HA capabilities defines one FlowHAClient
    object within the packet processing thread.  Each client is provided a
    handle bit-mask as part of the pending logic.  The client may set the
    pending bit to indicate the desire to add content to the HA message.
    A maximum of 17 clients may existing (numbers 1-16, corresponding to the
    16 bits of the pending flags.  Client 0 is the 'session', is always present,
    and is handled as a special case.  Client 0 is the fundamental session HA
    state sync functionality.  Other clients are optional.


09/25/2023
In response to the need for more nuanced management of different protocol
types within the `flow_cache`, we've leveraged the newly added feature in
xhash. This enhancement includes the introduction of Protocol-Based
Least Recently Used (LRU) caches, which substantially improves the handling
of multiple protocol types in the cache.
With this implementation, during various pruning sessions (idle,
excess, and timeout etc), we've adopted a round-robin method to cycle through
the LRU lists designated for each protocol type. This methodology ensures
that every protocol, irrespective of its inherent timeout characteristics,
is given an equal opportunity for pruning and retirement from the cache.
This adjustment to `flow_cache` not only addresses the previously observed
bias during pruning sessions, especially concerning UDP flows,
but lays down a framework for more advanced and controlled data management
within the cache moving forward.

10/25/2024
Allowlist LRU
To address the need for preserving flows that have been allowlisted and are
at risk of timing out, we've introduced a configurable Allowlist LRU cache
within the flow_cache. This enhancement enables the retention of flows marked
with a whitelist verdict, preventing them from being prematurely pruned due
to inactivity timeouts. This is particularly beneficial in scenarios where
Snort ceases to observe traffic for a flow after the whitelist decision,
especially if that flow is long-lived. Without this adjustment,
such flows may be pruned by the cache upon timeout, potentially impacting
event logging at the flow’s end-of-life (EOF) due to missing
pruned flow information.

The Allowlist LRU cache is disabled by default but can be enabled by adding
allowlist_cache = { enable = true } in the stream configuration.
Like the protocol-based LRUs, this allowlist functionality is an
additional LRU rather than a new hash_table, thereby maintaining
consistent performance with previous configurations.

04/07/2025
Changed the flow data hash to start with 7 hash buckets. This is trying to balance
between memory usage and speed.
The flow data hash load factor controls the when a hash table rebalance happens. We
don't ever want a hash rebalance to happen for flow data as that would mean allocating
a new bigger bucket array and rehashing all of the elements in the hash. So, a max
load factor of 5.0 was chosen. This means that the average depth for all buckets in
the hash would need to be 5 or more before a rebalance would happen. This is very
unlikely to happen.

05/06/2026
==== HA Reload

HighAvailabilityManager supports config reload. Two pieces cooperate:

  - HAReloadTuner (a ReloadSwapper registered from snort.cc::get_reload_config())
    runs first at tswap on each packet thread.  Its job is to delete and recreate
    the THREAD_LOCAL HighAvailability object so it matches the new config.

  - Each HA-using inspector (stream + stream_tcp/udp/icmp/ip and appid)
    registers its own ReloadSwapper from its configure().  These run after
    HAReloadTuner in FIFO order; each one calls its plugin's tterm()+tinit() to
    rebuild the plugin's FlowHAClient instances against the freshly created
    HighAvailability object.

During reload, InspectorManager::thread_reinit() calls each inspector's
tinit() before any ReloadSwapper::tswap() runs (see Analyzer::reinit() and
Swapper::apply()).  The plugin's per-thread HA client pointer therefore still
references the prior cycle's instance at the time tinit() fires, and the new
HighAvailability object hasn't been built yet.  Plugin tinit() must guard
against re-entry (typically: skip the new() when the pointer is non-null);
the per-plugin ReloadSwapper will tterm()+tinit() once HAReloadTuner::tswap()
has installed the new HighAvailability.  Without this guard, the prior
FlowHAClient leaks and FlowHAClient's ctor logs a spurious
"already-occupied index" error.

HighAvailabilityManager::configure() runs on the main thread and is called
unconditionally on every reload so the client-name registry is reset every
cycle. Data-path entrypoints gate only on the per-thread ha pointer, so a
reload aborted after configure() but before tswap is harmless.

FlowHAClient indices are assigned by sorting the registered names in
finalize_ha_client_indices(), called once per Snort::init() and once per
get_reload_config() (only when HA is configured).  Both peers in an HA pair
must register the same set of names so the sorted order, and therefore the
on-wire client indices, are identical.  Names are constructed by the API
(register_ha_client / FlowHAClient ctor) from the inspector's module name
plus an optional suffix; this guarantees uniqueness across plugins without
relying on convention.  Every client (including the session/stream client)
gets a regular index and handle = 1 << bit; the session client is identified
by HAClientInfo::is_session, not by any reserved handle value.  The
pending-update mask is 16 bits, so a maximum of 16 clients total are allowed.

When peers register divergent name sets the standby will hit unknown_client_idx
or client_consume_errors counters.

SideChannel reload limitation
-----------------------------
SideChannelManager is configured once at process startup (snort.cc calls
SideChannelManager::pre_config_init() and the per-thread set is built by
SideChannelManager::thread_init() during Analyzer startup); it does not
participate in reload.  HighAvailability::HighAvailability() looks up its
SideChannel via SideChannelManager::get_side_channel(port) against that
startup-time set.

Consequence: enabling HA on a reload, disabling it, or changing the HA
ports/min_session_lifetime/min_sync_interval/daq_channel settings is fully
supported, but adding, removing, or repointing a side_channel{} configuration
is not honored at reload time -- the new SideChannel definition will be
ignored and HA will continue to use (or fail to find) the startup-time
channel.  Changes to side_channel{} require a process restart.
