Lmod Internal Overview
~~~~~~~~~~~~~~~~~~~~~~

This document provides a concrete, step-by-step walkthrough of Lmod's most common
operation: loading a module. It follows a single ``module load`` command from
user input to the final environment modification, explaining the key functions
and components involved in that specific sequence.

To illustrate this process, we will trace how Lmod handles the command
``module load foo/1.0``, assuming the ``foo/1.0.lua`` modulefile contains the
following code:

.. code-block:: lua

    setenv("Foo", "Bar")
    prepend_path("PATH", "/home/user/bin")

For a higher-level, component-based view of the architecture (the "what" and "where"), 
please first read the :doc:`443_structure` document. We recommend understanding the 
high-level components before diving into this detailed procedural walkthrough.
Also Lmod coding conventions are discussed in :doc:`444_code_conventions`.

In Lmod's simplest terms, it takes commands from the user to change the state of the user's environment.  
It does this by loading and unloading modulefiles. When Lmod takes a command, it modifies an internal
table of key value pairs.   Finally, once the command has successfully
completed, Lmod will output the table of key value pairs to stdout.

The two core internal data structures Lmod uses to manage this state are:

-  **The Module Table (MT)**: An in-memory database of all known modules and their current state (active, inactive, etc.).
-  **The Variable Table (varT)**: An in-memory representation of the shell environment being modified.

These tables are snapshotted and managed by the :doc:`FrameStk <454_framestack_deepdive>` to allow for reversible operations.
You can learn more about these core components in the :doc:`Lmod Glossary <441_lmod_glossary>`.

The output is typically written as shell commands. The choice of shell is picked
by the user. The `module` command itself is a shell function (in bash/zsh) or
alias (in tcsh/csh) that uses `eval` to apply Lmod's output to the current
shell.

For bash and zsh, the shell function ``module`` can be defined in its
simpliest terms as:

.. code-block:: shell

   module () {
     eval "$($LMOD_CMD shell "$@")"
   }

The `eval` command is the key to Lmod's ability to modify the shell's
environment. The process works in three stages:

1.  **$LMOD_CMD shell "$@"**: First, the Lmod program runs. Its job is *not*
    to change the environment itself, but to **generate plain text** to its
    standard output. This text consists of the shell commands required to make
    the desired changes (e.g., ``export FOO=Bar;`` or ``unset
    PATH;``). Note that if Lmod sees **shell** as its first arguemnt
    it figures out what shell the user is running.
2.  **$(...)**: The shell's command substitution syntax captures this plain
    text output from the Lmod process.
3.  **eval "..."**: Finally, `eval` takes the captured string of commands
    and executes it in the context of the *current* shell. This is what allows
    Lmod, an external program, to define variables, aliases, and functions in
    your interactive session.

For tcsh/csh the shell alias is::

   alias module 'eval `$LMOD_CMD tcsh \!*`'

In all cases the second argument to `lmod` controls how the key value pairs are
expressed.  For bash/zsh **Foo => Bar** gets expanded to::

   Foo=Bar
   export Foo

And for tcsh/csh it gets expanded to::

   setenv Foo Bar

For the rest of this discussion, we will concentrate on what the Lmod
program does assuming that the output will be for bash/zsh users and
ignoring the evaluation step.

Internal Steps
--------------

The following steps trace the execution of the command **module load foo/1.0**.

#. The user issues the command **module load foo/1.0**
#. Lmod decides what command to run by using the second argument
   (namely **load**) and converts the word into a command.  It does
   this by searching the **lmodCmdA** table in **src/lmod.in.lua** for the
   user command (**load**).  The **lmodCmdA** table matches action
   value as **loadTbl** and the **loadTbl** has the comand
   **Load_Usr()** which is a function in **src/cmdfuncs.lua**
#. The function **Load_Usr()** calls the local function
   **l_usrLoad()** with the **check_must_load** set to true.  This
   means that Lmod will check to see if all the requested modulefiles
   were successfully loaded at the end.
#. The function **l_usrLoad()** takes the remaining arguments and
   breaks them down into two lists.  Any modules that lead with a
   minus are added to the unload list.  All other modules are added to
   the load list.
#. The **l_usrLoad** function converts the command line argument
   **foo/1.0** to an **MName** object. An :doc:`MName <441_lmod_glossary>` is Lmod's internal representation
   of a module, encapsulating its name, version, and the logic to find its file path.
   The complex details of this name-to-path resolution process are found in the
   :doc:`MName Resolution Deep Dive <450_mname_resolution_deepdive>`.
#. The module is ready to start the loading process. It uses a derived
   object called **mcp** (short for main control program, a nod to the
   movie Tron). The :doc:`mcp <441_lmod_glossary>` is Lmod's central conductor; it knows the current
   context (e.g., 'loading' vs. 'unloading') and dictates how modulefile commands
   should be interpreted. How this works is discussed in the :doc:`MCP Deep Dive <451_mcp_deepdive>`.
   In our case, the **mcp:load_usr(lA)** calls **M.load_usr()** in
   **src/MainControl.lua**.  After telling Lmod to register the list
   of loaded module, Lmod then calls **M.load()** still in
   **src/MainControl.lua** 
#. The function **M.load()** builds a hub singleton and calls
   **hub:load()** with the list of MName objects to load.  Note that a
   user might request more than one module to load.
#. The **M.load()** is found in **src/Hub.lua**.  Here Lmod has
   implemented many of its rules.  For example this routine checks to
   see if there is another "Foo" module loaded.  In that case the old
   Foo module is unloaded and the new one then loaded.  It check for 
   downstream conflicts.  Assuming that all is well, then the routine
   **loadModuleFile()** is called.
#. The function **loadModuleFile()** is found in **src/loadModuleFile.lua**
   This routine reads in the entire contents of the modulefile.  If
   the modulefile is a TCL module, then the conversion from TCL to
   Lua is done here with the **runTCLprog()** routine. Finally it
   takes the contents of the modulefile which in all cases is now a
   lua program and calls **sandbox_run()** to evaluate the modulefile.
#. The **sandbox_run()** routine is an interesting feature of Lua.  It
   allows Lmod to call the Lua interpreter and control what functions
   are available.  In particular, modulefiles can only call certain
   Lmod functions like **setenv()** and **prepend_path()** but not
   other internal Lmod functions. It also allows Lmod to capture any
   syntax or other errors that a modulefile might have. The sandbox mechanism
   is explained in detail in the :doc:`Sandbox Deep Dive <452_sandbox_deepdive>`.
#. Once the **sandbox_run()** function is called.  It is now Lua that
   controls the evaluation of the modulefile.  The only time that Lmod
   has control is when a function implemented in Lmod like
   **setenv()** or **prepend_path()** is called.  Any other Lua code
   inside a module is evaluated by Lua.
#. After all modulefile have been loaded, Lmod checks that all
   registered modules have been loaded. 
#. Finally, if there are no errors, Lmod then takes the internal key
   value pairs and output that text in the requested style, such as
   bash as text which is then evaluated by the shell function or shell
   alias.  This only happens for values that have changed.

Visual Summary of Internal Steps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following flowchart provides a high-level summary of the process described above.

.. code-block:: text

    User Shell: "module load foo"
           |
           v
    Lmod Process:
    1. Parse Command (`lmod.in.lua`)
    2. Create `MName` for "foo"
    3. Build an `mcp` object to orchestrate load
    4. `Hub` applies rules (conflicts, etc.)
    5. `loadModuleFile` reads file
    6. `sandbox` executes module code using the generated `mcp`
    7. Update internal state (`VarT`, `MT`)
    8. Generate shell code string (e.g., "export FOO=Bar;")
           |
           v
    User Shell: `eval` executes the string

Steps to evaluate a modulefile
------------------------------

The above steps show how Lmod takes a module file, evaluates it and
generates the output text.  In this section the steps necessary to
evaluate the module are discussed here.  Here we discuss how the line
**setenv("Foo", "Bar")** is evaluated.

#. Lua finds the function **setenv()** from the modulefile and calls
   this function in **src/modfuncs.lua**.
#. The **setenv()** function has to figure out what action it is
   supposed to take. For example this modulefile could be loading, in
   that case it calls **M.setenv()** in **src/MainControl.lua**. But
   if Lmod is unloading the module then **M.unsetenv()** is called.
   This is controlled by **mcp**.  See the :doc:`MCP Deep Dive <451_mcp_deepdive>` for more
   details.
#. The function **M.setenv()** store the name of the environment
   variable as the key and the next command line argument as the
   value.  In this case the key is "Foo" and the value is "Bar".  This
   key value pair is stored in the **varT** table. See the :doc:`varT Deep Dive <456_vart_deepdive>` for details.

The evaluation of **prepend_path("PATH","/home/user/bin")** works
similarly.

#. Lua finds the function **prepend_path()** from the modulefile and calls
   this function in **src/modfuncs.lua**.
#. The **prepend_path()** function has to figure out what action it is
   supposed to take. For example this modulefile could be loading, in
   that case it calls **M.prepend_path()** in **src/MainControl.lua**. But
   if Lmod is unloading the module then **M.remove_path()** is called.
   This is controlled by **mcp**.  See the :doc:`MCP Deep Dive <451_mcp_deepdive>` for more
   details.
#. The function **M.prepend_path()** store the name of the environment
   variable as the key and the next command line argument as the
   value.  In this case the key is "PATH" and "/home/user/bin" is
   prepended to "PATH".  These changes to the  key value pair is
   stored in the **varT** table.

Summary
-------

As we have seen, a single `module load` command initiates a chain of events:
parsing the user's request, resolving a module name to a file (:doc:`MName <441_lmod_glossary>`),
orchestrating the operation based on context (:doc:`mcp <441_lmod_glossary>`), enforcing loading rules
like conflict detection (:doc:`Hub <441_lmod_glossary>`), and finally evaluating the modulefile in a
secure `sandbox`. The entire process culminates in Lmod generating a string of
shell commands, which the user's shell then executes via `eval` to modify its
own environment.

