    configuration information, the nodes to be managed, information about
    how those nodes are grouped into partitions, and various scheduling
    parameters associated with those partitions. This file should be
    consistent across all nodes in the cluster.
    .LP
    The file location can be modified at execution time by setting the SLURM_CONF
    environment variable. The Slurm daemons also allow you to override
    both the built\-in and environment\-provided location using the "\-f"
    option on the command line.
    .LP
    The contents of the file are case insensitive except for the names of nodes
    and partitions. Any text following a "#" in the configuration file is treated
    as a comment through the end of that line.
    Changes to the configuration file take effect upon restart of
    Slurm daemons, daemon receipt of the SIGHUP signal, or execution
    of the command "scontrol reconfigure" unless otherwise noted.
    Changes to TCP listening settings will require a daemon restart.
    .LP
    If a line begins with the word "Include" followed by whitespace
    and then a file name, that file will be included inline with the current
    configuration file. For large or complex systems, multiple configuration files
    may prove easier to manage and enable reuse of some files (See INCLUDE
    MODIFIERS for more details).
    .LP
    Note on file permissions:
    .LP
    The \fIslurm.conf\fR file must be readable by all users of Slurm, since it
    is used by many of the Slurm commands.  Other files that are defined
    in the \fIslurm.conf\fR file, such as log files and job accounting files,
    may need to be created/owned by the user "SlurmUser" to be successfully
    accessed.  Use the "chown" and "chmod" commands to set the ownership
    and permissions appropriately.
    See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
    about the various files and directories used by Slurm.

    .SH "PARAMETERS"
    .LP
    The overall configuration parameters available include:

    .TP
    \fBAccountingStorageBackupHost\fR
    The name of the backup machine hosting the accounting storage database.
    If used with the accounting_storage/slurmdbd plugin, this is where the backup
    slurmdbd would be running.
    Only used with systems using SlurmDBD, ignored otherwise.
    .IP

    .TP
    \fBAccountingStorageEnforce\fR
    This controls what level of association\-based enforcement to impose
    on job submissions.  Valid options are any combination of
    \fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR,
    corresponding association exists in the system.  If \fIlimits\fR are
    enforced, users can be limited by association to whatever job size or run
    time limits are defined.

    If \fInojobs\fR is set, Slurm will not account for any jobs or steps on the
    system. Likewise, if \fInosteps\fR is set, Slurm will not account for any
    steps that have run.

    If \fIsafe\fR is enforced, a job will only be launched against an association
    or qos that has a TRES\-minutes limit set, if the job will be able to
    run to completion. Without this option set, jobs will be launched as long as
    their usage hasn't reached the TRES\-minutes limit. This can lead to jobs being
    launched but then killed when the limit is reached.
    With the 'safe' option set, a job won't be killed due to limits,
    even if the limits are changed after the job was started and the
    association or qos violates the updated limits.

    With \fIqos\fR and/or \fIwckeys\fR enforced jobs will not be scheduled unless
    a valid qos and/or workload characterization key is specified.
    .IP

    .TP
    \fBAccountingStorageExternalHost\fR
    A comma\-separated list of external slurmdbds (<host/ip>[:port][,...]) to
    register with. If no port is given, the \fBAccountingStoragePort\fR will be
    used.

    This allows clusters registered with the external slurmdbd to communicate with
    each other using the \fI\-\-cluster/\-M\fR client command options.

    The cluster will add itself to the external slurmdbd if it doesn't exist. If a
    non\-external cluster already exists on the external slurmdbd, the slurmctld
    will ignore registering to the external slurmdbd.
    .IP

    .TP
    \fBAccountingStorageHost\fR
    The name of the machine hosting the accounting storage database.
    Only used with systems using SlurmDBD, ignored otherwise.
    .IP

    .TP
    \fBAccountingStorageParameters\fR
    Comma\-separated list of key\-value pair parameters. Currently
    supported values include options to establish a secure connection to the
    database:
    .IP
    .RS
    .TP 2
    \fBSSL_CERT\fR
    The path name of the client public key certificate file.
    .IP
    .IP

    .TP
    \fBSSL_CIPHER\fR
    The list of permissible ciphers for SSL encryption.
    .RE
    .IP

    .TP
    \fBAccountingStoragePass\fR
    The password used to gain access to the database to store the
    accounting data.  Only used for database type storage plugins, ignored
    otherwise.  In the case of Slurm DBD (Database Daemon) with MUNGE
    authentication this can be configured to use a MUNGE daemon
    specifically configured to provide authentication between clusters
    while the default MUNGE daemon provides authentication within a
    cluster.  In that case, \fBAccountingStoragePass\fR should specify the
    named port to be used for communications with the alternate MUNGE
    daemon (e.g.  "/var/run/munge/global.socket.2"). The default value is
    NULL.
    .IP

    .TP
    \fBAccountingStoragePort\fR
    The listening port of the accounting storage database server.
    Only used for database type storage plugins, ignored otherwise.
    The default value is SLURMDBD_PORT as established at system
    build time. If no value is explicitly specified, it will be set to 6819.
    This value must be equal to the \fBDbdPort\fR parameter in the
    slurmdbd.conf file.
    .IP

    .TP
    \fBAccountingStorageTRES\fR
    Comma\-separated list of resources you wish to track on the cluster.
    These are the resources requested by the sbatch/srun job when it
    is submitted. Currently this consists of any GRES, BB (burst buffer) or
    license along with CPU, Memory, Node, Energy, FS/[Disk|Lustre], IC/OFED, Pages,
    and VMem. By default Billing, CPU, Energy, Memory, Node, FS/Disk, Pages and VMem
    are tracked. These default TRES cannot be disabled, but only appended to.
    AccountingStorageTRES=gres/craynetwork,license/iop1
    will track billing, cpu, energy, memory, nodes, fs/disk, pages and vmem along
    with a gres called craynetwork as well as a license called iop1. Whenever these
    resources are used on the cluster they are recorded. The TRES are automatically
    set up in the database on the start of the slurmctld.

    If multiple GRES of different types are tracked (e.g. GPUs of different types),
    then job requests with matching type specifications will be recorded.
    Given a configuration of
    "AccountingStorageTRES=gres/gpu,gres/gpu:tesla,gres/gpu:volta"
    Then "gres/gpu:tesla" and "gres/gpu:volta" will track only jobs that explicitly
    request those two GPU types, while "gres/gpu" will track allocated GPUs of any
    gres/gpumem and gres/gpuutil can be set individually when gres/gpu is not set.
    .IP

    .TP
    \fBAccountingStorageType\fR
    The accounting storage mechanism type.  Acceptable values at
    present "accounting_storage/slurmdbd".
    The "accounting_storage/slurmdbd" value indicates that accounting records
    will be written to the Slurm DBD, which manages an underlying MySQL
    database. See "man slurmdbd" for more information.  When this is not set it
    indicates that account records are not maintained.
    .IP

    .TP
    \fBAccountingStorageUser\fR
    The user account for accessing the accounting storage database.
    Only used for database type storage plugins, ignored otherwise.
    .IP

    .TP
    \fBAccountingStoreFlags\fR
    Comma separated list used to tell the slurmctld to store extra fields that may
    be more heavy weight than the normal job information.
    .IP
    .RS
    .TP
    Current options are:
    .IP

    .TP
    \fBjob_comment\fR
    Include the job's comment field in the job complete message sent to the Accounting Storage database.
    Note the AdminComment and SystemComment are always recorded in the database.
    .IP

    .TP
    \fBjob_env\fR
    Include a batch job's environment variables used at job submission in the job
    start message sent to the Accounting Storage database.
    .IP

    .TP
    \fBjob_extra\fR
    Include the job's extra field in the job complete message sent to the Accounting
    Storage database.
    .IP

    .TP
    \fBjob_script\fR
    Include the job's batch script in the job start message sent to the Accounting Storage database.
    .RE
    .IP
    .TP
    \fBAcctGatherEnergyType\fR
    Identifies the plugin to be used for energy consumption accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    energy consumption data for jobs and nodes. The collection of energy
    consumption data takes place on the node level, hence only in case of exclusive
    job allocation the energy consumption measurements will reflect the job's
    real consumption. In case of node sharing between jobs the reported consumed
    energy per job (through sstat or sacct) will not reflect the real energy
    consumed by the jobs. Default is nothing is collected.

    Configurable values at present are:
    .IP
    .RS
    .TP 20
    \fBacct_gather_energy/gpu\fR
    Energy consumption data is collected from the GPU management library (e.g. rsmi)
    for the corresponding type of GPU. Only available for rsmi at present.
    .IP

    .TP
    \fBacct_gather_energy/ipmi\fR
    Energy consumption data is collected from the Baseboard Management Controller
    (BMC) using the Intelligent Platform Management Interface (IPMI).
    .IP

    .TP
    \fBacct_gather_energy/pm_counters\fR
    Energy consumption data is collected from the Baseboard Management
    Controller (BMC) for HPE Cray systems.
    .IP

    .TP
    \fBacct_gather_energy/rapl\fR
    Energy consumption data is collected from hardware sensors using the Running
    Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
    execution of the command "sudo modprobe msr".
    .IP

    .TP
    \fBacct_gather_energy/xcc\fR
    Energy consumption data is collected from the Lenovo SD650 XClarity Controller
    (XCC) using IPMI OEM raw commands.
    .RE
    .IP

    .TP
    \fBAcctGatherInterconnectType\fR
    Identifies the plugin to be used for interconnect network traffic accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    network traffic data for jobs and nodes.
    The collection of network traffic data takes place on the node level,
    .IP

    .TP
    \fBacct_gather_interconnect/sysfs\fR
    Network traffic statistics are collected from the Linux sysfs
    pseudo\-filesystem for specific interfaces defined in
    \fBacct_gather.conf\fR(5).
    In order to account for per job network traffic, add the "ic/sysfs" TRES to
    \fIAccountingStorageTRES\fR.
    .RE
    .IP

    .TP
    \fBAcctGatherFilesystemType\fR
    Identifies the plugin to be used for filesystem traffic accounting.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    filesystem traffic data for jobs and nodes.
    The collection of filesystem traffic data takes place on the node level,
    hence only in case of exclusive job allocation the collected values will
    reflect the job's real traffic. In case of node sharing between jobs the reported
    filesystem traffic per job (through sstat or sacct) will not reflect the real
    filesystem traffic by the jobs.

    Configurable values at present are:
    .IP
    .RS
    .TP 20
    \fBacct_gather_filesystem/lustre\fR
    Lustre filesystem traffic data are collected from the counters found in
    /proc/fs/lustre/.
    In order to account for per job lustre traffic, add the "fs/lustre" TRES to
    \fIAccountingStorageTRES\fR.
    .RE
    .IP

    .TP
    \fBAcctGatherProfileType\fR
    Identifies the plugin to be used for detailed job profiling.
    The jobacct_gather plugin and slurmd daemon call this plugin to collect
    detailed data such as I/O counts, memory usage, or energy consumption for jobs
    and nodes. There are interfaces in this plugin to collect data as step start
    and completion, task start and completion, and at the account gather
    frequency. The data collected at the node level is related to jobs only in
    case of exclusive job allocation.

    Configurable values at present are:
    .IP
    .RS
    .TP 20
    \fBacct_gather_profile/hdf5\fR
    This enables the HDF5 plugin. The directory where the profile files
    If set to "YES", Slurm allows individual jobs to override node's configured
    CoreSpecCount value. For a job to take advantage of this feature,
    a command line option of \-\-core\-spec must be specified.  The default
    value for this option is "YES" for Cray systems and "NO" for other system types.
    .IP

    .TP
    \fBAuthAltTypes\fR
    Comma\-separated list of alternative authentication plugins that the slurmctld
    will permit for communication. Acceptable values at present include
    \fIauth/jwt\fR.

    \fBNOTE\fR: \fIauth/jwt\fR requires a jwt_hs256.key to be populated in the
    \fBStateSaveLocation\fR directory for \fBslurmctld\fR only. The jwt_hs256.key
    should only be visible to the SlurmUser and root. It is not suggested to place
    the jwt_hs256.key on any nodes but the controller running \fBslurmctld\fR.
    \fIauth/jwt\fR can be activated by the presence of the \fISLURM_JWT\fR
    environment variable.  When activated, it will override the default
    \fBAuthType\fR.
    .IP

    .TP
    \fBAuthAltParameters\fR
    Used to define alternative authentication plugins options. Multiple options may
    be comma separated.
    .IP
    .RS
    .TP 15
    \fBdisable_token_creation\fR
    Disable "scontrol token" use by non\-SlurmUser accounts.
    .TP
    \fBmax_token_lifespan\fR=<seconds>
    Set max lifespan (in seconds) for any token generated for user accounts. Limit
    applies to all users except SlurmUser. Sites wishing to have per user limits
    should generate tokens using JWT\-compatible tools, and\/or an authenticating
    proxy, instead of using \fIscontrol token\fR.
    .IP

    .TP
    \fBjwks=\fR
    Absolute path to JWKS file. Key should be owned by SlurmUser or root, must be
    readable by SlurmUser, with suggested permissions of 0400. It must not be
    writable by 'other'.
    Only RS256 keys are supported, although other key types may be listed in the
    file. If set, no HS256 key will be loaded by default (and token generation is
    disabled), although the jwt_key setting may be used to explicitly re\-enable
    HS256 key use (and token generation).
    .IP

    .TP
    \fBjwt_key=\fR
    Absolute path to JWT key file. Key must be HS256. Key should be owned by

    .TP
    \fBAuthInfo\fR
    Additional information to be used for authentication of communications
    between the Slurm daemons (slurmctld and slurmd) and the Slurm
    clients.  The interpretation of this option is specific to the
    configured \fBAuthType\fR.
    Multiple options may be specified in a comma\-delimited list.
    If not specified, the default authentication information will be used.
    .IP
    .RS
    .TP 14
    \fBcred_expire\fR
    Default job step credential lifetime, in seconds (e.g. "cred_expire=1200").
    It must be sufficiently long enough to load user environment, run prolog,
    deal with the slurmd getting paged out of memory, etc.
    This also controls how long a requeued job must wait before starting again.
    The default value is 120 seconds.
    .IP

    .TP
    \fBsocket\fR
    Path name to a MUNGE daemon socket to use
    (e.g. "socket=/var/run/munge/munge.socket.2").
    The default value is "/var/run/munge/munge.socket.2".
    Used by \fIauth/munge\fR and \fIcred/munge\fR.
    .IP

    .TP
    \fBttl\fR
    Credential lifetime, in seconds (e.g. "ttl=300").
    The default value is dependent upon the MUNGE installation, but is typically
    300 seconds.
    .IP

    .TP
    \fBuse_client_ids\fR
    Allow the \fIauth/slurm\fR plugin to authenticate users without relying on
    the user information from LDAP or the operating system.
    .RE
    .IP

    .TP
    \fBAuthType\fR
    The authentication method for communications between Slurm
    components.
    All Slurm daemons and commands must be terminated prior to changing
    the value of \fBAuthType\fR and later restarted.
    Changes to this value will interrupt outstanding job steps and prevent them
    from completing.
    Acceptable values at present:
    .RS
    Deprecated option, see \fBSlurmctldHost\fR.
    .IP

    .TP
    \fBBackupController\fR
    Deprecated option, see \fBSlurmctldHost\fR.

    The backup controller recovers state information from the
    \fBStateSaveLocation\fR directory, which must be readable and writable from both
    the primary and backup controllers.
    While not essential, it is recommended that you specify a backup controller.
    See  the \fBRELOCATING CONTROLLERS\fR section if you change this.
    .IP

    .TP
    \fBBatchStartTimeout\fR
    The maximum time (in seconds) that a batch job is permitted for
    launching before being considered missing and releasing the
    allocation. The default value is 10 (seconds). Larger values may be
    required if more time is required to execute the \fBProlog\fR, load
    user environment variables, or if the slurmd daemon gets paged from memory.
    .br
    .br
    \fBNOTE\fR: The test for a job being successfully launched is only performed when
    the Slurm daemon on the compute node registers state with the slurmctld daemon
    on the head node, which happens fairly rarely.
    Therefore a job will not necessarily be terminated if its start time exceeds
    \fBBatchStartTimeout\fR.
    This configuration parameter is also applied to launch tasks and avoid aborting
    \fBsrun\fR commands due to long running \fBProlog\fR scripts.
    .IP

    .TP
    \fBBcastExclude\fR
    Comma\-separated list of absolute directory paths to be excluded when
    autodetecting and broadcasting executable shared object dependencies through
    \fBsbcast\fR or \fBsrun \-\-bcast\fR. The keyword "\fInone\fR" can be used to
    indicate that no directory paths should be excluded. The default value is
    "\fI/lib,/usr/lib,/lib64,/usr/lib64\fR". This option can be overridden by
    \fBsbcast \-\-exclude\fR and \fBsrun \-\-bcast\-exclude\fR.
    .IP

    .TP
    \fBBcastParameters\fR
    Controls sbcast and srun \-\-bcast behavior. Multiple options can be specified
    in a comma separated list.
    Supported values include:
    .IP
    .RS
    .TP 15
    \fBDestDir=\fR
    Destination directory for file being broadcast to allocated compute nodes.
    dependencies to allocated compute nodes. The files are placed in a directory
    alongside the executable. For \fBsrun\fR only, the \fBLD_LIBRARY_PATH\fR is
    automatically updated to include this cache directory as well.
    This can be overridden with either \fBsbcast\fR or \fBsrun\fR
    \fB\-\-send\-libs\fR option. By default this is disabled.
    .RE
    .IP

    .TP
    \fBBurstBufferType\fR
    The plugin used to manage burst buffers. Acceptable values at present are:
    .IP
    .RS
    .TP
    \fBburst_buffer/datawarp\fR
    Use Cray DataWarp API to provide burst buffer functionality.
    .IP

    .TP
    \fBburst_buffer/lua\fR
    This plugin provides hooks to an API that is defined by a Lua script. This
    plugin was developed to provide system administrators with a way to do any task
    (not only file staging) at different points in a job's life cycle.
    .IP

    .TP
    \fBburst_buffer/none\fR
    .RE
    .IP

    .TP
    \fBCliFilterPlugins\fR
    A comma\-delimited list of command line interface option filter/modification
    plugins. The specified plugins will be executed in the order listed.
    No cli_filter plugins are used by default. Acceptable values at present are:
    .IP
    .RS
    .TP
    \fBcli_filter/lua\fR
    This plugin allows you to write your own implementation of a cli_filter
    using lua.
    .IP

    .TP
    \fBcli_filter/syslog\fR
    This plugin enables logging of job submission activities performed. All the
    salloc/sbatch/srun options are logged to syslog together with environment
    variables in JSON format. If the plugin is not the last one in the list it may
    log values different than what was actually sent to slurmctld.
    .IP

    .TP
    .TP
    \fBClusterName\fR
    The name by which this Slurm managed cluster is known in the
    accounting database.  This is needed distinguish accounting records
    when multiple clusters report to the same database. Because of limitations
    in some databases, any upper case letters in the name will be silently mapped
    to lower case. In order to avoid confusion, it is recommended that the name
    be lower case. The cluster name must be 40 characters or less in order to
    comply with the limit on the maximum length for table names in MySQL/MariaDB.
    .IP

    .TP
    \fBCommunicationParameters\fR
    Comma\-separated options identifying communication options.
    .IP
    .RS
    .TP 15
    \fBblock_null_hash\fR
    Require all Slurm authentication tokens to include a newer (20.11.9 and
    21.08.8) payload that provides an additional layer of security against
    credential replay attacks.  This option should only be enabled once all Slurm
    daemons have been upgraded to 20.11.9/21.08.8 or newer, and all jobs that were
    started before the upgrade have been completed.
    .TP
    \fBCheckGhalQuiesce\fR
    Used specifically on a Cray using an Aries Ghal interconnect.  This will check
    to see if the system is quiescing when sending a message, and if so, we wait
    until it is done before sending.
    .IP

    .TP
    \fBDisableIPv4\fR
    Disable IPv4 only operation for all slurm daemons (except slurmdbd). This
    should also be set in your \fBslurmdbd.conf\fR file.
    .IP

    .TP
    \fBEnableIPv6\fR
    Enable using IPv6 addresses for all slurm daemons (except slurmdbd). When
    using both IPv4 and IPv6, address family preferences will be based on your
    /etc/gai.conf file. This should also be set in your \fBslurmdbd.conf\fR file.
    .IP

    .TP
    \fBgetnameinfo_cache_timeout\fR
    When munge is used as AuthType slurmctld makes use of getnameinfo to obtain
    the hostname from IP address stored in munge credential. This parameter controls
    the number of seconds slurmctld should keep the IP to hostname resolution. When
    set to 0 cache is disabled. The default value is 60.
    .IP

    .TP
    Specifies how long sockets communications used between the srun command and its
    slurmstepd process are kept alive after disconnect. Longer values can be used
    to improve reliability of communications in the event of network failures.
    .IP

    .TP
    \fBNoCtldInAddrAny\fR
    Used to directly bind to the address of what the node resolves to running
    the slurmctld instead of binding messages to any address on the node,
    which is the default.
    .IP

    .TP
    \fBNoInAddrAny\fR
    Used to directly bind to the address of what the node resolves to instead
    of binding messages to any address on the node which is the default.
    This option is for all daemons/clients except for the slurmctld.
    .RE
    .IP

    .TP
    \fBCompleteWait\fR
    The time to wait, in seconds, when any job is in the COMPLETING state
    before any additional jobs are scheduled. This is to attempt to keep jobs on
    nodes that were recently in use, with the goal of preventing fragmentation.
    If set to zero, pending jobs will be started as soon as possible.
    Since a COMPLETING job's resources are released for use by other
    jobs as soon as the \fBEpilog\fR completes on each individual node,
    this can result in very fragmented resource allocations.
    To provide jobs with the minimum response time, a value of zero is
    recommended (no waiting).
    To minimize fragmentation of resources, a value equal to \fBKillWait\fR
    plus two is recommended.
    In that case, setting \fBKillWait\fR to a small value may be beneficial.
    The default value of \fBCompleteWait\fR is zero seconds.
    The value may not exceed 65533.

    \fBNOTE\fR: Setting \fBreduce_completing_frag\fR affects the behavior
    of \fBCompleteWait\fR.
    .IP

    .TP
    \fBControlAddr\fR
    Deprecated option, see \fBSlurmctldHost\fR.
    .IP

    .TP
    \fBControlMachine\fR
    Deprecated option, see \fBSlurmctldHost\fR.
    .IP

    .TP
    explicitly set with the \-\-cpu\-freq option. Acceptable values at present
    include one of the following governors:
    .IP
    .RS
    .TP 14
    \fBConservative\fR
    attempts to use the Conservative CPU governor
    .IP

    .TP
    \fBOnDemand\fR
    attempts to use the OnDemand CPU governor
    .IP

    .TP
    \fBPerformance\fR
    attempts to use the Performance CPU governor
    .IP

    .TP
    \fBPowerSave\fR
    attempts to use the PowerSave CPU governor
    .TP
    Default: Use system default. No attempt to set the governor is made if
    \-\-cpu\-freq option has not been specified.
    .RE
    .IP

    .TP
    \fBCpuFreqGovernors\fR
    List of CPU frequency governors allowed to be set with the salloc, sbatch, or
    srun option \-\-cpu\-freq.
    Acceptable values at present include:
    .IP
    .RS
    .TP 14
    \fBConservative\fR
    attempts to use the Conservative CPU governor
    .IP

    .TP
    \fBOnDemand\fR
    attempts to use the OnDemand CPU governor (a default value)
    .IP

    .TP
    \fBPerformance\fR
    attempts to use the Performance CPU governor (a default value)
    .IP

    .TP
    \fBPowerSave\fR
    .IP

    .TP
    \fBCredType\fR
    The cryptographic signature tool to be used in the creation of
    job step credentials.
    Acceptable values at present are:
    .RS
    .TP
    \fBcred/munge\fR
    Indicates that Munge is to be used (default).
    .IP

    .TP
    \fBcred/slurm\fR
    Use Slurm's internal credential format.
    .RE
    .IP

    .TP
    \fBDebugFlags\fR
    Defines specific subsystems which should provide more detailed event logging.
    Multiple subsystems can be specified with comma separators.
    Most DebugFlags will result in additional logging messages for the identified
    subsystems if \fBSlurmctldDebug\fR is at 'verbose' or higher.
    More logging may impact performance.

    \fBNOTE\fR: You can also set debug flags by having the \fBSLURM_DEBUG_FLAGS\fR
    environment variable defined with the desired flags when the process (client
    command, daemon, etc.) is started.
    The environment variable takes precedence over the setting in the slurm.conf.

    Valid subsystems available include:
    .IP
    .RS
    .TP 17
    \fBAccrue\fR
    Accrue counters accounting details
    .IP

    .TP
    \fBAgent\fR
    RPC agents (outgoing RPCs from Slurm daemons)
    .IP

    .TP
    \fBBackfill\fR
    Backfill scheduler details
    .IP

    .TP
    \fBBackfillMap\fR

    .TP
    \fBCPU_Bind\fR
    CPU binding details for jobs and steps
    .IP

    .TP
    \fBCpuFrequency\fR
    Cpu frequency details for jobs and steps using the \-\-cpu\-freq option.
    .IP

    .TP
    \fBData\fR
    Generic data structure details.
    .IP

    .TP
    \fBDependency\fR
    Job dependency debug info
    .IP

    .TP
    \fBElasticsearch\fR
    Elasticsearch debug info (deprecated). Alias of \fBJobComp\fR.
    .IP

    .TP
    \fBEnergy\fR
    AcctGatherEnergy debug info
    .IP

    .TP
    \fBFederation\fR
    Federation scheduling debug info
    .IP

    .TP
    \fBFrontEnd\fR
    Front end node details
    .IP

    .TP
    \fBGres\fR
    Generic resource details
    .IP

    .TP
    \fBHetjob\fR
    Heterogeneous job details
    .IP

    .TP
    .TP
    \fBJobComp\fR
    Job Completion plugin details
    .IP

    .TP
    \fBJobContainer\fR
    Job container plugin details
    .IP

    .TP
    \fBLicense\fR
    License management details
    .IP

    .TP
    \fBNetwork\fR
    Network details. \fBWarning\fR: activating this flag may cause logging of
    passwords, tokens or other authentication credentials.
    .IP

    .TP
    \fBNetworkRaw\fR
    Dump raw hex values of key Network communications. \fBWarning\fR: This flag
    will cause very verbose logs and may cause logging of passwords, tokens or
    other authentication credentials.
    .IP

    .TP
    \fBNodeFeatures\fR
    Node Features plugin debug info
    .IP

    .TP
    \fBNO_CONF_HASH\fR
    Do not log when the slurm.conf files differ between Slurm daemons
    .IP

    .TP
    \fBPower\fR
    Power management plugin and power save (suspend/resume programs) details
    .IP

    .TP
    \fBPriority\fR
    Job prioritization
    .IP

    .TP
    \fBProfile\fR
    AcctGatherProfile plugins details
    .IP
    .IP

    .TP
    \fBScript\fR
    Debug info regarding the process that runs slurmctld scripts such as
    PrologSlurmctld and EpilogSlurmctld
    .IP

    .TP
    \fBSelectType\fR
    Resource selection plugin
    .IP

    .TP
    \fBSteps\fR
    Slurmctld resource allocation for job steps
    .IP

    .TP
    \fBSwitch\fR
    Switch plugin
    .IP

    .TP
    \fBTimeCray\fR
    Timing of Cray APIs
    .IP

    .TP
    \fBTraceJobs\fR
    Trace jobs in slurmctld. It will print detailed job information
    including state, job ids and allocated nodes counter.
    .IP

    .TP
    \fBTriggers\fR
    Slurmctld triggers
    .IP

    .TP
    \fBWorkQueue\fR
    Work Queue details
    .RE
    .IP

    .TP
    \fBDefCpuPerGPU\fR
    Default count of CPUs allocated per allocated GPU. This value is used only if
    the job didn't specify \-\-cpus\-per\-task and \-\-cpus\-per\-gpu.
    .IP

    .TP
    exist on the core (or \-\-hint=nomultithread which implies
    \-\-threads\-per\-core=1), the job will be unable to use those extra threads on
    the core and those threads will not be included in the memory per CPU
    calculation. But if the job has access to all threads on the core, those threads
    will be included in the memory per CPU calculation even if the job did not
    explicitly request those threads.

    In the following examples, each core has two threads.

    In this first example, two tasks can run on separate hyperthreads
    in the same core because \-\-threads\-per\-core is not used. The
    third task uses both threads of the second core. The allocated
    memory per cpu includes all threads:

    .nf
    .ft B
    $ salloc \-n3 \-\-mem\-per\-cpu=100
    salloc: Granted job allocation 17199
    $ sacct \-j $SLURM_JOB_ID \-X \-o jobid%7,reqtres%35,alloctres%35
      JobID                             ReqTRES                           AllocTRES
    \-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
      17199     billing=3,cpu=3,mem=300M,node=1     billing=4,cpu=4,mem=400M,node=1
    .ft
    .fi

    In this second example, because of \-\-threads\-per\-core=1, each
    task is allocated an entire core but is only able to use one
    thread per core. Allocated CPUs includes all threads on each
    core. However, allocated memory per cpu includes only the
    usable thread in each core.

    .nf
    .ft B
    $ salloc \-n3 \-\-mem\-per\-cpu=100 \-\-threads\-per\-core=1
    salloc: Granted job allocation 17200
    $ sacct \-j $SLURM_JOB_ID \-X \-o jobid%7,reqtres%35,alloctres%35
      JobID                             ReqTRES                           AllocTRES
    \-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
      17200     billing=3,cpu=3,mem=300M,node=1     billing=6,cpu=6,mem=300M,node=1
    .ft
    .fi
    .IP

    .TP
    \fBDefMemPerGPU\fR
    Default real memory size available per allocated GPU in megabytes.
    The default value is 0 (unlimited).
    Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
    \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
    mutually exclusive.
    .IP

    .TP
    \fBDependencyParameters\fR
    Multiple options may be comma separated.
    .IP
    .RS
    .TP
    \fBdisable_remote_singleton\fR
    By default, when a federated job has a singleton dependency, each cluster in the
    federation must clear the singleton dependency before the job's singleton
    dependency is considered satisfied. Enabling this option means that only the
    origin cluster must clear the singleton dependency. This option must be set
    in every cluster in the federation.
    .IP

    .TP
    \fBkill_invalid_depend\fR
    If a job has an invalid dependency and it can never run terminate it
    and set its state to be JOB_CANCELLED. By default the job stays pending
    with reason DependencyNeverSatisfied.
    .IP

    .TP
    \fBmax_depend_depth=#\fR
    Maximum number of jobs to test for a circular job dependency. Stop testing
    after this number of job dependencies have been tested. The default value is
    10 jobs.
    .RE
    .IP

    .TP
    \fBDisableRootJobs\fR
    If set to "YES" then user root will be prevented from running any jobs.
    The default value is "NO", meaning user root will be able to execute jobs.
    \fBDisableRootJobs\fR may also be set by partition.
    .IP

    .TP
    \fBEioTimeout\fR
    The number of seconds srun waits for slurmstepd to close the TCP/IP
    connection used to relay data between the user application and srun
    when the user application terminates. The default value is 60 seconds.
    May not exceed 65533.
    .IP

    .TP
    \fBEnforcePartLimits\fR
    If set to "ALL" then jobs which exceed a partition's size and/or
    time limits will be rejected at submission time. If job is submitted to
    multiple partitions, the job must satisfy the limits on all the requested
    partitions. If set to "NO" then the job will be accepted and remain queued
    until the partition limits are altered(Time and Node Limits).
    If set to "ANY" a job must satisfy any of the requested partitions
    When more than one epilog script is configured, they are executed in reverse
    alphabetical order (z-a -> Z-A -> 9-0). The Epilog script(s) may be used
    to purge files, disable user login, etc.
    By default there is no epilog.
    See \fBProlog and Epilog Scripts\fR for more information.
    .IP

    .TP
    \fBEpilogMsgTime\fR
    The number of microseconds that the slurmctld daemon requires to process
    an epilog completion message from the slurmd daemons. This parameter can
    be used to prevent a burst of epilog completion messages from being sent
    at the same time which should help prevent lost messages and improve
    throughput for large jobs.
    The default value is 2000 microseconds.
    For a 1000 node job, this spreads the epilog completion messages out over
    two seconds.
    .IP

    .TP
    \fBEpilogSlurmctld\fR
    Fully qualified pathname of a program for the slurmctld to execute
    upon termination of a job allocation (e.g.
    "/usr/local/slurm/epilog_controller").
    The program executes as SlurmUser, which gives it permission to drain
    nodes and requeue the job if a failure occurs (See scontrol(1)).
    Exactly what the program does and how it accomplishes this is completely at
    the discretion of the system administrator.
    Information about the job being initiated, its allocated nodes, etc. are
    passed to the program using environment variables.
    See \fBProlog and Epilog Scripts\fR for more information.
    .IP

    .TP
    \fBFairShareDampeningFactor\fR
    Dampen the effect of exceeding a user or group's fair share of allocated
    resources. Higher values will provides greater ability to differentiate
    between exceeding the fair share at high levels (e.g. a value of 1 results
    in almost no difference between overconsumption by a factor of 10 and 100,
    while a value of 5 will result in a significant difference in priority).
    The default value is 1.
    .IP

    .TP
    \fBFederationParameters\fR
    Used to define federation options. Multiple options may be comma separated.
    .IP
    .RS
    .TP
    \fBfed_display\fR
    If set, then the client status commands (e.g. squeue, sinfo, sprio, etc.) will
    display information in a federated view by default. This option is functionally
    .TP
    \fBGetEnvTimeout\fR
    Controls how long the job should wait (in seconds) to load the user's
    environment before attempting to load it from a cache file.
    Applies when the salloc or sbatch \fI\-\-get\-user\-env\fR option is used.
    If set to 0 then always load the user's environment from the cache file.
    The default value is 2 seconds.
    .IP

    .TP
    \fBGresTypes\fR
    A comma\-delimited list of generic resources to be managed (e.g.
    \fIGresTypes=gpu,mps\fR).
    These resources may have an associated GRES plugin of the same name providing
    additional functionality.
    No generic resources are managed by default.
    Ensure this parameter is consistent across all nodes in the cluster for
    proper operation.
    .IP

    .TP
    \fBGroupUpdateForce\fR
    If set to a non\-zero value, then information about which users are members
    of groups allowed to use a partition will be updated periodically, even when
    there have been no changes to the /etc/group file.
    If set to zero, group member information will be updated only after the
    /etc/group file is updated.
    The default value is 1.
    Also see the \fBGroupUpdateTime\fR parameter.
    .IP

    .TP
    \fBGroupUpdateTime\fR
    Controls how frequently information about which users are members of
    groups allowed to use a partition will be updated, and how long user
    group membership lists will be cached.
    The time interval is given in seconds with a default value of 600 seconds.
    A value of zero will prevent periodic updating of group membership information.
    Also see the \fBGroupUpdateForce\fR parameter.
    .IP

    .TP
    \fBGpuFreqDef\fR=[<\fItype\fR]=\fIvalue\fR>[,<\fItype\fR=\fIvalue\fR>]
    Default GPU frequency to use when running a job step if it
    has not been explicitly set using the \-\-gpu\-freq option.
    This option can be used to independently configure the GPU and its memory
    frequencies.
    There is no default value. If unset, no attempt to change the GPU frequency
    is made if the \-\-gpu\-freq option has not been set.
    After the job is completed, the frequencies of all affected GPUs will be reset
    to the highest possible values.
    In some cases, system power caps may override the requested values.
    the lowest available frequency.
    .IP

    .TP
    \fBmedium\fR
    attempts to set a frequency in the middle of the available range.
    .IP

    .TP
    \fBhigh\fR
    the highest available frequency.
    .IP

    .TP
    \fBhighm1\fR
    (high minus one) will select the next highest available frequency.
    .RE
    .IP

    .TP
    \fBHealthCheckInterval\fR
    The interval in seconds between executions of \fBHealthCheckProgram\fR.
    The default value is zero, which disables execution.
    .IP

    .TP
    \fBHealthCheckNodeState\fR
    Identify what node states should execute the \fBHealthCheckProgram\fR.
    Multiple state values may be specified with a comma separator.
    The default value is ANY to execute on nodes in any state.
    .IP
    .RS
    .TP 12
    \fBALLOC\fR
    Run on nodes in the ALLOC state (all CPUs allocated).
    .IP

    .TP
    \fBANY\fR
    Run on nodes in any state.
    .IP

    .TP
    \fBCYCLE\fR
    Rather than running the health check program on all nodes at the same time,
    cycle through running on all compute nodes through the course of the
    \fBHealthCheckInterval\fR. May be combined with the various node state
    options.
    .IP

    .TP
    \fBIDLE\fR
    .TP
    \fBHealthCheckProgram\fR
    Fully qualified pathname of a script to execute as user root periodically
    on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
    program may be used to verify the node is fully operational and DRAIN the node
    or send email if a problem is detected.
    Any action to be taken must be explicitly performed by the program
    (e.g. execute
    "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
    to drain a node).
    The execution interval is controlled using the \fBHealthCheckInterval\fR
    parameter.
    Note that the \fBHealthCheckProgram\fR will be executed at the same time
    on all nodes to minimize its impact upon parallel programs.
    This program will be killed if it does not terminate normally within
    60 seconds.
    This program will also be executed when the slurmd daemon is first started and
    before it registers with the slurmctld daemon.
    By default, no program will be executed.
    .IP

    .TP
    \fBInactiveLimit\fR
    The interval, in seconds, after which a non\-responsive job allocation
    command (e.g. \fBsrun\fR or \fBsalloc\fR) will result in the job being
    terminated. If the node on which the command is executed fails or the
    command abnormally terminates, this will terminate its job allocation.
    This option has no effect upon batch jobs.
    When setting a value, take into consideration that a debugger using \fBsrun\fR
    to launch an application may leave the \fBsrun\fR command in a stopped state
    for extended periods of time.
    This limit is ignored for jobs running in partitions with the
    \fBRootOnly\fR flag set (the scheduler running as root will be
    responsible for the job).
    The default value is unlimited (zero) and may not exceed 65533 seconds.
    .IP

    .TP
    \fBInteractiveStepOptions\fR
    When LaunchParameters=use_interactive_step is enabled, launching salloc will
    automatically start an srun process with InteractiveStepOptions to launch
    a terminal on a node in the job allocation.
    The default value is "\-\-interactive \-\-preserve\-env \-\-pty $SHELL".
    The "\-\-interactive" option is intentionally not documented in the srun man
    page. It is meant only to be used in \fBInteractiveStepOptions\fR in order to
    create an "interactive step" that will not consume resources so that other
    steps may run in parallel with the interactive step.
    .IP

    .TP
    \fBJobAcctGatherType\fR
    The JobAcctGather plugin collects memory, cpu, io, interconnect, energy and gpu
    .IP

    .TP
    \fBjobacct_gather/linux\fR
    Collect cpu and memory statistics by reading procfs. The plugin will take all
    the pids of the task and for each of them will read /proc/<pid>/stats. If UsePSS
    is set it will also read /proc/<pid>/smaps, and if NoShare is set it will also
    read /proc/<pid>/statm (see \fBJobAcctGatherParams\fR for more information).

    This plugin carries a performance penalty on jobs with a large number of spawned
    processes since it needs to iterate over all the task pids and aggregate the
    stats into one single metric for the ppid, and then these values need to be
    aggregated to the task stats.
    .IP

    .TP
    \fBjobacct_gather/none\fR
    This is the default value. No accounting data is collected. \fBsstat\fR will
    not work.
    .RE
    .IP

    \fBNOTE\fR: Changing the plugin type when jobs are running in the cluster is
    possible. The already running steps will keep using the previous plugin
    mechanism, while new steps will use the new mechanism.
    .IP

    .TP
    \fBJobAcctGatherFrequency\fR
    The job accounting and profiling sampling intervals.
    The supported format is follows:
    .IP
    .RS
    .TP 12
    \fBJobAcctGatherFrequency=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
    where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
    interval for the jobacct_gather plugin or a
    sampling interval for a profiling type by the
    acct_gather_profile plugin. Multiple,
    comma\-separated \fI<datatype>\fR=\fI<interval>\fR intervals
    may be specified. Supported datatypes are as follows:
    .IP
    .RS
    .TP
    \fBtask=\fI<interval>\fR
    where \fI<interval>\fR is the task sampling interval in seconds
    for the jobacct_gather plugins and for task
    profiling by the acct_gather_profile plugin.
    .IP

    .TP
    \fBenergy=\fI<interval>\fR
    for filesystem profiling using the acct_gather_filesystem
    plugin.

    .RE
    The default value for task sampling interval
    is 30 seconds. The default value for all other intervals is 0.
    An interval of 0 disables sampling of the specified type.
    If the task sampling interval is 0, accounting
    information is collected only at job termination, which reduces Slurm
    interference with the job, but also means that the statistics about a job
    don't reflect the average or maximum of several samples throughout the life
    of the job, but just show the information collected in the single sample.
    .br
    .br
    Smaller (non\-zero) values have a greater impact upon job performance,
    but a value of 30 seconds is not likely to be noticeable for
    applications having less than 10,000 tasks.
    .br
    .br
    Users can independently override each interval on a per job basis using the
    \fB\-\-acctg\-freq\fR option when submitting the job.
    .RE
    .IP

    .TP
    \fBJobAcctGatherParams\fR
    Arbitrary parameters for the job account gather plugin.
    Acceptable values at present include:
    .IP
    .RS
    .TP 20
    \fBNoShared\fR
    Exclude shared memory from RSS. This option cannot be used with UsePSS.
    .IP

    .TP
    \fBUsePss\fR
    Use PSS value instead of RSS to calculate real usage of memory. The PSS value
    will be saved as RSS. This option cannot be used with NoShared.
    .IP

    .TP
    \fBOverMemoryKill\fR
    Kill processes that are being detected to use more memory than requested by
    steps every time accounting information is gathered by the JobAcctGather plugin.
    This parameter should be used with caution because a job exceeding its memory
    allocation may affect other processes and/or machine health.

    \fBNOTE\fR: If available, it is recommended to limit memory by enabling
    task/cgroup as a TaskPlugin and making use of ConstrainRAMSpace=yes in the
    cgroup.conf instead of using this JobAcctGather mechanism for memory
    enforcement. Using JobAcctGather is polling based and there is a
    .RE
    .IP

    .TP
    \fBJobCompHost\fR
    The name of the machine hosting the job completion database.
    Only used for database type storage plugins, ignored otherwise.
    .IP

    .TP
    \fBJobCompLoc\fR
    This option sets a string which has different meanings depending on
    \fBJobCompType\fR:
    .IP
    .RS
    .TP
    If \fBjobcomp/elasticsearch\fR:
    Instructs this plugin to send the finished job records information to the
    Elasticsearch server URL endpoint (including the port number and the target
    index) configured in this option. This string should typically take the form
    of \fI<host>:<port>/<target>/_doc\fR. There is no default value for
    JobCompLoc when this plugin is enabled.

    \fBNOTE\fR: Refer to <https://slurm.schedmd.com/elasticsearch.html> for more
    information.
    .IP

    .TP
    If \fBjobcomp/filetxt\fR:
    Instructs this plugin to send the finished job records information to a file
    configured in this option. This string should represent an absolute path to
    a file. The default value for this plugin is \fI/var/log/slurm_jobcomp.log\fR.
    .IP

    .TP
    If \fBjobcomp/kafka\fR:
    When this plugin is configured, finished job records information is sent to a
    Kafka server. The plugin makes use of \fBlibrdkafka\fR. This string represents
    an absolute path to a file containing 'key=value' pairs configuring the library
    behavior. For the plugin to work properly, this file needs to exist and least
    the \fIbootstrap.servers\fR \fBlibrdkafka\fR property needs to be configured
    in it. There is no default value for JobCompLoc when this plugin is enabled.

    \fBNOTE\fR: For a full list of \fBlibrdkafka\fR properties, please refer to
    the library documentation. You can also view the jobcomp_kafka page for more
    information: <https://slurm.schedmd.com/jobcomp_kafka.html>

    \fBNOTE\fR: The target Kafka topic and other plugin parameters can be configured
    via \fBJobCompParams\fR.
    .IP

    .TP
    .TP
    If \fBjobcomp/script\fR:
    The finished job record information is made available via environment variables
    and processed by a script with name configured by this option. This string
    should represent a path to a script. There is no default value for JobCompLoc
    when this plugin is enabled. It needs to be explicitly configured or the
    plugin will fail to initialize.
    .RE
    .IP

    .TP
    \fBJobCompParams\fR
    Pass arbitrary text string to job completion plugin.
    Also see \fBJobCompType\fR.
    .RS
    .IP

    .TP
    Optional comma-separated list for \fBjobcomp/kafka\fR:
    .RS
    .IP

    .TP
    \fBflush_timeout\fR=<milliseconds>
    Maximum time (in milliseconds) to wait for all outstanding produce requests,
    et.al, to be completed. This is passed as a timeout argument to the
    \fBlibrdkafka\fR flush API function, called on plugin termination. This is done
    prior to destroying the producer instance to make sure all queued and in-flight
    produce requests are completed before terminating.
    For non-blocking calls, set to 0.
    To wait indefinitely for an event, set to -1 (not recommended, since this is
    called on plugin fini and could block slurmctld graceful termination).
    Accepted values are [-1,2147483647].
    Defaults to 500 (milliseconds).
    .IP

    .TP
    \fBpoll_interval\fR=<seconds>
    Seconds between calls to \fBlibrdkafka\fR API poll function, which polls the
    provided Kafka handle for events. The plugin spawns a separate thread to perform
    this call at the configured interval.
    Accepted values are [0,4294967295].
    Defaults to 2 (seconds).
    .IP

    .TP
    \fBrequeue_on_msg_timeout\fR
    Instruct the delivery report callback to requeue messages that failed delivery
    because their time waiting for successful delivery reached the \fBlibrdkafka\fR
    property \fBmessage.timeout.ms\fR.
    Defaults to not set (don't requeue and thus discard these messages).
    .IP
    completion data.
    Only used for database type storage plugins, ignored otherwise.
    .IP

    .TP
    \fBJobCompPort\fR
    The listening port of the job completion database server.
    Only used for database type storage plugins, ignored otherwise.
    .IP

    .TP
    \fBJobCompType\fR
    The job completion logging mechanism type.
    Acceptable values at present include:
    .IP
    .RS
    .TP
    \fBjobcomp/none\fR
    Upon job completion, a record of the job is purged from the system.
    If using the accounting infrastructure this plugin may not be of interest
    since some of the information is redundant.
    .IP

    .TP
    \fBjobcomp/elasticsearch\fR
    Upon job completion, a record of the job should be written to an
    Elasticsearch server, specified by the \fBJobCompLoc\fR parameter.
    .br
    \fBNOTE\fR: More information is available at the Slurm web site
    ( https://slurm.schedmd.com/elasticsearch.html ).
    .IP

    .TP
    \fBjobcomp/filetxt\fR
    Upon job completion, a record of the job should be written to a text file,
    specified by the \fBJobCompLoc\fR parameter.
    .IP

    .TP
    \fBjobcomp/kafka\fR
    Upon job completion, a record of the job should be sent to a Kafka server,
    specified by the file path referenced in \fBJobCompLoc\fR and/or using other
    \fBJobCompParams\fR.
    .IP

    .TP
    \fBjobcomp/lua\fR
    Upon job completion, a record of the job should be processed by the
    \fIjobcomp.lua\fR script, located in the default script directory
    (typically the subdirectory \fIetc\fR of the installation directory.
    .IP

    \fBJobCompUser\fR
    The user account for accessing the job completion database.
    Only used for database type storage plugins, ignored otherwise.
    .IP

    .TP
    \fBJobContainerType\fR
    Identifies the plugin to be used for job tracking.
    \fBNOTE\fR: The \fBJobContainerType\fR applies to a job allocation, while
    \fBProctrackType\fR applies to job steps.
    Acceptable values at present include:
    .IP
    .RS
    .TP 20
    \fBjob_container/cncu\fR
    Used only for Cray systems (CNCU = Compute Node Clean Up)
    .IP

    .TP
    \fBjob_container/tmpfs\fR
    Used to create a private namespace on the filesystem for jobs, which houses
    temporary file systems (/tmp and /dev/shm) for each job. 'PrologFlags=Contain'
    must be set to use this plugin.
    .RE
    .IP

    .TP
    \fBJobFileAppend\fR
    This option controls what to do if a job's output or error file
    exist when the job is started.
    If \fBJobFileAppend\fR is set to a value of 1, then append to
    the existing file.
    By default, any existing file is truncated.
    .IP

    .TP
    \fBJobRequeue\fR
    This option controls the default ability for batch jobs to be requeued.
    Jobs may be requeued explicitly by a system administrator, after node
    failure, or upon preemption by a higher priority job.
    If \fBJobRequeue\fR is set to a value of 1, then batch jobs may be requeued
    unless explicitly disabled by the user.
    If \fBJobRequeue\fR is set to a value of 0, then batch jobs will not be requeued
    unless explicitly enabled by the user.
    Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
    option to change the default behavior for individual jobs.
    The default value is 1.
    .IP

    .TP
    \fBJobSubmitPlugins\fR
    These are intended to be site\-specific plugins which can be used to set
    Set default partition to all partitions on the cluster.
    .IP

    .TP
    \fBdefaults\fR
    Set default values for job submission or modify requests.
    .IP

    .TP
    \fBlogging\fR
    Log select job submission and modification parameters.
    .IP

    .TP
    \fBlua\fR
    Execute a Lua script implementing site's own job_submit logic. Only one Lua
    script will be executed. It must be named "job_submit.lua" and must be located
    in the default configuration directory (typically the subdirectory "etc" of the
    installation directory). Sample Lua scripts can be found with the Slurm
    distribution, in the directory contribs/lua. Slurmctld will fatal on startup if
    the configured lua script is invalid. Slurm will try to load the script for each
    job submission. If the script is broken or removed while slurmctld is running,
    Slurm will fallback to the previous working version of the script.
    \fBWarning\fR: slurmctld runs this script while holding internal locks, and
    only a single copy of this script can run at a time. This blocks most
    concurrency in slurmctld. Therefore, this script should run to completion as
    quickly as possible.
    .IP

    .TP
    \fBpartition\fR
    Set a job's default partition based upon job submission parameters and
    available partitions.
    .IP

    .TP
    \fBpbs\fR
    Translate PBS job submission options to Slurm equivalent (if possible).
    .IP

    .TP
    \fBrequire_timelimit\fR
    Force job submissions to specify a timelimit.
    .RE
    .IP

    \fBNOTE\fR: For examples of use see the Slurm code in "src/plugins/job_submit"
    and "contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.
    .IP

    .TP
    \fBKillOnBadExit\fR
    The default value is 30 seconds.
    The value may not exceed 65533.
    .IP

    .TP
    \fBMaxBatchRequeue\fR
    Maximum number of times a batch job may be automatically requeued before
    being marked as JobHeldAdmin. (Mainly useful when the \fBSchedulerParameters\fR
    option \fBnohold_on_prolog_fail\fR is enabled.)
    The default value is 5.
    .IP

    .TP
    \fBNodeFeaturesPlugins\fR
    Identifies the plugins to be used for support of node features which can
    change through time. For example, a node which might be booted with various
    BIOS setting. This is supported through the use of a node's active_features
    and available_features information.
    Acceptable values at present include:
    .IP
    .RS
    .TP
    \fBnode_features/knl_cray\fR
    Used only for Intel Knights Landing processors (KNL) on Cray systems.
    See https://slurm.schedmd.com/intel_knl.html for more information.
    .IP

    .TP
    \fBnode_features/knl_generic\fR
    Used for Intel Knights Landing processors (KNL) on a generic Linux system.
    See https://slurm.schedmd.com/intel_knl.html for more information.
    .IP

    .TP
    \fBnode_features/helpers\fR
    Used to report and modify features on nodes using arbitrary scripts or
    programs.
    See helpers.conf man page for more information:
    https://slurm.schedmd.com/helpers.conf.html
    .RE
    .IP

    .TP
    \fBLaunchParameters\fR
    Identifies options to the job launch plugin.
    Acceptable values include:
    .IP
    .RS
    .TP 24
    \fBbatch_step_set_cpu_freq\fR
    Set the cpu frequency for the batch step from given \-\-cpu\-freq, or
    slurm.conf CpuFreqDef, option.  By default only steps started with srun will
    .IP

    .TP 24
    \fBenable_nss_slurm\fR
    Permits passwd and group resolution for a job to be serviced by slurmstepd rather
    than requiring a lookup from a network based service. See
    https://slurm.schedmd.com/nss_slurm.html for more information.
    .IP

    .TP 24
    \fBlustre_no_flush\fR
    If set on a Cray XC cluster, then do not flush the Lustre cache on job step
    completion. This setting will only take effect after reconfiguring, and will
    only take effect for newly launched jobs.
    .IP

    .TP 24
    \fBmem_sort\fR
    Sort NUMA memory at step start. User can override this default with
    SLURM_MEM_BIND environment variable or \-\-mem\-bind=nosort command line option.
    .IP

    .TP
    \fBmpir_use_nodeaddr\fR
    When launching tasks Slurm creates entries in MPIR_proctable that are used by
    parallel debuggers, profilers, and related tools to attach to running process.
    By default the MPIR_proctable entries contain MPIR_procdesc structures where
    the host_name is set to NodeName by default. If this option is specified,
    NodeAddr will be used in this context instead.
    .IP

    .TP
    \fBdisable_send_gids\fR
    By default, the slurmctld will look up and send the user_name and extended gids
    for a job, rather than independently on each node as part of each task launch.
    This helps mitigate issues around name service scalability when launching jobs
    involving many nodes. Using this option will disable this functionality. This
    option is ignored if enable_nss_slurm is specified.
    .IP

    .TP 24
    \fBslurmstepd_memlock\fR
    Lock the slurmstepd process's current memory in RAM.
    .IP

    .TP
    \fBslurmstepd_memlock_all\fR
    Lock the slurmstepd process's current and future memory in RAM.
    .IP

    .TP
    \fBtest_exec\fR

    .TP
    \fBulimit_pam_adopt\fR
    When pam_slurm_adopt is used to join an external process into a job cgroup,
    RLIMIT_RSS is set, as is done for tasks running in regular steps.
    .RE
    .IP

    .TP
    \fBLicenses\fR
    Specification of licenses (or other resources available on all
    nodes of the cluster) which can be allocated to jobs.
    License names can optionally be followed by a colon
    and count with a default count of one.
    Multiple license names should be comma separated (e.g.
    "Licenses=foo:4,bar").
    Note that Slurm prevents jobs from being scheduled if their
    required license specification is not available.
    Slurm does not prevent jobs from using licenses that are
    not explicitly listed in the job submission specification.
    .IP

    .TP
    \fBLogTimeFormat\fR
    Format of the timestamp in slurmctld and slurmd log files. Accepted
    format values include "iso8601", "iso8601_ms", "rfc5424", "rfc5424_ms",
    "rfc3339", "clock", "short" and "thread_id". The values ending in "_ms" differ
    from the ones without in that fractional seconds with millisecond precision are
    printed. The default value is "iso8601_ms". The "rfc5424" formats are the same
    as the "iso8601" formats except that the timezone value is also shown.
    The "clock" format shows a timestamp in microseconds retrieved
    with the C standard clock() function. The "short" format is a short
    date and time format. The "thread_id" format shows the timestamp
    in the C standard ctime() function form without the year but
    including the microseconds, the daemon's process ID and the current thread name
    and ID.
    A special option "format_stderr" can be added to the format as a comma separated
    value (e.g. "LogTimeFormat=iso8601_ms,format_stderr"). It will change the
    default format of the logs on stderr stream by prepending the timestamp as
    specified by \fBLogTimeFormat\fR.
    .IP

    .TP
    \fBMailDomain\fR
    Domain name to qualify usernames if email address is not explicitly given
    with the "\-\-mail\-user" option. If unset, the local MTA will need to qualify
    local address itself. Changes to MailDomain will only affect new jobs.
    .IP

    .TP
    \fBMailProg\fR
    Fully qualified pathname to the program used to send email per user request.
    .TP
    \fBSLURM_JOB_STATE\fR
    The base state of the job when the MailProg is called.
    .RE
    .IP
    .RS
    .TP
    \fBSLURM_JOB_MAIL_TYPE\fR
    The mail type triggering the mail.
    .RE
    .RE
    .IP
    .RS
    .TP
    \fBBEGIN\fR
    .IP
    .RS
    .TP
    \fBSLURM_JOB_QEUEUED_TIME\fR
    The amount of time the job was queued.
    .RE
    .RE
    .IP
    .RS
    .TP
    \fBEND, FAIL, REQUEUE, TIME_LIMIT_*\fR
    .IP
    .RS
    .TP
    \fBSLURM_JOB_RUN_TIME\fR
    The amount of time the job ran for.
    .RE
    .RE
    .IP
    .RS
    .TP
    \fBEND, FAIL\fR
    .IP
    .RS
    .TP
    \fBSLURM_JOB_EXIT_CODE_MAX\fR
    Job's exit code or highest exit code for an array job.
    .RE
    .IP
    .RS
    .TP
    \fBSLURM_JOB_EXIT_CODE_MIN\fR
    Job's minimum exit code for an array job.
    .RE
    .IP
    .RS
    .TP
    .RE
    .IP

    .TP
    \fBMaxArraySize\fR
    The maximum job array task index value will be one less than MaxArraySize
    to allow for an index value of zero.
    Configure MaxArraySize to 0 in order to disable job array use.
    The value may not exceed 4000001.
    The value of \fBMaxJobCount\fR should be much larger than \fBMaxArraySize\fR.
    The default value is 1001.
    See also \fBmax_array_tasks\fR in SchedulerParameters.
    .IP

    .TP
    \fBMaxDBDMsgs\fR
    When communication to the SlurmDBD is not possible the slurmctld will queue
    messages meant to processed when the SlurmDBD is available again.
    In order to avoid running out of memory the slurmctld will only queue so many
    messages. The default value is 10000, or \fBMaxJobCount\fR * 2 + Node Count
    * 4, whichever is greater.  The value can not be less than 10000.
    .IP

    .TP
    \fBMaxJobCount\fR
    The maximum number of jobs slurmctld can have in memory at one time.
    Combine with \fBMinJobAge\fR to ensure the slurmctld daemon does not exhaust
    its memory or other resources. Once this limit is reached, requests to submit
    additional jobs will fail. The default value is 10000 jobs.
    \fBNOTE\fR: Each task of a job array counts as one job even though they will not
    occupy separate job records until modified or initiated.
    Performance can suffer with more than a few hundred thousand jobs.
    Setting per MaxSubmitJobs per user is generally valuable to prevent a single
    user from filling the system with jobs.
    This is accomplished using Slurm's database and configuring enforcement of
    resource limits.
    .IP

    .TP
    \fBMaxJobId\fR
    The maximum job id to be used for jobs submitted to Slurm without a specific
    requested value. Job ids are unsigned 32bit integers with the first 26 bits
    reserved for local job ids and the remaining 6 bits reserved for a cluster id
    to identify a federated job's origin. The maximum allowed local job id is
    67,108,863 (0x3FFFFFF). The default value is 67,043,328 (0x03ff0000).
    \fBMaxJobId\fR only applies to the local job id and not the federated job id.
    Job id values generated will be incremented by 1 for each subsequent job. Once
    \fBMaxJobId\fR is reached, the next job will be assigned \fBFirstJobId\fR.
    Federated jobs will always have a job ID of 67,108,865 or higher.
    Also see \fBFirstJobId\fR.
    .IP

    guaranteed due to the possibility of having heterogeneous configurations and
    multi\-partition/qos jobs. If this is a concern it is advised to use a job
    submit LUA plugin instead to enforce auto\-adjustments to your specific needs.
    .IP

    .TP
    \fBMaxMemPerNode\fR
    Maximum real memory size available per allocated node in megabytes.
    Used to avoid over\-subscribing memory and causing paging.
    \fBMaxMemPerNode\fR would generally be used if whole nodes
    are allocated to jobs (\fBSelectType=select/linear\fR) and
    resources are over\-subscribed (\fBOverSubscribe=yes\fR or
    \fBOverSubscribe=force\fR).
    The default value is 0 (unlimited).
    Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
    \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
    .IP

    .TP
    \fBMaxNodeCount\fR
    Maximum count of nodes which may exist in the controller. By default MaxNodeCount
    will be set to the number of nodes found in the slurm.conf. MaxNodeCount will
    be ignored if less than the number of nodes found in the slurm.conf. Increase
    MaxNodeCount to accommodate dynamically created nodes with dynamic node
    registrations and nodes created with scontrol.
    .IP

    .TP
    \fBMaxStepCount\fR
    The maximum number of steps that any job can initiate. This parameter
    is intended to limit the effect of bad batch scripts.
    The default value is 40000 steps.
    .IP

    .TP
    \fBMaxTasksPerNode\fR
    Maximum number of tasks Slurm will allow a job step to spawn
    on a single node. The default \fBMaxTasksPerNode\fR is 512.
    May not exceed 65533.
    .IP

    .TP
    \fBMCSParameters\fR
    MCS = Multi\-Category Security
    MCS Plugin Parameters.
    The supported parameters are specific to the \fBMCSPlugin\fR.
    Changes to this value take effect when the Slurm daemons are reconfigured.
    More information about MCS is available here
    fR(8) manual.
    .IP

    .TP
    \fBPowerParameters\fR
    System power management parameters.
    The supported parameters are specific to the \fBPowerPlugin\fR.
    Changes to this value take effect when the Slurm daemons are reconfigured.
    More information about system power management is available here
    fR(5)
    man page
    .IP

    .TP
    \fBtopology/default\fR
    default for other systems, best\-fit logic over one\-dimensional topology
    .IP

    .TP
    \fBtopology/tree\fR
    used for a hierarchical network, as described in the \fBtopology.conf\fR(5)
    man page
    \fBTreeWidth\fR specifies the width of the tree (i.e. the fanout).
    On architectures with a front end node running the slurmd daemon, the value
    must always be equal to or greater than the number of front end nodes which
    eliminates the need for message forwarding between the slurmd daemons.
    On other architectures the default value is 16, meaning each slurmd daemon can
    communicate with up to 16 other slurmd daemons. This value balances offloading
    slurmctld (max 16 threads running), time of communication, and node fault
    tolerance (4368 nodes can be contacted with three message hops). The default
    value will work well for most clusters however on bigger systems this value can
    be increased to avoid long timeouts and retransmissions in case of unresponsive
    nodes. The value may not exceed 65533.
    .IP

    .TP
    \fBUnkillableStepProgram\fR
    If the processes in a job step are determined to be unkillable for a period
    of time specified by the \fBUnkillableStepTimeout\fR variable, the program
    specified by \fBUnkillableStepProgram\fR will be executed.
    By default no program is run.

    See section \fBUNKILLABLE STEP PROGRAM SCRIPT\fR for more information.
    .IP

    .TP
    \fBUnkillableStepTimeout\fR
    The length of time, in seconds, that Slurm will wait before deciding that
    processes in a job step are unkillable (after they have been signaled with
    SIGKILL) and execute \fBUnkillableStepProgram\fR.
    The default timeout value is 60 seconds.
    If exceeded, the compute node will be drained to prevent future jobs from being
    scheduled on the node.

    \fBNOTE\fR: Ensure that UnkillableStepTimeout is at least 5 times larger than
    MessageTimeout, otherwise it can lead to unexpected draining of nodes.
    .IP

    .TP
    \fBUsePAM\fR
    If set to 1, PAM (Pluggable Authentication Modules for Linux) will be enabled.
    PAM is used to establish the upper bounds for resource limits. With PAM support
    enabled, local system administrators can dynamically configure system resource
    limits. Changing the upper bound of a resource limit will not alter the limits
    of running jobs, only jobs started after a change has been made will pick up
    the new limits.
    The default value is 0 (not to enable PAM support).
    Remember that PAM also needs to be configured to support Slurm as a service.
    For sites using PAM's directory based configuration option, a configuration
    file named \fBslurm\fR should be created. The module\-type, control\-flags, and
    module\-path names that should be included in the file are:
    .br
    auth        required      pam_localuser.so
    .br
    .IP

    .TP
    \fBVSizeFactor\fR
    Memory specifications in job requests apply to real memory size (also known
    as resident set size). It is possible to enforce virtual memory limits for
    both jobs and job steps by limiting their virtual memory to some percentage
    of their real memory allocation. The \fBVSizeFactor\fR parameter specifies
    the job's or job step's virtual memory limit as a percentage of its real
    memory limit. For example, if a job's real memory limit is 500MB and
    VSizeFactor is set to 101 then the job will be killed if its real memory
    exceeds 500MB or its virtual memory exceeds 505MB (101 percent of the
    real memory limit).
    The default value is 0, which disables enforcement of virtual memory limits.
    The value may not exceed 65533 percent.

    \fBNOTE\fR: This parameter is dependent on \fBOverMemoryKill\fR being
    configured in \fBJobAcctGatherParams\fR. It is also possible
    to configure the \fBTaskPlugin\fR to use \fBtask/cgroup\fR for memory
    enforcement. \fBVSizeFactor\fR will not have an effect on memory enforcement
    done through cgroups.
    .IP

    .TP
    \fBWaitTime\fR
    Specifies how many seconds the srun command should by default wait after
    the first task terminates before terminating all remaining tasks. The
    "\-\-wait" option on the srun command line overrides this value.
    The default value is 0, which disables this feature.
    May not exceed 65533 seconds.
    .IP

    .TP
    \fBX11Parameters\fR
    For use with Slurm's built\-in X11 forwarding implementation.
    .IP
    .RS
    .TP 8
    \fBhome_xauthority\fR
    If set, xauth data on the compute node will be placed in \fB~/.Xauthority\fR
    rather than in a temporary file under \fBTmpFS\fR.
    .RE
    .IP

    .SH "NODE CONFIGURATION"
    The configuration of nodes (or machines) to be managed by Slurm is
    also specified in \fB/etc/slurm.conf\fR.
    Changes in node configuration (e.g. adding nodes, changing their
    processor count, etc.) require restarting both the slurmctld daemon
    and the slurmd daemons.
    All slurmd daemons must know each node in the system to forward
    messages in support of hierarchical communications.
    .LP
    Default values can be specified with a record in which
    \fBNodeName\fR is "DEFAULT".
    The default entry values will apply only to lines following it in the
    configuration file and the default values can be reset multiple times
    in the configuration file with multiple entries where "NodeName=DEFAULT".
    Each line where \fBNodeName\fR is "DEFAULT" will replace or add to previous
    default values and will not reinitialize the default values.
    The "NodeName=" specification must be placed on every line
    describing the configuration of nodes.
    A single node name can not appear as a NodeName value in more than one line
    (duplicate node name records will be ignored).
    In fact, it is generally possible and desirable to define the
    configurations of all nodes in only a few lines.
    This convention permits significant optimization in the scheduling
    of larger clusters.
    In order to support the concept of jobs requiring consecutive nodes
    on some architectures,
    node specifications should be place in this file in consecutive order.
    No single node name may be listed more than once in the configuration
    file.
    Use "DownNodes=" to record the state of nodes which are temporarily
    in a DOWN, DRAIN or FAILING state without altering permanent
    configuration information.
    A job step's tasks are allocated to nodes in order the nodes appear
    in the configuration file. There is presently no capability within
    Slurm to arbitrarily order a job step's tasks.
    .LP
    Multiple node names may be comma separated (e.g. "alpha,beta,gamma")
    and/or a simple node range expression may optionally be used to
    specify numeric ranges of nodes to avoid building a configuration
    file with large numbers of entries.
    The node range expression can contain one  pair of square brackets
    with a sequence of comma\-separated numbers and/or ranges of numbers
    separated by a "\-" (e.g. "linux[0\-64,128]", or "lx[15,18,32\-33]").
    Note that the numeric ranges can include one or more leading
    zeros to indicate the numeric portion has a fixed number of digits
    (e.g. "linux[0000\-1023]").
    Multiple numeric ranges can be included in the expression
    (e.g. "rack[0\-63]_blade[0\-41]").
    If one or more numeric expressions are included, one of them
    must be at the end of the name (e.g. "unit[0\-31]rack" is invalid),
    but arbitrary names can always be used in a comma\-separated list.
    .LP
    The node configuration specified the following information:

    .TP
    \fBNodeName\fR
    Name that Slurm uses to refer to a node.
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    follows the configuration for "NodeName=baker" they will be
    considered adjacent in the computer.
    \fBNOTE\fR: If the \fBNodeName\fR is "ALL" the process parsing the configuration
    will exit immediately as it is an internally reserved word.
    .IP

    .TP
    \fBNodeHostname\fR
    Typically this would be the string that "/bin/hostname \-s" returns.
    It may also be the fully qualified domain name as returned by "/bin/hostname \-f"
    (e.g. "foo1.bar.com"), or any valid domain name associated with the host
    through the host database (/etc/hosts) or DNS, depending on the resolver
    settings.  Note that if the short form of the hostname is not used, it
    may prevent use of hostlist expressions (the numeric portion in brackets
    must be at the end of the string).
    A node range expression can be used to specify a set of nodes.
    If an expression is used, the number of nodes identified by
    \fBNodeHostname\fR on a line in the configuration file must
    be identical to the number of nodes identified by \fBNodeName\fR.
    By default, the \fBNodeHostname\fR will be identical in value to
    \fBNodeName\fR.
    .IP

    .TP
    \fBNodeAddr\fR
    Name that a node should be referred to in establishing
    a communications path.
    This name will be used as an
    argument to the getaddrinfo() function for identification.
    If a node range expression is used to designate multiple nodes,
    they must exactly match the entries in the \fBNodeName\fR
    (e.g. "NodeName=lx[0\-7] NodeAddr=elx[0\-7]").
    \fBNodeAddr\fR may also contain IP addresses.
    By default, the \fBNodeAddr\fR will be identical in value to
    \fBNodeHostname\fR.
    .IP

    .TP
    \fBBcastAddr\fR
    Alternate network path to be used for sbcast network traffic to a given node.
    This name will be used as an argument to the getaddrinfo() function.
    If a node range expression is used to designate multiple nodes,
    they must exactly match the entries in the \fBNodeName\fR
    (e.g. "NodeName=lx[0\-7] BcastAddr=elx[0\-7]").
    \fBBcastAddr\fR may also contain IP addresses.
    By default, the \fBBcastAddr\fR is unset, and sbcast traffic will be routed
    to the \fBNodeAddr\fR for a given node.
    Note: cannot be used with CommunicationParameters=NoInAddrAny.
    .IP

    .TP
    \fBBoards\fR
    explicitly requests less than the configured CoreSpecCount.
    If this option and \fBCpuSpecList\fR are both designated for a
    node, an error is generated.  For information on the algorithm used by Slurm
    to select the cores refer to the core specialization documentation
    ( https://slurm.schedmd.com/core_spec.html ).
    .IP

    .TP
    \fBCoresPerSocket\fR
    Number of cores in a single physical processor socket (e.g. "2").
    The CoresPerSocket value describes physical cores, not the
    logical number of processors per socket.
    \fBNOTE\fR: If you have multi\-core processors, you will likely
    need to specify this parameter in order to optimize scheduling.
    The default value is 1.
    .IP

    .TP
    \fBCpuBind\fR
    If a job step request does not specify an option to control how tasks are bound
    to allocated CPUs (\-\-cpu\-bind) and all nodes allocated to the job have the same
    \fBCpuBind\fR option the node \fBCpuBind\fR option will control how tasks are
    bound to allocated resources. Supported values for \fBCpuBind\fR are "none",
    "socket", "ldom" (NUMA), "core" and "thread".
    .IP

    .TP
    \fBCPUs\fR
    Number of logical processors on the node (e.g. "2").
    It can be set to the total
    number of sockets(supported only by select/linear), cores or threads.
    This can be useful when you want to schedule only the cores on a hyper\-threaded
    node. If \fBCPUs\fR is omitted, its default will be set equal to the product of
    \fBBoards\fR, \fBSockets\fR, \fBCoresPerSocket\fR, and \fBThreadsPerCore\fR.
    .IP

    .TP
    \fBCpuSpecList\fR
    A comma\-delimited list of Slurm abstract CPU IDs reserved for system use.
    The list will be expanded to include all other CPUs, if any, on the same cores.
    Depending upon the \fBTaskPluginParam\fR option of \fBSlurmdOffSpec\fR,
    the Slurm daemon slurmd may either be confined to these
    resources (the default) or prevented from using these resources.
    Isolation of slurmd from user jobs may improve application performance.
    A job can use these cores if AllowSpecResourcesUsage=yes and the user
    explicitly requests less than the number of CPUs in this list.
    If this option and \fBCoreSpecCount\fR are both designated for a node,
    an error is generated.
    This option has no effect unless cgroup job confinement is also configured
    (i.e. the \fItask/cgroup\fR \fBTaskPlugin\fR is enabled and
    \fBConstrainCores=yes\fR is set in cgroup.conf).
    .IP
    Boolean operations.
    .IP

    .TP
    \fBGres\fR
    A comma\-delimited list of generic resources specifications for a node.
    The format is: "<name>[:<type>][:no_consume]:<number>[K|M|G]".
    The first field is the resource name, which matches the GresType configuration
    parameter name.
    The optional type field might be used to identify a model of that generic
    resource.
    It is forbidden to specify both an untyped GRES and a typed GRES with the same
    fR(5), \fBgetaddrinfo\fR(3),
    \fBgetrlimit\fR(2), \fBgres.conf\fR(5), \fBgroup\fR(5), \fBhostname\fR(1),
    \fBscontrol\fR(1), \fBslurmctld\fR(8), \fBslurmd\fR(8),
    \fBslurmdbd\fR(8), \fBslurmdbd.conf\fR(5), \fBsrun\fR(1),
    \fBspank\fR(8), \fBsyslog\fR(3), \fBtopology.conf\fR(5)