QSUB(1)

NAME
qsub - submit an NQS batch request.

SYNOPSIS
qsub [ flags ] [ script-file ]

DESCRIPTION
Qsub submits a batch request to the Network Queueing System (NQS).

If no script-file is specified, then the set of commands to be executed as a batch request is taken directly from the standard input file (stdin). In all cases however, the script file is spooled, so that later changes will not affect previously queued batch requests.

All of the flags that can be specified on the command line can also be specified within the first comment block inside the batch request script file as embedded default flags. Such flags appearing in the batch request script file set default characteristics for the batch request. If the same flag is specified on the command line, then the command line flag (and any associated value) takes precedence over the embedded flag. See the section entitled: LONG DESCRIPTION for more information on embedded default flags.

What follows is a terse definition of the flags implemented by the Qsub command (see the section: LONG DESCRIPTION for the complete definition and syntax used for each of these flags).

    -a - run request after stated time
    -bb - broadcast message when the request begins execution
    -be - broadcast message when the request ends execution
    -d - delete the script file after spooled
    -e - direct stderr output to stated destination
    -eo - direct stderr output to the stdout destination
    -h - print short synopsis of switches
    -ke - keep stderr output on the execution machine
    -ko - keep stdout output on the execution machine
    -lc - establish per-process corefile size limit
    -ld - establish per-process data-segment size limits
    -lf - establish per-process permanent-file size limits
    -lF - establish per-request permanent-file space limits
    -lm - establish per-process memory size limits
    -lM - establish per-request memory space limits
    -ln - establish per-process nice execution value limit
    -ls - establish per-process stack-segment size limits
    -lt - establish per-process CPU time limits
    -lT - establish per-request CPU time limits
    -lv - establish per-process temporary-file size limits
    -lV - establish per-request temporary-file space limits
    -lw - establish per-process working set limit
    -mb - send mail when the request begins execution
    -me - send mail when the request ends execution
    -mr - send mail when the request is restarted
    -mt - send mail when the request is transferred to the execution machine
    -mu - send mail for the request to the stated user
    -nr - declare that batch request is not restartable
    -o - direct stdout output to the stated destination
    -p - specify intra-queue request priority
    -q - queue request in the stated queue
    -r - assign stated request name to the request
    -re - remotely access the stderr output file
    -ro - remotely access the stdout output file
    -rs - specify that a request is restartable
    -s - specify shell to interpret the batch request script
    -v - print the current Qsub version number
    -x - export all environment variables with request
    -z - submit the request silently

LONG DESCRIPTION
As described above, it is possible to specify default flags within the batch request script file that configure the default behavior of the batch request. The algorithm used to scan for such embedded default flags within an NQS batch request script file is as follows:

    1.  Read the first line of the script file.

    2.  If the current line contains only whitespace characters, or the first non-whitespace character of the line is ":", then goto step 7.

    3.  If the first non-whitespace character of the current line is not a "#" character, then goto step 8.

    4.  Skip over whitespace until the next token is reached. If the token is not "QSUB" and not "@$" , then goto step 7.

    5.  If no "-" is present as the first non-whitespace character following the "@$" sequence, then goto step 8.

    6.  Process the embedded flag, stopping the parsing process upon reaching the end of the line, or upon reaching the first unquoted "#" character.

    7.  Read the next script file line. Goto step 2.

    8.  End. No more embedded flags will be recognized.

Here is an example of the use of embedded flags within the script file.
  #
  # Batch request script example:
  #
  # QSUB -a "11:30pm EDT" -lt "21:10, 20:00"
  # # Run request after 11:30 EDT by default,
  # # and set a maximum per-process CPU time
  # # limit of 21 minutes and ten seconds.
  # # Send a warning signal when any process
  # # of the running batch request consumes
  # # more than 20 minutes of CPU time.
  # QSUB -lT 1:45:00
  # # Set a maximum per-request CPU time limit
  # # of one hour, and 45 minutes. (The
  # # implementation of CPU time limits is
  # # completely dependent upon the UNIX
  # # implementation at the execution
  # # machine.)
  # QSUB -mb -me # Send mail at beginning and end of
  # # request execution.
  # QSUB -q batch1 # Queue request to queue: batch1 by
  # # default.
  # QSUB # No more embedded flags.
  #
  make all

The following paragraphs give the detailed descriptions of the flags supported by the Qsub command.

-a date-time
    Do not run the batch request before the specified date and/or time. If a date-time specification is comprised of two or more tokens separated by whitespace characters, then the date-time specification must be placed within double quotes as in: -a "July, 4, 2026 12:31-EDT", or otherwise escaped such that Qsub and the shell will interpret the entire date-time specification as a single character string. This restriction also applies when an embedded default -a flag with accompanying date-time specification appears within the batch request script file.

    The syntax accepted for the date-time parameter is relatively flexible. Unspecified date and time values default to an appropriate value (e.g. if no date is specified, then the current month, day, and year are assumed).

    A date may be specified as a month and day (current year assumed), or the year can also be explicitly specified. It is also possible to specify the date as a weekday name (e.g. "Tues"), or as one of the strings: "today", or "tomorrow". Weekday names and month names can be abbreviated by any three character (or longer) prefix of the actual name. An optional period can follow an abbreviated month or day name.

    Time of day specifications can be given using a twenty-four hour clock, or "am" and "pm" specifications may be used alternatively. In the absence of a meridian specification, a twenty-four hour clock is assumed.

    It should be noted that the time of day specification is interpreted using the precise meridian definitions whereby "12am" refers to the twenty-four hour clock time of 0:00:00, "12m" refers to noon, and "12-pm" refers to 24:00:00. Alternatively, the phrases "midnight" and "noon" are accepted as time of day specifications, where "midnight" refers to the time of 24:00:00.

    A timezone may also appear at any point in the date-time specification. Thus, it is legal to say: "April 1, 1987 13:01-PDT". In the absence of a timezone specification, the local timezone is assumed, with daylight savings time being inferred when appropriate, based on the date specified.

    All alphabetic comparisons are performed in a case insensitive fashion such that both "WeD" and "weD" refer to the day of Wednesday.

    Some valid date-time examples are:

        01-Jan-1986 12am, PDT
        Tuesday, 23:00:00
        11pm tues.
        tomorrow 23-MST

-bb Broadcast a message when the request begins execution. This message will be delivered to all terminals the user has within the NQS domain. It includes such information as the request sequence number, the request name, the execution host and the time. The message is broadcast to the list of hosts in the NQS domain. The default list is in /usr/lib/nqs/nqs-domain. This list can be overridden by a list in a file in the user's home directory on the execution machine called .nqs-domain. This file must have the same format as the system default file, but may contain different system names.

-be Broadcast a message when the request ends execution. This message will be delivered to all terminals the user has within the NQS domain. It includes such information as the request sequence number, the request name, the execution host, and the time. The message is broadcast to the list of hosts in the NQS domain. The default list is in /usr/lib/nqs/nqs-domain. This list can be overridden by a list in a file in the user's home directory on the execution machine called .nqs-domain. This file must have the same format as the system default file, but may contain different system names.

-d  If the script file is successfully spooled, delete the original copy specified on the command line.

-e [machine:][[/]path/] stderr-filename
    Direct output generated by the batch request which is sent to the stderr file to the named [machine:][[/]path/] stderr-filename.

    The brackets "[" and "]" enclose optional portions of the stderr destination machine, path, and stderr-filename.

    If no explicit machine destination is specified, then the destination machine defaults to the machine that originated the batch request, or to the machine that will eventually run the request, depending on the respective absence, or presence of the -ke flag.

    If no machine destination is specified, and the path/filename does not begin with a "/", then the current working directory is prepended to create a fully qualified path name, provided that the -ke (keep stderr) flag is also absent. In all other cases, any partial path/filename is interpreted relative to the user's home directory on the stderr destination machine.

    This flag cannot be specified when the -eo flag option is also present.

    If the -eo and -e [machine:][[/]path/] stderr-filename flag options are not present, then all stderr output for the batch request is sent to the file whose name consists of the first seven characters of the request-name followed by the characters: ".e", followed by the request sequence number portion of the request-id discussed below. In the absence of the -ke flag, this default stderr output file will be placed on the machine that originated the batch request in the current working directory, as defined when the batch request was first submitted. Otherwise, the file will be placed in the user's home directory on the execution machine.

    If the stderr-filename contains any # characters, they will be replaced by the numeric part of the request's sequence number. This provides extra flexibility in naming the stderr output file. The # character is ignored in all other parts of the file path.

-eo Direct all output that would normally be sent to the stderr file to the stdout file for the batch request. This flag cannot be specified when the -e [machine:][[/]path/] stderr-filename flag option is also present.

-h  Print a short summary of the various Qsub switches. The -h switch is ignored when found within a script.

-ke In the absence of an explicit machine destination for the stderr file produced by a batch request, the machine destination chosen for the stderr output file is the machine that originated the batch request. In some cases however, this behavior may be undesirable, and so the -ke flag can be specified which instructs NQS to leave any stderr output file produced by the request on the machine that actually executed the batch request.

    This flag is meaningless if the -eo flag is specified, and cannot be specified if an explicit machine destination is given for the stderr parameter of the -e flag.

-ko In the absence of an explicit machine destination for the stdout file produced by a batch request, the machine destination chosen for the stdout output file is the machine that originated the batch request. In some cases however, this behavior may be undesirable, and so the -ko flag can be specified which instructs NQS to leave any stdout output file produced by the request on the machine that actually executed the batch request.

    This flag cannot be specified if an explicit machine destination is given for the stdout parameter of the -o flag.

-lc per-process corefile size limit
    Set a per-process maximum core file size limit for all processes that constitute the running batch request. If any process comprising the running request attempts to exit creating a core file whose size would exceed the maximum per-process core file size limit for the request, then the core file image of the aborting process will be reduced to the necessary size by an algorithm dependent upon the underlying UNIX implementation.

    Not all UNIX implementations support per-process corefile size limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process corefile size limit.

-ld per-process data-segment size limit [ , warn-limit ]
    Set a per-process maximum and an optional warning data-segment size limit for all processes that constitute the running batch request. If any process comprising the running request exceeds the maximum per-process data-segment size-limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process data-segment warning size limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -ld flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-process data-segment size limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process data-segment size limit.

-lf per-process permanent-file size limit [ , warn-limit ]
    Set a per-process maximum and an optional warning permanent-file size limit for all processes that constitute the running batch request. If any process comprising the running request attempts to write to a permanent file such that the file size would increase beyond the maximum per-process permanent-file size limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process warning permanent-file size limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lf flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-process permanent-file size limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    At the time of this writing, the author was unaware of any UNIX implementation that made a distinction at the kernel level, between permanent, and temporary files. While it is certainly possible to construct a pseudo-temporary file by first creating it, and then unlinking its pathname, the disk space allocated for such a file will be allocated from the same pool of disk space that all other UNIX files are allocated from. Furthermore, such a file will be subject to the same quota enforcement mechanisms, if any are provided by the underlying UNIX implementation, that all other UNIX files are created under.

    For all UNIX implementations that do not support a distinction between permanent, and temporary files at the kernel level, this limit is interpreted as a per-process file size limit, with the word permanent removed from the definition.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process permanent-file size limit.

-lF per-request permanent-file space limit [ , warn-limit ]
    Set a per-request maximum and an optional warning cumulative permanent-file space limit for all processes that constitute the running batch request. If any process comprising the running request attempts to write to a permanent file such that the file space consumed by all permanent files opened for writing by all of the processes in the batch request, would increase beyond the maximum per-request permanent-file space limit for the request, then all of the processes in the request will be terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-request warning permanent-file space limits. When such a warning limit is exceeded, a signal is delivered to one or more of the processes comprising the running request, depending upon the underlying UNIX implementation.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lF flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-request permanent-file space limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    At the time of this writing, the author was unaware of any UNIX implementation that made a distinction at the kernel level, between permanent, and temporary files. While it is certainly possible to construct a pseudo-temporary file by first creating it, and then unlinking its pathname, the disk space allocated for such a file will be allocated from the same pool of disk space that all other UNIX files are allocated from. Furthermore, such a file will be subject to the same quota enforcement mechanisms, if any are provided by the underlying UNIX implementation, that all other UNIX files are created under.

    For all UNIX implementations that do not support a distinction between permanent, and temporary files at the kernel level, this limit is interpreted as a per-request file space limit, with the word permanent removed from the definition.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-request permanent-file space limit.

-lm per-process memory size limit [ , warn-limit ]
    Set a per-process maximum and an optional warning memory size limit for all processes that constitute the running batch request. If any process comprising the running request exceeds the maximum per-process memory size limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process warning memory size limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lm flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-process memory size limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process memory size limit.

-lM per-request memory space limit [ , warn-limit ]
    Set a per-request maximum and an optional warning cumulative memory space limit for all processes that constitute the running batch request. If the sum of all memory consumed by all of the processes comprising the running request exceeds the maximum per-request memory space limit for the request, then all of the processes in the request will be terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-request warning memory size limits. When such a warning limit is exceeded, a signal is delivered to one or more of the processes comprising the running request, depending upon the underlying UNIX implementation.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lM flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-request memory space limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-request memory space limit.

-ln per-process nice value limit
    Set a per-process nice value for all processes comprising the running batch request.

    At present, all UNIX implementations support the use of an integer called the nice value, which determines the execution-time priority of a process relative to all other processes in the system. By letting the user set a limit on the nice value for all processes comprising the running request, a user can cause a request to consume less (or more) of the CPU resource presented by the execution machine.

    This is particularly useful when a user wishes to execute a CPU intensive batch request on a machine running interactive processes. By setting a low execution-time priority, a user can make a long running batch request give way to more interactive processes during the daytime, while the coming of the nighttime hours with typically smaller process loads will allow such a request to gain more and more of the CPU resource. In this way, long running batch requests can be polite to their more transient, interactive neighbor processes.

    The only quirk associated with this flag results from the peculiar choice of nice values, implemented by the standard UNIX implementations. In general, increasingly negative nice values cause the relative execution priority of a process to increase, while increasingly positive nice values causes the relative priority to decrease! Thus, a nice value limit specification of: "-ln -10" is greedier than a nice value limit specification of: "-ln 0".

    Since varying UNIX implementations often support a different finite range of nice values, NQS allows the specification of nice values that can eventually turn out to be outside the limits for the UNIX implementation running at the execution machine. In such cases, NQS will simply bind the specified nice value limit to within the necessary range as appropriate.

    Lastly, any nice value specified by the use of this flag must be acceptable to the batch queue in which the request is ultimately placed (see the section entitled LIMITS for more information).

-ls per-process stack-segment size limit [ , warn-limit ]
    Set a per-process maximum and an optional warning stack-segment size limit for all processes that constitute the running batch request. If any process comprising the running request exceeds the maximum per-process stack-segment size limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process warning stack-segment size limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -ls flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-process stack-segment size limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process stack-segment size limit.

-lt per-process CPU time limit [ , warn-limit ]
    Set a per-process maximum and an optional warning CPU time limit for all processes that constitute the running batch request. If any process comprising the running request exceeds the maximum per-process CPU time limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process CPU warning time limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lt flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-process CPU time limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process CPU time limit.

-lT per-request CPU time limit [ , warn-limit ]
    Set a per-request maximum and an optional warning cumulative CPU time limit for all of the processes that constitute the running batch request. If the sum of the CPU times consumed by all of the processes in the batch request exceeds the maximum per-request CPU time limit for the request, then all of the processes in the request will be terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-request CPU warning time limits. When such a warning limit is exceeded, a signal is delivered to one or more of the processes comprising the running request, depending upon the underlying UNIX implementation.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lT flag with its associated limit value(s) appears within the batch request script file.

    Not all UNIX implementations support per-request CPU time limits. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-request CPU time limit.

-lv per-process temporary file size limit [ , warn-limit ]
    Set a per-process maximum and an optional warning temporary (volatile) file size limit for all processes that constitute the running batch request. If any process comprising the running request attempts to write to a temporary file such that the file size would increase beyond the maximum per-process temporary-file size limit for the request, then that process is terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-process warning temporary-file size limits. When a warning limit is exceeded, a signal as determined by the underlying UNIX implementation is delivered to the offending process.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lv flag with its associated limit value(s) appears within the batch request script file.

    At the time of this writing, no UNIX operating system known to the author supported a distinction at the kernel level between permanent and temporary files. Certainly, a pseudo-temporary file can be constructed by creating it, and then unlinking its pathname. However, the file space allocated for such a file will be allocated from the same pool of disk space that all other UNIX files are allocated from.

    Until a mechanism is implemented in the kernel that knows about permanent and temporary files, this limit cannot be supported in the sense most useful for batch requests, namely the strict enforcement of disk quotas for permanent versus temporary files.

    Until such a time, this limit will simply be ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process temporary-file size limit.

-lV per-request temporary file space limit [ , warn-limit ]
    Set a per-request maximum and an optional warning cumulative temporary (volatile) file space limit for all processes that constitute the running batch request. If any process comprising the running request attempts to write to a temporary file such that the file space consumed by all temporary files opened for writing by all of the processes in the batch request would increase beyond the maximum per-request temporary-file space limit for the request, then all of the processes in the request will be terminated by a signal chosen by the underlying UNIX implementation.

    The ability to specify an optional warning limit exists for those UNIX operating systems that support per-request warning temporary-file space limits. When such a warning limit is exceeded, a signal is delivered to one or more of the processes comprising the running request, depending upon the underlying UNIX implementation.

    If a maximum limit (and optional warning limit) specification is comprised of two or more tokens separated by whitespace characters, then the specification must be enclosed within double quotes, or otherwise escaped such that Qsub and the shell will interpret the entire specification as a single character string token. This caveat also applies when an embedded default -lV flag with its associated limit value(s) appears within the batch request script file.

    At the time of this writing, no UNIX operating system known to the author supported a distinction at the kernel level between permanent and temporary files. Certainly, a pseudo-temporary file can be constructed by creating it, and then unlinking its pathname. However, the file space allocated for such a file will be allocated from the same pool of disk space that all other UNIX files are allocated from.

    Until a mechanism is implemented in the kernel that knows about permanent and temporary files, this limit cannot be supported in the sense most useful for batch requests, namely the strict enforcement of disk quotas for permanent versus temporary files.

    Until such a time, this limit will simply be ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a temporary-file space limit.

-lw per-process working set size limit
    Set a per-process maximum working set size limit for all processes that constitute the running batch request.

    Not all UNIX implementations support per-process working set size limits, and such a limit only makes sense in the context of a paged virtual memory machine. If a batch request specifies this limit, and the machine upon which the batch request is eventually run does not support the enforcement of this limit, then the limit is simply ignored.

    See the section entitled LIMITS for more information on the implementation of batch request limits, and for a description of the precise syntax of a per-process working set size limit.

-mb Send mail to the user on the originating machine when the request begins execution. If the -mu flag is also present, then mail is sent to the user specified for the -mu flag instead of to the invoking user.

-me Send mail to the user on the originating machine when the request has ended execution. If the -mu flag is also present, then mail is sent to the user specified for the -mu flag instead of to the invoking user.

-mr Send mail to the user on the originating machine when the request is restarted. If the -mu flag is also present, then mail is sent to the user specified for the -mu flag instead of to the invoking user.

-mt Send mail to the user on the originating machine when the request is transferred to the execution machine. If the -mu flag is also present, then mail is sent to the user specified for the -mu flag instead of to the invoking user.

-mu user-name
    Specify that any mail concerning the request should be delivered to the user user-name. User-name may be formatted either as user (containing no `@' characters), or as user@machine. In the absence of this flag, any mail concerning the request will be sent to the invoker on the originating machine.

-nr Declare that the request is non-restartable. If this flag is specified, then the request will not be restarted by NQS upon system boot if the request was running at the time of an NQS shutdown or system crash.

    Requests that are not running are always preserved across host crashes and NQS shutdowns for later requeueing, with or without this flag.

    When NQS is shutdown via an operator command to the qmgr(1M) NQS control program, a SIGTERM signal is sent to all processes comprising all running NQS requests on the local host, and all queued NQS requests are barred from beginning execution. After a finite number of seconds have elapsed (typically sixty, but this value can be overridden by the operator), all remaining processes comprising all remaining running NQS requests are killed by the signal: SIGKILL.

    For an NQS request to be properly restarted after an NQS shutdown, the -nr flag must not be specified, and the spawned batch request shell must ignore SIGTERM signals (which is done by default). The spawned batch request shell must also not exit before the final SIGKILL arrives. Since the batch request shell is simply spawning commands and programs, waiting for their completion, this implies that the commands and programs being executed by the batch request shell must also be immune to SIGTERM signals, saving state as appropriate before being killed by the final SIGKILL signal.

    See the CAVEATS section below for more discussion concerning the restartability of batch requests.

-o [machine:][[/]path/] stdout-filename
    Direct output generated by the batch request which is sent to the stdout file to the named [machine:][[/]path/] stdout-filename.

    The brackets "[" and "]" enclose optional portions of the stdout destination machine, path, and stdout-filename.

    If no explicit machine destination is specified, then the destination machine defaults to the machine that originated the batch request, or to the machine that will eventually run the request, depending on the respective absence, or presence of the -ko flag.

    If no machine destination is specified, and the path/filename does not begin with a "/", then the current working directory is prepended to create a fully qualified path name, provided that the -ko (keep stdout) flag is also absent. In all other cases, any partial path/filename is interpreted relative to the user's home directory on the stdout destination machine.

    If no -o [machine:][[/]path/] stdout-filename flag is specified, then all stdout output for the batch request is sent to the file whose name consists of the first seven characters of the request-name followed by the characters: ".o", followed by the request sequence number portion of the request-id discussed below. In the absence of the -ko flag, this default stdout output file will be placed on the machine that originated the batch request in the current working directory, as defined when the batch request was first submitted. Otherwise, the file will be placed in the user's home directory on the execution machine.

    If the stdout-filename contains any # characters, they will be replaced by the numeric part of the request's sequence number. This provides extra flexibility in naming the stdout output file. The # character is ignored in all other parts of the file path.

-p priority
    Explicitly assign an intra-queue priority to the request. The specified priority must be an integer, and must be in the range [0..63], inclusive. A value of 63 defines the highest intra-queue request priority, while a value of 0 defines the lowest. This priority does not determine the execution priority of the request. This priority is only used to determine the relative ordering of requests within a queue.

    When a request is added to a queue, it is placed at a specific position within the queue such that it appears ahead of all existing requests whose priority is less than the priority of the new request. Similarly, all requests with a higher priority will remain ahead of the new request when the queueing process is complete. When the priority of the new request is equal to the priority of an existing request, the existing request takes precedence over the new request.

    If no intra-queue priority is chosen by the user, then NQS assigns a default value.

-q queue-name
    Specify the queue to which the batch request is to be submitted. If no -q queue-name specification is given, then the user's environment variable set is searched for the variable: QSUB_QUEUE. If this environment variable is found, then the character string value for QSUB_QUEUE is presumed to name the queue to which the request should be submitted. If the QSUB_QUEUE environment variable is not found, then the request will be submitted to the default batch request queue, if defined by the local system administrator. Otherwise, the request cannot be queued, and an appropriate error message is displayed to this effect.

-r request-name
    Assign the specified request-name to the request. In the absence of an explict -r request-name specification, the request-name defaults to the name of the script file (leading path name removed) given on the command line. If no script file was given, then the default request-name assigned to the request is STDIN.

    In all cases, if the request-name is found to begin with a digit, then the character 'R' is prepended to prevent a request-name from beginning with a digit. All request-names are truncated to a maximum length of 15 characters.

-re By default, all output generated by a batch request sent to the stderr file is temporarily into a file residing in a protected directory on the machine that executes the request. When the batch request completes execution, this file is then spooled to its final destination, possibly on a remote machine.

    This default spooling of the stderr output file is done to reduce the network traffic costs incurred by the submitter (owner) of a batch request which is to return its stderr output to a remote machine upon completion. In some cases, this behavior is not desired. If it is necessary to override this behavior, then the -re flag can be specified which says that stderr output produced by the request is to be immediately written to the final destination file, as output is generated, no matter what the networking cost.

    Circumstances may not allow a given NQS implementation to support this flag, in which case it will be ignored, and the stderr output file will simply be spooled as it ordinarily would without this flag.

-ro By default, all output generated by a batch request sent to the stdout file is temporarily spooled into a file residing in a protected directory on the machine that executes the request. When the batch request completes execution, this file is then spooled to its final destination, possibly on a remote machine.

    This default spooling of the stdout output file is done to reduce the network traffic costs incurred by the submitter (owner) of a batch request which is to return its stdout output to a remote machine upon completion. In some cases, this behavior is not desired. If it is necessary to override this behavior, then the -ro flag can be specified which says that stdout output produced by the request is to be immediately written to the final destination file, as output is generated, no matter what the networking cost.

    Circumstances may not allow a given NQS implementation to support this flag, in which case it will be ignored, and the stdout output file will simply be spooled as it ordinarily would without this flag.

-rs Indicate that the request is restartable. By default requests are not restartable. If this switch is used, it is the responsibility of the user to ensure that the request will execute correctly if restarted, by the use of appropriate programming techniques.

-s shell-name
    Specify the absolute path name of the shell which will be used to interpret the batch request script. This flag unconditionally overrides any shell strategy configured on the execution machine for selecting which shell to spawn in order to interpret the batch request script.

    In the absence of this flag, the NQS system at the execution machine will use one of three (3) distinct shell choice strategies for the execution of the batch request. Any one of the three strategies can be configured by a system administrator for each NQS machine.

    The three shell strategies are called:

        fixed,
        free, and
        login.

    These shell strategies respectively cause the configured fixed shell to be exec'd to interpret all batch requests, cause the user's login shell as defined in the password file to be exec'd which in turn chooses and spawns the appropriate shell for interpreting the batch request script, or cause only the user's login shell to be exec'd to interpret the script.

    A shell strategy of fixed means that the same shell (as configured by the system administrator), will be used to execute all batch requests.

    A shell strategy of free will run the batch request script exactly as would an interactive invocation of the script, and is the default NQS shell strategy.

    The strategies of fixed and login exist for host systems that are short on available free processes. In these two strategies, a single shell is exec'd, and that same shell is the shell that executes all of the commands in the batch request script.

    The shell strategy configured for a particular NQS system can be determined by the qlimit(1) command.

-v  Print the current version of the Qsub program.

-x  Export all environment variables. When a batch request is submitted, the current values of the environment variables: HOME, SHELL, PATH, LOGNAME (not all systems), USER (not all systems), MAIL, and TZ are saved for later recreation when the batch request is spawned, as the respective environment variables: QSUB_HOME, QSUB_SHELL, QSUB_PATH, QSUB_LOGNAME, QSUB_USER, QSUB_MAIL, and QSUB_TZ. Unless the -x flag is specified, no other environment variables will be exported from the originating host for the batch request. If the -x flag option is specified, then all remaining environment variables whose names do not conflict with the automatically exported variables, are also exported with the request. These additional environment variables will be recreated under the same name when the batch request is spawned.

-z  Submit the batch request silently. If the request is submitted successfully, then no messages are displayed indicating this fact. Error messages will, however, always be displayed.

If the batch request is successfully submitted and the -z flag has not been specified, the request-id of the request is displayed to the user. A request-id is always of the form: seqno.hostname where seqno refers to the sequence number assigned to the request by NQS, and hostname refers to the name of originating local machine. This identifier is used throughout NQS to uniquely identify the request, no matter where it is in the network.

The following events take place in the following order when an NQS batch request is spawned:

    The process that will become the head of the process group for all processes comprising the batch request is created by NQS.

    Resource limits are enforced.

    The real and effective group-id of the process is set to the group-id as defined in the local password file for the request owner.

    The real and effective user-id of the process is set to the real user-id of the batch request owner.

    The user file creation mask is set to the value that the user had on the originating machine when the batch request was first submitted.

    It the user explicitly specified a shell by use of the -s flag (discussed above), then that user-specified shell is chosen as the shell that will be used to execute the batch request script. Otherwise, a shell is chosen based upon the shell strategy as configured for the local NQS system (see the earlier discussion of the -s flag for a description of the possible shell strategies that can be configured for an NQS system).

    The environment variables of HOME, SHELL, PATH, LOGNAME (not all systems), USER (not all systems), and IL are set from the user's password file entry, as though the user had logged directly into the execution machine.

    The environment string: ENVIRONMENT=BATCH is added to the environment so that shell scripts (and the user's .profile (Bourne shell) or .cshrc and .login (C-shell) scripts), can test for batch request execution when appropriate, and not (for example) perform any setting of terminal characteristics, since a batch request is not connected to an input terminal.

    The environment variables of QSUB_WORKDIR, QSUB_HOST, QSUB_REQNAME, and QSUB_REQID are added to the environment. These environment variables equate to the obvious respective strings of the working directory at the time that the request was submitted, the name of the originating host, the name of the request, and the request request-id.

    All of the remaining environment variables saved for recreation when the batch request is spawned are added at this point to the environment. When a batch request is initially submitted, the current values of the environment variables: HOME, SHELL, PATH, LOGNAME (not all systems), USER (not all systems), MAIL, and TZ are saved for later recreation when the batch request is spawned. When recreated however, these variables are added to the environment under the respective names: QSUB_HOME, QSUB_SHELL, QSUB_PATH, QSUB_LOGNAME, QSUB_USER, QSUB_MAIL, and QSUB_TZ, to avoid the obvious conflict with the local version of these environment variables. Additionally, all environment variables exported from the originating host by the -x option are added to the environment at this time.

    The current working directory is then set to the user's home directory on the execution machine, and the chosen shell is exec'd to execute the batch request script with the environment as constructed in the steps outlined above.

In all cases, the chosen shell is exec'd as though it were the login shell. If the Bourne shell is chosen to execute the script, then the .profile file is read. If the C-shell is chosen, then the .cshrc and .login scripts are read.

If the user did not specify a specific shell for the batch request, then NQS chooses which shell should be used to execute the shell script, based on the shell strategy as configured by the system administrator (see the earlier discussion of the -s flag).

In such a case, a free shell strategy instructs NQS to execute the login shell for the user (as configured in the password file). The login shell is in turn instructed to examine the shell script file, and fork another shell of the appropriate type to interpret the shell script, behaving exactly as an interactive invocation of the script.

Otherwise no additional shell is spawned, and the chosen fixed or login shell sequentially executes the commands contained in the shell script file until completion of the batch request.

QUEUE TYPES
NQS supports four different queue types that serve to provide four very different functions. These four queue types are known as batch, device, pipe, and network.

The queue type of batch can only be used to execute NQS batch requests. Only NQS batch requests created by the qsub(1) command can be placed in a batch queue.

The queue type of device can only be used to execute NQS device requests. Only NQS device requests created by the qpr(1) command can be placed in a device queue.

Queues of type pipe are used to send NQS requests to other pipe queues, or to request destination queues of type batch or device, as appropriate for the request type. In general, pipe queues, in combination with network queues, act as the mechanism that NQS uses to transport both batch and device requests to distant queues on other remote machines. It is also perfectly legal for a pipe queue to transport requests to queues on the same machine.

When a pipe queue is defined, it is given a destination set which defines the set of possible destination queues for requests entered in that pipe queue. In this manner, it is possible for a batch or device request to pass through many pipe queues on its way to its ultimate destination, which must eventually be a queue of type batch or device (matching the request type).

Each pipe queue has an associated server. For each request handled by a pipe queue, the associated server is spawned which must select a queue destination for the request being handled, based on the characteristics of the request, and upon the characteristics of each queue in the destination set defined for the pipe queue.

Since a different server can be configured for each pipe queue, and batch and device queues can be endowed with the pipeonly attribute that will only admit requests queued via another pipe queue, it is possible for respective NQS installations to use pipe queues as a request class mechanism, placing requests that ask for different resource allocations in different queues, each of which can have different associated limits and priorities.

It is also completely possible for a pipe queue server, when handling a request, to discover that no destination queue will accept the request, for various reasons which can include insufficient resource limits to execute the request, or a lack of a corresponding account or privilege for queueing at a remote queue. In such circumstances, the request will be deleted, and the user will be notified by mail (see mail(1)).

The queue type of network, as alluded to earlier, is implicitly used by pipe queues to pass NQS requests between machines, and is also used to handle queued file transfer operations.

QUEUE ACCESS
NQS supports queue access restrictions. For each queue of queue type other than network, access may be either unrestricted or restricted. If access is unrestricted, any request may enter the queue. If access is restricted, a request can only enter the queue if the requester or the requester's login group has been given access to that queue (see qmgr(1M)). Requests submitted by root are an exception; they are always queued, even if root has not explicitly been given access.

Use qstat(1) to determine who has access to a particular queue.

LIMITS
NQS supports many batch request resource limit types that can be applied to an NQS batch request. The existence of configurable resource limits allows an NQS user to set resource limits within which his or her request must execute. In many instances, smaller limit values can result in a more favorable scheduling policy for a batch request.

The syntax used to specify a limit-value for one of the limit-flags (-llimit-letter-type), is quite flexible, and describes values for two general limit categories. These two general categories respectively deal with time related limits, and those limits are not time related.

For finite CPU time limits, the limit-value is expressed in the reasonably obvious format:

        [[hours :] minutes : ] seconds [.milliseconds]

Whitespace can appear anywhere between the principal tokens, with the exception that no whitespace can appear around the decimal point.

Example time limit-values are:

        1234 : 58 : 21.29    - 1234 hrs 58 mins 21.290 secs
        12345    - 12345 seconds
        121.1    - 121.100 seconds
        59:01    - 59 minutes and 1 second

For all other finite limits (with the exclusion of the nice limit-value -ln), the acceptable syntax is:

        .fraction [units]

or

        integer [.fraction] [units]

where the integer and fraction tokens represent strings of up to eight decimal digits, denoting the obvious values. In both cases, the units of allocation may also be specified as one of the case insensitive strings:

        b    - bytes
        w    - words
        kb    - kilobytes (2^10 bytes)
        kw    - kilowords (2^10 words)
        mb    - megabytes (2^20 bytes)
        mw    - megawords (2^20 words)
        gb    - gigabytes (2^30 bytes)
        gw    - gigawords (2^30 words)

In the absence of any units specification, the units of bytes are assumed.

For all limit types with the exception of the nice limit-value (-ln), it is possible to state that no limit should be applied. This is done by specifying a limit-value of "unlimited", or any initial substring thereof. Whenever an infinite limit-value is specified for a particular resource type, then the batch request operates as though no explicit limits have been placed upon the corresponding resource, other than by the limitations of the physical hardware involved.

The complications caused by batch request resource limits first show up when queueing a batch request in a batch queue. This operation is described in the following paragraphs.

If a batch request specifies a limit that cannot be enforced by the underlying UNIX implementation, then the limit is simply ignored, and the batch request will operate as though there were no limit (other than the obvious physical maximums), placed upon that resource type. (See the qlimit(1) command to find out what limits are supported by a given machine.)

For each remaining finite limit that can be supported by the underlying UNIX implementation that is not a CPU time-limit or UNIX execution-time nice-value-limit, the limit-value is internally converted to the units of bytes or words, whichever is more appropriate for the underlying machine architecture.

As an example, a per-process memory size limit value of 321 megabytes would be interpreted as 321 x 2^20 bytes, provided that the underlying machine architecture was capable of directly addressing single bytes. Thus the original limit coefficient of 321 would become 321 x 2^20. On a machine that was only capable of addressing words, the appropriate conversion of 321 x 2^20 bytes / #of-bytes-per-word would be performed.

If the result of such a conversion would cause overflow when the coefficient was represented as a signed-long integer on the supporting hardware, then the coefficient is replaced with the coefficient of: of 2^N-1 where N is equal to the number of bits of precision in a signed long integer. For typical 32-bit machines, this default extreme limit would therefore be 2^31-1 bytes. For word addressable machines in the supercomputer class supporting 64-bit long integers, the default extreme limit would be 2^63-1 words.

Lastly, some implementations of UNIX reserve coefficients of the form: 2^N-1 as synonymous with infinity, meaning no limit is to be applied. For such UNIX implementations, NQS further decrements the default extreme limit so as not to imply infinity.

The identical internal conversion process as described in the preceding paragraphs is also performed for each finite limit-value configured for a particular batch queue using the qmgr(1M) program.

After all of the applicable limit-values have been converted as described above, each such resulting limit-value is then compared against the corresponding limit-value as configured for the destination batch queue. If, for every type of limit, the batch queue limit-value is greater than or equal to the corresponding batch request limit-value, then the request can be successfully queued, provided that no other anomalous conditions occur. For request infinity limit-values, the corresponding queue limit-value must also be configured as infinity.

These resource limit checks are performed irrespective of the batch request arrival mechanism, either by a direct use of the qsub(1) command, or by the indirect placement of a batch request into a batch queue via a pipe queue. It is impossible for a batch request to be queued in an NQS batch queue if any of these resource limit checks fail.

Finally, if a request fails to specify a limit-value for a resource limit type that is supported on the execution machine, then the corresponding limit-value configured for the destination queue becomes the limit-value for the unspecified request limit.

Upon the successful queueing of a request in a batch queue, the set of limits under which the request will execute is frozen, and will not be modified by subsequent qmgr(1M) commands that alter the limits of the containing batch queue.

CAVEATS
When an NQS batch request is spawned, a new process-group is established such that all processes of the request exist in the same process-group. If the qdel(1) command is used to send a signal to an NQS batch request, the signal is sent to all processes of the request in the created process-group. However, should one or more processes of the request choose to successfully execute a setpgrp(2) system call, then such processes will not receive any signals sent by the qdel(1) command. This can lead to "rogue" requests whose constituent processes must be killed by other means such as the kill(1) command. However, NQS takes advantage of any UNIX implementations that provide a mechanism of "locking" a process, and all of its subsequent children in a particular process-group. For such UNIX implementations, this problem does not occur.

It is extremely wise for all processes of an NQS request to catch any SIGTERM signals. By default, the receipt of a SIGTERM signal causes the receiving process to die. NQS sends a SIGTERM signal to all processes in the established process-group for a batch request as a notification that the request should be prepared to be killed, either because of an abort queue command issued by an operator using the qmgr(1M) program, or because it is necessary to shutdown NQS and all running requests as part of a general shutdown procedure of the local host.

It must be understood that the spawned shell ignores SIGTERM signals. If the current immediate child of the shell does not ignore or catch SIGTERM signals, then it will be killed by the receipt of such, and the shell will go on to execute the next command from the script (if there is one). In any case, the shell will not be killed by the SIGTERM signal, though the executing command will have been killed.

After receiving a SIGTERM signal delivered from NQS, a process of a batch request typically has sixty seconds to get its "house in order" before receiving a SIGKILL signal (though the sixty second duration can be changed by the operator).

All batch requests terminated because of an operator NQS shutdown request that did not specify the -nr flag are considered restartable by NQS, and are requeued (provided that the batch request shell process is still present at the time of the SIGKILL signal broadcast as discussed above), so that when NQS is rebooted, such batch requests will be respawned to continue execution. It is however, up to the user to make the request restartable by the appropriate programming techniques. NQS simply spawns the request again as though it were being spawned for the first time.

Upon completion of a batch request, a mail message can be sent to the submitter (see the discussion of the -me flag above). In many instances, the completion code of the spawned Bourne or C-Shell will be displayed. This is merely the value returned by the shell through the exit(2) system call.

Lastly, there is no good way to echo commands executed by unmodified versions of the Bourne and C shells. While the C-shell can be spawned in such a fashion as to echo the commands it executes, it is often very difficult to tell an echoed command from genuine output produced by the batch request, because no "magic" character such as a '$' is displayed in front of the echoed command. The Bourne shell does not support any echo option whatsoever.

Thus, one of the better ways to write the shell script for a batch request is to place appropriate lines in the shell script of the form:

        echo "explanatory-message"

where the echoed message should be a meaningful message chosen by the user.

LIMITATIONS AND IMPLEMENTATION NOTES
Network queues have not yet been implemented.

The original "@$" syntax used to introduce embedded argument flags was chosen because it rarely conflicts with anything else present in a shell script file.

SEE ALSO
mail(1), qcat(1), qdel(1), qdev(1), qlimit(1), qpr(1), qstat(1),
kill(2), setpgrp(2), signal(2) in the NPSN UNIX System Programmer Reference Manual.
qmgr(1M) in the NPSN UNIX System Administrator Reference Manual.

NPSN HISTORY
Origin: Sterling Software Incorporated

August 1985 - Brent Kingsbury, Sterling Software
Original release.

May 1986
Second release.

August 1994 - John Roman, Monsanto Company
Release 3.36