SPLITJOB(1)

NAME
splitjob - splits data through parallel invocations of jobs

SYNOPSIS
splitjob [options] [commands]

DESCRIPTION
splitjob Reads from stdin, splits and sends to multiple parallel invocations of commands and concatenates their output to stdout.

OPTIONS
A summary of options is included below.

-j <n>
    Set number of parallel jobs (default this will be the given number of commands)

-b <size>
    Set block size for each job (default 1 MB). The size is given in bytes but may end with suffixes k (*1024), M (*1048576) or G (*1073741824).

-m <hex>
    Set magic bytes to separate blocks (default none). The string of magic bytes is a hexadecimal number but might contain wildcards * , example of magic bytes to separate blocks in xz compressed data: fd377a585a00

    For best performance when magic bytes are used to separate blocks of compressed data a block size should be given with flag -b which is at least as big as the block size used for parallel compression.

-B  Use bzip2 magic bytes to separate blocks (same as -m 425a68**314159265359)

-X  Use xz magic bytes to separate blocks (same as -m fd377a585a00)

-L  Use lzip magic bytes to separate blocks (same as -m 4c5a4950)

-r <n>
    Set number of retries for failed jobs (default 3).

-h  Show summary of options.

-v  Show program version and copyright.

EXAMPLES
Use multiple local cores:

splitjob -j 4 bzip2 < bigfile > bigfile.bz2

Use remote machines:

splitjob "ssh host1 gzip" "ssh host2 gzip" < f > f.gz

Use bigger blocks to reduce overhead:

splitjob -j 2 -b 10M gzip < file > file.gz

Parallel decompression of a file which was compressed in parallel with splitjob:

splitjob -X -r 10 -j 5 -b 384M "xz -d -" < file.xz > file

ENVIRONMENT
The variables LANG and LC_ALL have the usual meaning, however there are not many translations available.

SIGNALS
Signals can be used to adjust the number of parallel jobs of a running splitjob process. However, the number of running jobs might not become adjusted until the currently longest running job has been finished.

SIGUSR1 Decreases number of parallel jobs by 1.

SIGUSR2 Increases number of parallel jobs by 1.

BUGS
Splitjob does its best to detect and avoid any problems. If some sub command fails it will by default make some retries before giving up and exiting with a non-zero return value. However, like pbzip2, mpibzip2 and bzip2smp I would like to say: Use at your own risk! Verify the contents of compressed files before relying on them. If splitjob exits with any other return value than 0 its output should be discarded!

Parallel decmpression is a really cool feature, but it has some limitations:

*   Performance gain is limited as splitjob itself requires performance in a single thread to search for magic bytes and most compression algorithms are quicker at decompression than compression.

*   Unless a block size is used which is at least as big as the block size used for compression you will lose performance as the jobs will be unable to complete in parallel.

*   If the compressed data itself happens to contain the magic string parallel decompression with splitjob might fail! Check the return value of splitjob to make sure decompression was successful. Also look for error messages sent to stderr. Since version 3.1 of splitjob attempts to avoid failure are made by merging with data from next job at retry when a failure is detected and magic bytes are used to separate blocks. These attempts might still end in failure if:

*   A single block of compressed data contains more occurances of the magic bytes than the selected number of retries. This will give the error message "Failed again, giving up!" and can be avoided by increasing the number of retries with the -r switch.

*   A job has already sent some of its data to stdout and no longer keeps it in its buffer. This will give the error message "Got too much data and failed!" and can be avoided by increasing the block size with the -b switch.

*   Experimental version 2.2 of splitjob also had the option:
    -G Use gzip magic bytes to separate blocks (same as -m 1f8b) However, option -G was removed in stable version 3.0 as tests showed that gzip could silently fail when attempting parallel decompression.

AUTHOR
Splitjob is mostly written by Henrik Carlqvist, the file CONTRIBUTORS in the source archive contains a complete list of contributors.

SEE ALSO
gzip(1) bzip2(1) xz(1)