UXD(1)

NAME
uxd - UTF-8 hex dumper

SYNOPSIS

uxd [-n] [-c colors] [-d data] [-l length] [-o offset] [[-s | -S] seekpos] [-[1aAbcijmnprtTuvw] ...] [file | -]

uxd --help | -h | --version | -v

DESCRIPTION

uxd is a hex dump utility that's aware of UTF-8 multibyte sequence semantics, and uses colorized output to indicate which byte sequences go with which human-readable characters.

Input is read from file, or standard input if file is missing or given as -. The input is treated as UTF-8 encoded Unicode. Since ASCII is a subset, uxd works fine on plain ASCII files too. Other encodings such as UTF-16, ISO-8859-*, Shift-JIS, etc, can be used, but uxd won't handle these any better than a regular hex-dump utility such as xxd.

Output is written to standard output, which is normally a terminal. It's assumed that the terminal supports ANSI-style color and UTF-8. See TERMINAL SUPPORT below. If you want to pipe the output to a pager, try less -R.

Note: this man page requires your man command to support UTF-8 embedded in the page. If it looks mangled, try viewing the source (uxd.rst) in a text editor.

OPTIONS

These options can be used on the command line, and/or in the UXD_OPTS environment variable. The command line takes precedence over the environment.

Options can be bundled: -ubc1234 is the same as -u -b -c 1234. The one exception is the -n option, which should appear by itself.

The options that accept numbers (-l, -o, -s, and -S) allow decimal, or hex (with 0x prefix). You can use the suffixes k, m, g, and t for power-of-2 based kilobytes, megabytes, gigabytes, or terabytes (e.g. 1k is 1024 bytes), as well as K, M, G, and T for power-of-10 based (e.g. 1K is 1000 bytes). Also, a decimal point can be used: 1.5K is 1500 bytes, 1.5k is 1536 bytes.

--  No more options; whatever comes after this is a filename, even if it begins with -.

-1  Don't alternate colors.

-a  Don't dump lines that consist entirely of ASCII characters (codepoints U+00 to U+7F).

-A  ASCII output. Unprintable characters and UTF-8 sequences are rendered as periods (like xxd does). Also, spaces are rendered as spaces rather than the ␣ character. Color highlighting is still done.

-b  Bold output. This may be more or less readable, depending on your terminal and its color settings. Ignored if -m given.

-c nnnnn
    Set the colors to use. Must be 1 to 5 digits, from 0 to 7. These are standard ANSI colors:
    0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
    black | red | green | yellow | blue | purple | cyan | white

    The first 2 digits are the alternating colors for normal characters, the 3rd and 4th (optional) are the alternating colors for non-printable and space characters, and the 5th (optional) is for invalid UTF-8 sequences.

    Default colors are 23561. If fewer than 5 colors are supplied, the remaining colors keep their default values.

    Note that the default color set doesn't include white or black: usually one of these is the terminal's background color. Also, it avoids blue, because blue text is hard to read on many terminals.

    This option also disables a prior -m option.

-d data
    Dump this data, instead of reading from a file or stdin. If data contains spaces or shell metacharacters, make sure you remember to quote it. Only one -d option can be given.

-h, --help
    Print built-in usage message and exit.

-i  After dumping, print information about the input: number of bytes, characters, ASCII (one-byte) characters, multi-byte characters, and bad sequences.

-j  Java mode (aka MUTF-8). Identical to UTF-8 except the overlong 0xc0 0x80 encoding for codepoint U+0000 (aka NUL), is highlighted in purple and not counted as an error. This may be useful for looking at serialized data created by Java programs.

-l length
    Stop dumping after length bytes (not characters). If the limit is reached in the middle of a multibyte character, the entire character will be dumped. Negative length doesn't make sense, and is an error.

-m  Monochrome mode. Uses underline, bold, reverse video instead of color. Use this if you have trouble distinguishing the colors, or if they look too much like angry fruit salad. Disables prior -b, -c options.

-n  Ignore UXD_OPTS environment variable. This option should not be bundled with other options (e.g. use -n -u, not -nu).

-o offset
    Add this amount to the hex offsets (left column). May be negative, if you can think of a reason to want it to be.

-p  Permissive mode. Turns off error highlighting for overlongs, codepoints above U+10FFFF, and surrogates. Only malformed sequences will be highlighed in red.

-r  Don't highlight multi-byte sequences in reverse video.

-s pos
    Seek in input before starting to dump. pos is bytes, not characters. Positive pos means seek from the start of the input. Negative pos only works on files (not standard input); it means seek backward from EOF.

-S pos
    Same as -s, but the displayed offsets start at 0 rather than the position after seeking. -S 100 is the same as -s 100 -o -100. Works with negative pos, too.

-t  Put terminal in UTF-8 mode, if possible. Prints ESC % G sequence, which may or may not be supported by your terminal (works for xterm(1); does not work for urxvt(1)). To avoid surprises, this option also takes the terminal out of UTF-8 mode before uxd exits, on the theory that you won't need this option if the terminal is normally running in UTF-8 mode. If this assumption turns out to be wrong, you can use the -T option instead.

-T  Same as -T, but leaves the terminal in UTF-8 mode on exit. Can be used to recover from a previous misuse of -t, like so:

    uxd -T -dd

-u  Use uppercase hex digits A-F. Default is lowercase.

-v, --version
    Print version number and exit.

-w  WTF-8 mode. Surrogates U+D800 to U+D8FF will be highlighted in purple and not counted as errors.

OUTPUT FORMAT

The output is designed to fit in an 80-column terminal. If you want HTML output, you might have a look at aha(1).

Each line of output consists of eighteen columns: the offset from the start of the file (in hex; minimum 4 digits), 16 bytes of hex data (or empty cells, if the last line of the dump is for fewer than 16 bytes), and the human-readable form of the same data.

The hex bytes and human-readable data are colorized to make it obvious which bytes make up each character. Since UTF-8 is a variable-width encoding, this means that one character may be composed of up to 4 bytes.

The hex bytes that make up one character are displayed in the same color, which alternates between yellow and green for successive characters. In addition, they have dashes instead of spaces between them. An example would be c3-b1 (for an ñ character).

The 16-byte hex display always has an extra "spacer" column in the center. Normally this is a space, but if a multibyte character spans it, it will be a dash (so there'll be two dashes: c3--b1).

Since the output lines are always 16 hex bytes, multibyte characters can span two lines. When this happens, the character itself will be printed on the first line, along with the first byte(s) in hex. The last hex byte will be followed by a dash, and the next line of hex dump will have the remaining bytes (in the same color as the first byte(s) and character). This sounds complicated, but it's easy to understand once you see it a few times.

EXAMPLE

It's hard to give a proper example, since man pages don't support color. You'll have to use your imagination.

Example copied from the Japanese ls(1) man page:

$ echo デフォル | ./uxd
0000: e3-83-87 e3-83-95 e3-82--a9 e3-83-ab 0a           デフォル↵
      GGGGGGGG YYYYYYYY GGGGGGGGG YYYYYYYY PP           G Y G Y P

The colors are indicated by G/Y/P, for green, yellow, and purple. The character above each letter is displayed in that color.

From the colorization, and from the dashes between the bytes, it's obvious that "e3 83 87" is the hex representation of the first character, and that the 2nd is represented by "e3 83 95.

The newline is displayed in purple because it's not a regular printable character. Its human-readable representation is ↵. Note that if a regular ↵ character appears in the input, it'll be rendered in either green or yellow (so you can tell it's not just another newline).

COLORS

The colors in this description are the default ones. They can be changed with the -c option (see above).

green, yellow
    Printable characters (except the space, U+0020) alternate between green and yellow.

purple, cyan
    Spaces and unprintable characters ("control" characters, newlines, tabs, etc) alternate between purple and cyan. These are printed as "visible" characters, e.g. ␣ for the space, ↵ for a newline. Hopefully this is an improvement over the usual practice of printing these as periods, like standard hex dumpers do. The Unicode BOM (byte order marker, U+FEFF) is printed as a purple letter B.

red Invalid UTF-8 sequences. These are rendered as � (U+0FFD) with a red background, to make them stand out. Invalid sequences are:

•   Prefix bytes (>= 0x80) which are not followed by the correct number of continuation bytes (with their high 2 bits set to 10).

•   Continuation bytes that aren't preceded by a valid prefix byte.

•   Truncated UTF-8 sequence at EOF.

    Also, there are sequences that are valid UTF-8 encodings, but not valid Unicode. These are normally rendered with a red background.

•   UTF-16 surrogates (codepoints U+D800 to U+DFFF) [*]. Rendered as S.

•   Codepoints above U+10FFFF, which are disallowed by the Unicode standard [*]. Rendered as >.

•   Overlong encodings (e.g. codepoints U+0000 to U+007F encoded as 2 or more bytes) [*]. Rendered as O.

    Each error-highlighted sequence will increment the "Bad sequences" count, if the -i option is used.

    For items marked with [*], the -j, -p, and/or -w options can disable error highlighting for this type of error. They will be displayed in purple or cyan rather than red.

TERMINAL SUPPORT

uxd should work with any modern terminal that supports color, ANSI-style escape sequences, Unicode, and UTF-8 rendering.

The author's testing is done primarily with urxvt(1). Other terminals aren't tested as often. Some terminals may need UTF-8 enabled, if it's not on by default (either in the terminal's settings or using the -t option to uxd).

Known to work: urxvt, xterm, st, xfce4-terminal, gnome-terminal, kitty, konsole, the Linux console (but see FONTS, below).

Known not to work: rxvt (doesn't support Unicode at all), and its derivatives such as aterm.

uxd also builds and runs correctly on a Mac running a recent version of OSX with Terminal.app.

FONTS

For the human-readable column to display correctly, you'll need a font with lots of glyphs. Try Deja Vu Sans Mono, Symbola, or Quivira (although it's not really a terminal font). If you use urxvt, it searches for glyphs in multiple fonts, so you can use all of the above at once.

Any glyph your font lacks, you'll see as a dotted box, or perhaps a solid block. This isn't something uxd can do anything about; you'll have to use a different font, or (if you use urxvt) add another font to your URxvt*font resource.

The Linux console is capable of rendering UTF-8, but it's incapable of displaying more than 512 glyphs. Most console fonts only define 256, since using more than 256 means the console won't be able to do bold. Expect to see lots of solid or dotted boxes. This isn't specifically a problem with uxd.

FILES

uxd doesn't read any files other than the input file, and doesn't write to any files other than standard output. There's no config file.

ENVIRONMENT

UXD_OPTS
    If this is set, its value is treated as a set of options, which get applied before any command-line options (unless the command-line options inclue -n).

NO_COLOR
    If this is set (to any value), uxd runs in monochrome mode, just as though the -m option were given. This variable is also respected by xxd.

    It's not necessary to have a UTF-8 locale set in e.g. LANG or LC_ALL. Also, the TERM variable is not used.

EXIT STATUS

Zero for success, non-zero for failure.

Failure status will only be returned if uxd failed to open the input file. Invalid input (non-UTF-8) doesn't count as an error; it'll just have lots of red in the output.

LIMITATIONS

There are not bugs, because they're part of the design.

Only UTF-8 and a couple of variants (WTF-8, MUTF-8) are supported. There is no support for UTF-16, UTF-32, UTF-EBCDIC, or any other non-UTF-8 encoding.

There's no support for any output number base except hex.

The input is read one byte at a time, so a search or regex match option would be difficult or impossible to implement.

Seeking backwards from the end of the file is impossible when reading from standard input. The only way to fake this would be to read the whole file into memory at startup, which uxd doesn't do.

BUGS

Combining characters are not handled well. Or at all, really: the 2 characters being combined will have an ANSI color code in between. urxvt at least ignores the color code, so the composite character displays in the color of the first (non-combining) character. I'm not sure what a better solution would be...

COPYRIGHT

Licensed under the WTFPL. See http://www.wtfpl.net/txt/copying/ for details.

AUTHORS

B.  Watson <urchlay@slackware.uk>.

SEE ALSO

xxd(1), bvi(1), utf-8(7), unicode(7), console_codes(4)