Elvis-2.2_0 Regular Expressions

# 5. REGULAR EXPRESSIONS

Elvis uses regular expressions for searching and substitutions. A regular expression is a text string in which some characters have special meanings. This is much more powerful than simple text matching.

## 5.1 Syntax

Elvis' regexp package treats the following one- or two-character strings (called meta-characters) in special ways:

\(subexpression\)
  The `\(` and `\)` metacharacters are used to delimit subexpressions. When the regular expression matches a particular chunk of text, Elvis will remember which portion of that chunk matched the subexpression. The :s/regexp/newtext/ command makes use of this feature.
^
  The `^` metacharacter matches the beginning of a line. If, for example, you wanted to find "foo" at the beginning of a line, you would use a regular expression such as `/^foo/.` Note that `^` is only a metacharacter if it occurs at the beginning of a regular expression; practically anyplace else, it is treated as a normal character. (Exception: It also has a special meaning inside a [character-list] metacharacter, as described below.)
$
  The `$` metacharacter matches the end of a line. It is only a metacharacter when it occurs at the end of a regular expression; elsewhere, it is treated as a normal character. For example, the regular expression `/$$/` will search for a dollar sign at the end of a line.
$name
${name}
  This inserts the value of the named option into the regular expression. If the value contains any metacharacters, they'll be interpreted as expected. For example, after...

      	:set w="[[:alnum:]_:']"

  ...the command `/$w/` will search for the next letter, digit, underscore, colon, or apostrophe.

  Note: This only works if the magicname option is set.

\<
  The `\<` metacharacter matches a zero-length string at the beginning of a word. A word is considered to be a string of 1 or more letters, digits, or underscores. A word can begin at the beginning of a line or after 1 or more non-alphanumeric characters.
\>
  The `\>` metacharacter matches a zero-length string at the end of a word. A word can end at the end of the line or before 1 or more non-alphanumeric characters. For example, `/\<end\>/` would find any instance of the word "end", but would ignore any instances of e-n-d inside another word such as "calendar".
\b, \B, \h, \H
  The `\h` metacharacter matches a zero-length string at either end of a word. The `\H` metacharacter matches a zero-length string which is not at either end of a word; it can be either in the middle of a word, or between words.

  The `\b` and `\B` metacharacters are the same as `\h` and `\H`, but only if the magicperl option is set. Otherwise `\b` is treated as a backspace.

\@
  When you're performing a search in visual mode, and the cursor is on a word before you start typing the search command, then `\@` matches the word at the cursor.
\=
  Ordinarily, the visual mode search command leaves the cursor on the first character of the matching text that it finds. If your regular expression includes a `\=` metacharacter, then it will leave the cursor at the position that matched the `\=`. For example, if you place `\=` at the end of your regular expression, then the cursor will be left after the matching text instead of at the start of it.
.
  The `.` metacharacter matches any single character.

  NOTE: If the magic option is turned off, then `.` is treated as an ordinary, literal character. You should use `\.` to get the meta-character version in this case.

[character-list]
  This matches any single character from the character-list. Inside the character-list, you can denote a span of characters by writing only the first and last characters, with a hyphen between them. If the character-list is preceded by a `^` character, then the list is inverted -- it will match any character that isn't mentioned in the list. For example, `/[a-zA-Z]/` matches any ASCII letter, and `/[^ ]/` matches anything other than a blank.

  NOTE: If the magic option is turned off, then the opening [ is treated as an ordinary, literal character. To get the meta-character behavior, you should use \[character-list] in this case.

  There is no way to quote the ']' or '-' characters, which means that if you want to include those characters as members of the list, you must place them in positions where they couldn't be mistaken for the end of the list or a range. Specifically, ']' can appear only as the first character in the list (immediately after the "[" or "[^" that starts the list) or as the last character in a range. '-' can appear there too, or immediately after the last character of a range. For example, `[])}]` matches a closing bracket, parentheses, or curly brace. `[^-+]` matches any character except '+' or '-'. Probably the trickiest example, `[]-]-]` matches a closing bracket or a '-'. (Note that the range "]-]" matches a single bracket; we wrote it this way so that the following "-" would be in a context where it couldn't be mistaken for a range and so must be a literal '-' character.)

  There are also special cases for some common character lists. When one of the following special symbols appears in a character list, the list will include all appropriate characters for that symbol including the non-ascii characters as indicated by the digraph table. Note that the brackets around these symbols are in addition to the brackets around the whole class. For example, `/[[:alpha:]]/` matches any single letter, and `/[[:alpha:]_][[:alnum:]_]*/` matches any C identifier.

          .----------------.-------------------------------------------.
          | SPECIAL SYMBOL | INCLUDED CHARACTERS                       |
          |----------------|-------------------------------------------|
          |   [:alnum:]    | all letters and digits                    |
          |   [:alpha:]    | all letters                               |
          |   [:ascii:]    | all ASCII characters                      |
          |   [:blank:]    | the space and tab characters              |
          |   [:cntrl:]    | ASCII control characters                  |
          |   [:digit:]    | all digits                                |
          |   [:graph:]    | all printable characters excluding space  |
          |   [:lower:]    | all lowercase letters                     |
          |   [:print:]    | all printable characters including space  |
          |   [:punct:]    | all punctuation characters                |
          |   [:space:]    | all whitespace characters except linefeed |
          |   [:upper:]    | all uppercase characters                  |
          |   [:xdigit:]   | all hexadecimal digits                    |
          ^----------------^-------------------------------------------^

\s, \S, \d, \D, \w, \W, \p, and \P
  These are all shortcuts for certain character lists. The lowercase `\s`, `\d`, `\w`, and `\p` symbols match (respectively) any whitespace character, digit, alphanumeric character, or any printable character. The uppercase versions are the opposites; they match any single character that the lowercase versions don't match.
\I, \i
  These are shortcuts for character lists that are useful to describe character lists. Uppercase `\I` is any character which can appear appear at the front of an identifier, and `\i` is any character which can appear later in the identifier. As presently implemented, these character lists are hardcoded to use character lists that are useful for C/C++, however I expect to make them sensitive the startword and inword lines in the elvis.syn file eventually.
\0, \a, \b, \f, \r, and \t
  These are control characters, just as they would be in C strings. Note that there is no `\n`.
\{n\} or \{n}
  This is a closure operator, which means that it repeats the subexpression that precedes it for a controlled number of times. Closure operators have a high precedence, so normally it'll apply to an expression that matches a single character; if you want it to apply to more than that, you'll need to enclose the preceding expression in \(...\) parentheses.

  The \{n\} or \{n} closure operator, in particular, means that the preceding expression should be repeated exactly n times. For example, `/^-\{80\}$/` matches a line of eighty hyphens, and `/\<[[:alpha:]]\{4}\>/` matches any four-letter word.

\{n,m\} or \{n,m}
  This is a closure operator. It indicates that the preceding expression should be repeated between n and m times, inclusive. If the m is omitted (but the comma is present) then m is taken to be infinity. For example, `/"[^"]\{3,5\}"/` matches any pair of quotes which contains three, four, or five non-quote characters. `/.\{81,}/` matches any line which contains more than 80 characters.
*
  The * metacharacter is a closure operator. It indicates that the preceding expression can be repeated zero or more times. It is equivalent to \{0,\}. For example, `/.*/` matches a whole line.

  NOTE: If the magic option is turned off, then * is treated as an ordinary, literal character. You should use \* to get the meta-character version in this case.

\+
  The \+ metacharacter is a closure operator. It indicates that the preceding expression can be repeated one or more times. It is equivalent to \{1,\}. For example, `/.\+/` matches a whole line, but only if the line contains at least one character. It doesn't match empty lines.
\?
  The \? metacharacter is a closure operator. It indicates that the preceding expression is optional -- that is, that it can occur 0 or 1 times. It is equivalent to \{0,1\}. For example, `/no[ -]\?one/` matches "no one", "no-one", or "noone".
\Q, \V, \E
  These change the way that any following metacharacters are parsed. After \Q, all metacharacters require a leading backslash; i.e., any text that doesn't involve backslashes is considered to be literal text. After \V, the traditional vi metacharacters are recognized when used without backslashes but all others do require backslashes. After \E, normal parsing rules are used, under the influence of the magicchar, magicperl, and magicname options. If nomagic is set then these metacharacters have no effect.

Anything else is treated as a normal character which must exactly match a character from the scanned text. The special strings may all be preceded by a backslash to force them to be treated normally.

Normally, the closure operators (`\{`m`,`n`}`, `*`, `\+`, and `\?`) are "greedy", meaning they try to match as much text as possible. You can convert any of them into a "non-greedy" version (matching as few as possible) by placing an extra `\?` metacharacter after the closure operator. For example, in the text "one2three4five6seven8nine", the greedy regular expression `/\d.*\d/` matches "2three4five6seven8", while the non-greedy `/\d.*\?\d/` matches just "2three4".

## 5.2 Substitutions

The :s command has at least two arguments: a regular expression, and a substitution string. The text that matched the regular expression is replaced by text which is derived from the substitution string.

You can use any punctuation character to delimit the regular expression and the replacement text. The first character after the command name is used as the delimiter. Most folks prefer to use a slash character most of the time, but if either the regular expression or the replacement text uses a lot of slashes, then some other punctuation character may be more convenient.

Most other characters in the substitution string are copied into the text literally but a few have special meaning:

    .-------.----------------------------------------------------------.
    |SYMBOL | MEANING                                                  |
    |-------|----------------------------------------------------------|
    |  ^M   | Insert a newline (instead of a carriage-return)          |
    |   &   | Insert a copy of the original text                       |
    |   ~   | Insert a copy of the previous replacement text           |
    |  \1   | Insert a copy of that portion of the original text which |
    |       |      matched the first set of \( \) parentheses          |
    | \2-\9 | Do the same for the second (etc.) pair of \( \)          |
    |  \U   | Convert following characters to uppercase                |
    |  \L   | Convert following characters to lowercase                |
    |  \E   | End the effect of \U or \L                               |
    |  \u   | Convert the next character to uppercase                  |
    |  \l   | Convert the next character to lowercase                  |
    |  \#   | Insert the line number, as a string of digits            |
    |  \0   | Insert a nul character                                   |
    |  \a   | Insert a bell character                                  |
    |  \b   | Insert a backspace character                             |
    |  \f   | Insert a form-feed character                             |
    |  \n   | Insert a line-feed character                             |
    |  \r   | Insert a carriage-return character                       |
    |  \t   | Insert a tab character                                   |
    | $name | Value of name option (only if magicname is set)          |
    |${name}| Alternative form of $name                                |
    ^-------^----------------------------------------------------------^

These may be preceded by a backslash to force them to be treated normally. The delimiting character can also be preceded by a backslash to include it in either the regular expression or the substitution string.

Traditionally `\0` was a synonym for the `&` symbol -- they both inserted a copy of the matching text. Elvis breaks from tradition here to make `\0` insert a NUL character because there would otherwise be no way to have a substitution insert a NUL character.

## 5.3 Options

Elvis has several options which affect the way regular expressions are used. These options may be examined or set via the :set command.

The first option is called "[no]magic". This is a boolean option, and it is "`magic`" (TRUE) by default. It selects between two different notations for metacharacters in a regular expression. While in `nomagic` mode, all of the metacharacters except `^` and `$` loose their special meaning unless they're prefixed with a backslash. In the normal `magic` mode, metacharacters listed in the magicchar option's value don't require a leading backslash but all others do.

The "magicchar" option is intended to allow you to alter Elvis' regular expression syntax to mimic that of other utilities. Vi is a very old editor, and its regular expression syntax is rather archaic.

The "[no]magicperl" option changes the meanings of some metacharacters to be more Perl-like. Specifically, it causes `\b` to mean "edge of a word" instead of "backspace". In later versions of Elvis, this is expected to change other metacharacters as well.

The "[no]magicname" option enables or disables the use of `$`name in regular expressions. Normally it is disabled, so you can search for dollar signs easily. Aliases will often use ":local magicname" and then compute and use a regular expression. Here's a simple example showing how this might be used:

    	alias findup {
    	  " Search for uppercase version of current word
    	  local magicname n
    	  let n = toupper(current("word"))
    	  /$n/
    	}

The "[no]ignorecase" and "[no]smartcase" options control whether searches and substitutions will distinguish between uppercase and lowercase letters. These are both Boolean options, and both are false by default which causes all searches to be case sensitive. Setting `ignorecase` (while leaving `smartcase` unset) causes all searches to be case insensitive, except in a character list metacharacter. Setting both `ignorecase` and `smartcase` will cause searches to be case insensitive unless the search expression contains some uppercase letters, in which case the search will be case sensitive.

Also, the "[no]wrapscan" and "[no]autoselect" options affect searches.

## 5.4 Examples

This example changes every occurrence of "utilize" to "use":

    :%s/utilize/use/g

This example deletes all whitespace that occurs at the end of a line anywhere in the file.

    :%s/\s\+$//

This example converts the current line to uppercase:

    :s/.*/\U&/

This example underlines each letter in the current line, by changing it into an "underscore backspace letter" sequence. (The ^H is entered as "control-V backspace".):

    :s/[a-zA-Z]/_^H&/g

This example locates the last colon in a line, and swaps the text before the colon with the text after the colon. The first \( \) pair is used to delimit the stuff before the colon, and the second pair delimit the stuff after. In the substitution text, \1 and \2 are given in reverse order to perform the swap:

    :s/\(.*\):\(.*\)/\2:\1/