= Writing Pattern Files for Gaupol =

<<TableOfContents(3)>>

== Introduction ==

Gaupol's text correction assistant handles four types of automatic corrections to subtitle texts that are based on pattern files: correcting common errors, recognizing preferred line-break points, capitalizing texts and removing texts meant for the hearing impaired. The assistant does regular expression search and replace operations based on patterns defined in desktop-style pattern files. Patterns are written in separate files based on whether they are general, script-, language- or country-specific. Most automatic corrections that you may want to make are specific to either a script or a language. Some correction patterns for the Latin script and English language are shipped with Gaupol, but for other scripts and languages, or for special needs, you may want to write your own. Writing such correction patterns does not require any programming, but does require knowledge of regular expressions.

Pattern files are placed under `~/.local/share/gaupol/patterns` on Unix or `C:\Documents and Settings\USERNAME\Application Data\Gaupol\patterns` on Windows XP or `C:\Users\USERNAME\AppData\Roaming\Gaupol\patterns` on Windows Vista and 7. The text correction assistant will read pattern files from that directory and handle making the actual corrections. Files are named as SCRIPT-[LANGUAGE[-COUNTRY]].EXTENSION, where SCRIPT is a four-letter ISO 15924 code, LANGUAGE a two-letter ISO 639 code, COUNTRY a two-letter ISO 3166 code and EXTENSION one of `common-error`, `line-break`, `capitalization` or `hearing-impaired`. General corrections are written under the general `Zyyy` script. See the [[https://github.com/otsaloma/gaupol/tree/master/data/patterns|development repository]] for an example of the filenames.

Regular expressions in the pattern files are Perl-style regular expressions that follow the [[http://docs.python.org/lib/re-syntax.html|Python syntax]]. As flags, you'll usually want to use DOTALL, MULTILINE and UNICODE.

[[http://home.gna.org/subtitleeditor/|Subtitle Editor]] uses functionally similar, but syntactically different (XML file format and [[http://library.gnome.org/devel/glib/unstable/glib-regex-syntax.html|GRegex regular expressions]]) pattern files. Patterns are shared between Subtitle Editor and Gaupol development teams and you may do the same with the patterns you write or wish to copy from Subtitle Editor hackers.

== Common Fields in Pattern Files ==

There are some fields that are common for all types of pattern files.

 * '''`Name`''' and '''`Description`''' define how the pattern is presented to the user in the text correction assistant.

 * '''`Policy`''' is either `Replace` or `Append` and tells how to handle multiple patterns with the same name. All patterns with the same name are shown to the user as one and the user enables or disables them all at once. The default `Policy` value, `Append`, keeps the existing patterns and adds the new pattern to those. `Replace` discards the existing patterns and adds the new pattern. Pattern files are parsed in order of decreasing generality, i.e. general first, then script, then language and finally country with global patterns parsed before local ones. Thus, the choice given is whether to discard or keep already parsed patterns in more general levels. The main purpose of the `Policy` option is to allow more specific patterns to override less specific patterns, e.g. to allow an English pattern to override a Latin one.

== Types of Pattern Files ==

=== Common Error Correction ===

Common error patterns are for correcting recognizable syntactic errors made by either sloppy humans or by inaccurate image recognition (OCR) software when converting image based subtitles to text. Let's take a look at an example from file `Zyyy.common-error`, i.e. from general corrections that can be used across all scripts, languages and countries.

{{{
[Common Error Pattern]
Name=Multiple consecutive spaces
Description=Replace multiple consecutive spaces with only one
Classes=Human;OCR;
Pattern= {2,}
Flags=DOTALL;MULTILINE;UNICODE;
Replacement=\040
Repeat=False
}}}

 * '''`Classes`''' defines if the pattern is valid for human errors, OCR errors or both. The text correction assistant lets the user choose which of these classes are used to minimize the amount false positives. Especially the OCR corrections may be too aggressive and erroneous if used for human-written text.

 * '''`Pattern`''' and '''`Flags`''' define what to search for as a regular expression.

 * '''`Replacement`''' defines what is substituted for matches of pattern.

 * '''`Repeat`''' can be `True` to do substitutions again and again as long as matches of pattern are found in the text. This is useful if one match of pattern partially overlaps another. Use this with caution as it can cause Gaupol to hang due to an eternal loop.

=== Line-Break ===

Line-break patterns define elegant locations in text which to use for a line-break if the text contains lines too long. Usually these locations are between dialogue or between clauses or between different type of text within one subtitle. Below is an example of a simple line-break pattern for dialogue separated by a single dash. `Pattern`, `Flags` and `Replacement` fields are as in common error patterns, except now the replacement should contain a single newline (\n) in place of a single space in pattern.

{{{
[Line Break Pattern]
Name=Dialogue
Description=Break according to dialogue separated by dashes
Pattern= (- )
Flags=DOTALL;MULTILINE;UNICODE;
Replacement=\n\1
}}}

=== Capitalization ===

Capitalization patterns define which characters should be changed to upper case assuming that the text contains lower case in place of proper capitalization. Below is an example from the English language, where the first person pronoun "I" should always be capitalized. The `Capitalize` field defines whether the capitalization should be done at the start of the pattern (`Start`) or immediately following the pattern (`After`). The former can be used to match words that should start in upper case and the latter to match sentence termination, after which the next sentence should start in upper case.

{{{
[Capitalization Pattern]
Name=First person pronoun
Description=Capitalize the first person pronoun "I"
Capitalize=Start
Pattern=\bi\b
Flags=DOTALL;MULTILINE;UNICODE;
}}}

=== Hearing Impaired Text Removal ===

Hearing impaired patterns are used to remove descriptions of non-verbal sounds used in subtitles for deaf people. While the descriptions of sounds are useful for the intended audience they are often far too verbose for others. Below is a pattern for removing anything between square brackets, which is one common notation used. The fields are similar to those for common error patterns, with the replacement field now usually being empty. After running these removals, Gaupol does a lot of clean-up, e.g. removing lines that have no alphanumeric characters left, so it should be enough to keep these patterns to a recognizable minimum.

{{{
[Hearing Impaired Pattern]
Name=Sound in brackets
Description=Remove description of a sound between square brackets
Pattern=\[.*?\]
Flags=DOTALL;MULTILINE;UNICODE;
Replacement=
}}}
