# Introduction to Internationalization

Internationalization
is the designing of computer systems and applications for users around the world. Such users have different languages and may have different requirements for the functionality and user interface of the systems they operate. In spite of these differences, users want to be able to implement enterprise-wide applications that run at their sites worldwide. These
applications must be able to interoperate across country boundaries, run on a variety of hardware configurations from multiple vendors, and be localized to meet local users' needs. This open, distributed computing environment is the reasoning behind
common open software environments. The internationalization technology identified within this specification provides these benefits to a global market.

## Overview of Internationalization

Multiple environments may exist within a common open system for support of different national languages. Each of these national
environments is called a locale, which considers the language, its characters, fonts, and the customs used to input and format data. The Common Desktop Environment is fully internationalized such that any application can run using any locale installed in the system.

A locale defines the behavior of a program at run time according to the language and cultural conventions of a user's geographical area. Throughout the system,
locales affect the following:

- Encoding and processing of text data

- Identifying the language and encoding of resource files and their text values

- Rendering and layout of text strings

- Interchanging text that is used for interclient text communication

- Selecting the input method (which code set will be generated) and the processing of text data

- Encoding and decoding for interclient text communication

- Bitmap/icon files

- Actions and file types

- User Interface Definition (UID) files

An internationalized application contains no code that is dependent on the user's locale, the characters needed to represent that locale, or any formats (such as date and currency) that the user expects to see and interact with. The desktop accomplishes this by separating language- and culture-dependent information from the application and saving it outside the application.

Figure 1-1 shows the kinds of information that should be external to an application to simplify internationalization.

### Information external to the application

By keeping the language- and culture-dependent information separate from the application source code, the application does not need to be rewritten or recompiled to be marketed in different countries. Instead, the only requirement is for the external information to be localized to accommodate local language and customs.

An internationalized application is also adaptable to the requirements of different native languages, local customs, and character-string encodings. The process of adapting the operation to a particular native language, local custom, or string encoding is called
localization. A
goal of internationalization is to permit localization without program source modifications or recompilation.

For a quick overview of internationalization, refer to X/Open CAE Specification System Interface Definition, Issue 4, X/Open Company Ltd., 1992, ISBN: 1- 872630-46-4.

### Current State of Internationalization

Previously, the industry supplied many variants of internationalization from proprietary functions to the new set of standard functions published by X/Open. Also, there have been different levels of enabling, such as simple ASCII support, Latin/European support, Asian multibyte support, and Arabic/Hebrew bidirectional support.

The interfaces defined
within the X/Open specification are capable of supporting a large set of languages and territories,
including:

`Script` | Description
Latin Language | Americas, Eastern/Western European
Greek | Greece
Turkish | Turkey
East Asia | Japanese, Korean, and Chinese
Indic | Thai
Bidirectional | Arabic and Hebrew

Furthermore, the
goal of the Common Desktop Environment is that localization of these technologies (translation of messages and documentation and other adaptation for local needs) be done in a consistent way, so that a supported user anywhere in the world will find the same common localized environment from vendor to vendor.
End users and administrators can expect a consistent set of localization features that provide a complete application environment for support of global software.

### Internationalization Standards

Through the work of many companies, the functionality of the internationalization application program interface has been standardized over time to include additional requirements and languages, particularly those of East Asia. This work has been centered primarily in the Portable Operating System Interface for Computer Environments (POSIX) and
X/Open specifications. The original X/Open specification was published in the second edition of the X/Open Portability Guide (XPG2) and was based on the Native Language Support product released by Hewlett-Packard. The latest published X/Open internationalization standard is referred to as XPG4.

It is important that each layer within the desktop use the proper set of
standards interfaces defined for internationalization to ensure end users get a consistent, localized interface. The definition of a locale and the common open set of locale-dependent functions are based on the following
specifications:

- X Window System, The Complete Reference to Xlib, Xprotocol, ICCCM, XLFD - X Version, Release 5, Digital Press, 1992, ISBN 1-55558-088-2.

- ANSI/IEEE Standard Portable Operating System Interface for Computer Environments, IEEE.

- OSF™ Motif 1.2 Programmer' Reference, Revision 1.2, Open Software Foundation, Prentice Hall, 1992, ISBN 0-13-643115-1.

- X/Open CAE Specification Commands and Utilities, Issue 4, X/Open Company Ltd., 1992, ISBN 1-872630-48-0.

Within this environment, software developers can expect to develop worldwide applications that are portable, can interoperate across distributed systems (even from different vendors), and can meet the diverse language and cultural requirements of multinational users supported by the desktop standard locales.

### Common Internationalization System

IPG.intro.mkr.5 shows a view of how
internationalization is pervasive across a specific single-host system. The goal is that the applications (clients) are built to be shipped worldwide for the set of locales supported in the underlying system. Using standard interfaces improves access to global markets and minimizes the amount of localization work needed by application developers. In addition, country representatives can be ensured of consistent localization within systems adhering to the principles of the desktop.

#### Common internationalized system

## Locales

Most single-display clients operate in a single locale that is determined at run time from the setting of the environment variable, which is usually `$LANG` or the `xnlLanguage` resource. Locale environment variables, such as `LC_ALL`, `LC_CTYPE`, and `LANG`, can be used to control the environment.

The `LC_CTYPE` category of the locale is used by the environment to identify the locale-specific features used at run time. The fonts and input method loaded by the toolkit are determined by the `LC_CTYPE` category.

Programs that are enabled for internationalization are expected to call the `XtSetLanguageProc()` function (which calls `setlocale()` by default) to set the locale desired by the user. None of the libraries call the `setlocale()` function to set the locale, so it is the responsibility of the application to call `XtSetLanguageProc()` with either a specific locale or some value loaded at run time. If applications are internationalized and do not use `XtSetLanguageProc()`, obtain the locale name from one of the following prioritized sources to pass it to the `setlocale()` function:

- A command-line option

- A resource

- The empty string (“ ”)

The empty string makes the `setlocale()` function use the `$LC_*` and `$LANG` environment variables to determine locale settings. Specifically, setlocale (`LC_ALL`, “ ”) specifies that the locale should be checked and taken from environment variables in the order shown in Table 1-1 for the various locale categories.

## Locale Categories

`Category` | `1st Env. Var.` | `2nd Env. Var.` | `3rd Env. Var.`
LC_CTYPE: | LC_ALL | LC_TYPE | LANG
LC_COLLATE: | LC_ALL | LC_COLLATE | LANG
LC_TIME: | LC_ALL | LC_TIME | LANG
LC_NUMERIC: | LC_ALL | LC_NUMERIC | LANG
LC_MONETARY: | LC_ALL | LC_MONETARY | LANG
LC_MESSAGES: | LC_ALL | LC_MESSAGES | LANG

The toolkit already defines a standard command-line option ( `-lang`) and a resource (`xnlLanguage`). Also, the resource value can be set in the server `RESOURCE_MANAGER`, which may affect all clients that connect to that server.

## Fonts, Font Sets, and Render Tables

All X clients use fonts for drawing text. The basic object used in drawing text is `XFontStruct`, which identifies the font that contains the images to be drawn.

The desktop already supports fonts by way of the `XFontStruct` data structure defined by Xlib; yet, the encoding of the characters within the font must be known to an internationalized application. To communicate this information, the program expects that all fonts at the server are identified by an X Logical Font Description (XLFD) name. The XLFD name enables users to describe both the base characteristics and the charset (encoding of font glyphs). The term charset is used to denote the encoding of glyphs within the font, while the term code set means the encoding of characters within the locale. The charset for a given font is determined by the CharSetRegistry and CharSetEncoding fields of the XLFD name. Text and symbols are drawn as defined by the codes in the fonts.

A font set (for example, an `XFontSet` data structure defined by Xlib) is a collection of one or more fonts that enables all characters defined for a given locale to be drawn. Internationalized applications may be required to draw text encoded in the code sets of the locale where the value of an encoded character is not identical to the glyph index. Additionally, multiple fonts may be required to render all characters of the locale using one or more fonts whose encodings may be different than the code set of the locale. Since both code sets and charsets may vary from locale to locale, the concept of a font set is introduced through `XFontSet`.

While fonts are identified by their XLFD name, font sets are identified by a list of XLFD names. The list can consist of one or more XLFD names with the exception that only the base characteristics are significant; the encoding of the desired fonts is determined from the locale. Any charsets specified in the XLFD base name list are ignored and users need only concentrate on specifying the base characteristics, such as point size, style, and weight. A font set is said to be locale-sensitive and is used to draw text that is encoded in the code set of the locale. Internationalized applications should use font sets instead of font structs to render text data.

Render tables are collections of renditions that specify how text is to be rendered. They are summarized elsewhere in this section.

### Font Specification

The font specification can be either an X Logical Function Description (XLFD) name or an alias for the XLFD name. For example, the following are valid font specifications for a 14-point font:

    -dt-application-medium-r-normal-serif-*-*-*-*-p-*-iso8859-1

OR

    -*-r-*-14-*iso8859-1

### Font Set Specification

The font set specification is a list of names (XLFD names or their aliases) and is sometimes called a base name list. All names are separated by commas, with any blank spaces before or after the comma being ignored. Pattern-matching (wildcard) characters can be specified to help shorten XLFD names.

Remember that a font set specification is determined by the locale that is running. For example, the ja_JP Japanese locale defines three fonts (character sets) necessary to display all of its characters; the following identifies the set of Gothic fonts needed.

- Example of full XLFD name list:

      -dt-mincho-medium-r-normal--14-*-*-m-*-jisx0201.1976-0,-dt-mincho-medium-r-normal--28-*-*-*-m-*-jisx0208.1983-0:

- Example of single XLFD pattern name:

      -dt-*-medium-*-24-*-m-*:

The preceding two cases can be used with a Japanese locale as long as fonts exist that match the base name list.

### Base Font Name List Specification

The base font name list is a list of base font names associated with a font set as defined by the locale. The base font names are in a comma-separated list and are assumed to be characters from the portable character set; otherwise, the result is undefined. Blank space immediately on either side of a separating comma is ignored.

Use of XLFD font names permits international applications to obtain the fonts needed for a variety of locales from a single locale-independent base font name. The single base font name specifies a family of fonts whose members are encoded in the various charsets needed by the locales of interest.

An XLFD base font name can explicitly name the font's charset needed for the locale. This enables the user to specify an exact font for use with a charset required by a locale, fully controlling the font selection.

If a base font name is not an XLFD name, an attempt is made to obtain an XLFD name from the font properties for the font.

The following algorithm is used to select the fonts that are used to display text with font sets.

For each charset required by the locale, the base font name list is searched for the first of the following cases that names a set of fonts that exist at the server.

- The first XLFD-conforming base font name that specifies the required charset or a superset of the required charset in its CharSetRegistry and CharSetEncoding fields.

- The first set of one or more XLFD-conforming base font names that specify one or more charsets that can be remapped to support the required charset. The Xlib implementation can recognize various mappings from a required charset to one or more other charsets and use the fonts for those charsets. For example, JIS Roman is ASCII with the ~ (tilde) and \ (backslash) characters replaced by the yen and overbar characters; Xlib can load an ISO8859-1 font to support this character set if a JIS Roman font is not available.

- The first XLFD-conforming font name, or the first non-XLFD font name for which an XLFD font name can be obtained, combined with the required charset (replacing the CharSetRegistry and CharSetEncoding fields in the XLFD font name). In the first instance, the implementation can use a charset that is a superset of the required charset.

- The first font name that can be mapped in some locale-dependent manner to one or more fonts that support imaging text in the charset.

For example, assume a locale requires the following charsets:

- ISO8859-1

- JISX0208.1983

- JISX0201.1976

- GB2312-1980.0

You can supply a base font name list that explicitly specifies the charsets, ensuring that specific fonts are used if they exist, as shown in the following example:

    “-dt-mincho-Medium-R-Normal-*-*-*-*-*-M-*-JISX0208.1983-0,\
    -dt-mincho-Medium-R-Normal-*-*-*-*-*-M- \
    *-JISX0201.jisx0201\.1976-1,\
    -dt-song-Medium-R-Normal-*-*-*-*-*-M-*-GB2312-1980.0,\
    -*-default-Bold-R-Normal-*-*-*-*-M-*-ISO8859-1“

You can supply a base font name list that omits the charsets, which selects fonts for each required code set, as shown in the following example:

    “-dt-Fixed-Medium-R-Normal-*-*-*-*-*-M-*,\
    -dt-Fixed-Medium-R-Normal-*-*-*-*-*-M-*,\
    -dt-Fixed-Medium-R-Normal-*-*-*-*-*-M-*,\
    -*-Courier-Bold-R-Normal-*-*-*-*-M-*”

Alternatively, the user can supply a single base font name that selects from all available fonts that meet certain minimum XLFD property requirements, as shown in the following example:

    “-*-*-*-R-Normal--*-*-*-*-*-M-*”

### Render Tables

A render table consists of one or more entries called renditions.
Each rendition is tagged with a name that is used when drawing a compound string. For internationalized applications, renditions and render tables should be specified in resource files to ensure the independence of application binaries from the different needs of various locales. For detailed information about renditions and render tables, refer to &MotifProgGd;.

## Text Drawing

The desktop provides various functions for rendering localized text, including simple text, compound strings, and some widgets. These include functions within the Xlib and Motif libraries.

## Input Methods

The Common Desktop Environment provides the ability to enter localized input for an internationalized application that is using the Motif Toolkit. Specifically, the `XmText[Field]` widgets are enabled to interface with input methods provided by each locale. In addition, the `dtterm` client is enabled to use input methods. For detailed information on input methods, refer to Motif Programmer's Guide.

By default, each internationalized client that uses the Motif Toolkit uses the input method associated with a locale specified by the user. The `XmNinputMethod` resource is provided as the input method portion of the locale modifiers to allow a user to specify any alternative input method.

The user interface of the input method consists of several elements. The need for these areas is dependent on the input method being used. They are usually needed by input methods that require complex input processing and dialogs. See Figure 1-3 for an illustration of these areas.

### Example of VendorShell widget with auxiliary (Japanese)

The `VendorShell` may contain the `XmNinputPolicy` resource.
This dictates whether its children widgets share input contexts or not. When using a root window input method style, for example, the input context would probably be shared by several widgets, while in an off-the-spot input method style, the input context might be shared between more than one widget, although it might not. However, with an over-the-spot input method style, an input context would almost certainly belong to a single widget. The possible values of `XmNinputPolicy` are `XmPER_WIDGET`, which will provide a new input context for each widget, and `XmPER_SHELL`, which will cause the children widgets of a common shell to share a single input context.

### Preedit Area

A preedit area is used to display the string being preedited. An input method can support the following modes of preediting: OffTheSpot, OnTheSpot (default) OverTheSpot, Root, and None.

A string that has been committed cannot be reconverted. The status of the string is moved from the preedit area to the location where the user is entering characters.

#### OffTheSpot

In OffTheSpot mode preediting using an input method, the location of preediting is usually inside the application window on the right side of the status area as shown in IPG.intro.mkr.13. A Japanese input method is used for the example.

##### Example of OffTheSpot preediting with the VendorShell widget (Japanese)

When preediting using an input method, the preedit string being preedited may be highlighted in some form depending on the input method.

To use OffTheSpot mode, set the `XmNpreeditType` resource of the `VendorShell` widget either with the `XtSetValues()` function or with a resource file. The `XmNpreeditType` resource can also be set as the resource of a `TopLevelShell`, `ApplicationShell`, or `DialogShell` widget, all of which are subclasses of the `VendorShell` widget class.

#### OverTheSpot

In OverTheSpot mode, the location of the preedit area is set to where the user is trying to enter characters (for example, the insert cursor position of the `Text` widget that has the current focus). The characters in a preedit area are displayed at the cursor position as an overlay window, and they can be highlighted depending on the input method.

Although a preedit area may consist of multiple lines in OverTheSpot mode. The preedit area is always within the MainWindow area and cannot cross its edges in any direction.

Keep in mind that although the preedit string under construction may be displayed as though it were part of the `Text` widget's text, it is not passed to the client and displayed in the underlying edit screen until preedit ends. See IPG.intro.mkr.14 for an illustration.

To use OverTheSpot mode explicitly, set the `XmNpreeditType` resource of the `VendorShell` widget either with the `XtSetValues()` function or with a resource file. The `XmNpreeditType` resource can be set as the resource of a `TopLevelShell`, `ApplicationShell`, or `DialogShell` widget because these are subclasses of the `VendorShell` widget class.

##### Example of OverTheSpot preediting with the VendorShell widget (Japanese)

#### OnTheSpot (Default)

In OnTheSpot mode, the preedit string is displayed in the text widget window. The preedit string is considered part of the text widget value, and its integrity can be ensured by the verify callbacks of the text widget (the verify callbacks are controlled by the `verifyPreedit` resource, which defaults to `False`). If the verify callbacks of the text widget do not accept any part of the preedit buffer, the preedit string is committed (for information on user actions that cause the preedit string to be committed, refer to &MotifProgGd;).

When preediting using an input method, the preedit string being preedited may be highlighted in some form depending on the input method.

To use OnTheSpot mode, set the `XmNpreeditType` resource of the `VendorShell` widget either with the `XtSetValues()` function or with a resource file. The `XmNpreeditType` resource can also be set as the resource of a `TopLevelShell,` `ApplicationShell`, or `DialogShell` widget, all of which are subclasses of the `VendorShell` widget class.

#### Root

In Root mode, the preedit and status areas are located separate from the client's window. The Root mode behavior is similar to OffTheSpot. See IPG.intro.mkr.15 for an illustration.

##### Example of Root preediting with the VendorShell widget (Japanese)

### Status Area

A status area reports the input or keyboard status of the input method to the users. For OverTheSpot and OffTheSpot styles, the status area is located at the lower left corner of the VendorShell window.

- If Root style, the status area is placed outside the client window.

- If the preedit style is OffTheSpot mode, the preedit area is displayed to the right of the status area.

The `VendorShell` widget provides geometry management so that a status area is rearranged at the bottom corner of the VendorShell window.

### Auxiliary Area

An auxiliary area helps the user with preediting. Depending on the particular input method, an auxiliary area can be created. The Japanese input method in IPG.intro.mkr.11 creates the following types of auxiliary areas:

- ZENKOUHO

- JIS NUMBER

- Switching conversion method

  - SAKIYOMI-REN-BUNSETSU

  - IKKATSU-REN-BUNSETSU

  - TAN-BUNSETSU

  - FUKUGOU-GO

### Focus Area

A focus area is any descendant widget under the `VendorShell` widget subtree that currently has focus. The Motif application programmer using existing widgets does not need to worry about the focus area. The important information to remember is that only one widget can have input method processing at a time. The input method processing moves to the window (widget) that currently has the focus.

### Layout Direction

Layout direction refers to the direction that is used to display visual elements such as widget children, widget components, and text (controlled by the `VendorShell` resource, `XmNlayoutDirection`). In general, this direction matches the direction that people use when reading or writing in a particular language. Languages such as English, French, German, and Swedish are read and written from left to right. Therefore, when users working in those languages enter characters from a computer keyboard, each new character is displayed to the right of the preceding one. These same users would also expect the layout of other visual elements to be displayed from left to right. For example, in a menu bar, the cascade buttons would be laid out from left to right so that a simple menu bar would position the "File" cascade button in the upper left corner, and the "Help" cascade button would appear in the upper right corner of the menu bar.

Languages such as Arabic and Hebrew are read and written from right to left. To display text correctly in these languages on the screen, each successive character that a user enters must appear to the left of the preceding character. Using the example above for layout of other visual elements, these users would expect a menu bar to lay out cascade buttons from right to left. The result would typically position the "File" cascade button in the upper right corner and the "Help" cascade button in the upper left corner of the menu bar. For more information, on layout direction, refer to &MotifProgGd;.

### Vertical Writing

In some Asian languages, texts are drawn vertically. When the `VendorShell` resource `XmNlayoutDirection` is set to `XmTOP_TO_BOTTOM`, the vertical writing feature is enabled. In addition to drawing texts vertically, this feature adapts the text widget in other ways appropriate for the user. For example, when word wrapping is turned on, the text wraps from the bottom of one column to the top of the next column. For more information on vertical writing, refer to &MotifProgGd;.

## Interclient Communications Conventions (ICCC)

The Interclient Communications Conventions (ICCC) defines the mechanism used to pass text between clients. Because the system is capable of supporting multiple code sets, it may be possible that two applications that are communicating with each other are using different code sets. ICCC defines how these two clients agree on how the data is passed between them. If two clients have incompatible character sets (for example, Latin1 and Japanese (JIS)), some data may be lost when characters are transported.
`
`

However, if two clients have different code sets but compatible character sets, ICCC enables these clients to pass information with no data lost. If code sets of the two clients are not identical, CompoundText encoding is used as the interchange with the `COMPOUND_TEXT` atom used. If data being communicated involves only portable characters (7-bit, ASCII, and others) or the ISO8859-1 code set, the data is communicated as is with no conversion by way of the `XA_STRING` atom.

Titles and icon names need to be communicated to the Window Manager using the `COMPOUND_TEXT` atom if nonportable characters are used; otherwise, the `XA_STRING` atom can be used.

Motif, for example, uses its own atoms to transfer textual data:

- `_MOTIF_COMPOUND_STRING` transfers data in XmString format.

- `_MOTIF_RENDER_TABLE` transfers the value of its render table as type STRING

For more information, refer to Motif Widget Writer's Guide.

Any other encoding is limited to the ability to convert to the locale of the Window Manager. The Window Manager runs in a single locale and supports only titles and icon names that are convertible to the code set of the locale under which it is running.

The Motif library and all desktop clients should follow these conventions.