Author: Bob Jacobsen
Keywords: JMRI technical code xml schema usage

# JMRI: XML Schema Usage

JMRI uses XML for a number of purposes: to hold decoder definitions, for its persistance system for configuration and panel information, and to create parts of the web site from other files. This page describes how we specify the content of those files using XML schema.

For examples (not a tutorial!) on the structure of our schema, see the examples page.

The current schema can be seen online in the schema directory. The most commonly used one is the layout.xsd schema for panel files. See below on how it's organized.

## Access to Schema Definitions

JMRI uses XML Schema to define the format of its files.

Those XML Schema may need to be available to the program when it reads the files, as they define the default values of missing attributes and other needed information.

In the JMRI distributions, these are stored in the xml/schema directory. Note that they are not stored in each directory alongside the XML files. There are just too many locations to keep such a set of schema definition files up to date. JMRI itself, via the jmri.jmrit.XmlFile class, provides support for locating those files when the XML parser needs them.

## Modifying JMRI Schema

This section talks about how to handle small changes to existing schema, for example added or removing an attribute or element. For more extensive changes, including creating entirely new types or new file formats, see the next section on "Developing JMRI Schema".

Every time you change what JMRI writes (and therefore reads) in an XML file, you must also do the following:

1. You need to change the code that does the reading and writing.
2. You need to change the schema file(s) so that the XML format can be properly checked.
3. You have to provide new test XML files to make sure nothing has been broken, and in some cases have to adjust old ones.

Please don't skip the latter steps. They matter a lot to the long-term stability of JMRI code.

If possible, it's best to make changes by addition, so that existing files can continue to be read unchanged. JMRI strongly values backward compatibility, where any newer version of JMRI can still load and use a file written by an older version.

If you can make a change that's just an addition, then the process is:

1. Change your code
2. Add the new elements and attributes to the most recent version of the schema file(s).
3. Run "ant headlesstest" to make sure that the older files already present in the tests can still be processed. Fix anything that breaks. (You may discover at this point that you didn't actually make a backward compatible change, in which case either improve that or see the section on "Schema Versioning" below).
4. Create a test file with the new content. Ideally, this won't require using the screen, so it can actually be loaded and stored as part of headlesstest. In that case, put your file in a "load" test subdirectory (see below). At a minimum, though, put your file in a "verify" test subdirectory so that your schema changes will be tested. For more on this, see below.

### Schema Versioning

"Versioning" schema allows us to have different schema for older and newer files. This lets newer versions of JMRI continue to check and read files that were written by older versions of JMRI. This backward compatibility is an important JMRI feature which we don't want to lose.

In practical terms, versioning consists of having multiple-but-related versions of the schema definition files which are labeled by the first JMRI version that can read them.

When do you need to create a new version?

- You do not need to create a new version of the schema if you add or change things such that existing files will continue to validate.

  In that case, just make your schema changes in the current schema document, and commit them back to the JMRI code repository.

- You do need to version a schema when you make a change to the schema such that previous files will no longer validate with the current schema.

  In this case, the steps to version the schema are:

  1. Copy the current schema file to a new one with the number of the next JMRI release. E.g. copy types/turnouts-2-9-6.xsd to types/turnouts-3.7.3.xsd if you're doing this before JMRI 3.7.3 is released. Make your changes and commit that new version.
  2. If this is a subfile, such as the types/turnouts-2-9-6.xsd, that is included in a main schema such as layout-2-9-6.xsd, that main file also needs to be copied, have the include changed, and commited. Now you've got a new layout-3-7-3.xsd schema for output files to reference.
  3. Then, change the Java code that writes the schema reference to the top of output files to use the new filename. For example, layout (panel) files are written by `src/jmri/configurexml/ConfigXmlManager.java`. Look for the
     `static final public String schemaVersion = "-3-7-3"`
     line and change it to your new version number suffix.
  4. Run "ant headlesstest" and when tests fail, fix them. If a LoadAndStoreTest flags a compare inLine/outLine error, update the schema version in the header of the reference file. For example in `java/test/jmri/configurexml/loadref/BlockAndSignalMastTest.xml` point to the most recent layout.xsd version in line 3:
     `<layout-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://jmri.org/xml/schema/layout-4-7-2.xsd">`
  5. If there's an associated XML stylesheet(s), its name has to be changed in a coordinated way. (You should also update the stylesheet to show your new XML content, of course).

In either case, it's important to include sufficient test files that the unit tests catch any problems with the new and old schema. See the test section below.

Note that the unlabeled schema is the primordial, oldest, now-obsolete schema. For example layout.xsd is older than layout-2-9-6.xsd, and therefore not to be used for new files anymore. Don't assume that layout.xsd is the default for new files!

## Checking JMRI Schema

It's important that the JMRI schema definitions be kept semantically correct. If we let too many problems build up, we'll eventually have a lot of back-fitting to do. The W3C online schema validation tool is a good tool for checking that changes to the JMRI schema are still technically correct. You should check your changes with it before commiting them to the repository. Unfortunately, it doesn't seem to check compliance with nested schema elements, e.g. from DocBook (see below) or JMRI schema, but it's still a very useful check.

Using the JMRI "Validate XML File" tool in the "Debug" menu to validate a .xml file ("instance file") that uses your new or updated schema is an important check of both. Use it often during development. You can also use it from the command line via e.g.:

    ./runtest.csh apps/jmrit/XmlFileValidateRunner xml/decoders/0NMRA.xml

There is also a PowerShell script available, see the JUnit page for more information.

For a quick file check, Linux and Mac OS X users can validate from the command line with e.g.

     cd xml
     xmllint -schema schema/aspecttable.xsd -noout signals/sample-aspects.xml

`xmllint` can't check schema files themselves, unfortunately, because their schema isn't something it can handle.

Your schema docs should point to our standard stylesheet in their head matter:

    <?xml-stylesheet href="schema2xhtml.xsl" type="text/xsl"?>

Stylesheets turn XML code, like this schema, into a human-readable form when the XML is parsed and displayed by a browser. For an example of that, click on this link to the aspecttable.xsd schema file. Our standard stylesheet is pretty basic. It just shows basic structure. If anybody knows of a better stylesheet, we can certainly switch to it.

### JUnit testing

You should also add a JUnit test that checks a typical file. There are three kinds of checks that can be done:

1. You should always have a class that validates a typical file against the schema.

   That's done by having a SchemaTest class in your test-tree package (see e.g. test/jmri/configurexml/SchemaTest.java for an example) that checks all the XML files kept there. If that's in place, just put a copy of a (new) typical XML file in the existing "verify" subdirectory.

   To more extensively check your schema, you can check that it fails XML files that you think are not valid. There are lots of ways to be not valid, and you don't need to check them all, but if there something specific you want to be sure of, put an example of that in the "invalid" subdirectory. Those files are expected to fail for some specific reason. You should document that reason via comments in the file itself so your colleagues can figure it out later.

   If there's no "verify" subdirectory, create one and add it to the end of the SchemaTest class in that package. If there's no SchemaTest class, create one by replicating an existing one, see link above. Don't forget to add it to PackageTest so it gets invoked!

2. If you're working on configurexml files (panel files), and your new code isn't involved in the actual display of panels (e.g. can run as part of headlesstest), you should add a test that a sample file can be loaded and stored back successfully. Beyond just checking the schema, this also checks that all the load and store plumbing is working. (We run these tests headless as part of Jenkins, so please don't add tests that pop screen windows; they'll just cause errors).

   That's done by having a LoadAndStoreTest class in your test-tree package (see e.g. test/jmri/configurexml/LoadAndStoreTest.java for an example) that checks all the XML files kept there. If that's in place, just put a copy of a (new) typical XML file in the existing "load" subdirectory.

   If there's no "load" subdirectory, create one and add it to the end of the LoadAndStoreTest class in that package. If there's no LoadAndStoreTest class, create one by replicating an existing one, see link above. Don't forget to add it to PackageTest so it gets invoked!

   When LoadAndStoreTest runs, it loads the files in the load directory one-by-one, storing each back out to the "temp" directory within the local preferences directory, and then compares the input and output files. Sometimes this load-and-store process results in something that's in a different order, or contains more info (e.g. attributes missing from the input file are written with default values in the output file). If the comparison fails, but the output file is still OK when you manually inspect it, copy that output file to the "loadref" directory (create it if needed) within your test package. See test/jmri/configurexml/loadref for an example. LoadAndStoreTest will compare to files it finds here, instead of to the original file in the "load" subdirectory.

   If your changes to the code cause this testing to fail with older versions of the file, do not change the older file version. Instead, either put an updated output reference in the "loadref" directory, version the schema to allow the older file to continue to load, or improve your code. Backward compatibility is important!

3. You can also add a custom JUnit test that reads your sample file and makes sure that the proper objects have been created, that they have the correct data and state, etc. This could be anything from a "load and check that new beans exist in the new manager" to something quite more extensive.

At a minimum, please do the schema checks. They're easy, and will save lots of trouble in the future. If your new features don't involve displaying on the screen, adding load-and-store checks is also valuable, and isn't so hard.

Note: Do not remove or modify any existing XML file checks. Those keep old versions of the files working! If your new code and/or schema breaks the processing of the existing files, you need to either fix your code or version the schema to allow multiple formats to coexist.

## Developing JMRI Schema

For some examples of the XML schema structures described here, see the separate XML Schema Examples page/

Our preferred organization for XML schema is based on the structure of the underlying code: A particular *Xml class is the unit of reuse.

Lots of classes descend from jmri.configurexml.XmAdapter: (see Javadoc)

By convention, provide <xsd:annotation><xsd:appinfo> element containing the class name that reads/writes the element:

`<xs:annotation> <xs:documentation> Some human readable docs go here </xs:documentation> <xs:appinfo> <jmri:usingclass configurexml="false">jmri.managers.DefaultSignalSystemManager</jmri:usingclass> </xs:appinfo> </xs:annotation>`

### The Venetian Blinds Pattern

We are moving toward structuring our XML using the "Venetian Blinds pattern". In this style, the top level elements that are written by classes have types defined for them. Any elements that are within those are defined anonymously, within those elements. For an example of this, see the types/sensors.xsd file, which defines a type for the "sensors" element written for SensorManagers. Within that, there is included a definition of a "Sensor" element, and a "comment" element within that.

This limits the number of types, and keeps the schema files roughly aligned with the classes that do the reading and writing.

There are a few items (elements and attribute groups) that extend across multiple types. They are defined in the types/general.xsd file.

More information on XML schema design patterns is available at DeveloperWorks and the Oracle Java site.

### On Elements vs Attributes

When defining how to store new classes or updates to classes, the basic questions are:

- Are we storing data? In that case, it should be stored in an element of its own. Comments, speed values, user and system names are all examples of data that should be separately stored.
- Is this a modifier that provides information about the data in the element? In that case, it's OK to store the modifier information in an attribute. Color of a label, whether a turnout is inverted, which icon of the following list to load are examples of modifiers.

JMRI XML originally leaned heavily to attributes due to limitations in the JDOM library. These limitations are long gone, and we're now moving toward using elements in the proper way.

### Available Defined Types

The JMRI schema provides a large number of pre-defined data types. These (generally) check their content, and will be maintained in the future as valid content changes, so it's best to use these where possible instead of defining your own.

A partial list of the pre-defined types:

systemNameType
  System names, to eventually be tightened up to a real test of validity
userNameType
  User names, not including the empty name
nullUserNameType
  User names, with an empty value allowed
beanNameType
  Either user or system name
turnoutStateType
  closed, thrown
signalColorType
  red, yellow, etc
trueFalseType
  true, false
yesNoType
  yes, no
yesNoMaybeType
  yes, no, maybe

For others, browse the general types schema.

## External Standards and Future Work

The OASIS collaboration defines a number of schema and schema elements that have become well-known standards. Were possible, we should use those standard elements to improve inter-operability. The first ones of interest are:

- DockBook defines elements for several concepts we use:
  - author (http://www.docbook.org/tdg/en/html/author.html)
  - address (http://www.docbook.org/tdg/en/html/address.html)
  - revision history (http://www.docbook.org/tdg/en/html/revhistory.html)
  See
  - http://www.docbook.org/specs/docbook-5.0-spec-cs-01.html
  - http://www.docbook.org/xml/5.0/xsd/
  - http://www.docbook.org/xml/5.0/xsd/docbook.xsd

  We have our own DocBook subset that we use, because the full DocBook 5.0 schema takes a very long time to parse, and isn't fully consistent with versions of other software that we use. We use the normal DocBook 5.0 namespace, so we can later easily convert to a more complete schema transparently. Our smaller schema is located at http://jmri.org/xml/schema/docbook/docbook.xsd (our usual schema location). It is only referenced from JMRI schema files, not instance files, so that we can later convert with finite work.

- UBL, though aimed at business transactions, defines elements to represent parties (companies, people), devices, model numbers, etc.
- OpenDocument (OODF) defines a set of elements and structures for computations as part of its spreadsheet module. (But they provide Relax-NG schema, not W3C XML Schema, so that doesn't help so much)

Learning to use these will require some work, as we can't assume that computers using JMRI have internet access, so can't just reference the entire schema as remote entities.

## Copyright, Author and Revision Information

For various reasons, we need to move to DocBook format for Copyright, Author and Revision information in our XML files (instance files).

Sample XML:

`<db:copyright> <db:year>2009</db:year> <db:year>2010</db:year>< db:holder>JMRI</db:holder></db:copyright> <db:authorgroup> <db:author> <db:personname><db:firstname>Sample</db:firstname><db:surname>Name</db:surname></db:personname> <db:email>name@com.domain</db:email> </db:author> </db:authorgroup> <db:revhistory> <db:revision> <db:revnumber>1</db:revnumber> <db:date>2009-12-28</db:date> <db:authorinitials>initials</db:authorinitials> </db:revision> </db:revhistory>`

Sample schema description: (But see the real one, which is provided in schema/docbook)

`<xs:element ref="docbook:copyright" minOccurs="1" maxOccurs="1" > <xs:annotation><xs:documentation> DocBook element(s) providing copyright information in standard form. Must be present. </xs:documentation></xs:annotation> </xs:element> <xs:element ref="docbook:authorgroup" minOccurs="1" maxOccurs="unbounded" > <xs:annotation><xs:documentation> DocBook element(s) describing the authors in standard form </xs:documentation></xs:annotation> </xs:element> <xs:element ref="docbook:revhistory" minOccurs="1" maxOccurs="unbounded" > <xs:annotation><xs:documentation> DocBook element(s) describing the revision history in standard form </xs:documentation></xs:annotation> </xs:element>`