//----------------------------------------------------------------------------
//
// TSDuck - The MPEG Transport Stream Toolkit
// Copyright (c) 2005-2025, Thierry Lelegard
// BSD-2-Clause license, see LICENSE.txt file or https://tsduck.io/license
//
//----------------------------------------------------------------------------

[#addpsi]
=== Adding PSI/SI tables or descriptors

Adding support for new PSI/SI tables or descriptors is a welcome contribution to TSDuck.
Users from various continents, using different standards, or participating in standardization processes,
are in the best position to implement new tables or descriptors.

We recommend to release these contributions in open source, as part of the TSDuck project.

This section summarizes the main steps when implementing new tables or descriptors.

See xref:chap-contribution[xrefstyle=short] for more details on the contribution process.

==== Code base selection

The main recommendation to start with is: do not develop a new table or descriptor from scratch.
Use an existing and proven one as code base and adapt to your new structure.

To identify the right existing structure as code base, use some of these criteria:

* Short single-section table vs. long multi-section table.
* Flat vs. structured section, with descriptors or not, including substructures with descriptors (e.g. PMT).
* Use of strings, DVB strings, ATSC strings, ISDB strings.
* Public descriptor, DVB private descriptor, table-specific descriptor.

===== Classification of descriptors

Currently, TSDuck implements nearly 300 different descriptors from various standards.
Because the number of possible descriptor _tags_ (a.k.a. descriptor ids) is limited to 256 values,
there is no room for all possible descriptors.
For this reason, the various standard organizations use tricks such as _extended descriptors_,
_private descriptors_, or _table-specific descriptors_.

In a MPEG/DVB context, the allocation of descriptor ids is the following:

[.compact-table]
[cols="<1m,<1",frame=none,grid=none,stripes=none,options="autowidth,noheader"]
|===
|0x00-0x3E |MPEG-defined descriptors
|0x3F |MPEG extension descriptor
|0x40-0x7E |DVB-defined descriptors
|0x7F |DVB extension descriptor
|0x80-0xFE |Private descriptors
|0xFF |Reserved, not allocated, typically used as "null" value
|===

===== Extended descriptors

MPEG and DVB separately define the concept of _extended descriptors_.
Because of the shortage of descriptor ids, each of the two standards has defined an _extension_descriptor_,
typically using their last allocated descriptor id.
This descriptor is a generic envelope for specialized descriptors.

The first byte of the descriptor payload is an _extended_descriptor_id_ which identifies the actual descriptor type.
This allows the definition of up to 256 additional descriptors.

TSDuck does not use any specific type for the generic _extended_descriptor_.
Instead, there is a distinct type for each form of _extended_descriptor_.
Each of them has its own XML element and {cpp}, just like any other descriptor.
The extended descriptor mechanism is only considered as a binary serialization detail,
not a different type of descriptor.

If you have to implement a MPEG-defined extended descriptor,
you may use the _HEVC_timing_and_HRD_descriptor_ as code base.

If you have to implement a DVB-defined extended descriptor,
you may use the _supplementary_audio_descriptor_ as code base.

===== Private descriptors (DVB)

In the DVB standard, descriptor ids 0x80 to 0xFE are "private".
They are reserved for use by private entities, typically TV operators,
broadcast equipment vendors, Conditional Access System (CAS) vendors.

To determine which semantics should be associated with a given descriptor id in that range,
DVB defines a _private_data_specifier_descriptor_ which contains a 32-bit _private_data_specifier_ (PDS).
DVB allocates a unique PDS to any private organization which requests it.
See xref:tsduck.html#ETSI-101-162[ETSI-101-162] and https://www.dvbservices.com/identifiers/[the DVB services]
for more details on allocated PDS values.

In a descriptor list, all private descriptors which come after a _private_data_specifier_descriptor_
are defined by the private organization which is identified in the _private_data_specifier_descriptor_.
If several private descriptors from distinct defining entities must be placed in the same descriptor list,
several _private_data_specifier_descriptor_ are allowed to switch from one entity to another.

In a DVB context, a descriptor with a tag in the range 0x80-0xFE and no preceding
_private_data_specifier_descriptor_ is illegal.

That being said, in practice, the experience has exhibited families of bugs:

* Rogue signalization: Some organizations define their own private descriptors,
  with descriptor ids in the range 0x80-0xFE, without officially allocated PDS value.
  They use their own arbitrarily defined PDS value, hoping that no one else will use the same PDS value.
* Signalization bugs: Some broadcasters "forget" to insert the right _private_data_specifier_descriptor_
  before a well-defined private descriptor.
  (This is the reason why the option `--default-pds` exists in several TSDuck commands and plugins).
* Implementation bugs: Some implementers of receivers (set-top box, TV set) ignore the PDS rule
  or forget to check the previous PDS. They blindly interpret some private descriptor based on some
  expected descriptor id in the range 0x80-0xFE. If the same private descriptor id is used in the
  context of another PDS, the receiver incorrectly interprets the binary descriptor.

All these bugs are real and were regularly found during the development and usage of TSDuck.
If you implement a private descriptor, be sure to follow the rules.

You may use the EACEM-defined _eacem_stream_identifier_descriptor_ as code base.

**Logical channel number descriptors:**
Most of the commonly used private descriptors are some forms of _logical_channel_number_descriptor_.
The Logical Channel Number (LCN) is the usual concept of TV channel number,
the oldest and most traditional way of identifying a TV channel.
Surprisingly, neither MPEG nor DVB defined it.
Therefore, operators or equipment vendors have to define their own way of identifying LCN's.
For this reason, there is a wide range of variants of private _logical_channel_number_descriptor_
which all contain the same kind of information.
If you implement such a descriptor, with a similar implementation,
your class should be a subclass of `AbstractLogicalChannelDescriptor`.
Use _eacem_logical_channel_number_descriptor_ as code base.

===== Table-specific descriptors

So-called _table-specific descriptors_ are specific descriptors which exist only in the context of a couple of specific tables.
They usually re-use the tag of a standard descriptor, typically in the MPEG-defined range.
Of course, it is assumed that the standard descriptor, the tag of which has been hijacked,
will never be used in those specific tables to avoid ambiguities.

Let's take an example, the _target_IP_address_descriptor_.
This is a DVB-defined descriptor which can be used only inside an INT or a UNT, two DVB-defined tables.
The _target_IP_address_descriptor_ uses tag 0x09, which is normally used by a MPEG-defined _CA_descriptor_.
When TSDuck analyzes a descriptor list and encounters a tag 0x09, it usually starts to analyze a _CA_descriptor_,
except when the table is an INT or a UNT, in which case it analyzes a _target_IP_address_descriptor_.

This situation is supported by TSDuck.
If you have to implement such a table-specific descriptor, use _target_IP_address_descriptor_ as code base.

==== Affiliation to a standard

Each table or descriptor is defined either by a standard body or an organization, committee or private company.
Check if other PSI/SI from this organization is already implemented in TSDuck.
This is important because source files for PSI/SI are organized by standard.

Tables are implemented in the directory `src/libtsduck/dtv/tables`, using one subdirectory per standard.
The current subdirectories are `atsc`, `dvb`, `isdb`, `mpeg`, `scte`.
Currently, only renown standard bodies define tables.

Descriptors are implemented in the directory `src/libtsduck/dtv/descriptors`, using one subdirectory per standard.
The current subdirectories are the same as tables, plus various organizations such as `eacem`, `dtg`, `aom`, or `uwa`,
plus private companies which define private DVB descriptors such as `eutelsat`, or `sky`.

Try to find the right subdirectory for your new structure.
Create another directory if required.

In that subdirectory, you will have to create four or five files (the last is optional).
For instance, the MPEG-defined _ISO_639_language_descriptor_ is implemented as:

[source,text]
----
src/libtsduck/dtv/descriptors/mpeg:

    tsISO639LanguageDescriptor.xml
    tsISO639LanguageDescriptor.adoc
    tsISO639LanguageDescriptor.h
    tsISO639LanguageDescriptor.cpp
    tsISO639LanguageDescriptor.names
----

More details follow in the next sections.

==== Declaring identifiers

Your table or descriptor must have a 8-bit identifier.
You need to add it in the TSDuck source code.

Table ids are defined in file `src/libtsduck/dtv/signalization/tsTID.h`, in enum list `TID`.
Descriptor ids are defined in file `src/libtsduck/dtv/signalization/tsDID.h`, in enum list `DID`.
The ids are grouped by standard, be sure to add it at the right place.

In the case of a table, if that table is expected on some predefined PID, also add this PID
in file `src/libtsduck/dtv/transport/tsTS.h`, in the enum list `PID`.

In the case of a private DVB descriptor, your descriptor is valid only after a _private_data_specifier_descriptor_
which contains the _private_data_specifier_ (PDS) of the organization which defines the descriptor.
Check if that PDS value is present in file `src/libtsduck/dtv/signalization/tsPDS.h`, in enum list `PDS`.
Add it if not present.

For TSDuck to display meaningful identifiers, the source tree contains _names files_, with a `.names` extensions.
These files associate a unique value with a name.
There are several sections (for PID, TID, DID, for instance).
In each section, a value can be present only once and values must be declared in ascending order.

Add the table, descriptor, or PDS name in the files `src/libtsduck/dtv/signalization/tsTID.names`,
`tsDID.names`, or `tsPDS.names`. Carefully read the comments at the beginning of each section.
They explain the encoding of each unique value.

For table ids, the value includes the standard and the optional _CAS_id_ (useful for ECM and EMM only).

For descriptor ids, the value includes the standard, the PDS for private descriptors,
or the _table_id_ for table-specific descriptors.

If you implement a MPEG-defined or DVB-defined extended descriptor, add the corresponding
_extended_descriptor_id_ in `src/libtsduck/dtv/signalization/tsDID.h`,
in enum lists with `XDID_MPEG_` and `XDID_DVB_` symbols.

==== XML definition

You must define an XML representation for your table or descriptor in a `.xml` file.
Use the selected code base as reference.

This XML file is an _XML model file_, as defined in the TSDuck User's Guide.

A table shall be defined as one XML element inside the following envelope:

[source,xml]
----
<?xml version="1.0" encoding="UTF-8"?>
<tsduck>
  <_tables>
    <my_table_name ...>
      <_any in="_metadata"/>
      ...
    </my_table_name>
  </_tables>
</tsduck>
----

Note the mandatory `<_any in="_metadata"/>`.

A descriptor shall be defined as one XML element inside the following envelope:

[source,xml]
----
<?xml version="1.0" encoding="UTF-8"?>
<tsduck>
  <_descriptors>
    <my_descriptor_name ...>
      ...
    </my_descriptor_name>
  </_descriptors>
</tsduck>
----

For attributes and element names, preferably use the exact same names as defined in the standard
for your table or descriptor.

Do not blindly copy the binary structure in the XML description.
Define an XML equivalent representation.

For instance, a common pattern for optional fields in binary structures is to define a one-bit _foo_flag_
and a subsequent optional _foo_ field. The _foo_ field is typically present only when _foo_flag_ is 1.
Do not define _foo_flag_ in the XML structure.
Just define a _foo_ attribute and document it as optional.
It is the work of your serialization and deserialization functions to interpret
the presence or absence of the XML attribute _foo_ as a _foo_flag_ value.

The template value of XML attributes is a short informal type declaration.
For integer values, always start the description string with `uint__N__` or `int__N__`,
when _N_ is the size in bits of the binary field.
This `uint__N__` or `int__N__` is used by the automatic XML-to-JSON translation
to generate a JSON number instead of a JSON string.

Your `.xml` file will be automatically grabbed by the TSDuck build system and integrated
into the final configuration files.

==== Documentation file

You must describe the documentation for your table or descriptor in
https://asciidoctor.org[asciidoc] format in a `.adoc` file.
This file will be automatically grabbed by the TSDuck build system and integrated
into the TSDuck User's Guide, in HTML and PDF format.

Even if you do not know this format, this is pretty simple.
All `.adoc` files for the signalization use the same pattern.
The following is the file for the _ISO_639_language_descriptor_.

[source,adoc]
------
==== ISO_639_language_descriptor

Defined by MPEG in <<ISO-13818-1>>.

[source,xml]
----
<ISO_639_language_descriptor>
  <!-- One per language -->
  <language code="char3, required" audio_type="uint8, required"/>
</ISO_639_language_descriptor>
----
------

The first line starts with `====` and contains the table or descriptor name.
This is the section title in the user's guide.

The sentence _"Defined by XXX in \<<YYY>>"_ is mandatory and indicates where to
find the reference documentation for the structure.
The syntax `\<<YYY>>` is a reference in the bibliography.
There must be a corresponding `* +++[[[YYY]]]+++` entry in the file `doc/user/20F-app-references.adoc`,
for instance:

[source,adoc]
----
* [[[ISO-13818-1]]] ISO/IEC 13818-1:2018 | ITU-T Recommendation H.222 (2017):
  "Generic coding of moving pictures and associated audio information: Systems" (also known as "MPEG-2 System Layer").
----

If the reference does not exist yet in the bibliography, add it.
Keep the references sorted in alphabetical order.

Add a copy of the XML description of the table or descriptor.

[.compact-list]
* Remove the enclosing `<tsduck>`, `<_tables>`, `<_descriptors>` structures, just keep your structure.
* Make sure to reindent its top-level tag to the first column.
* In tables, remove the `<_any in="_metadata"/>`. It is meaningless for the user.
* Add any comment or formatting which makes the result more informative to the user.

==== {cpp} class

The {cpp} header (`.h`) and body (`.cpp`) files for the table or descriptor class are mandatory.
Start with the selected code base and carefully replace the structure names.

===== Structure registration

In the `.cpp` file, there is a fundamental macro:
`TS_REGISTER_TABLE()` for tables and `TS_REGISTER_DESCRIPTOR()` for descriptors.
This is a {cpp} trick which automatically registers your structure in the PSI/SI repository
during the initialization of the module.
If you omit this macro, your table will not be recognized.

The registration macro may take various forms depending on the type of structure
(standard descriptor, table-specific descriptor, extended descriptor, etc.)
Be careful to select a code base with the same characteristics in order to copy the same type of registration.

===== Programming reference

All public structures and fields in the {cpp} header file must be documented using Doxygen tags.
See examples in existing structures.
This is the way your structure will become documented in the TSDuck Programming Reference (doxygen format).

No public element shall be left undocumented.
To verify this, generate the documentation and check any error.
Undocumented elements are reported.

[.compact-list]
* On {unix}, run `make doxygen`.
* On Windows systems, run the PowerShell script `doc\doxy\build-doxygen.ps1`.

In the initial descriptor of your {cpp} class, make sure it is properly identified with the right group and standard.
For instance:

[source,c++]
----
//!
//! Representation of a Program Association Table (PAT).
//! @see ISO/IEC 13818-1, ITU-T Rec. H.222.0, 2.4.4.3
//! @ingroup libtsduck table
//!
----

or:

[source,c++]
----
//!
//! Representation of an ISO_639_language_descriptor
//! @see ISO/IEC 13818-1, ITU-T Rec. H.222.0, 2.6.18.
//! @ingroup libtsduck descriptor
//!
----

The directive `@ingroup` is used by Doxygen to assign the class in the right group.
The group name `libtsduck` indicates that the class belongs to the TSDuck library, and not the TSCore library.
The group names `table` and `descriptor` are self-explanatory.

The directive `@see` is important in three ways.

[.compact-list]
1. It is included in the Doxygen documentation.
2. It helps the future maintainers of the code to find the right documentation and directly the section number where to look.
3. It is also used in the automatic generation of the xref:chap-sigref[xrefstyle=short] of this document.

==== Names file

If necessary, you may provide a `.names` file.
This is useful when a field of your structure can get distinct values with distinct meanings.
When displaying a structure, it is more convenient for the user to get a meaningful name
rather than a value.

A `.names` file is organized in several sections.
By convention, use section names which start with the XML name of your structure, followed by a dot.

Example of the file `tsISO639LanguageDescriptor.names`, for the _ISO_639_language_descriptor_:

[source,test]
----
[ISO_639_language_descriptor.audio_type]
Bits = 8
0x00 = undefined
0x01 = clean effects
0x02 = hearing impaired
0x03 = visual impaired commentary
----

In the {cpp} source file, use the inherited static method `DataName()` to retrieve
a meaningful name, with optional formatting of the value before or after the name.

Example of the file `tsISO639LanguageDescriptor.cpp`:

[source,c++]
----
void ts::ISO639LanguageDescriptor::DisplayDescriptor(TablesDisplay& disp, PSIBuffer& buf, const UString& margin, DID did, TID tid, PDS pds)
{
    ...
    disp << ", Type: " << DataName(MY_XML_NAME, u"audio_type", buf.getUInt8(), NamesFlags::FIRST) << std::endl;
----

Your `.names` file will be automatically grabbed by the TSDuck build system and integrated
into the final configuration files.

==== Tests

There are lots of traps and pitfalls in the coding of a table or descriptor.
It is crucial to test it thoroughly.

First, become familiar with the TSDuck test suite as described in xref:testtools[xrefstyle=short].

Once you have cloned your forked versions of the two repositories, `tsduck` and `tsduck-test`,
side by side in the same parent directory, you can implement a test for your table or descriptor.

This kind of test is standardized.
The idea is to start from an XML file containing several samples of your table or descriptor.
Then, invoke the common script `standard-si-test.sh`.

This standard test compiles the XML file in binary, decompiles it to generate XML and JSON,
recompiles the output, inject the tables in a transport stream, extract them in text form, etc.
All intermediate results are kept as reference.

This kind of test is interesting in two ways. First, during the initial test, after development,
it is a good tool to debug the serialization, deserialization, binary and XML. Second, the
reference outputs will track any future regression.

For instance, the test 027 is the reference test for SCTE 35 tables and descriptors.
All tested structures are in the file `tsduck-test/input/test-027.xml`.
The test script `tsduck-test/tests/test-027.sh` is very simple:

[source,shell]
----
#!/usr/bin/env bash
source $(dirname $0)/../common/testrc.sh
test_cleanup "$SCRIPT.*"
source "$COMMONDIR"/standard-si-test.sh $SCRIPT.xml
----

NOTE: In practice, _all_ test scripts for that kind of PSI/SI test are identical.
Only the input `.xml` file changes.

If your table or descriptor belongs to a set of structures which are already tested in an existing test,
you may simply add your tested XML definitions in the existing test and update its reference output.

Otherwise, especially if you plan to implement several structures, you may create a new test.
Just use existing tests with `standard-si-test.sh` as a starting point.

Pay attention to the XML structures you want to test.
Keep in mind that you test one given structure in all possible ways, regardless of real applications.
Your tested structures do not need to carry meaningful values.
You test the _syntax_ of your table or descriptor, not its _semantics_.
You just want to test code, nothing else.

Here are some guidelines:

* If you test a descriptor, your don't care about which table it is in.
  Use a `<CAT>` for instance, a table which only contains descriptors and nothing else.
* If you test a table which contains descriptors, use any kind of simple descriptors,
  _ISO_639_language_descriptor_ for instance. You do not care if such a descriptor does
  not make sense in your table.
* If you test a table which contains descriptors, test each descriptor list with zero, one, two descriptors.
* Test optional fields in structures where they are present and in other structures where
  they are omitted.
* More generally, when your code takes different steps or branches in the presence of different
  forms of input, test all possible forms of input.
* Test adjacent fields with different values. If two flags are in consecutive bits in
  the binary structure, test once with a `true`/`false` combination and once with a
  `false`/ `true` combination.
* Use integer values which use the full width of a binary field to detect incorrect
  truncation or size errors. For instance, in a `uint32` field, use value 0xDEADBEEF,
  for instance, not 0 or 1.

When the result is satisfactory, submit a pull request for each repository, `tsduck` and `tsduck-test`.
See xref:contributor[xrefstyle=short] for more details on that.
