//----------------------------------------------------------------------------
//
// TSDuck - The MPEG Transport Stream Toolkit
// Copyright (c) 2005-2025, Thierry Lelegard
// BSD-2-Clause license, see LICENSE.txt file or https://tsduck.io/license
//
//----------------------------------------------------------------------------

[#testing]
=== Testing TSDuck

[#testoverview]
==== Testing overview

TSDuck is highly flexible, allowing an unlimited number of configurations.
Testing it is consequently challenging.
Moreover, some use cases are difficult to automate or require specific hardware.
Testing live TS reception using all possible DVB tuners or Dektec devices requires a lot of material,
human and time resources which are far beyond the capabilities of a free open-source project.

From a strict industrial standpoint, TSDuck is consequently not fully tested.

However, on a pragmatic standpoint, test suites have been setup to extract the
best of what could be done with limited resources.

Thus, although it is impossible to guarantee that a given release of TSDuck is bug-free,
we may be relatively confident that no major regression has been introduced.

[#testorg]
==== Organization of the tests

The code of TSDuck is divided in two parts, a large {cpp} library (`tsduck.dll`, `libtsduck.so`, or `.dylib`)
and a collection of small command line tools and plugins.

Similarly, the tests for TSDuck are divided in two parts.

* The TSDuck library has its own unitary test suite based on a custom framework named "TSUnit".
  This test suite is part of the {repo}[main tsduck] repository
  for TSDuck in directory `src/utest`.
* The tools and plugins are less easy to test.
  They work on large transport stream files which would clutter the TSDuck repository.
  The repository {repotest}[tsduck-test]
  contains those tests and the relevant scripts and data files.

The two test suites are fully automated.

[#testlib]
==== The TSDuck library test suite

In the main TSDuck repository, the directory `src/utest` contains the source file for one single program named `utest`.
This program is divided in many source files (or test suites).
Each source file contains many unitary tests.
The test infrastructure is based on a custom framework named "TSUnit".

NOTE: The structure of TSUnit is freely inspired by equivalent frameworks such as CppUnit and JUnit.
It has been adapted to TSDuck specificities and focuse on fast and easy development of test suites.

The `utest` executable is built twice, once using the TSDuck shared library and once using
the static library. On {unix}, both versions of the test suite are built and run using
`make test` as illustrated below

[source,shell]
----
$ make -j10 test
  .....
  [CXX] utestZlib.cpp
  [LD] /home/tsduck/bin/release-x86_64-vmubuntu/utest
  [LD] /home/tsduck/bin/release-x86_64-vmubuntu/utest_static
  [UTEST] /home/tsduck/bin/release-x86_64-vmubuntu/utest 

OK (666 tests, 32317 assertions)

  [UTEST] /home/tsduck/bin/release-x86_64-vmubuntu/utest_static 

OK (663 tests, 32294 assertions)

$
----

The number of tests and assertions changes with new versions of TSDuck.
When new features are added, the corresponding new tests are added.

Note that the statically linked version contains slightly less tests.
The missing tests are related to plugin activations and they can run only in a shared library environment.

On Windows, the Visual Studio project builds two executables named `utests-tsduckdll.exe` and `utests-tsducklib.exe`.
They can be run manually or from Visual Studio:

[source,powershell]
----
PS D:\tsduck> .\bin\Release-x64\utests-tsduckdll.exe

OK (666 tests, 32369 assertions)

PS D:\tsduck> .\bin\Release-x64\utests-tsducklib.exe

OK (663 tests, 32346 assertions)

PS D:\tsduck>
----

Note that the number of assertions is slightly different on Windows
because some system-specific tests are different.

By default, the test program runs all tests and reports failures only.
But `utest` also accepts a few command options which make it appropriate to debug individual features.

The available command line options are:

[.compact-table]
[cols="<1m,<1",frame=none,grid=none,stripes=none,options="autowidth,noheader"]
|===
|-d |Debug messages are output on standard error
|-l |List all tests but do not execute them.
|-t _name_ |Run only one test or test suite.
|===

First, the option `-l` is used to list all available tests:

[source,shell]
----
$ utest -l
AlgorithmTest
    AlgorithmTest::EnumerateCombinations
ArgsTest
    ArgsTest::Accessors
    ArgsTest::AmbiguousOption
........
    XMLTest::Validation
Xoshiro256ssTest
    Xoshiro256ssTest::Random
ZlibTest
    ZlibTest::AllLevels
    ZlibTest::Reference1
    ZlibTest::Reference4
    ZlibTest::Reference9
$
----

The left-most names are test suite names.
They represent one source file in `src/utest`.
It is possible to run all tests in a test suite or one specific test.
For instance:

[source,shell]
----
$ utest -t ArgsTest
$ utest -t XMLTest::Validation
----

The option `-d` is used to produce debug message (see examples in the test source files).

Thus, `utest` alone can be used as an automated fairly complete non-regression test suite for the TSDuck library
or as a debug environment for a given feature under development
(in all good "test-driven development" approaches, the code is written at the same time as its unitary test).

[#testtools]
==== The TSDuck tools and plugins test suite

The Git repository {repotest}[tsduck-test]
contains high-level tests for the TSDuck tools and plugins.

This test suite is fully automated and works on test files only.
No specific hardware (DVB tuner or Dektec device), no live stream is tested.
This is a known limitation.

The principle is simple.
Reference input files are stored in the repository, mainly transport stream files which were once captured from real live sources.
Test scripts contain deterministic commands which should always produce the same outputs (data, logs, etc).
These reference outputs are also stored in the repository.

The test suite runs using a given version of TSDuck and its outputs are compared with the reference outputs.
If differences are detected, there is a potential problem.

Since different versions of TSDuck may produce slightly different outputs,
a given version of the test suite formally applies to one version of TSDuck only.
Git tags are aligned in both repositories (or should be...) to indicate the target version.

[#teststruct]
===== Structure of the test suite

In short, execute the script `run-all-tests.sh` to run the complete test suite.

The repository contains the following subdirectories:

[cols="<1m,<1",frame=none,grid=none,stripes=none,options="autowidth,noheader"]
|===

|tests
|Contains one script per test or set of tests.
 The name for test _NNN_ is `test-NNN.sh`.
 Each test script can be executed individually.
 All tests are executed using the script `run-all-tests.sh`.

|common
|Contains utilities and common script.

|input
|Contains input data files for the tests.

|reference
|Contains reference output files for the various tests.
 There is one subdirectory `test-NNN` per test which contains all output files for that test.

|tmp
|Contains output files which are created by the execution of the tests.
 These files are typically compared against reference output files in `reference`.
 These files are temporary by definition.
 The subdirectory `tmp` is present on test machines only and is excluded from the Git repository.

|===

[#testadd]
===== Adding new tests

To add a new test:

* Allocate a new test number and document the purpose of the new test in the file `README.md`.
* Add input files in subdirectory `input`.
  For test _NNN_, all input files should be named `test-NNN.*`.
  There is generally zero or one input file per test, sometimes more.
* Create the script `test-NNN.sh` in subdirectory `tests`.
  Use other existing test scripts as templates.
* Run the command `tests/test-NNN.sh --init`.
  If the test is properly written,
  this creates the reference output files in the subdirectory `reference/test-NNN`.
  Manually check the created files, verify that they are correct.
  Be careful with this step since these files will be used as references.
* Run the same command without the `--init` option.
  This time, the output files are created in `tmp` and are compared with files in `reference`.
  Verify that all tests pass.
  Errors may appear if the test script is not properly written or if the output files contain unique,
  non-deterministic, time-dependent, system-dependent or file-system-dependent information.
  Make sure the output files are totally reproduceable in all environments.
  At worst, add code in the test script to remove any information from the output files
  which is known to be non-reproduceable.

Sometimes, TSDuck is modified in such in a way that an output file is modified on purpose.
Usually, this starts with a failed test.
When analysing the test failure, it appears that the modification of the output is intentional.
In that case, re-run the command `tests/test-NNN.sh --init` to update the reference output files.
Do not forget to manually validate them since they will act as the new reference.

TIP: The reference output files are stored in the Git repository.
Therefore, the best way to have a quick overiew of what changed in the output reference files is simply `git diff`.

[#testdev]
===== Testing a development version

By default, the test suite uses the TSDuck command from the system path.
Typically, it will use the installed version.

To test a development version, the two Git repositories `tsduck` and `tsduck-test`
shall be checked out at the same level, side by side in the same parent directory.
First, TSDuck shall be rebuilt in its repository.

Then, when the option `--dev` is specified to a test script or to `run-all-tests.sh`,
the test suite automatically uses the TSDuck executables from the development repository.

[source,shell]
------
$ ./run-all-tests.sh --dev
----  Testing TSDuck - The MPEG Transport Stream Toolkit - version 3.40-4133 from /home/tsduck/bin/release-x86_64-vmubuntu
----  test-001: Test test-001.tsanalyze.full.txt passed
----  test-001: Test test-001.tsanalyze.full.log passed
........
----  test-179: Test test-179.tstabdump.txt passed
----  test-179: Test test-179.tstabdump.log passed
----  ALL 2545 tests passed
$
------

The test suite is made of `bash` shell scripts.
On Windows systems, the suite can be run from a "Git Bash" window for instance.

On {unix}, in the `tsduck` project, the `make` target `test-suite` runs
the test suite on the binaries which were just built.
It assumes that the work areas of the git repositories `tsduck` and `tsduck-test` are
located side by side in the same parent directory.
This target updates the `tsduck-test` repository from its remote origin and runs the test suite.

[source,shell]
------
$ make test-suite
Already up to date.
----  Testing TSDuck - The MPEG Transport Stream Toolkit - version 3.40-4133 from /home/tsduck/bin/release-x86_64-vmubuntu
----  test-001: Test test-001.tsanalyze.full.txt passed
----  test-001: Test test-001.tsanalyze.full.log passed
........
----  test-179: Test test-179.tstabdump.txt passed
----  test-179: Test test-179.tstabdump.log passed
----  ALL 2545 tests passed
$
------

[#testlargefiles]
===== Large files

The `tsduck-test` repository contains large files, typically transport stream files.

Initially, these files were not stored inside the regular GitHub repository.
Instead, they used the https://git-lfs.github.com[Git Large File Storage] (LFS) feature of GitHub.
However, using LFS on GitHub happended to be a pain, as experienced by others and explained in
https://medium.com/@megastep/github-s-large-file-storage-is-no-panacea-for-open-source-quite-the-opposite-12c0e16a9a91[this article].

As a consequence, the transport stream files were re-integrated into the Git repository as regular files.
But we now limit their size to 20 MB.
