Next: Hints, Previous: Releasing GDB, Up: Top

## 22 Testsuite

The testsuite is an important component of the gdb package. While it is always worthwhile to encourage user testing, in practice this is rarely sufficient; users typically use only a small subset of the available commands, and it has proven all too common for a change to cause a significant regression that went unnoticed for some time.

The gdb testsuite uses the DejaGNU testing framework. The tests themselves are calls to various `Tcl` procs; the framework runs all the procs and summarizes the passes and fails.

### 22.1 Using the Testsuite

To run the testsuite, simply go to the gdb object directory (or to the testsuite's objdir) and type `make check`. This just sets up some environment variables and invokes DejaGNU's `runtest` script. While the testsuite is running, you'll get mentions of which test file is in use, and a mention of any unexpected passes or fails. When the testsuite is finished, you'll get a summary that looks like this:

                         === gdb Summary ===

         # of expected passes            6016
         # of unexpected failures        58
         # of unexpected successes       5
         # of expected failures          183
         # of unresolved testcases       3
         # of untested testcases         5

To run a specific test script, type:

         make check RUNTESTFLAGS='tests'

where tests is a list of test script file names, separated by spaces.

If you use GNU make, you can use its `-j` option to run the testsuite in parallel. This can greatly reduce the amount of time it takes for the testsuite to run. In this case, if you set `RUNTESTFLAGS` then, by default, the tests will be run serially even under `-j`. You can override this and force a parallel run by setting the `make` variable `FORCE_PARALLEL` to any non-empty value. Note that the parallel `make check` assumes that you want to run the entire testsuite, so it is not compatible with some dejagnu options, like `--directory`.

The ideal test run consists of expected passes only; however, reality conspires to keep us from this ideal. Unexpected failures indicate real problems, whether in gdb or in the testsuite. Expected failures are still failures, but ones which have been decided are too hard to deal with at the time; for instance, a test case might work everywhere except on AIX, and there is no prospect of the AIX case being fixed in the near future. Expected failures should not be added lightly, since you may be masking serious bugs in gdb. Unexpected successes are expected fails that are passing for some reason, while unresolved and untested cases often indicate some minor catastrophe, such as the compiler being unable to deal with a test program.

When making any significant change to gdb, you should run the testsuite before and after the change, to confirm that there are no regressions. Note that truly complete testing would require that you run the testsuite with all supported configurations and a variety of compilers; however this is more than really necessary. In many cases testing with a single configuration is sufficient. Other useful options are to test one big-endian (Sparc) and one little-endian (x86) host, a cross config with a builtin simulator (powerpc-eabi, mips-elf), or a 64-bit host (Alpha).

If you add new functionality to gdb, please consider adding tests for it as well; this way future gdb hackers can detect and fix their changes that break the functionality you added. Similarly, if you fix a bug that was not previously reported as a test failure, please add a test case for it. Some cases are extremely difficult to test, such as code that handles host OS failures or bugs in particular versions of compilers, and it's OK not to try to write tests for all of those.

DejaGNU supports separate build, host, and target machines. However, some gdb test scripts do not work if the build machine and the host machine are not the same. In such an environment, these scripts will give a result of “UNRESOLVED”, like this:

         UNRESOLVED: gdb.base/example.exp: This test script does not work on a remote host.

### 22.2 Testsuite Parameters

Several variables exist to modify the behavior of the testsuite.

- `TRANSCRIPT`

  Sometimes it is convenient to get a transcript of the commands which the testsuite sends to gdb. For example, if gdb crashes during testing, a transcript can be used to more easily reconstruct the failure when running gdb under gdb.

  You can instruct the gdb testsuite to write transcripts by setting the DejaGNU variable `TRANSCRIPT` (to any value) before invoking `runtest` or `make check`. The transcripts will be written into DejaGNU's output directory. One transcript will be made for each invocation of gdb; they will be named `transcript.n`, where n is an integer. The first line of the transcript file will show how gdb was invoked; each subsequent line is a command sent as input to gdb.

                make check RUNTESTFLAGS=TRANSCRIPT=y

  Note that the transcript is not always complete. In particular, tests of completion can yield partial command lines.

- `GDB`

  Sometimes one wishes to test a different gdb than the one in the build directory. For example, one may wish to run the testsuite on `/usr/bin/gdb`.

                make check RUNTESTFLAGS=GDB=/usr/bin/gdb

- `GDBSERVER`

  When testing a different gdb, it is often useful to also test a different gdbserver.

                make check RUNTESTFLAGS="GDB=/usr/bin/gdb GDBSERVER=/usr/bin/gdbserver"

- `INTERNAL_GDBFLAGS`

  When running the testsuite normally one doesn't want whatever is in `~/.gdbinit` to interfere with the tests, therefore the test harness passes `-nx` to gdb. One also doesn't want any windowed version of gdb, e.g., ‘`gdb -tui`’, to run. This is achieved via `INTERNAL_GDBFLAGS`.

                set INTERNAL_GDBFLAGS "-nw -nx"

  This is all well and good, except when testing an installed gdb that has been configured with `--with-system-gdbinit`. Here one does not want `~/.gdbinit` loaded but one may want the system `.gdbinit` file loaded. This can be achieved by pointing `$HOME` at a directory without a `.gdbinit` and by overriding `INTERNAL_GDBFLAGS` and removing `-nx`.

                cd testsuite
                HOME=`pwd` runtest \
                  GDB=/usr/bin/gdb \
                  GDBSERVER=/usr/bin/gdbserver \
                  INTERNAL_GDBFLAGS=-nw

There are two ways to run the testsuite and pass additional parameters to DejaGnu. The first is with `make check` and specifying the makefile variable ‘`RUNTESTFLAGS`’.

         make check RUNTESTFLAGS=TRANSCRIPT=y

The second is to cd to the `testsuite` directory and invoke the DejaGnu `runtest` command directly.

         cd testsuite
         make site.exp
         runtest TRANSCRIPT=y

### 22.3 Testsuite Configuration

It is possible to adjust the behavior of the testsuite by defining the global variables listed below, either in a `site.exp` file, or in a board file.

- `gdb_test_timeout`

  Defining this variable changes the default timeout duration used during communication with gdb. More specifically, the global variable used during testing is `timeout`, but this variable gets reset to `gdb_test_timeout` at the beginning of each testcase, making sure that any local change to `timeout` in a testcase does not affect subsequent testcases.

  This global variable comes in handy when the debugger is slower than normal due to the testing environment, triggering unexpected `TIMEOUT` test failures. Examples include when testing on a remote machine, or against a system where communications are slow.

  If not specifically defined, this variable gets automatically defined to the same value as `timeout` during the testsuite initialization. The default value of the timeout is defined in the file `gdb/testsuite/config/unix.exp` that is part of the gdb test suite1.

### 22.4 Testsuite Organization

The testsuite is entirely contained in `gdb/testsuite`. While the testsuite includes some makefiles and configury, these are very minimal, and used for little besides cleaning up, since the tests themselves handle the compilation of the programs that gdb will run. The file `testsuite/lib/gdb.exp` contains common utility procs useful for all gdb tests, while the directory `testsuite/config` contains configuration-specific files, typically used for special-purpose definitions of procs like `gdb_load` and `gdb_start`.

The tests themselves are to be found in `testsuite/gdb.*` and subdirectories of those. The names of the test files must always end with `.exp`. DejaGNU collects the test files by wildcarding in the test directories, so both subdirectories and individual files get chosen and run in alphabetical order.

The following table lists the main types of subdirectories and what they are for. Since DejaGNU finds test files no matter where they are located, and since each test file sets up its own compilation and execution environment, this organization is simply for convenience and intelligibility.

`gdb.base`
  This is the base testsuite. The tests in it should apply to all configurations of gdb (but generic native-only tests may live here). The test programs should be in the subset of C that is valid K&R, ANSI/ISO, and C`++` (`#ifdef`s are allowed if necessary, for instance for prototypes).
`gdb.lang`
  Language-specific tests for any language lang besides C. Examples are `gdb.cp` and `gdb.java`.
`gdb.platform`
  Non-portable tests. The tests are specific to a specific configuration (host or target), such as HP-UX or eCos. Example is `gdb.hp`, for HP-UX.
`gdb.compiler`
  Tests specific to a particular compiler. As of this writing (June 1999), there aren't currently any groups of tests in this category that couldn't just as sensibly be made platform-specific, but one could imagine a `gdb.gcc`, for tests of gdb's handling of GCC extensions.
`gdb.subsystem`
  Tests that exercise a specific gdb subsystem in more depth. For instance, `gdb.disasm` exercises various disassemblers, while `gdb.stabs` tests pathways through the stabs symbol reader.

### 22.5 Writing Tests

In many areas, the gdb tests are already quite comprehensive; you should be able to copy existing tests to handle new cases.

You should try to use `gdb_test` whenever possible, since it includes cases to handle all the unexpected errors that might happen. However, it doesn't cost anything to add new test procedures; for instance, `gdb.base/exprs.exp` defines a `test_expr` that calls `gdb_test` multiple times.

Only use `send_gdb` and `gdb_expect` when absolutely necessary. Even if gdb has several valid responses to a command, you can use `gdb_test_multiple`. Like `gdb_test`, `gdb_test_multiple` recognizes internal errors and unexpected prompts.

Do not write tests which expect a literal tab character from gdb. On some operating systems (e.g. OpenBSD) the TTY layer expands tabs to spaces, so by the time gdb's output reaches expect the tab is gone.

The source language programs do not need to be in a consistent style. Since gdb is used to debug programs written in many different styles, it's worth having a mix of styles in the testsuite; for instance, some gdb bugs involving the display of source lines would never manifest themselves if the programs used GNU coding style uniformly.

Some testcase results need more detailed explanation:

`KFAIL`
  Known problem of gdb itself. You must specify the gdb bug report number like in these sample tests:

                kfail "gdb/13392" "continue to marker 2"

  or

                setup_kfail gdb/13392 "*-*-*"
                kfail "continue to marker 2"

`XFAIL`
  Known problem of environment. This typically includes gcc but it includes also many other system components which cannot be fixed in the gdb project. Sample test with sanity check not knowing the specific cause of the problem:

                # On x86_64 it is commonly about 4MB.
                if {$stub_size > 25000000} {
                    xfail "stub size $stub_size is too large"
                    return
                }

  You should provide bug report number for the failing component of the environment, if such bug report is available:

                if {[test_compiler_info {gcc-[0-3]-*}]
                   || [test_compiler_info {gcc-4-[0-5]-*}]} {
                   setup_xfail "gcc/46955" *-*-*
                }
                gdb_test "python print ttype.template_argument(2)" "&C::c"

### 22.6 Board settings

In gdb testsuite, the tests can be configured or customized in the board file by means of Board Settings. Each setting should be consulted by test cases that depend on the corresponding feature.

Here are the supported board settings:

`gdb,cannot_call_functions`
  The board does not support inferior call, that is, invoking inferior functions in gdb.
`gdb,can_reverse`
  The board supports reverse execution.
`gdb,no_hardware_watchpoints`
  The board does not support hardware watchpoints.
`gdb,nofileio`
  gdb is unable to intercept target file operations in remote and perform them on the host.
`gdb,noinferiorio`
  The board is unable to provide I/O capability to the inferior.
`gdb,nosignals`
  The board does not support signals.
`gdb,skip_huge_test`
  Skip time-consuming tests on the board with slow connection.
`gdb,skip_float_tests`
  Skip tests related to float points on target board.
`gdb,use_precord`
  The board supports process record.
`gdb_server_prog`
  The location of GDBserver. If GDBserver somewhere other than its default location is used in test, specify the location of GDBserver in this variable. The location is a file name of GDBserver that can be either absolute or relative to testsuite subdirectory in build directory.
`in_proc_agent`
  The location of in-process agent. If in-process agent other than its default location is used in test, specify the location of in-process agent in this variable. The location is a file name of in-process agent that can be either absolute or relative to testsuite subdirectory in build directory.
`noargs`
  gdb does not support argument passing for inferior.
`no_long_long`
  The board does not support type `long long`.
`use_gdb_stub`
  The tests are running with gdb stub.

#### Footnotes

[1] If you are using a board file, it could override the test-suite default; search the board file for "timeout".