Dakota Reference Manual: The Dakota Restart Utility

| Dakota Reference Manual Version 6.15 Explore and Predict with Confidence

- Main Page
- Related Pages

All Pages
The Dakota Restart Utility

The Dakota restart utility program provides a variety of facilities for managing restart files from Dakota executions. The executable program name is `dakota_restart_util` and it has the following options, as shown by the usage message returned when executing the utility without any options:

    Usage:
      dakota_restart_util command <arg1> [<arg2> <arg3> ...] --options
        dakota_restart_util print <restart_file>
        dakota_restart_util to_neutral <restart_file> <neutral_file>
        dakota_restart_util from_neutral <neutral_file> <restart_file>
        dakota_restart_util to_tabular <restart_file> <text_file>
          [--custom_annotated [header] [eval_id] [interface_id]]
          [--output_precision <int>]
        dakota_restart_util remove <double> <old_restart_file> <new_restart_file>
        dakota_restart_util remove_ids <int_1> ... <int_n> <old_restart_file> <new_restart_file>
        dakota_restart_util cat <restart_file_1> ... <restart_file_n> <new_restart_file>
    options:
      --help                       show dakota_restart_util help message
      --custom_annotated arg       tabular file options: header, eval_id,
                                   interface_id
      --freeform                   tabular file: freeform format
      --output_precision arg (=10) set tabular output precision

Several of these functions involve format conversions. In particular, the binary format used for restart files can be converted to ASCII text and printed to the screen, converted to and from a neutral file format, or converted to a tabular format for importing into 3rd-party plotting programs. In addition, a restart file with corrupted data can be repaired by value or id, and multiple restart files can be combined to create a master database.

# Print Command

The `print` option is useful to show contents of a restart file, since the binary format is not convenient for direct inspection. The restart data is printed in full precision, so that exact matching of points is possible for restarted runs or corrupted data removals. For example, the following command

    dakota_restart_util print
    dakota.rst

results in output similar to the following:

        ------------------------------------------
        Restart record    1  (evaluation id    1):
        ------------------------------------------
        Parameters:
                              1.8000000000000000e+00 intake_dia
                              1.0000000000000000e+00 flatness

        Active response data:
        Active set vector = { 3 3 3 3 }
                             -2.4355973813420619e+00 obj_fn
                             -4.7428486677140930e-01 nln_ineq_con_1
                             -4.5000000000000001e-01 nln_ineq_con_2
                              1.3971143170299741e-01 nln_ineq_con_3
         [ -4.3644298963447897e-01  1.4999999999999999e-01 ] obj_fn gradient
         [  1.3855136437818300e-01  0.0000000000000000e+00 ] nln_ineq_con_1 gradient
         [  0.0000000000000000e+00  1.4999999999999999e-01 ] nln_ineq_con_2 gradient
         [  0.0000000000000000e+00 -1.9485571585149869e-01 ] nln_ineq_con_3 gradient

        ------------------------------------------
        Restart record    2  (evaluation id    2):
        ------------------------------------------
        Parameters:
                              2.1640000000000001e+00 intake_dia
                              1.7169994018008317e+00 flatness

        Active response data:
        Active set vector = { 3 3 3 3 }
                             -2.4869127192988878e+00 obj_fn
                              6.9256958799989843e-01 nln_ineq_con_1
                             -3.4245008972987528e-01 nln_ineq_con_2
                              8.7142207937157910e-03 nln_ineq_con_3
         [ -4.3644298963447897e-01  1.4999999999999999e-01 ] obj_fn gradient
         [  2.9814239699997572e+01  0.0000000000000000e+00 ] nln_ineq_con_1 gradient
         [  0.0000000000000000e+00  1.4999999999999999e-01 ] nln_ineq_con_2 gradient
         [  0.0000000000000000e+00 -1.6998301774282701e-01 ] nln_ineq_con_3 gradient

        ...<snip>...

        Restart file processing completed: 11 evaluations retrieved.

# Neutral File Format

A Dakota restart file can be converted to a neutral file format using a command like the following:

        dakota_restart_util to_neutral dakota.rst dakota.neu

which results in a report similar to the following:

        Writing neutral file dakota.neu
        Restart file processing completed: 11 evaluations retrieved.

Similarly, a neutral file can be returned to binary format using a command like the following:

        dakota_restart_util from_neutral dakota.neu dakota.rst

which results in a report similar to the following:

        Reading neutral file dakota.neu
        Writing new restart file dakota.rst
        Neutral file processing completed: 11 evaluations retrieved.

The contents of the generated neutral file are similar to the following (from the first two records for the Cylinder example in[4]).

        6 7 2 1.8000000000000000e+00 intake_dia 1.0000000000000000e+00 flatness 0 0 0 0
        NULL 4 2 1 0 3 3 3 3 1 2 obj_fn nln_ineq_con_1 nln_ineq_con_2 nln_ineq_con_3
          -2.4355973813420619e+00 -4.7428486677140930e-01 -4.5000000000000001e-01
           1.3971143170299741e-01 -4.3644298963447897e-01  1.4999999999999999e-01
           1.3855136437818300e-01  0.0000000000000000e+00  0.0000000000000000e+00
           1.4999999999999999e-01  0.0000000000000000e+00 -1.9485571585149869e-01 1
        6 7 2 2.1640000000000001e+00 intake_dia 1.7169994018008317e+00 flatness 0 0 0 0
        NULL 4 2 1 0 3 3 3 3 1 2 obj_fn nln_ineq_con_1 nln_ineq_con_2 nln_ineq_con_3
          -2.4869127192988878e+00 6.9256958799989843e-01 -3.4245008972987528e-01
           8.7142207937157910e-03 -4.3644298963447897e-01  1.4999999999999999e-01
           2.9814239699997572e+01  0.0000000000000000e+00  0.0000000000000000e+00
           1.4999999999999999e-01  0.0000000000000000e+00 -1.6998301774282701e-01 2

This format is not intended for direct viewing (`print` should be used for this purpose). Rather, the neutral file capability has been used in the past for managing portability of restart data across platforms (recent use of more portable binary formats has largely eliminated this need) or for advanced repair of restart records (in cases where the `remove` command was insufficient).

# Tabular Format

Conversion of a binary restart file to a tabular format enables convenient import of this data into 3rd-party post-processing tools such as Matlab, TECplot, Excel, etc. This facility is nearly identical to the output activated by the tabular_data keyword in the Dakota input file specification, but with two important differences:

1. No function evaluations are suppressed as they are with `tabular_data` (i.e., any internal finite difference evaluations are included).
2. The conversion can be performed later, i.e., for Dakota runs executed previously.

An example command for converting a restart file to tabular format is:

        dakota_restart_util to_tabular dakota.rst dakota.m

which results in a report similar to the following:

        Writing tabular text file dakota.m
        Restart file processing completed: 10 evaluations tabulated.

The contents of the generated tabular file are similar to the following (from the example in the Restart section of[4]). Note that while evaluations resulting from numerical derivative offsets would be reported (as described above), derivatives returned as part of the evaluations are not reported (since they do not readily fit within a compact tabular format):

    %eval_id interface             x1             x2         obj_fn nln_ineq_con_1 nln_ineq_con_2
    1            NO_ID            0.9            1.1         0.0002           0.26           0.76
    2            NO_ID        0.90009            1.1 0.0001996404857   0.2601620081       0.759955
    3            NO_ID        0.89991            1.1 0.0002003604863   0.2598380081       0.760045
    4            NO_ID            0.9        1.10011 0.0002004407265       0.259945   0.7602420121
    5            NO_ID            0.9        1.09989 0.0001995607255       0.260055   0.7597580121
    6            NO_ID     0.58256179   0.4772224441   0.1050555937   0.1007670171 -0.06353963386
    7            NO_ID   0.5826200462   0.4772224441   0.1050386469   0.1008348962 -0.06356876195
    8            NO_ID   0.5825035339   0.4772224441   0.1050725476   0.1006991449 -0.06351050577
    9            NO_ID     0.58256179   0.4772701663   0.1050283245    0.100743156 -0.06349408333
    10           NO_ID     0.58256179   0.4771747219   0.1050828704   0.1007908783 -0.06358517983
    ...

Controlling tabular format: The command-line options `--freeform` and `--custom_annotated` give control of headers in the resulting tabular file. Freeform will generate a tabular file with no leading row nor columns (variable and response values only). Custom annotated format accepts any or all of the options:

- `header:` include %-commented header row with labels
- `eval_id:` include leading column with evaluation ID
- `interface_id:` include leading column with interface ID

For example, to recover Dakota 6.0 tabular format, which contained a header row, leading column with evaluation ID, but no interface ID:

    dakota_restart_util to_tabular dakota.rst dakota.m --custom_annotated header eval_id

Resulting in

    %eval_id             x1             x2         obj_fn nln_ineq_con_1 nln_ineq_con_2
    1                   0.9            1.1         0.0002           0.26           0.76
    2               0.90009            1.1 0.0001996404857   0.2601620081       0.759955
    3               0.89991            1.1 0.0002003604863   0.2598380081       0.760045
    ...

Finally, `--output_precision integer` will generate tabular output with the specified integer digits of precision.

# Concatenation of Multiple Restart Files

In some instances, it is useful to combine restart files into a single master function evaluation database. For example, when constructing a data fit surrogate model, data from previous studies can be pulled in and reused to create a combined data set for the surrogate fit. An example command for concatenating multiple restart files is:

        dakota_restart_util cat dakota.rst.1 dakota.rst.2 dakota.rst.3 dakota.rst.all

which results in a report similar to the following:

        Writing new restart file dakota.rst.all
        dakota.rst.1 processing completed: 10 evaluations retrieved.
        dakota.rst.2 processing completed: 110 evaluations retrieved.
        dakota.rst.3 processing completed: 65 evaluations retrieved.

The `dakota.rst.all` database now contains 185 evaluations and can be read in for use in a subsequent Dakota study using the `-read_restart` option to the `dakota` executable.

# Removal of Corrupted Data

On occasion, a simulation or computer system failure may cause a corruption of the Dakota restart file. For example, a simulation crash may result in failure of a post-processor to retrieve meaningful data. If 0's (or other erroneous data) are returned from the user's `analysis_driver`, then this bad data will get recorded in the restart file. If there is a clear demarcation of where corruption initiated (typical in a process with feedback, such as gradient-based optimization), then use of the `-stop_restart` option for the `dakota` executable can be effective in continuing the study from the point immediately prior to the introduction of bad data. If, however, there are interspersed corruptions throughout the restart database (typical in a process without feedback, such as sampling), then the `remove` and `remove_ids` options of `dakota_restart_util` can be useful.

An example of the command syntax for the `remove` option is:

        dakota_restart_util remove 2.e-04 dakota.rst dakota.rst.repaired

which results in a report similar to the following:

        Writing new restart file dakota.rst.repaired
        Restart repair completed: 65 evaluations retrieved, 2 removed, 63 saved.

where any evaluations in `dakota.rst` having an active response function value that matches `2.e-04` within machine precision are discarded when creating `dakota.rst.repaired`.

An example of the command syntax for the `remove_ids` option is:

        dakota_restart_util remove_ids 12 15 23 44 57 dakota.rst dakota.rst.repaired

which results in a report similar to the following:

        Writing new restart file dakota.rst.repaired
        Restart repair completed: 65 evaluations retrieved, 5 removed, 60 saved.

where evaluation ids `12`, `15`, `23`, `44`, and `57` have been discarded when creating `dakota.rst.repaired`. An important detail is that, unlike the `-stop_restart` option which operates on restart record numbers, the `remove_ids` option operates on evaluation ids. Thus, removal is not necessarily based on the order of appearance in the restart file. This distinction is important when removing restart records for a run that contained either asynchronous or duplicate evaluations, since the restart insertion order and evaluation ids may not correspond in these cases (asynchronous evaluations have ids assigned in the order of job creation but are inserted in the restart file in the order of job completion, and duplicate evaluations are not recorded which introduces offsets between evaluation id and record number). This can also be important if removing records from a concatenated restart file, since the same evaluation id could appear more than once. In this case, all evaluation records with ids matching the `remove_ids` list will be removed.

If neither of these removal options is sufficient to handle a particular restart repair need, then the fallback position is to resort to direct editing of a neutral file to perform the necessary modifications. <!-----------------------------------------—>