\section{dprepro and pyprepro}\label{interfaces:dprepro-and-pyprepro}

Dakota is packaged with two template processing tools that are
intended for use in the preprocessing phase of analisys drivers.

The first tool, \texttt{pyprepro}, features simple parameter 
substitution, setting of immutable (fixed) variable names, and 
provides full access to all of the Python programming language
within templates. As such, templates can contain loops, 
conditionals, arrays (lists), dictionaries, and other Python language 
features.

The second tool, \texttt{dprepro}, uses the same template engine
as \texttt{pyprepro}, and in addition undertands Dakota's parameter
file formats. In particular, when using \texttt{dprepro}, Dakota variables 
become available for use within templates. \texttt{dprepro} is also integrated
with the \texttt{dakota.interfacing} module to provide direct access
to \texttt{Parameters} and \texttt{Results} objects within templates 
(see Section~\ref{interfacing:params-and-results}) and to provide template 
processing capability within Python scripts that import 
\texttt{dakota.interfacing}.

The \texttt{dprepro} described in this section is a replacement for the existing
\texttt{dprepro} script that shipped with Dakota releases prior to 6.8. The 
new \texttt{dprepro} maintains as much backward compatibility with the old 
\texttt{dprepro} as possible. An important difference between the old 
\texttt{dprepro} and the new \texttt{d/pyprepro} is that the former was written 
in Perl, and the latter are in Python. Although the old \texttt{dprepro} was deprecated 
as of the 6.8 release of Dakota, it is still available in Dakota's \texttt{bin/} folder 
under the name \texttt{dprepro.perl}.

\subsection{Usage}\label{interfaces:dprepro-usage}

Users who are familiar with the existing Perl version of dprepro should
experience few or no differences with the new Python version. There are,
however, a number of new options and features, especially in the kinds of
expressions that are permitted in templates. These are described in
the following section.

Figure~\ref{advint:dprepro_usage} shows the result of running
\texttt{dprepro --help}. The three required positional arguments are:
1. \textt{include}: The name of a Dakota parameters file,
2. \textt{infile}: The name of a template file (or a dash if the template 
is provided on \texttt{stdin}), and 
3. \textff{outfile}: The name of the output file, which is the result of
processing the template. This argument is optional, and output is written
to \texttt{stdout} if it is missing.

The remaining options are used to
* Set custom delimiters for Python code lines (\texttt{--code}) and blocks 
(\texttt{--code-block}) inline statements that print (\texttt{--inline}).
The last of these is equivalent to Perl \texttt{dprepro}'s 
\texttt{--left-delimiter} and \texttt{--right-delimter} switches, which also
are permitted. They default to \texttt{"\{ \}"}.
* Insert additional parameters for substitution, either from a JSON file
(\texttt{--json-include}) or directly on the command line (\texttt{--var}).
* Silence warnings (\texttt{--no-warn})
* Set the default numerical output format (\texttt{--output-format}).

\begin{figure}
  \centering
  \begin{bigbox}
    \begin{small}
      \verbatimtabinput[8]{dprepro_usage}
    \end{small}
  \end{bigbox}
  \caption{\texttt{dprepro} usage}
  \label{advint:dprepro_usage}
\end{figure}

The \texttt{pyprepro} script accepts largely the same command line options.
The primary differences are that \texttt{pyprepro} does not require or accept
Dakota format parameters files, and it has just two positional command line
arguments, the \texttt{infile} and \texttt{outfile}, both defined as above.
In addition, \texttt{pyprepro} accepts one or more \texttt{--include} files.
These may be used to set parameters and execute arbitrary Python scripting 
before template processing occurs.

\subsection{Template Expressions}\label{interfaces:template-expressions}

This section describes the expressions that are permitted in templates. All 
examples, except where otherwise noted, use the default delimiters \texttt{\{\ \ \}} 
for inline printed expressions, \texttt{\%} for single-line 
Python statements, and \texttt{\{\%\ \%\}} for Python code blocks.

Expressions can be of three different forms (with defaults)

\begin{itemize}
\tightlist
\item
  Inline single-line expressions (rendered) {[} \texttt{\{expression\}}
  {]}
\item
  Python code single-line (silent) {[} \texttt{\%\ expression} {]}
\item
  Python code multi-line blocks (silent) {[}
  \texttt{\{\%\ expression\ (that\ can\ be\ over\ many\ line)\ \%\}} {]}
\end{itemize}

Expressions can contain just about any valid Python code. The only
important difference is that indentation is ignored and blocks must end 
with \texttt{end}. See the examples below.

\subsubsection{Inline Expressions}\label{interfaces:inline-expressions}

Inline expressions are delineated with \texttt{\{expression\}} and
\textbf{always display}.

Consider:

\begin{verbatim}
param1 = {param1 = 10}
param2 = {param1 + 3}
param3 = {param3 = param1**2}
\end{verbatim}

Returns:

\begin{verbatim}
param1 = 10
param2 = 13
param3 = 100
\end{verbatim}

In this example, the first and third line both display a value
\emph{and} set the parameter.

\subsubsection{Python Single Line Code}\label{interfaces:python-single-line-code}

A \texttt{\%} at the start of a line is used to begin a single-line code expression.
These are non-printing.
Consider the following example.

\begin{verbatim}
% param1 = pi/4
The new value is {sin(param1)}
\end{verbatim}

It returns:

\begin{verbatim}
The new value is 0.7071067812
\end{verbatim}

Furthermore, single lines can be used for Python logic and loops. This example 
demonstrates looping over an array, which is explained in further detail below.
As stated previously, unlike ordinary Python, indentation is not required and is 
ignored. Blocks of Python code are concluded with \texttt{end}.

\begin{verbatim}
% angles = [0,pi/4,pi/2,3*pi/4,pi]
% for angle in angles:
cos({angle}) = { cos(angle)}
% end
\end{verbatim}

Returns:

\begin{verbatim}
cos(0) = 1
cos(0.7853981634) = 0.7071067812
cos(1.570796327) = 6.123233996e-17
cos(2.35619449) = -0.7071067812
cos(3.141592654) = -1
\end{verbatim}

\subsubsection{Code Blocks}\label{interfaces:code-blocks}

Finally, multi-line code blocks may be specified without prepending each 
Python statement with \texttt{\%}. Instead, the entire block is enclosed
in \texttt{\{\% \%\}}. (Indentation is ignored within code blocks.)

\begin{verbatim}
{%
# Can have comments too!
txt = ''
for ii in range(10):
    txt += ' {}'.format(ii)
end
%}
txt: {txt}
\end{verbatim}

returns:

\begin{verbatim}
txt:  0 1 2 3 4 5 6 7 8 9
\end{verbatim}

\subsubsection{Changing delimiters}\label{interfaces:changing-delimiters}

As noted in the \texttt{-\/-help} for \texttt{dprepro} and \texttt{pyprepro},
the delimiters for single-line Python statements, code blocks, and inline 
printed expressions can be changed. This is useful when the defaults are 
reserved characters in the output format.

For code blocks (default \texttt{\{\%\ \%\}}), the innermost characters
cannot be any of ``\texttt{\{\}{[}{]}()}''.

\subsubsection{Escaping delimiters}\label{interfaces:escaping-delimiters}

All delimiters can be escaped with a leading \texttt{\textbackslash{}}.
A double \texttt{\textbackslash{}\textbackslash{}} followed by the
delimiter will return \texttt{\textbackslash{}}. For example:

\begin{verbatim}
{A=5}
\{A=5\}
\\{A=5\\}
\end{verbatim}

Returns:

\begin{verbatim}
5
{A=5}
\{A=5\}  
\end{verbatim}

Note that escaping the trailing delimiter (e.g.
\texttt{\textbackslash{}\}}) is optional.

\subsubsection{Immutable Variables}\label{interfaces:immutable-variables}

Variables can be fixed such that they cannot be redefined (without
explicitly allowing it).

In this example, the attempted reassignemnt of \texttt{param} to 20 is
ignored,

\begin{verbatim}
% param = Immutable(10)
% param = 20 
{param}
\end{verbatim}

and the output is

\begin{verbatim}
10
\end{verbatim}

because \texttt{param} is \texttt{Immutable}. To explicitly make a
variable mutable again, call it with \texttt{Mutable()}:

\begin{verbatim}
set             : \{ param = Immutable(10) \} : { param = Immutable(10) }           
try to reset    : \{ param = 20 \}            : { param = 20 }          
make mutable    : \{ param = Mutable(21) \}   : { param = Mutable(21) } 
reset           : \{ param = 20 \}            : { param = 20 }         
\end{verbatim}

Returns:

\begin{verbatim}
set             : { param = Immutable(10) } : 10
try to reset    : { param = 20 }            : 10
make mutable    : { param = Mutable(21) }   : 21
reset           : { param = 20 }            : 20
\end{verbatim}

Note that any variable set via an \texttt{-\/-include} file or on the 
command line using \{\textt{-\/-var} will be \textbf{set as Immutable}.
This is useful for overriding defaults within templates.

\texttt{MyTemplate.inp}:

\begin{verbatim}
param1 = {param1 = 10}
param2 = {param2 = pi}
\end{verbatim}

If called directly:

\begin{verbatim}
param1 = 10
param2 = 3.141592654
\end{verbatim}

However, if called
\texttt{pyprepro\ -\/-var\ "param1=30"\ \textless{}inputfile\textgreater{}}:

\begin{verbatim}
param1 = 30
param2 = 3.141592654
\end{verbatim}

Or, with an optional \texttt{-\/-include} file:

\texttt{MyInclude.inp}:

\begin{verbatim}
{param1 = 32}
\end{verbatim}

And call
\texttt{pyprepro\ -\/-include\ MyInclude.inp\ MyTemplate.inp}:

\begin{verbatim}
param1 = 32
param2 = 3.141592654
\end{verbatim}

There is one caveat to variable immutability. While the variable name is 
reserved, the value can still be changed if it is a mutable Python object 
(``mutable'' has different meanings for Python objects than is used in 
\texttt{pyprepro} and \texttt{dprepro} templates). For example:

\begin{verbatim}
% param = Immutable( [1,2,3])
% param.append(4)   # This will work because it is modifying the object
% param = ['a','b','c']   # This won't because it is redefining
{param}
\end{verbatim}

Will output:

\begin{verbatim}
[1, 2, 3, 4]
\end{verbatim}

\\subsubsection{DakotaParams and DakotaResults}\label{interfaces:params-and-results}

If the \texttt{dakota} Python package (see 
Section~\ref{interfaces:dakota.interfacing}) is available for import, (e.g. has
been added to the \texttt{PYTHONPATH}) then \texttt{Parameters} and 
\texttt{Results} objects generated from the Dakota parameters file provided 
on the comnmand line to \texttt{dprepro} will be available for use in templates.
The \texttt{Parameters} and \texttt{Results} objects are named \texttt{DakotaParams} and
\texttt{DakotaResults}, respectively.

This implies that all of these objects' attributes are available within
templates, such as the evaluation ID (\texttt{DakotaParams.eval\_id} and the 
active set vector entries (\texttt{DakotaResults[0].asv.function}). Dakota 
variables also become available not only directly within the template, but as 
members of \texttt{DakotaParams}. That is, if  \texttt{x1} is a Dakota variable,
it will be available within a template both by the name \texttt{x1}, and as 
\texttt{DakotaParams["x1"]}. In this way, variables that are disallowed by the
naming conventions of Python (explained in the following section) can still
be accessed using their original names.

\subsubsection{Unicode Support}\label{interfaces:unicode}

Variables must obey the naming conventions for the version of Python that is
used to run \texttt{d/pyprepro}. For Python 2, only ASCII alpanumeric
characters and the underscore are permitted, and identifiers must not begin
with a number. In Python 3, this requirement is relaxed considerably, and
many Unicode characters are permitted in identifiers.

Because Dakota itself has few such restrictions on variable names,
\texttt{d/pyprepro} "mangles" noncompliant names in the following ways
before making them available in templates:

* Variables/paramters that begin with a number are prepended by the lowercase
letter 'i'.
* Disallowed characters such as # are replaced by underscores (\texttt{\_}).
* In Python 2, non-ASCII letters are normalized to their rough ASCII
equivalents (e.g. ñ is replaced by n).

As stated in the previous section, when using \texttt{dprepro} with 
\texttt{dakota.interfacing}, the original variable names are always available
via the \texttt{DakotaParams} object.

\subsection{General Coding}\label{interfaces:general-coding}

The language of pyprepro is Python with a single, slight modification.
In normal Python, indentation delineates blocks. However, in pyprepro,
indentation is ignored and blocks must have an \texttt{end} whether they
are part of multi-line code (\texttt{\{\%\ \%\}}) or part of single line
operation (\texttt{\%}).

\subsubsection{Python Coding Tips.}\label{python-coding-tips.}

For the most part, if you are familiar with other interpreted languages
such as Matlab, coding is Python is very similar.

The key notes are:

\begin{itemize}
\tightlist
\item
  Blocks have \texttt{:} at the end of their statement. For example,
  \texttt{if\ CONDITION:} \emph{with} the colon
\item
  Indentation \textbf{USUALLY} matters but \textbf{DOESN'T} in pyprepo

  \begin{itemize}
  \tightlist
  \item
    As such, you must include a \texttt{end} statement
  \end{itemize}
\item
  Arrays are zero-based
\item
  Exponentiation is double \texttt{**}. Example: \texttt{x**y} (``x to
  the y'')
\end{itemize}

\subsubsection{Conditionals}\label{conditionals}

Python has the standard set of conditionals. Recall the conditional
block declaration must end with a \texttt{:} and the entire block must
have an \texttt{end} statement (again, this is not in normal Python).
Consider the following example:

\begin{verbatim}
% param = 10.5
% if param == 10.0:
param is 10! See: {param}
% else:
param does not equal 10, it is {param}
% end

% if 10 <= param <= 11:
param ({param}) is between 10 and 11
% else:
param is out of scope
% end
\end{verbatim}

results in:

\begin{verbatim}
param does not equal 10, it is 10.5

param (10.5) is between 10 and 11
\end{verbatim}

Boolean operations are also possible using simple \texttt{and},
\texttt{or}, and \texttt{not} syntax

\begin{verbatim}
% param = 10.5
% if param >= 10 and param <= 11:
param is in [10 11]
% else:
param is NOT in [10,11]
% end
\end{verbatim}

returns:

\begin{verbatim}
param is in [10 11]
\end{verbatim}

\subsubsection{Loops}\label{loops}

Python contains \texttt{for} loops that iterate over arbitrary arrays or
with an index. As with conditionals, the declaration must end with
\texttt{:} and the block must have an \texttt{end}.

To iterate over an index, from 0 to 4, use the \texttt{range} command

\begin{verbatim}
% for ii in range(5):
{ii}
% end
\end{verbatim}

will return:

\begin{verbatim}
0
1
2
3
4
\end{verbatim}

You can also iterate over objects in an array (list):

\begin{verbatim}
% animals = ['cat','mouse','dog','lion']
% for animal in animals:
I want a {animal}
%end
\end{verbatim}

results in

\begin{verbatim}
I want a cat
I want a mouse
I want a dog
I want a lion
\end{verbatim}

\subsubsection{Arrays}\label{arrays}

Arrays are \textbf{zero indexed} and can also have negative indices
representing the end of the array. They are references by
\texttt{myarray{[}index{]}}

Consider:

\begin{verbatim}
% animals = ['cat','mouse','dog','lion']
{animals[0]}
{animals[-1]}
\end{verbatim}

will result in

\begin{verbatim}
cat
lion
\end{verbatim}

Note that pyprepro will \emph{try} to nicely format arrays for printing.
For certain types, it may not work well.

\begin{verbatim}
{theta = [0,45,90,135,180,225,270,315]}
\end{verbatim}

(with \texttt{\{\ \}} to print input) results in

\begin{verbatim}
[0, 45, 90, 135, 180, 225, 270, 315]
\end{verbatim}

\subsubsubsection{Math on arrays}\label{math-on-arrays}

Unlike some tools (e.g.~Matlab) you cannot do math on all elements of
arrays directly. However, there are two possibilities.

The first will \emph{always} work:

\begin{verbatim}
% theta = [0,45,90,135,180,225,270,315] 
{ [ sin(pi*th/180) for th in theta ] }
\end{verbatim}

results in

\begin{verbatim}
[0, 0.7071067812, 1, 0.7071067812, 1.224646799e-16, -0.7071067812, -1, -0.7071067812]
\end{verbatim}

The alternative is to use NumPy \emph{if} it is installed.

\begin{verbatim}
% theta = [0,45,90,135,180,225,270,315]
% import numpy as np
% theta = np.array(theta) # Redefine as numpy array
{ np.sin(pi*theta/180) }
\end{verbatim}

will return:

\begin{verbatim}
[0, 0.7071067812, 1, 0.7071067812, 1.224646799e-16, -0.7071067812, -1, -0.7071067812]
\end{verbatim}

\subsubsection{Strings}\label{strings}

Python has extremely powerful and extensive string support. Strings can
be initialized in any of the following ways:

\begin{verbatim}
{mystring1="""
multi-line
string inline
"""}
{mystring1}
{% mystring2 = '''
another multi-line example
but in a block
''' %}
mystring2: {mystring2}

Single quotes: {'singe'}
Double quotes: {'double'}
\end{verbatim}

and it returns

\begin{verbatim}
multi-line
string inline


multi-line
string inline

mystring2:
another multi-line example
but in a block


Single quotes: singe
Double quotes: double
\end{verbatim}

The choice of single \texttt{\textquotesingle{}} or \texttt{"} can be
driven by convenience.

Strings can be joined by adding them:

\begin{verbatim}
{%
a = 'A'
b = 'B'
%}
{a + ' ' + b}
\end{verbatim}

returns:

\begin{verbatim}
A B
\end{verbatim}

\subsubsubsection{Custom Functions}\label{custom-functions}

You can define any arbitrary functions using either \texttt{def} or
\texttt{lambda}

Consider the following: (note, we use indentation here for readability
but indentation \emph{is ignored} and the function definition is
terminated with \texttt{end}):

\begin{verbatim}
{%
def myfun1(param):
    return (param + 1) ** 2 + 3
end

myfun2 = lambda param: (param + 1) ** 2 + 5
%}
{myfun1(1.2)}
{myfun2(1.2)}
{ [ myfun1(x) for x in [1,2,3,4] ] }
\end{verbatim}

returns:

\begin{verbatim}
7.84
9.84
[7, 12, 19, 28]
\end{verbatim}

\subsection{Auxiliary Functions}\label{interfaces:auxiliary-functions}

There are a series of auxiliary functions to help. The primary one is
the \texttt{include}

\subsubsection{Include}\label{include}

Using

\begin{verbatim}
% include('path/to/include.txt')
\end{verbatim}

Will insert the contents of
\texttt{\textquotesingle{}path/to/include.txt\textquotesingle{}}. Inside
\texttt{\textquotesingle{}path/to/include.txt\textquotesingle{}}, there
can be new variable definitions and/or it can access older ones. Note
that unlike the command-line \texttt{-\/-include}, there is \emph{no
(im)mutability assigned} for these unless explicit for each parameter!

The code will search for the include text first in the path of the
original template file and then in the path where \texttt{pyprepro} is
executed.

\subsubsection{Immutable and Mutable}\label{immutable-and-mutable}

As explained elsewhere, variables can be defined as
\texttt{Immutable(value)} or \texttt{Mutable(value)}. If a variable is
Immutable, the value cannot be reset unless \emph{explicitly} made
mutable.

Note: files called with the command line \texttt{-\/-include} have all
of their variables be immutable. But this does not affect those with the
\texttt{include()} function

\subsubsection{Print all variables}\label{print-all-variables}

\texttt{all\_vars()} and \texttt{all\_var\_names()} print out all
\emph{defined} variables. Consider the following that also demonstrates
setting a comment string (two ways)

\begin{verbatim}
% param1 = 1
{param2 = 'two'}
all variables and values: {all_vars()}
all varables: {all_var_names()}

{all_var_names(comment='//')}
// {all_var_names()} <--- Don't do this
\end{verbatim}

returns:

\begin{verbatim}
two
all variables and values: {'param1': 1, 'param2': u'two'}
all varables: ['param2', 'param1']

// ['param2', 'param1']
// ['param2', 'param1'] <--- Don't do this
\end{verbatim}

Notice the empty \texttt{()} at the end of \texttt{all\_vars} and
\texttt{all\_var\_names}. If possible, it is \emph{better} to use
\texttt{comment=\textquotesingle{}//\textquotesingle{}} syntax since the
result of these can be multiple lines.

\subsubsection{Set global print format}\label{set-global-print-format}

As discussed elseware, the print format can be set on a per item basis
by manually converting to a string. Alternatively, it can be (re)set
globally inside the template (as well as at the command line).

\begin{verbatim}
{pi}
% setfmt('%0.3e')
{pi}
% setfmt() # resets
{pi}
\end{verbatim}

returns:

\begin{verbatim}
3.141592654
3.142e+00
3.141592654
\end{verbatim}

\subsubsubsection{Aside: Set output format
individually}\label{aside-set-output-format-individually}

The following demonstrates setting the output for a specific line. The
key is to use Python to convert the value into a string which is then
displayed:

\begin{verbatim}
{pi}
{ '%0.3f' % pi }
\end{verbatim}

Will output:

\begin{verbatim}
3.141592654
3.142
\end{verbatim}

\subsubsection{Using Defaults undefined
parameters}\label{using-defaults-undefined-parameters}

Directly calling undefined parameters will result in an error. There is
no \emph{universal} default value. However, there are the following
functions:

\begin{itemize}
\tightlist
\item
  \texttt{get} -- get param with optional default
\item
  \texttt{defined} -- determine if the variable is defined
\end{itemize}

The usage is explained in the following examples:

\begin{verbatim}
Defined Parameter:
% param1 = 'one'
{ get('param1') } <-- one
{ get('param1','ONE') } <-- one

Undefined Paramater
{ get('param2') } <-- *blank*
{ get('param2',0) } <-- 0

Check if defined: { defined('param2') }

% if defined('param2'):
param2 is defined: {param2}
% else:
param2 is undefined
% end
\end{verbatim}

returns:

\begin{verbatim}
Defined Parameter:
one <-- one
one <-- one

Undefined Paramater
 <-- *blank*
0 <-- 0

Check if defined: False

param2 is undefined
\end{verbatim}

But notice if you have the following:

\begin{verbatim}
{param3}
\end{verbatim}

you will get the following error:

\begin{verbatim}
Error occurred:
    NameError: name 'param3' is not defined
\end{verbatim}

\subsubsection{Mathematical Functions}\label{mathematical-functions}

All of the Python \texttt{math} module in imported with the functions:

\begin{verbatim}
  acos       degrees     gamma   radians  
  acosh      erf         hypot   sin      
  asin       erfc        isinf   sinh      
  asinh      exp         isnan   sqrt      
  atan       expm1       ldexp   tan       
  atan2      fabs        lgamma  tanh      
  atanh      factorial   log     trunc     
  ceil       floor       log10   
  copysign   fmod        log1p   
  cos        frexp       modf             
  cosh       fsum                               
\end{verbatim}

Also included are the following constants

\begin{longtable}[]{@{}ll@{}}
\toprule
Name & value\tabularnewline
\midrule
\endhead
\texttt{pi},\texttt{PI} & 3.141592654\tabularnewline
\texttt{e},\texttt{E} & 2.718281828\tabularnewline
\texttt{tau} (\texttt{2*pi}) & 6.283185307\tabularnewline
\texttt{deg} (\texttt{180/pi}) & 57.29577951\tabularnewline
\texttt{rad} (\texttt{pi/180}) & 0.01745329252\tabularnewline
\texttt{phi} (\texttt{(sqrt(5)+1\ )/2}) & 1.618033989\tabularnewline
\bottomrule
\end{longtable}

Note that all trig functions are assuming radians. See
\href{https://docs.Python.org/3/library/math.html}{Python's
\texttt{math} library} for more details. To compute based on degrees,
convert first:

\begin{verbatim}
{ tan( radians(45) )}
{ tan( 45*rad)}
{ degrees( atan(1) )}
{ atan(1) * deg }
\end{verbatim}

returns:

\begin{verbatim}
1
1
45
45
\end{verbatim}

\subsubsection{Other Functions}\label{other-functions}

Other functions can be imported. All of Python is available. For
example, to get a random number, you can do:

\begin{verbatim}
% from random import random,seed
% seed(1)
{A = random()}
\end{verbatim}

Returns (may depend on the system)

\begin{verbatim}
0.1343642441
\end{verbatim}
