Executable Evaluator

Click here to view the API reference

The executable evaluator is an excellent option if there is not an existing evaluator in the library that fits your needs. It gives you the ability to directly interface your desired application with Python and use it as an evaluator. There is plenty of flexibility built into it that enables you to interface with numerous applications. This tutorial will walk you through the inner workings of the evaluator through examples to help you get started with building your own interface.

Here is the general order of operations that the executable evaluator performs.

        flowchart LR
    start((Start)) --> write_csv[Write CSV] & write[Input writer] --> run[Run executable]
    run --> read_csv[Read CSV] & read[Output reader] --> return(((Return)))

By default the evaluator will write the inputs to a CSV file and pass the absolute path of this file to the executable as a command line argument. Similarly, the default output is a CSV file whose absolute path is also passed as a command line argument. When constructing the evaluator you may provide functions to the input_writer and output_reader arguments to override this default behavior. However, the file names cannot be changed and your functions are expected to write/read the contents of the given files. If your executable requires specific input/output file names and it disregards the paths given as arguments then you are also free to use these functions to write/read these specific files.

Note

You can use any delimiter besides a comma by providing it as an argument to the delimiter parameter in the constructor. This delimiter will be used when generating the input and expected when reading the output.

Basic usage

To get a feel for how this evaluator operates let’s analyze a simple test case. In this example our “application” will simply add two variables. The file below will act as our application

add.py

import sys
import numpy as np

# Read input
in_file = sys.argv[1]
print('in_file: ', in_file)

x1, x2 = np.loadtxt(in_file, delimiter=',', skiprows=1, unpack=True, ndmin=2)
print(f'x1: {x1}\nx2: {x2}')

# Compute
result = x1 + x2

# Write output
out_file = sys.argv[2]
print('out_file: ', out_file)

np.savetxt(out_file, result, delimiter='\n', header='sum', comments='')

This is a fairly simple script. We read in the file which will have data formatted as a comma delimited text file where the first row contains the columns names. After adding the values together we write the sums to the designated output file. Making sure to add the column name in the first row.

Now we construct our evaluator and run some data through it.

import standard_evaluator as se
import pandas as pd

# Create evaluator
eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=se.EvaluatorInfo(
        name='add',
        inputs=[
            se.FloatVariable(name='x1'),
            se.FloatVariable(name='x2'),
        ],
        outputs=[
            se.FloatVariable(name='sum'),
        ],
    ),
    pre_args=['exe_files/add.py'],
)

# Data points to evaluate
data = pd.DataFrame({
    'x1': [1, 2],
    'x2': [5, 8]
})

# Do evaluation
eval(data)

data

As expected the evaluator returns the sum of x1 and x2. Let’s dive a little deeper into what is going on here. If we were going to be running add.py as a standalone file we would use something like this in a terminal window:

python exe_files/add.py input.csv output.csv

Which would run our script that lives in the subdirectory exe_files using input.csv as input and output.csv as output. Now the command that the evaluator builds will have the following format

\[{\underbrace{\texttt{path/to/program}}_{\texttt{executable\_name}}}\texttt{ }{\underbrace{\texttt{arg1 arg2}}_\texttt{pre\_args}}~\texttt{ input.in output.out }{\underbrace{\texttt{arg3 arg4}}_\texttt{post\_args}}\]

So to match this pattern we use python as the executable name/path (Note that since it is already in my PATH variable I do not need to provide the full path to the executable). Since Python expects a path to the desired script file before anything else use the pre_args parameter to set the path to our script which resides in the exe_files subdirectory.

Finally, you may have noticed that none of the print statements in our script had any affect. This is because the evaluator captures the stdout and stderr streams and supresses them. Otherwise your console might get flooded with data. If this information is useful to you or you simply want to use it then you can use the output_reader parameter. This parameter expects to take in a function of the following form

def reader(out_path: pathlib.Path, responses: list[str], stdout: str) -> pandas.DataFrame:
    ...

Here is a brief description what what each parameter will contain:

out_path: Path to the generated output file
responses: Names of each of the responses that should be returned
stdout: String containing the captured output in the stdout and stderr streams

This function should then return a pandas DataFrame with the results parsed from the output file. Let’s construct a reader to view the captured output.

import pandas as pd
import pathlib

def add_reader(out_file: pathlib.Path, responses: list[str], stdout: str) -> pd.DataFrame:
    print(stdout)
    return pd.read_csv(out_file)

This reader is super simple. It prints the captured output and reads the generated CSV as a pandas DataFrame. To use it we simply provide this function as the argument to the output_reader parameter.

# Create evaluator
eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=se.EvaluatorInfo(
        name='add',
        inputs=[
            se.FloatVariable(name='x1'),
            se.FloatVariable(name='x2'),
        ],
        outputs=[
            se.FloatVariable(name='sum'),
        ],
    ),
    pre_args=['exe_files/add.py'],
    output_reader=add_reader,
)

# Data points to evaluate
data = pd.DataFrame({
    'x1': [1, 2],
    'x2': [5, 8]
})

# Do evaluation
eval(data)

data

We again get the correct result but this time we can also see the captured output we placed in our script. As you likely noticed, our script is run twice–once for each row. The next section explains why this happens and how you can control it.

Controlling how data is processed

As we saw in the previous example the executable was run once for each data point. The executable evaluator provides parameters that allow you to control how data gets fed into your application.

Note

When both mult_evals and run_parallel are set to True priority is given to mult_evals and run_parallel is ignored!

`mult_evals`

This parameter when set to True will place all data points into the same file and call your application once. Likewise, the evaluator expects your application to return outputs for each of the points in the same file. Let’s run our previous example with this flag enabled to see it in action!

# Create evaluator
eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=se.EvaluatorInfo(
        name='add',
        inputs=[
            se.FloatVariable(name='x1'),
            se.FloatVariable(name='x2'),
        ],
        outputs=[
            se.FloatVariable(name='sum'),
        ],
    ),
    pre_args=['exe_files/add.py'],
    output_reader=add_reader,
    mult_evals=True,
)

# Data points to evaluate
data = pd.DataFrame({
    'x1': [1, 2, 3, 4, 5],
    'x2': [5, 8, 1, 4, 7]
})

# Do evaluation
eval(data)

data

As expected, we see that our script is only run once and all of the data points are evaluated at once.

`run_parallel`

An alternative approach if your application can only process one input at a time is to run it in parallel. Setting this parameter to True will use dask to run multiple instances of your application at once. In the code below we use our trusty addition script and compare the run times in serial and parallel execution.

import time

add_interface = se.EvaluatorInfo(
    name='add',
    inputs=[
        se.FloatVariable(name='x1'),
        se.FloatVariable(name='x2'),
    ],
    outputs=[
        se.FloatVariable(name='sum'),
    ],
)

# Serial
eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=add_interface,
    pre_args=['exe_files/add.py'],
    run_parallel=False,
)

data = pd.DataFrame({
    'x1': [1, 2, 3, 4, 5],
    'x2': [5, 8, 1, 4, 7]
})

start = time.time()
eval(data)
end = time.time()

print(f'Serial: {end - start:.3f}')


# Parallel
eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=add_interface,
    pre_args=['exe_files/add.py'],
    run_parallel=True,
)

data = pd.DataFrame({
    'x1': [1, 2, 3, 4, 5],
    'x2': [5, 8, 1, 4, 7]
})

start = time.time()
eval(data)
end = time.time()

print(f'Parallel: {end - start:.3f}')

With run_parallel set to True we can clearly see that there is a reduction in runtime.

Array Variables

The executable evaluator supports ArrayVariable inputs and outputs. When your interface includes array variables, the evaluator automatically handles the conversion between the internal rolled format (NumPy arrays in DataFrame columns) and the flat CSV format that external executables expect.

How it works

Input (writing): Array variables are automatically unrolled into individual scalar columns using bracket notation before writing the CSV. For example, an ArrayVariable named x with shape (3,) becomes columns x[0], x[1], x[2].
Output (reading): After reading the output CSV, columns matching the bracket pattern are automatically rolled back into a single column containing NumPy arrays.
2D arrays: A variable with shape (2,3) produces columns x[0,0], x[0,1], x[0,2], x[1,0], x[1,1], x[1,2].

Custom readers/writers with arrays

When you provide a custom input_writer or output_reader, the unrolling/rolling still happens automatically:

Your input_writer receives the unrolled DataFrame with scalar columns (e.g., x[0], x[1], x[2]) — not the rolled NumPy arrays.
Your output_reader should return a DataFrame with unrolled column names — the evaluator will roll them back for you.

This means your custom I/O code only needs to deal with flat tabular data, regardless of whether arrays are involved.

Disabling automatic unrolling

If your custom I/O format natively handles arrays (e.g., HDF5, JSON, binary formats), you can disable the automatic unrolling by setting unroll_arrays=False:

eval = se.evaluators.ExecutableEvaluator(
    'my_exe',
    interface=my_interface,
    input_writer=my_writer,
    output_reader=my_reader,
    unroll_arrays=False,
)

With unroll_arrays=False:

Your input_writer receives the rolled DataFrame with NumPy arrays in cells.
Your output_reader must return a rolled DataFrame (NumPy arrays in cells, column names matching the variable names).
You are fully responsible for serializing and deserializing the array data in whatever format your executable expects.

Example

In this example our script (array_demo.py) computes y[i] = x[i] * 2 for each element of the input array, and total = sum(x) + s1. The evaluator handles all the unrolling and rolling automatically — we just pass NumPy arrays in and get NumPy arrays back.

array_demo.py

import sys
import numpy as np


in_file = sys.argv[1]
out_file = sys.argv[2]

# Read input CSV
data = np.loadtxt(in_file, delimiter=',', skiprows=1, ndmin=2)
header = open(in_file).readline().strip().split(',')

# Find x columns and s1
x_cols = sorted(
    [i for i, col in enumerate(header) if col.startswith('x[')],
)
s1_idx = header.index('s1')

# Compute outputs
out_header = [f'y[{i}]' for i in range(len(x_cols))] + ['total']
results = []

for row in data:
    x_vals = [row[i] for i in x_cols]
    s1_val = row[s1_idx]
    y_vals = [x * 2.0 for x in x_vals]
    total = sum(x_vals) + s1_val
    results.append(y_vals + [total])

# Write output CSV
with open(out_file, 'w') as f:
    f.write(','.join(out_header) + '\n')
    for row in results:
        f.write(','.join(str(v) for v in row) + '\n')

import numpy as np

# Define an interface with array variables
array_interface = se.EvaluatorInfo(
    name='array_demo',
    inputs=[
        se.ArrayVariable(name='x', shape=(3,), bounds=(-10, 10)),
        se.FloatVariable(name='s1', bounds=(-10, 10)),
    ],
    outputs=[
        se.ArrayVariable(name='y', shape=(3,)),
        se.FloatVariable(name='total'),
    ],
)

# The executable receives CSV with columns: x[0], x[1], x[2], s1
# and is expected to produce CSV with columns: y[0], y[1], y[2], total
array_eval = se.evaluators.ExecutableEvaluator(
    'python',
    interface=array_interface,
    pre_args=['exe_files/array_demo.py'],
)

# Build input with NumPy arrays in the DataFrame
sites = pd.DataFrame({
    'x': [np.array([1.0, 2.0, 3.0]), np.array([4.0, 5.0, 6.0])],
    's1': [10.0, 20.0],
})

array_eval(sites)

# Output arrays are automatically rolled back into NumPy arrays
sites

Note

For 2D arrays, column names in the CSV use comma-separated indices (e.g., x[0,0], x[0,1]). If your delimiter is also a comma, pandas will automatically quote these column names in the CSV. Make sure your executable handles quoted CSV headers correctly (Python’s csv module does this by default).

Executable Evaluator

Basic usage

Controlling how data is processed

mult_evals

run_parallel

Array Variables

How it works

Custom readers/writers with arrays

Disabling automatic unrolling

Example

`mult_evals`

`run_parallel`