Evaluators

This page documents the evaluator class hierarchy in the Standard Evaluator library. The abstract Evaluator base class defines the common API, with concrete implementations for different execution backends.

Base Class

class Evaluator

Bases: ABC

Defines behavior common to all evaluators and what behaviors should be defined by the evaluators themselves.

Attributes:

`opt_problem`	(readonly) The OptProblem defining the pre-defined optimization problem.
`interface`	(readonly) The EvaluatorInfo defining the inputs and outputs of the evaluator.
`inputs`	(readonly) The inputs of the evaluator.
`outputs`	(readonly) The outputs of the evaluator.
`nind`	Number of independent variables
`ndep`	Number of dependent variables
`name`	(readonly) Name given to evaluator for easier identification.
`comp_cost`	How costly this evaluator is to calculate responses.
`full_options_names`	Names of all options returned by full_options().

Methods:

`__init__`([name, comp_cost, cache, ...])	Initialize the problem and set the variables and responses.
`__call__`(sites, **kwargs)	Calling method to evaluate the actual function.
`eval_np`(sites[, names])	Method to allow calling the function with a numpy array
`eval_list`(sites[, names])	Method to allow calling the function with the DE5 evaluator class
`default_site`()	Provide a DataFrame that contains all inputs and their default values
`initial_guess`()	Provide an initial guess for an optimizer
`required_options`()	Return an instance of the options model with default values.
`required_options_names`()	Return the set of option names defined by this evaluator class.
`full_options`()	Return all valid options for this evaluator instance.
`lookup_option_value`(name)	Look up the current value of an option by name.
`current_options`()	Return the current options as a combined model instance.
`evaluate_gradient`(sites, opt_problem)	Public method: return the numeric gradient array for `sites`.
`evaluate_jacobian`(sites, opt_problem)	Public method: return the constraint Jacobian array for `sites`.

property opt_problem: OptProblem

(readonly) The OptProblem defining the pre-defined optimization problem.

Note that for some evaluators you will only get information for variables and responses since no optimization problem is defined.

Type:: OptProblem

property interface: EvaluatorInfo

(readonly) The EvaluatorInfo defining the inputs and outputs of the evaluator.

Type:: EvaluatorInfo

property inputs: List[str]

(readonly) The inputs of the evaluator.

Type:: list[str]

property outputs: List[str]

(readonly) The outputs of the evaluator.

Type:: list[str]

property nind: int

Number of independent variables

Type:: int

property ndep: int

Number of dependent variables

Type:: int

property name: str

(readonly) Name given to evaluator for easier identification. Defaults to the name of the class. A user provided name will be appended.

Type:: str

__init__(name=None, comp_cost=100, cache=None, cache_options=None, logging=False, interface=None, opt_problem=None, options=None, **kwargs)

Initialize the problem and set the variables and responses.

Parameters:

name (str, optional) – Name to give the evaluator. Defaults to evaluator type name. If a value is provided, this will be appended to the default.
comp_cost (float, optional) – Computational cost estimate. Defaults to 100.
cache (str, optional) – Path to SQLite database where cached sites are stored, by default None. Enabling this option will also make the get_cache function available. This funtion will allow you access the sites that are currently defined in the cache.
cache_options (dict, optional) –
Options to modify caching behavior, by default None. The below options are available:
- table_name (str): Name of table in database where sites are stored, by default ‘Design_Data’
- delta (float): Unscaled distance used to determine if two sites are duplicate, by default 1e-8
- auto_update (bool): When True the database will be re-read before evaluating sites, by default False. This can be useful if another process is modifying the database concurrently.
logging (bool, optional) – When set to True the evaluator will keep track of every site that is passed to it, by default False. The logged sites can be access through the get_log function.
interface (EvaluatorInfo)
opt_problem (OptProblem)
options (BaseModel | None)

Return type:

None

property comp_cost: float

How costly this evaluator is to calculate responses. Defaults to 100.

Type:: float

__call__(sites, **kwargs)

Calling method to evaluate the actual function.

Note

The passed dataframe will be modified in place.

Parameters:: sites (DataFrame) – The dataframe that contains the input values, and is updated with the responses
Return type:: None

eval_np(sites, names=None, **kwargs)

Method to allow calling the function with a numpy array

Parameters:

sites (Numpy array) – The sites to be evaluated (as an np.ndarray)
names (List | None) – Optional list of names of responses to use

Type:

list

Returns:

Numpy array with the response values for all sites

Return type:

Numpy array

eval_list(sites, names=None, **kwargs)

Method to allow calling the function with the DE5 evaluator class

Parameters:

sites (List of lists) – The list sites to be evaluated (implemented as a list of list of unrolled data)
names (List | None) – Optional list of names of responses to use

Type:

list

Returns:

List of list of unrolled response values for each site

Return type:

List of lists

default_site()

Provide a DataFrame that contains all inputs and their default values

A developer can overwrite the default values, but in general these values are based on the default values of the interface inputs.

Returns:: DataFrame containing default values of all inputs
Return type:: pd.DataFrame

Note: This is just providing a different name for the initial_guess method.

initial_guess()

Provide an initial guess for an optimizer

Returns:: DataFrame providing the initial guess to use with an optimizer
Return type:: pd.DataFrame

classmethod required_options()

Return an instance of the options model with default values.

Returns:: An instance of the model returned by _define_options().
Return type:: BaseModel

classmethod required_options_names()

Return the set of option names defined by this evaluator class.

Returns:: Names of all fields in the options model.
Return type:: set[str]

full_options()

Return all valid options for this evaluator instance.

By default this delegates to required_options(). Subclasses may override this when the full set of options depends on runtime state.

Returns:: An instance of the options model.
Return type:: BaseModel

property full_options_names: set

Names of all options returned by full_options().

Return type:: set[str]

lookup_option_value(name)

Look up the current value of an option by name.

Checks self._options first, then falls back to full_options() defaults.

Parameters:: name (str) – The option name to look up.
Returns:: The current value of the option.
Return type:: Any
Raises:: KeyError – If the name is not a known option.

current_options()

Return the current options as a combined model instance.

Merges full_options() defaults with the user-overridden values in self._options using combine_instances.

Returns:: Combined options instance.
Return type:: BaseModel

evaluate_gradient(sites, opt_problem)

Public method: return the numeric gradient array for sites.

This is a convenience wrapper that calls _evaluate_partials and returns only the gradient portion.

Parameters:

sites (pandas.DataFrame) – DataFrame of sites (one row per site).
opt_problem (OptProblem) – OptProblem instance containing gradient mapping information.

Returns:

Gradient array shaped (num_sites, num_objs_total, num_flat_vars).

Return type:

numpy.ndarray

Notes:

Uses cached partials when available to avoid redundant finite-difference work.
Preconditions and exceptions mirror those of _evaluate_partials.

evaluate_jacobian(sites, opt_problem)

Public method: return the constraint Jacobian array for sites.

This convenience wrapper calls _evaluate_partials and returns only the Jacobian portion.

Parameters:

sites (pandas.DataFrame) – DataFrame of sites (one row per site).
opt_problem (OptProblem) – OptProblem instance containing Jacobian mapping information.

Returns:

Jacobian array shaped (num_sites, num_cons_total, num_flat_vars).

Return type:

numpy.ndarray

Notes:

Uses cached partials when available to avoid redundant finite-difference work.
Preconditions and exceptions mirror those of _evaluate_partials.

Concrete Evaluators

class NumpyEvaluator

Bases: Evaluator

An abstract class to allow us to define evaluators that implement the calling sequence with numpy

Methods:

`eval_np`(sites[, names])	Evaluate the function with a numpy array.
`dataframe_to_float_ndarray`(df)	Convert a DataFrame to a float ndarray.

abstractmethod eval_np(sites, names=None, **kwargs)

Evaluate the function with a numpy array.

Parameters:

sites (ndarray[tuple[Any, ...], dtype[float64]]) – The sites to be evaluated.
names (list) – Which responses are computed. Defaults to None (all).
**kwargs – Additional keyword arguments.

Returns:

2D array with response values for all sites.

Return type:

NDArray[np.float64]

dataframe_to_float_ndarray(df)

Convert a DataFrame to a float ndarray.

Parameters:: df (DataFrame) – The DataFrame to be converted.
Returns:: Array representing the DataFrame values as floats.
Return type:: NDArray[np.float64]

class ExecutableEvaluator

Bases: Evaluator

Evaluator that can run an executable. Site values are passed to/from the executable using input/output files in whatever format is needed. Command line arguments may also be passed along with the input/output files. By default the input/output files will be formatted as a CSV file with headers.

Methods:

__init__(executable_name, *[, interface, ...])

Initialize the ExecutableEvaluator.

__init__(executable_name, *, interface=None, opt_problem=None, run_dir=None, pre_args='', post_args='', input_prepend='', output_prepend='', delimiter=',', mult_evals=False, run_parallel=False, unroll_arrays=True, input_writer=None, output_reader=None, name=None, comp_cost=100, cache=None, cache_options=None, logging=False, **kwargs)

Initialize the ExecutableEvaluator.

Parameters:

executable_name (Path | str) – Path to or name of the executable to run.
interface (EvaluatorInfo | None) – EvaluatorInfo defining inputs and outputs.
opt_problem (OptProblem | None) – OptProblem defining variables and responses.
run_dir (Path | str | None) – Directory to run executable in.
pre_args (str | list[str]) – Arguments placed before the input/output file paths.
post_args (str | list[str]) – Arguments placed after the input/output file paths.
input_prepend (str) – String to prepend to the input file name.
output_prepend (str) – String to prepend to the output file name.
delimiter (str) – Single character used to delimit values in I/O files. Defaults to ‘,’.
mult_evals (bool) – If True, all sites are passed at once. Defaults to False.
run_parallel (bool) – If True, sites are evaluated in parallel. Defaults to False.
unroll_arrays (bool) – If True (default), array variables are unrolled to scalar columns before writing and rolled back after reading. If False, custom I/O functions receive raw rolled DataFrames.
input_writer (Callable[[Path, list[str], DataFrame], None] | None) – Custom function to write the input file.
output_reader (Callable[[Path, list[str], str], DataFrame] | None) – Custom function to read the output file.
name (str | None) – Name for identifying evaluator.
comp_cost (float) – Cost of running this evaluator. Defaults to 100.
cache (str | None) – Path to SQLite database for caching.
cache_options (dict | None) – Options to modify caching behavior.
logging (bool) – Enable logging of evaluated sites. Defaults to False.
**kwargs – Additional keyword arguments forwarded to the base class.

Return type:

None

class ShiftScaleEvaluator

Bases: Evaluator

Evaluator that applies mapping between shifted / scaled space and the original space for another evaluator.

Methods:

`__init__`(evaluate[, interface, opt_problem])	Initialize the problem and set the variables and responses.
`optimizer_to_design_space`(sites)	Transform a DataFrame from optimizer space to design space.
`design_to_optimizer_space`(sites)	Transform a DataFrame from design space to optimizer space.

__init__(evaluate, interface=None, opt_problem=None)

Initialize the problem and set the variables and responses.

Parameters:

name (str, optional) – Name to give the evaluator. Defaults to evaluator type name. If a value is provided, this will be appended to the default.
comp_cost (float, optional) – Computational cost estimate. Defaults to 100.
cache (str, optional) – Path to SQLite database where cached sites are stored, by default None. Enabling this option will also make the get_cache function available. This funtion will allow you access the sites that are currently defined in the cache.
cache_options (dict, optional) –
Options to modify caching behavior, by default None. The below options are available:
- table_name (str): Name of table in database where sites are stored, by default ‘Design_Data’
- delta (float): Unscaled distance used to determine if two sites are duplicate, by default 1e-8
- auto_update (bool): When True the database will be re-read before evaluating sites, by default False. This can be useful if another process is modifying the database concurrently.
logging (bool, optional) – When set to True the evaluator will keep track of every site that is passed to it, by default False. The logged sites can be access through the get_log function.
evaluate (Evaluator)
interface (EvaluatorInfo)
opt_problem (OptProblem)

Return type:

None

optimizer_to_design_space(sites)

Transform a DataFrame from optimizer space to design space.

Parameters:: sites (DataFrame) – DataFrame with points in the optimizer space.
Returns:: DataFrame with the points in the design space.
Return type:: pd.DataFrame

design_to_optimizer_space(sites)

Transform a DataFrame from design space to optimizer space.

Parameters:: sites (DataFrame) – DataFrame with points in the design space.
Returns:: DataFrame with the points in the optimizer space.
Return type:: pd.DataFrame

class OpenMDAOEvaluator

Bases: Evaluator

A class to expose an OpenMDAO problem as an evaluator

Methods:

__init__(om_prob[, comp_cost, scan_model, ...])

Initialize the evaluator, saving relevant information

__init__(om_prob, comp_cost=100, scan_model=True, use_defined_problem=True, opt_problem=None)

Initialize the evaluator, saving relevant information

Parameters:

om_prob (om.Problem) – The OpenMDAO problem
comp_cost (float, optional) – Computational costs of the evaluator. Defaults to 100.
scan_model (bool, optional) – Flag whether or not to scan the whole model for inputs and outputs. Defaults to True.
use_defined_problem (bool, optional) – Flag to decide whether or not to use the optimization problem defined in the OpenMDAO model. Defaults to True.
opt_problem (OptProblem, optional) – An explicitly provided OptProblem. When supplied, scan_model and use_defined_problem are ignored and this problem is used directly. Defaults to None.

Raises:

ValueError – Scan_model and use_defined_problem cannot both be false at the same time (when opt_problem is not provided).

Return type:

None

class ExcelEvaluator

Bases: Evaluator

Evaluator class that interfaces with an Excel spreadsheet to perform evaluations.

This class manages the lifecycle of an Excel application instance, opens a specified workbook, and maps inputs and outputs to specific Excel sheets and cells. It supports executing macros at different stages of the evaluation process and optionally shows the Excel UI.

_spreadsheet

The spreadsheet model containing the file path.

Type:: SpreadsheetModel

_input_mapping

Mapping of input variable names to Excel sheet and cell locations.

Type:: Dict[str, Dict[str, str]]

_output_mapping

Mapping of output variable names to Excel sheet and cell locations.

Type:: Dict[str, Dict[str, str]]

_excel

COM object representing the Excel application instance.

Type:: CDispatch

_workbook

COM object representing the opened Excel workbook.

Type:: CDispatch

_macros

Macros to be executed at various stages of evaluation.

Type:: MacroDefinition

_logger

Logger instance for logging events and errors.

Type:: logging.Logger

Methods:

`__init__`(spreadsheet[, name, comp_cost, ...])	Initialize the ExcelEvaluator.
`__del__`()	Destructor to clean up Excel resources.

__init__(spreadsheet, name=None, comp_cost=100, cache=None, cache_options=None, logging=False, interface=None, opt_problem=None, show_excel=False, macros=None, num_threads=1, **kwargs)

Initialize the ExcelEvaluator.

Parameters:

spreadsheet (Union[str, SpreadsheetModel]) – Path to the Excel spreadsheet file or a SpreadsheetModel instance.
name (str, optional) – Name to assign to the evaluator. Defaults to the evaluator type name. If provided, this value will be appended to the default name.
comp_cost (float, optional) – Estimated computational cost of the evaluator. Defaults to 100.
cache (str, optional) – Path to an SQLite database for caching evaluated sites. Enables caching functionality. Defaults to None.
cache_options (dict, optional) –
Dictionary of options to customize caching behavior. Defaults to None. Supported options include:
- table_name (str): Name of the database table storing cached sites (default: ‘Design_Data’).
- delta (float): Unscaled distance threshold to identify duplicate sites (default: 1e-8).
- auto_update (bool): If True, reloads the database before each evaluation (default: False).
logging (bool, optional) – Enable logging of all evaluated sites. Defaults to False.
interface (EvaluatorInfo, optional) – Interface information describing inputs and outputs. Defaults to None.
opt_problem (OptProblem, optional) – Optimization problem context. Takes priority over interface definition. Defaults to None.
show_excel (bool, optional) – Whether to make the Excel application visible during evaluation. Defaults to False.
macros (MacroDefinition, optional) – Macros to execute at different stages of the evaluation process. Defaults to an empty MacroDefinition instance.
**kwargs – Additional keyword arguments passed to the base Evaluator class.
num_threads (int)

Return type:

None

__del__()

Destructor to clean up Excel resources.

Closes the workbook without saving changes and quits the Excel application. Suppresses exceptions that may occur during interpreter shutdown or if resources are already released.