Evaluator Hierarchy

This notebook introduces the core abstraction in the Standard Evaluator library: the Evaluator class hierarchy. You will learn how the abstract Evaluator base class defines a common API for wrapping analysis codes, explore the available concrete subclasses, and see how decorators add caching and logging capabilities. By the end, you will be able to create evaluators, call them with DataFrames or NumPy arrays, and leverage built-in caching and logging.

The Evaluator Class Hierarchy

The Evaluator abstract base class defines the standard interface that all evaluators share:

__call__(sites: DataFrame) — Evaluate sites provided as a pandas DataFrame. The DataFrame is modified in-place to include response columns.
eval_np(sites: ndarray) — Evaluate sites provided as a NumPy array with unrolled scalar columns (excludes fixed variables and unrolls ArrayVariables).
eval_list(sites: List) — Same as eval_np but accepts and returns Python lists.

Concrete Subclasses

Class	Purpose
`NumpyEvaluator`	Wraps a user-defined function that operates on NumPy arrays
`ExecutableEvaluator`	Wraps an external executable (command-line tool)
`ShiftScaleEvaluator`	Applies shift/scale transformations before delegating to another evaluator
`OpenMdaoEvaluator`	Wraps an OpenMDAO component as an evaluator
`ExcelEvaluator`	Wraps an Excel workbook as an evaluator
`MatlabEvaluator`	Wraps a MATLAB function via the MATLAB Engine API

Decorators

The evaluator constructor supports two decorator parameters:

cache — Pass a path to a SQLite database file to enable caching of evaluated sites. Previously computed sites are looked up rather than re-evaluated.
logging — Set to True to record every evaluation call. The logged history is accessible via get_log().

Setup

We begin by importing the library using the standard alias convention.

import standard_evaluator as se
import pandas as pd
import numpy as np

matlabengine not available.

Creating and Calling an Evaluator

We create a Sphere benchmark evaluator with 3 design variables. This is a NumpyEvaluator subclass representing the sphere function \(f(\mathbf{x}) = \sum x_i^2\). We then build an input DataFrame with 3 rows and call the evaluator using __call__.

# Create a Sphere benchmark evaluator with 3 design variables
evaluator = se.evaluators.Sphere(num_independent=3)

print("Inputs:", evaluator.inputs)
print("Outputs:", evaluator.outputs)
print("Variable bounds:")
for var in evaluator.opt_problem.variables:
    print(f"  {var.name}: {var.bounds}")

Inputs: ['x0', 'x1', 'x2']
Outputs: ['f']
Variable bounds:
  x0: (-10.0, 10.0)
  x1: (-10.0, 10.0)
  x2: (-10.0, 10.0)

Now we build an input DataFrame with 3 sample sites and call the evaluator. The __call__ method modifies the DataFrame in-place, appending the response column f.

# Build an input DataFrame with 3 rows of sample sites
sites = pd.DataFrame({
    "x0": [1.0, 2.0, -3.0],
    "x1": [0.5, -1.0, 4.0],
    "x2": [2.0, 3.0, -1.0],
})

# Call the evaluator — modifies sites in-place
evaluator(sites)

# Display the result with both input and response columns
sites

	x0	x1	x2	f
0	1.0	0.5	2.0	5.25
1	2.0	-1.0	3.0	14.00
2	-3.0	4.0	-1.0	26.00

The DataFrame now contains the original input columns (x0, x1, x2) along with the computed response column f, where each row’s value equals \(x_0^2 + x_1^2 + x_2^2\).

Using `eval_np` and `eval_list`

The eval_np and eval_list methods provide a convenience interface that works with raw NumPy arrays or Python lists instead of DataFrames. Key differences from __call__:

Fixed variables are excluded from the input — you only provide varying design variables.
ArrayVariables are unrolled into individual scalar columns (e.g., a shape (3,) array becomes 3 separate values).
Only the __call__ method preserves array variables as arrays within DataFrame cells.

These methods are useful when integrating with optimization libraries that operate on flat numeric arrays.

# eval_np: pass a 2D NumPy array (rows=sites, cols=unrolled inputs)
sites_np = np.array([
    [1.0, 0.5, 2.0],
    [2.0, -1.0, 3.0],
    [-3.0, 4.0, -1.0],
])

results_np = evaluator.eval_np(sites_np)
print("eval_np results (NumPy array):")
print(results_np)

eval_np results (NumPy array):
[[ 5.25]
 [14.  ]
 [26.  ]]

The eval_list method works identically but accepts and returns Python lists of lists.

# eval_list: pass a list of lists
sites_list = [
    [1.0, 0.5, 2.0],
    [2.0, -1.0, 3.0],
]

results_list = evaluator.eval_list(sites_list)
print("eval_list results (Python lists):")
print(results_list)

eval_list results (Python lists):
[[5.25], [14.0]]

Both eval_np and eval_list return only the response values (here, the f column as a single-column array or nested list).

Caching Decorator

By passing a cache parameter (a path to a SQLite database file) when constructing an evaluator, the caching decorator is applied. Previously evaluated sites are retrieved from the cache instead of being re-computed. After evaluation, you can inspect the cache using get_cache().

import tempfile
import os

# Create a temporary database file for caching
cache_file = os.path.join(tempfile.gettempdir(), "sphere_cache_demo.db")

# Create evaluator with caching enabled
cached_evaluator = se.evaluators.Sphere(num_independent=2, cache=cache_file)

# Evaluate two sites
sites_cached = pd.DataFrame({"x0": [1.0, 2.0], "x1": [3.0, 4.0]})
cached_evaluator(sites_cached)
print("Evaluation result:")
print(sites_cached)

# Inspect the cache
print("\nCached sites:")
print(cached_evaluator.get_cache())

Evaluation result:
    x0   x1     f
0  1.0  3.0  10.0
1  2.0  4.0  20.0

Cached sites:
    x0   x1     f
0  1.0  3.0  10.0
1  2.0  4.0  20.0

The cache stores all evaluated sites. On subsequent calls, any site matching a cached entry (within a configurable tolerance) will retrieve its response from the database rather than re-evaluating.

Logging Decorator

Setting logging=True when constructing an evaluator enables the logging decorator. Every evaluation call is recorded, and the full history can be retrieved using get_log(). Each logged site includes a call_num column indicating which evaluation call it belongs to.

# Create evaluator with logging enabled
logged_evaluator = se.evaluators.Sphere(num_independent=2, logging=True)

# First evaluation call
sites_a = pd.DataFrame({"x0": [1.0, 2.0], "x1": [3.0, 4.0]})
logged_evaluator(sites_a)

# Second evaluation call
sites_b = pd.DataFrame({"x0": [5.0], "x1": [6.0]})
logged_evaluator(sites_b)

# Retrieve the full evaluation log
log = logged_evaluator.get_log()
print("Evaluation log:")
log

Evaluation log:

	x0	x1	f	call_num
0	1.0	3.0	10.0	1.0
1	2.0	4.0	20.0	1.0
2	5.0	6.0	61.0	2.0

The log shows all 3 evaluated sites across both calls. The call_num column distinguishes which invocation produced each row — sites from the first call have call_num=1 and the site from the second call has call_num=2.