Evaluator Interface

This notebook explains how to define the interface for an evaluator in the Standard Evaluator library. You will learn the two supported approaches: OptProblem for optimization problems with objectives and constraints, and EvaluatorInfo for general-purpose evaluators that only describe inputs and outputs. By the end, you will know how to create both interface types, pass them to a NumpyEvaluator, and access the resulting input/output metadata.

Two Interface Approaches

Every evaluator in the Standard Evaluator library requires an interface definition that describes its inputs and outputs. There are two ways to provide this:

1. OptProblem — Used when your evaluator represents an optimization problem. It defines typed input variables, response variables, and additionally specifies which responses serve as objectives and which serve as constraints. This is the right choice when you plan to optimize the evaluator.

2. EvaluatorInfo — Used for general-purpose evaluators that only need input/output descriptions without optimization semantics. It defines named, typed inputs and outputs but does not include objectives or constraints.

Both approaches use the same variable types (FloatVariable, IntVariable, ArrayVariable, CategoricalVariable) to describe individual inputs and outputs with names, bounds, defaults, and other metadata.

Setup

We begin by importing the library using the standard alias convention.

import standard_evaluator as se
from standard_evaluator.evaluators.numpy_evaluator import NumpyEvaluator
import numpy as np
import pandas as pd

matlabengine not available.

Creating an OptProblem

An OptProblem defines the full optimization context: input variables with bounds, response variables, objectives to minimize, and constraints to satisfy. Here we create a problem with two float inputs, two responses, one objective, and one constraint.

# Create an OptProblem with 2 inputs, 2 responses, 1 objective, 1 constraint
opt_problem = se.OptProblem(
    name="demo_problem",
    variables=[
        se.FloatVariable(name="x", bounds=(-5.0, 5.0)),
        se.FloatVariable(name="y", bounds=(0.0, 10.0)),
    ],
    responses=[
        se.FloatVariable(name="f"),
        se.FloatVariable(name="g"),
    ],
    objectives=["f"],
    constraints=["g"],
)

print("Problem name:", opt_problem.name)
print("Variables:", opt_problem.variable_names())
print("Responses:", opt_problem.response_names())
print("Objectives:", opt_problem.objectives)
print("Constraints:", opt_problem.constraints)

Problem name: demo_problem
Variables: ['x', 'y']
Responses: ['f', 'g']
Objectives: ['f']
Constraints: ['g']

The OptProblem validates that all objective and constraint names reference existing responses (or variables for objectives). The bounds on each FloatVariable define the feasible design space.

Creating an EvaluatorInfo

An EvaluatorInfo describes inputs and outputs without optimization semantics. It supports the same variable types as OptProblem but has no concept of objectives or constraints. Here we create an interface with a FloatVariable and an IntVariable as inputs, and a FloatVariable as output.

# Create an EvaluatorInfo with 2 typed inputs and 1 output
eval_info = se.EvaluatorInfo(
    name="weighted_sum",
    inputs=[
        se.FloatVariable(name="a", bounds=(0.0, 100.0)),
        se.IntVariable(name="n", bounds=(1, 10)),
    ],
    outputs=[
        se.FloatVariable(name="result"),
    ],
)

print("Interface name:", eval_info.name)
print("Inputs:", eval_info.input_names())
print("Outputs:", [v.name for v in eval_info.outputs])

Interface name: weighted_sum
Inputs: ['a', 'n']
Outputs: ['result']

Instantiating a NumpyEvaluator with EvaluatorInfo

To create a NumpyEvaluator, you subclass it and implement the eval_np method. You can pass an EvaluatorInfo as the interface parameter to define what the evaluator expects as inputs and produces as outputs.

# Define a NumpyEvaluator that computes result = a * n
class WeightedEvaluator(NumpyEvaluator):
    """Computes result = a * n."""

    def eval_np(self, sites, **kwargs):
        a = sites[:, 0]
        n = sites[:, 1]
        return (a * n).reshape(-1, 1)


# Instantiate using the interface parameter
weighted_eval = WeightedEvaluator(interface=eval_info)

print("Evaluator inputs:", weighted_eval.inputs)
print("Evaluator outputs:", weighted_eval.outputs)

Evaluator inputs: ['a', 'n']
Evaluator outputs: ['result']

The evaluator now knows it expects columns a and n as inputs and produces a column result as output. Internally, the EvaluatorInfo is converted to an OptProblem (without objectives or constraints) so that both interface types share the same underlying data model.

Instantiating a NumpyEvaluator with OptProblem

When you have an optimization problem definition, pass it directly as the opt_problem parameter. This gives the evaluator access to objective and constraint metadata in addition to the variable definitions.

# Define a NumpyEvaluator that computes f = x^2 + y^2, g = x + y
class OptEvaluator(NumpyEvaluator):
    """Computes f = x^2 + y^2 and g = x + y."""

    def eval_np(self, sites, **kwargs):
        x = sites[:, 0]
        y = sites[:, 1]
        f = x**2 + y**2
        g = x + y
        return np.column_stack([f, g])


# Instantiate using the opt_problem parameter
opt_eval = OptEvaluator(opt_problem=opt_problem)

# Call the evaluator with a DataFrame
sites = pd.DataFrame({"x": [1.0, -2.0, 3.0], "y": [2.0, 5.0, 1.0]})
opt_eval(sites)

print("Inputs property:", opt_eval.inputs)
print("Outputs property:", opt_eval.outputs)
print("\nEvaluation results:")
sites

Inputs property: ['x', 'y']
Outputs property: ['f', 'g']

Evaluation results:

	x	y	f	g
0	1.0	2.0	5.0	3.0
1	-2.0	5.0	29.0	3.0
2	3.0	1.0	10.0	4.0

The inputs property returns the list of input variable names (['x', 'y']) and the outputs property returns the list of response names (['f', 'g']). The DataFrame is modified in-place with the computed response columns appended.

When to Use OptProblem vs EvaluatorInfo

Choose the interface type based on your use case:

Use Case	Interface Type	Reason
Running optimization with an optimizer	OptProblem	Defines objectives and constraints needed by optimization algorithms
Wrapping analysis code for general evaluation	EvaluatorInfo	Only input/output descriptions are needed, no optimization semantics
Integration with EvaluatorOpenMdaoComponent	OptProblem	The OpenMDAO component uses objectives and constraints for driver setup
Building surrogate models	Either	Surrogate training only needs input/output structure

Summary:

• Use OptProblem when your evaluator will be used in an optimization context and you need to specify which responses are objectives to minimize and which are constraints to satisfy.

• Use EvaluatorInfo when you only need to describe the evaluator’s inputs and outputs without optimization semantics — for example, when wrapping a general-purpose analysis tool or building a data pipeline.