{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "17e94ce6",
   "metadata": {},
   "source": [
    "# Executable Evaluator\n",
    "[Click here to view the API reference](project:/reference/evaluators/executable.rst)\n",
    "\n",
    "The executable evaluator is an excellent option if there is not an existing evaluator in the library that fits your needs. It gives you the ability to directly interface your desired application with Python and use it as an evaluator. There is plenty of flexibility built into it that enables you to interface with numerous applications. This tutorial will walk you through the inner workings of the evaluator through examples to help you get started with building your own interface.\n",
    "\n",
    "Here is the general order of operations that the executable evaluator performs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b38f45fe",
   "metadata": {},
   "source": [
    "```{mermaid}\n",
    "flowchart LR\n",
    "    start((Start)) --> write_csv[Write CSV] & write[Input writer] --> run[Run executable]\n",
    "    run --> read_csv[Read CSV] & read[Output reader] --> return(((Return)))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763ee416",
   "metadata": {},
   "source": [
    "By default the evaluator will write the inputs to a CSV file and pass the absolute path of this file to the executable as a command line argument. Similarly, the default output is a CSV file whose absolute path is also passed as a command line argument. When constructing the evaluator you may provide functions to the `input_writer` and `output_reader` arguments to override this default behavior. However, the file names cannot be changed and your functions are expected to write/read the contents of the given files. If your executable requires specific input/output file names and it disregards the paths given as arguments then you are also free to use these functions to write/read these specific files.\n",
    "\n",
    "```{note}\n",
    "You can use any delimiter besides a comma by providing it as an argument to the `delimiter` parameter in the constructor. This delimiter will be used when generating the input and expected when reading the output.\n",
    "```\n",
    "\n",
    "## Basic usage\n",
    "To get a feel for how this evaluator operates let's analyze a simple test case. In this example our \"application\" will simply add two variables. The file below will act as our application"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dac2111",
   "metadata": {},
   "source": [
    "```{code-block}\n",
    ":name: add\n",
    ":caption: add.py\n",
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "# Read input\n",
    "in_file = sys.argv[1]\n",
    "print('in_file: ', in_file)\n",
    "\n",
    "x1, x2 = np.loadtxt(in_file, delimiter=',', skiprows=1, unpack=True, ndmin=2)\n",
    "print(f'x1: {x1}\\nx2: {x2}')\n",
    "\n",
    "# Compute\n",
    "result = x1 + x2\n",
    "\n",
    "# Write output\n",
    "out_file = sys.argv[2]\n",
    "print('out_file: ', out_file)\n",
    "\n",
    "np.savetxt(out_file, result, delimiter='\\n', header='sum', comments='')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0353f75d",
   "metadata": {},
   "source": [
    "This is a fairly simple script. We read in the file which will have data formatted as a comma delimited text file where the first row contains the columns names. After adding the values together we write the sums to the designated output file. Making sure to add the column name in the first row.\n",
    "\n",
    "Now we construct our evaluator and run some data through it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "813da9c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import standard_evaluator as se\n",
    "import pandas as pd\n",
    "\n",
    "# Create evaluator\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=se.EvaluatorInfo(\n",
    "        name='add',\n",
    "        inputs=[\n",
    "            se.FloatVariable(name='x1'),\n",
    "            se.FloatVariable(name='x2'),\n",
    "        ],\n",
    "        outputs=[\n",
    "            se.FloatVariable(name='sum'),\n",
    "        ],\n",
    "    ),\n",
    "    pre_args=['exe_files/add.py'],\n",
    ")\n",
    "\n",
    "# Data points to evaluate\n",
    "data = pd.DataFrame({\n",
    "    'x1': [1, 2],\n",
    "    'x2': [5, 8]\n",
    "})\n",
    "\n",
    "# Do evaluation\n",
    "eval(data)\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2541df9",
   "metadata": {},
   "source": [
    "As expected the evaluator returns the sum of `x1` and `x2`. Let's dive a little deeper into what is going on here. If we were going to be running `add.py` as a standalone file we would use something like this in a terminal window:\n",
    "\n",
    "```console\n",
    "python exe_files/add.py input.csv output.csv\n",
    "```\n",
    "\n",
    "Which would run our script that lives in the subdirectory `exe_files` using `input.csv` as input and `output.csv` as output. Now the command that the evaluator builds will have the following format\n",
    "\n",
    "$${\\underbrace{\\texttt{path/to/program}}_{\\texttt{executable\\_name}}}\\texttt{ }{\\underbrace{\\texttt{arg1 arg2}}_\\texttt{pre\\_args}}~\\texttt{ input.in output.out }{\\underbrace{\\texttt{arg3 arg4}}_\\texttt{post\\_args}}$$\n",
    "\n",
    "So to match this pattern we use `python` as the executable name/path (Note that since it is already in my PATH variable I do not need to provide the full path to the executable). Since Python expects a path to the desired script file before anything else use the `pre_args` parameter to set the path to our script which resides in the `exe_files` subdirectory.\n",
    "\n",
    "Finally, you may have noticed that none of the `print` statements in our script had any affect. This is because the evaluator captures the `stdout` and `stderr` streams and supresses them. Otherwise your console might get flooded with data. If this information is useful to you or you simply want to use it then you can use the `output_reader` parameter. This parameter expects to take in a function of the following form\n",
    "\n",
    "```python\n",
    "def reader(out_path: pathlib.Path, responses: list[str], stdout: str) -> pandas.DataFrame:\n",
    "    ...\n",
    "```\n",
    "\n",
    "Here is a brief description what what each parameter will contain:\n",
    "- `out_path`: Path to the generated output file\n",
    "- `responses`: Names of each of the responses that should be returned\n",
    "- `stdout`: String containing the captured output in the `stdout` and `stderr` streams\n",
    "\n",
    "This function should then return a pandas DataFrame with the results parsed from the output file. Let's construct a reader to view the captured output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83103817",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import pathlib\n",
    "\n",
    "def add_reader(out_file: pathlib.Path, responses: list[str], stdout: str) -> pd.DataFrame:\n",
    "    print(stdout)\n",
    "    return pd.read_csv(out_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec1687b8",
   "metadata": {},
   "source": [
    "This reader is super simple. It prints the captured output and reads the generated CSV as a pandas DataFrame. To use it we simply provide this function as the argument to the `output_reader` parameter.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fce4eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create evaluator\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=se.EvaluatorInfo(\n",
    "        name='add',\n",
    "        inputs=[\n",
    "            se.FloatVariable(name='x1'),\n",
    "            se.FloatVariable(name='x2'),\n",
    "        ],\n",
    "        outputs=[\n",
    "            se.FloatVariable(name='sum'),\n",
    "        ],\n",
    "    ),\n",
    "    pre_args=['exe_files/add.py'],\n",
    "    output_reader=add_reader,\n",
    ")\n",
    "\n",
    "# Data points to evaluate\n",
    "data = pd.DataFrame({\n",
    "    'x1': [1, 2],\n",
    "    'x2': [5, 8]\n",
    "})\n",
    "\n",
    "# Do evaluation\n",
    "eval(data)\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30fa8ae4",
   "metadata": {},
   "source": [
    "We again get the correct result but this time we can also see the captured output we placed in our script. As you likely noticed, our script is run twice–once for each row. The next section explains why this happens and how you can control it.\n",
    "\n",
    "## Controlling how data is processed\n",
    "As we saw in the previous example the executable was run once for each data point. The executable evaluator provides parameters that allow you to control how data gets fed into your application.\n",
    "\n",
    "```{note}\n",
    "When both `mult_evals` and `run_parallel` are set to `True` priority is given to `mult_evals` and `run_parallel` is ignored!\n",
    "```\n",
    "\n",
    "### `mult_evals`\n",
    "This parameter when set to `True` will place all data points into the same file and call your application once. Likewise, the evaluator expects your application to return outputs for each of the points in the same file. Let's run our previous example with this flag enabled to see it in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03e51de7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create evaluator\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=se.EvaluatorInfo(\n",
    "        name='add',\n",
    "        inputs=[\n",
    "            se.FloatVariable(name='x1'),\n",
    "            se.FloatVariable(name='x2'),\n",
    "        ],\n",
    "        outputs=[\n",
    "            se.FloatVariable(name='sum'),\n",
    "        ],\n",
    "    ),\n",
    "    pre_args=['exe_files/add.py'],\n",
    "    output_reader=add_reader,\n",
    "    mult_evals=True,\n",
    ")\n",
    "\n",
    "# Data points to evaluate\n",
    "data = pd.DataFrame({\n",
    "    'x1': [1, 2, 3, 4, 5],\n",
    "    'x2': [5, 8, 1, 4, 7]\n",
    "})\n",
    "\n",
    "# Do evaluation\n",
    "eval(data)\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34f0e36a",
   "metadata": {},
   "source": [
    "As expected, we see that our script is only run once and all of the data points are evaluated at once.\n",
    "\n",
    "### `run_parallel`\n",
    "An alternative approach if your application can only process one input at a time is to run it in parallel. Setting this parameter to `True` will use `dask` to run multiple instances of your application at once. In the code below we use our trusty addition script and compare the run times in serial and parallel execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2f444f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "add_interface = se.EvaluatorInfo(\n",
    "    name='add',\n",
    "    inputs=[\n",
    "        se.FloatVariable(name='x1'),\n",
    "        se.FloatVariable(name='x2'),\n",
    "    ],\n",
    "    outputs=[\n",
    "        se.FloatVariable(name='sum'),\n",
    "    ],\n",
    ")\n",
    "\n",
    "# Serial\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=add_interface,\n",
    "    pre_args=['exe_files/add.py'],\n",
    "    run_parallel=False,\n",
    ")\n",
    "\n",
    "data = pd.DataFrame({\n",
    "    'x1': [1, 2, 3, 4, 5],\n",
    "    'x2': [5, 8, 1, 4, 7]\n",
    "})\n",
    "\n",
    "start = time.time()\n",
    "eval(data)\n",
    "end = time.time()\n",
    "\n",
    "print(f'Serial: {end - start:.3f}')\n",
    "\n",
    "\n",
    "# Parallel\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=add_interface,\n",
    "    pre_args=['exe_files/add.py'],\n",
    "    run_parallel=True,\n",
    ")\n",
    "\n",
    "data = pd.DataFrame({\n",
    "    'x1': [1, 2, 3, 4, 5],\n",
    "    'x2': [5, 8, 1, 4, 7]\n",
    "})\n",
    "\n",
    "start = time.time()\n",
    "eval(data)\n",
    "end = time.time()\n",
    "\n",
    "print(f'Parallel: {end - start:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7102790",
   "metadata": {},
   "source": [
    "With `run_parallel` set to `True` we can clearly see that there is a reduction in runtime.\n",
    "\n",
    "## Array Variables\n",
    "\n",
    "The executable evaluator supports `ArrayVariable` inputs and outputs. When your interface includes array variables, the evaluator automatically handles the conversion between the internal rolled format (NumPy arrays in DataFrame columns) and the flat CSV format that external executables expect.\n",
    "\n",
    "### How it works\n",
    "\n",
    "- **Input (writing):** Array variables are automatically *unrolled* into individual scalar columns using bracket notation before writing the CSV. For example, an `ArrayVariable` named `x` with shape `(3,)` becomes columns `x[0]`, `x[1]`, `x[2]`.\n",
    "- **Output (reading):** After reading the output CSV, columns matching the bracket pattern are automatically *rolled back* into a single column containing NumPy arrays.\n",
    "- **2D arrays:** A variable with shape `(2,3)` produces columns `x[0,0]`, `x[0,1]`, `x[0,2]`, `x[1,0]`, `x[1,1]`, `x[1,2]`.\n",
    "\n",
    "### Custom readers/writers with arrays\n",
    "\n",
    "When you provide a custom `input_writer` or `output_reader`, the unrolling/rolling still happens automatically:\n",
    "- Your `input_writer` receives the **unrolled** DataFrame with scalar columns (e.g., `x[0]`, `x[1]`, `x[2]`) — not the rolled NumPy arrays.\n",
    "- Your `output_reader` should return a DataFrame with **unrolled** column names — the evaluator will roll them back for you.\n",
    "\n",
    "This means your custom I/O code only needs to deal with flat tabular data, regardless of whether arrays are involved.\n",
    "\n",
    "### Disabling automatic unrolling\n",
    "\n",
    "If your custom I/O format natively handles arrays (e.g., HDF5, JSON, binary formats), you can disable the automatic unrolling by setting `unroll_arrays=False`:\n",
    "\n",
    "```python\n",
    "eval = se.evaluators.ExecutableEvaluator(\n",
    "    'my_exe',\n",
    "    interface=my_interface,\n",
    "    input_writer=my_writer,\n",
    "    output_reader=my_reader,\n",
    "    unroll_arrays=False,\n",
    ")\n",
    "```\n",
    "\n",
    "With `unroll_arrays=False`:\n",
    "- Your `input_writer` receives the **rolled** DataFrame with NumPy arrays in cells.\n",
    "- Your `output_reader` must return a **rolled** DataFrame (NumPy arrays in cells, column names matching the variable names).\n",
    "- You are fully responsible for serializing and deserializing the array data in whatever format your executable expects.\n",
    "\n",
    "### Example\n",
    "\n",
    "In this example our script (`array_demo.py`) computes `y[i] = x[i] * 2` for each element of the input array, and `total = sum(x) + s1`. The evaluator handles all the unrolling and rolling automatically — we just pass NumPy arrays in and get NumPy arrays back."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "array-demo-source",
   "metadata": {},
   "source": [
    "```{code-block}\n",
    ":name: array_demo\n",
    ":caption: array_demo.py\n",
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "in_file = sys.argv[1]\n",
    "out_file = sys.argv[2]\n",
    "\n",
    "# Read input CSV\n",
    "data = np.loadtxt(in_file, delimiter=',', skiprows=1, ndmin=2)\n",
    "header = open(in_file).readline().strip().split(',')\n",
    "\n",
    "# Find x columns and s1\n",
    "x_cols = sorted(\n",
    "    [i for i, col in enumerate(header) if col.startswith('x[')],\n",
    ")\n",
    "s1_idx = header.index('s1')\n",
    "\n",
    "# Compute outputs\n",
    "out_header = [f'y[{i}]' for i in range(len(x_cols))] + ['total']\n",
    "results = []\n",
    "\n",
    "for row in data:\n",
    "    x_vals = [row[i] for i in x_cols]\n",
    "    s1_val = row[s1_idx]\n",
    "    y_vals = [x * 2.0 for x in x_vals]\n",
    "    total = sum(x_vals) + s1_val\n",
    "    results.append(y_vals + [total])\n",
    "\n",
    "# Write output CSV\n",
    "with open(out_file, 'w') as f:\n",
    "    f.write(','.join(out_header) + '\\n')\n",
    "    for row in results:\n",
    "        f.write(','.join(str(v) for v in row) + '\\n')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "array-demo-1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define an interface with array variables\n",
    "array_interface = se.EvaluatorInfo(\n",
    "    name='array_demo',\n",
    "    inputs=[\n",
    "        se.ArrayVariable(name='x', shape=(3,), bounds=(-10, 10)),\n",
    "        se.FloatVariable(name='s1', bounds=(-10, 10)),\n",
    "    ],\n",
    "    outputs=[\n",
    "        se.ArrayVariable(name='y', shape=(3,)),\n",
    "        se.FloatVariable(name='total'),\n",
    "    ],\n",
    ")\n",
    "\n",
    "# The executable receives CSV with columns: x[0], x[1], x[2], s1\n",
    "# and is expected to produce CSV with columns: y[0], y[1], y[2], total\n",
    "array_eval = se.evaluators.ExecutableEvaluator(\n",
    "    'python',\n",
    "    interface=array_interface,\n",
    "    pre_args=['exe_files/array_demo.py'],\n",
    ")\n",
    "\n",
    "# Build input with NumPy arrays in the DataFrame\n",
    "sites = pd.DataFrame({\n",
    "    'x': [np.array([1.0, 2.0, 3.0]), np.array([4.0, 5.0, 6.0])],\n",
    "    's1': [10.0, 20.0],\n",
    "})\n",
    "\n",
    "array_eval(sites)\n",
    "\n",
    "# Output arrays are automatically rolled back into NumPy arrays\n",
    "sites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "array-note",
   "metadata": {},
   "source": [
    "```{note}\n",
    "For 2D arrays, column names in the CSV use comma-separated indices (e.g., `x[0,0]`, `x[0,1]`). If your delimiter is also a comma, pandas will automatically quote these column names in the CSV. Make sure your executable handles quoted CSV headers correctly (Python's `csv` module does this by default).\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "mystnb": {
   "execution_mode": "off"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}