Evaluate an Action

An Eval runs an Action against a Dataset and scores each result with one or more evaluators. Use Evals to compare prompt or model versions against a fixed test set.

Before you begin

You need an App with:

• an Action to evaluate
• a Dataset containing at least one usable item
• one or more evaluators available in the workspace

Create or import the Dataset from Datasets, or save filtered production logs from Logs.

Create an Eval in the UI

1. Open the App and select Evaluate in the sidebar.
2. Select Add Eval.
3. Under Select an Action to evaluate, choose the Action.
4. Under Select evaluations, add at least one evaluator and complete any evaluator-specific variables.
5. Choose a Dataset. You can create an empty Dataset from this flow, but you must add items before the Eval can run.
6. Enter the Eval name and create it.

Klu opens the Eval after creation. An Eval is tied to its Action, Dataset, selected Action version, and evaluator configuration. Edit the Eval when you need to change the version, evaluators, sampling rate, or failure alert behavior.

Run and inspect an Eval

Select Run Eval from the Eval page or its card on the Evaluate page. Starting a run is asynchronous: Eval run started confirms that work was queued, not that scoring has completed.

Remain on the Eval page or return later to inspect the run. Each completed run records the Action version and evaluator results. Use the run detail to inspect individual Dataset items, actual outputs, expected outputs, and pass/fail results. Use the comparison view to compare runs across Action versions.

The run action is disabled when no evaluators are configured. An empty Dataset is shown explicitly and cannot produce a meaningful run. Provider, Action, or evaluator failures appear on the run rather than turning a queued confirmation into a successful result.

Manage Evals with the Python SDK

The Python SDK exposes Evals as klu.evals. Its calls are asynchronous. The current TypeScript SDK does not expose an Eval client, so create and run Evals in the UI or use the Python SDK.

First list the evaluator types available to the workspace. Their GUIDs, names, and metadata requirements are workspace-specific.

import asyncio

from klu import Klu


async def main() -> None:
    klu = Klu("YOUR_API_KEY")

    eval_types = await klu.evals.get_eval_types()
    for eval_type in eval_types:
        print(eval_type.guid, eval_type.name, eval_type.metadata)


asyncio.run(main())

Create the Eval with an Action GUID, Dataset GUID, and evaluator configuration returned by that workspace:

import asyncio

from klu import Klu


async def main() -> None:
    klu = Klu("YOUR_API_KEY")
    evaluation = await klu.evals.create(
        name="Support answer quality",
        action_guid="YOUR_ACTION_GUID",
        eval_types=[
            {
                "guid": "YOUR_EVAL_TYPE_GUID",
                "name": "Your evaluator name",
                "metadata": {},
            }
        ],
        dataset="YOUR_DATASET_GUID",
        version=None,
        sampling_rate=None,
        alert_on_fail=False,
    )

    await klu.evals.run(evaluation.guid)
    runs = await klu.evals.get_eval_runs(evaluation.guid)

    for run in runs:
        print(run.guid, run.run_number, run.metadata)


asyncio.run(main())

run starts evaluation work and returns before the run is necessarily complete. Poll get_eval_runs and inspect the selected run's metadata or retrieve it with get_eval_run(eval_guid, run_guid). Do not interpret the string returned by run as completed scoring.

Keep comparisons valid

Use the same Dataset and evaluators when comparing Action versions. Change one material variable at a time, review item-level failures, and keep expected outputs representative of the behavior you want. If the Dataset changes between runs, the aggregate pass rates no longer describe the same test population.