Optimize an Action

Use Klu's optimization workflow to turn production logs into a test Dataset, establish an Eval baseline, compare Action variants, and promote only changes that improve the behavior you care about.

Choose the right workflow

• Use Logs to find failures and collect ratings, issues, and corrections.
• Use Datasets to preserve a reproducible training or evaluation set.
• Use Evaluate to score one Action version against a Dataset.
• Use Experiments to split prompts between two Actions and compare live behavior.
• Use Finetune to submit a Dataset to a supported model provider.

Use Finetune for provider training and Experiments for traffic comparisons.

Build an evaluation loop

1. Open Logs and filter for negative feedback, issues, or representative traffic.
2. Inspect the rows and add corrections where you know the desired output.
3. Select Save Dataset and choose an evaluation split.
4. Open Evaluate, select Add Eval, and bind the Dataset to the Action and evaluators.
5. Run the Eval to establish a baseline.
6. Change the prompt, model, or model settings in a new Action version.
7. Run the same Eval again and compare runs.

Keep the Dataset and evaluator configuration fixed while comparing versions. If both change, you cannot attribute the result difference to the Action change.

Run an A/B experiment in the UI

Experiments compare two Actions. Create both Actions before opening the Experiments page.

1. Select Add Experiment.
2. Enter Experiment Name.
3. Choose Select a Primary Action and Select a Secondary Action. They must be different Actions in the same App.
4. Select Create.
5. Open the experiment to inspect primary and secondary results, feedback, and date filters.

Select Add Experiment to begin.

Create and prompt an experiment with an SDK

Both SDKs expose Experiments and use Action GUIDs. Calls are asynchronous.

import asyncio

from klu import Klu


async def main() -> None:
    klu = Klu("YOUR_API_KEY")
    experiment = await klu.experiments.create(
        name="Support prompt comparison",
        app_guid="YOUR_APP_GUID",
        action_primary_guid="YOUR_PRIMARY_ACTION_GUID",
        action_secondary_guid="YOUR_SECONDARY_ACTION_GUID",
    )

    result = await klu.experiments.prompt(
        experiment_guid=experiment.guid,
        input="How do I change my billing email?",
    )
    print(result.msg, result.feedback_url)


asyncio.run(main())

Use force_action in Python or forceAction in TypeScript only when you intentionally need one arm, such as a deterministic verification. For normal experiment traffic, leave it unset so the experiment selects an Action.

Run experiment prompts in the background

The background methods return a result URL before generation completes. Poll that URL until the response contains the completed message.

import asyncio

from klu import Klu


async def main() -> None:
    klu = Klu("YOUR_API_KEY")
    started = await klu.experiments.async_prompt(
        experiment_guid="YOUR_EXPERIMENT_GUID",
        input="Summarize the refund policy",
    )

    if started.result_url is None:
        raise RuntimeError("Klu did not return a result URL")

    for _ in range(30):
        result = await klu.experiments.get_async_prompt_result(started.result_url)
        if result.status == "SUCCESS":
            print(result.msg)
            return
        if result.status == "FAILURE":
            raise RuntimeError(result.msg)
        await asyncio.sleep(2)

    raise TimeoutError("Experiment prompt did not finish within 60 seconds")


asyncio.run(main())

The first result read can report that processing is still in progress. Keep polling bounded, increase the delay for longer jobs, and stop on authentication, validation, or not-found errors instead of retrying them indefinitely.

Decide what to ship

Use Eval results for repeatable offline quality and experiment feedback for observed live behavior. Check latency, token use, provider errors, and cost alongside quality. Promote a version only after the evidence relevant to its intended traffic is complete; a queued Eval, running fine-tune, or newly created experiment is not a successful optimization result.