LLM operations

LLM operations is the practice of shipping model-backed behavior with controlled versions, measurable quality, observable failures, and a safe path to change. Provider catalogs, model names, prices, and benchmark rankings change too quickly to serve as durable operating guidance. Use the models configured in your workspace and validate them against your own Actions and data.

Use a production lifecycle

A practical Klu lifecycle has six stages:

  1. Define the Action's prompt, model configuration, tools, output contract, and optional Context.
  2. Test representative inputs in Studio and save a version.
  3. Evaluate that version against a dataset and review failures, latency, and cost.
  4. Deploy the selected version to Preview, Staging, or Production.
  5. Observe production data, feedback, errors, provider behavior, and asynchronous jobs.
  6. Improve from evidence, save a new version, and repeat the same gates.

Deployments assign Action versions to environments. Keep the previous known-good version available so you can restore it when a new version fails its production checks.

Separate the operating concerns

Quality

Model output varies with the prompt, model configuration, Context, tools, and production input distribution. Build eval datasets from representative and difficult cases. Define pass criteria before comparing versions, then inspect individual failures alongside aggregate scores.

Use Evals for repeatable checks and Optimize for an evidence-driven improvement loop.

Grounding and data freshness

Retrieval quality depends on source freshness, successful indexing, chunk settings, and query relevance. Verify source status and retrieval results before attributing a bad answer to the model.

Use Context sources to add sources, verify Documents, reindex safely, and troubleshoot failed ingestion.

Latency and availability

Measure end-to-end latency at the user boundary. Separate queue delay, retrieval, model time, tool calls, and streaming behavior where your telemetry allows it. Streaming improves perceived responsiveness while total completion time and downstream failures still need monitoring.

Choose synchronous, streaming, or asynchronous execution based on the product interaction.

Cost and limits

Track input and output volume, retries initiated by your product, retrieval size, tool calls, and background-job fan-out. Provider limits and pricing belong to the provider account configured for the workspace. Confirm those values at change time.

Review the limits shown in Klu and by your configured model provider before launch.

Access and exposure

Workspace keys can access every App and Action in their workspace. Use one named key per production service and revoke unused credentials. Treat shareable public Actions as unauthenticated entry points and test their data, tools, cost, and abuse boundaries before distribution.

Use Workspace administration for members, roles, keys, billing, deployments, and public access.

Define production checks

Before deploying an Action version, record:

  • The version and target environment.
  • The eval dataset and acceptance result.
  • Representative manual tests, including expected failure cases.
  • Context source status and retrieval checks when the Action uses Context.
  • Expected latency, output size, and provider or plan limits.
  • The rollback version and the operator authorized to restore it.

After deployment, verify the actual environment with an authenticated API request. Verify a public URL separately when one is enabled. Watch fresh production records and feedback instead of treating deployment completion as proof of correct behavior.

Respond to incidents

Start from the failing boundary:

  • Authentication failures: verify the workspace key, its owner, and whether it was revoked through member removal or key deletion.
  • Action-not-found or wrong-version behavior: verify the Action GUID or slug, environment, deployment, and public-version conditions.
  • Empty or irrelevant retrieval: inspect source status, Documents, indexing time, and Retrieval Test.
  • Provider errors or unexpected latency: review request logs and test the same deployed version with a minimal input.