Operate Context sources

A Context library turns files, database results, or API responses into retrieval-ready Documents. Klu stores the source definition with the workspace, processes it asynchronously, splits the loaded content, generates embeddings, and writes the resulting Documents to the documents database.

This page covers the Context Library UI and its indexing lifecycle. For programmatic Context and Document operations, use the Context API reference.

Before you begin

You need an active workspace membership and a Context library. Open Context and select Add Context if you need to create one. Context libraries are available across Apps in the workspace; an Action uses a library only after you connect that Context to the Action.

File ingestion also requires object-storage configuration in the Klu deployment. Database sources require a configured database provider. All source processing requires a running background worker and an embedding service.

Add a source

Open Context, select a library, then open Sources.

Upload files

1. Select Add Files or drop files into the upload control.
2. Wait for the upload to finish.
3. Watch the source status until it reads Ready for Search.

The web application uploads each file to the configured S3-compatible storage and stores its object URL on the Context source. Downloads from the source table go through an authenticated workspace-member route that returns a signed object URL.

The upload control does not declare a file-size or extension limit in the checked-in UI. Actual acceptance depends on the object store and the document loader. Validate representative files in your environment before committing to a production format or maximum size.

Add a database source

1. Select Add Source.
2. Choose Database under Context Source.
3. Select an existing Provider, or create one from the provider selector.
4. Enter the source query or filter under Context Configuration.
5. Select Add Source.

The current UI defines configuration forms for PostgreSQL, MySQL, Elastic, Redis, Pinecone, Snowflake, and MongoDB providers. This list shows implemented configuration shapes; successful ingestion still depends on the corresponding loader, credentials, network access, and query validity in your deployment.

Use a read-only database credential and a bounded query. Source processing executes the saved query again during reindexing, so an unbounded query can increase indexing time, document volume, and embedding cost.

Add an API source

1. Select Add Source.
2. Choose API under Context Source.
3. Enter url, method, and headers in Context Configuration.
4. Select Add Source.

The checked-in form exposes those values as source configuration. Keep tokens in an appropriate secret-bearing deployment configuration where your operating model supports it; the repository does not establish that arbitrary loader configuration receives dedicated secret-manager storage.

Understand the indexing lifecycle

Adding a source queues a processContextSource background job. The worker then:

1. Loads content using the source's loader and saved configuration.
2. Splits content using the source settings, or the Context settings when the source has no override.
3. Generates an embedding for every resulting chunk.
4. Deletes the source's previously generated Documents.
5. Inserts the replacement Documents and embeddings into the documents database.
6. Updates the source's index timestamp and status.

Source statuses have these operational meanings:

The source processor records FAILED and catches its processing error. Queue completion by itself is therefore insufficient proof of successful indexing. Use the source status and a retrieval test as the completion checks.

Configure retrieval and document splitting

Open the library's Settings tab. The current form enforces these values:

Select Save after editing settings. Saved splitting changes do not alter existing Documents. Use Refresh Document Embedding to recreate the library with the new settings.

The checked-in service falls back to a chunk size of 512 when no source or Context chunk size is available. The UI description currently mentions 256 as a default. Treat the service behavior as authoritative for self-hosted operation and set an explicit value when reproducibility matters.

Verify retrieval

Wait until every required source reads Ready for Search, then:

1. Open Documents and confirm that Documents exist and contain expected text and metadata.
2. Use the search box for literal search, or enable Semantic search for vector similarity.
3. Open Retrieval Test and submit representative production questions.
4. Confirm the returned Documents are relevant before connecting or redeploying an Action.

An overall Context badge can read Ready after processing. Source-level statuses and the Documents list give you finer evidence when a library contains several sources.

Refresh, reindex, and delete

Use the refresh control beside an individual source to rebuild only that source. Select several rows and choose Bulk Actions → Reindex Selected to process a batch.

Use Settings → Refresh Document Embedding to rebuild all sources with the saved Context settings. This deletes generated Documents before replacing them and removes any custom edits made directly to those Documents.

Deleting a source deletes its generated Documents from the documents database and removes the source record. The checked-in source-delete service does not delete the uploaded object from S3. Apply your storage-retention policy separately when physical object deletion is required.

Deleting the Context marks the Context deleted and removes its Action links. The implementation does not establish immediate physical deletion of every related uploaded object or document row. Treat application deletion and storage erasure as separate operational checks.

Troubleshoot indexing

A source remains PROCESSING

Confirm the worker process is running and connected through GRAPHILE_DATABASE_URL. Check worker logs for process-context-source and the source ID. A stopped worker leaves queued work unavailable to the web and API processes.

A source reads FAILED

Inspect the first error from the source's worker run. Common failure domains are source credentials, network reachability, invalid queries or headers, unsupported content, embedding-service configuration, and documents-database writes. Correct the cause, then use the source refresh control.

A batch completes with missing Documents

Inspect each source status. The worker can complete a queue task after the source processor has recorded FAILED. Confirm Ready for Search, inspect Documents, and run Retrieval Test.

Reindexing removes manual edits

This is expected repository behavior. Reindexing deletes the source's generated Documents and creates replacements from the source. Put durable corrections in the original file, query, or API response.

Retrieval returns stale content

Confirm the source's Indexed At value changed, then inspect the generated Documents. Reindexing clears Context caches after persistence; stale source data can still originate from an unchanged upstream response or query result.