Agent Studio¶
Agent Studio is Latent's remote platform for storing versioned eval datasets, conversations, and scored reports. latent-py talks to its HTTP API so your eval pipelines can pull a frozen dataset, score it, and push the resulting StatisticalReport back — closing the eval loop without leaving Python.
There are three ways to integrate:
| Surface | Use it for |
|---|---|
Catalog (agent_studio.Dataset / agent_studio.Report) |
Declarative inputs/outputs inside a @flow |
CLI (latent studio ...) |
Ad-hoc browsing, downloads, uploads, and tagging from a shell |
Client (StudioClient) |
Programmatic access from scripts and notebooks |
All three resolve their base URL and API key from the same AGENT_STUDIO_URL / AGENT_STUDIO_API_KEY settings (see Authentication).
In flow catalogs¶
Reference Agent Studio directly from a flow's catalog.yaml so datasets load and reports submit automatically. See Data Catalog for general catalog semantics and Eval Flows for the flows that consume these.
agent_studio.Dataset (input)¶
Fetches a dataset by slug and resolves it to a pandas DataFrame. Versions are cached on disk, so re-runs don't re-download.
# flows/<flow_name>/catalog.yaml
eval_set:
type: agent_studio.Dataset
slug: support-qa
version: latest # integer or "latest" (default: latest)
# base_url / api_key are optional per-call overrides;
# default to AGENT_STUDIO_URL / AGENT_STUDIO_API_KEY
| Key | Required | Default | Notes |
|---|---|---|---|
slug |
yes | — | Dataset slug in Agent Studio |
version |
no | latest |
Integer or "latest" |
base_url |
no | AGENT_STUDIO_URL |
Override the API base URL |
api_key |
no | AGENT_STUDIO_API_KEY |
Override the bearer token |
agent_studio.Report (output)¶
Submits a StatisticalReport (or pre-serialized dict) to the reports API instead of writing to disk. The target dataset is inferred from the flow's agent_studio.Dataset inputs when not given explicitly.
results_report:
type: agent_studio.Report
dataset_slug: support-qa # default: inferred from agent_studio.Dataset inputs
dataset_version: latest # default: inferred from agent_studio.Dataset inputs
# other optional keys: report_type, flow_name, name, idempotency_key,
# base_url, api_key
idempotency_key defaults to a per-output, per-minute key so repeated runs dedupe server-side; set it to null in the catalog to disable dedup.
The closed eval loop¶
Pull a versioned dataset, score it with judge_flow, and push the report back — all declared in one catalog:
# flows/qa_eval/catalog.yaml
eval_set:
type: agent_studio.Dataset
slug: support-qa
version: latest
qa_report:
type: agent_studio.Report
dataset_slug: support-qa
import pandas as pd
from latent.prefect import flow, task
from latent.flows import judge_flow
from latent.agents import Judge
@task("score", input="eval_set", output="qa_report")
async def score_task(eval_set: pd.DataFrame):
# eval_set is pulled from Agent Studio; returned report is pushed back
result = judge_flow(eval_data=eval_set, judge=Judge("scorer", ...))
return result["report"]
@flow("qa_eval")
async def run():
await score_task()
Conversations are not a catalog type
Agent Studio conversations are consumed via the CLI or
client. Download them to JSONL and load locally with
the conversation.JSONL catalog type. The old agent_studio.JSONL type was
removed — migrate to agent_studio.Dataset (same slug/version config).
From the CLI¶
The latent studio command group browses, downloads, and writes both datasets and conversations. See CLI Tools for the wider CLI.
latent studio datasets¶
# List datasets (optionally filter by type)
latent studio datasets ls --type generic --limit 50 [--json]
# Download a version to a directory
latent studio datasets download support-qa --version latest -o ./data [--csv|--jsonl]
# Create an empty dataset (one-time setup before the first upload)
latent studio datasets create support-qa \
--display-name "Support QA" --type generic [--source upload] \
[--description ...] [--tag ...]
# Upload a JSONL file as a new published version (optionally tag it)
latent studio datasets upload records.jsonl --slug support-qa \
[--tag production --exclusive] [--published-by cli]
# Tag / untag a specific version
latent studio datasets tag support-qa --version 3 --tag production [--exclusive]
latent studio datasets untag support-qa --version 3 --tag production
# Promote a version (exclusive tag, default 'production')
latent studio datasets promote support-qa --version 3 [--tag production]
--type accepts conversation, qa_pair, generic, or file_bundle; --source accepts upload, curated, disk, or mixed. Use generic for eval-results rows.
latent studio conversations¶
# List conversations with filters
latent studio conversations ls \
[--mode qa|conversation|all] [--from 2026-04-01] [--to 2026-05-01] \
[--feedback good|bad|any|none] [--min-score 1] [--max-score 5] \
[--limit 50] [--json]
# Download conversations as JSONL with full message content
latent studio conversations download \
[--mode ...] [--from ...] [--to ...] [--feedback ...] \
[--min-score ...] [--max-score ...] [--limit 5000] -o ./data
Dates are ISO 8601. download writes conversations-<timestamp>.jsonl, which you can then reference via a local conversation.JSONL catalog entry.
Programmatically¶
StudioClient is a thin HTTP client over the Agent Studio API. With no arguments it reads AGENT_STUDIO_URL / AGENT_STUDIO_API_KEY; pass base_url / api_key to override.
from latent.studio_client import StudioClient
client = StudioClient() # or StudioClient(base_url="https://...", api_key="sk-...")
# List datasets -> list[DatasetSummary]
for ds in client.list_datasets(type="generic"):
print(ds.name, ds.latest_version, ds.record_count)
# Resolve and download a version (returns an httpx.Response)
version = client.resolve_version("support-qa")
resp = client.download_dataset("support-qa", version, fmt="jsonl")
# Or fetch whatever version currently holds a tag
resp = client.download_dataset_by_tag("support-qa", "production", fmt="jsonl")
Writing data:
# Create once, then upload records as a new published version
dataset_id = client.create_dataset(
"support-qa", "Support QA", "generic", description="QA eval rows"
)
new_version = client.upload_records("support-qa", [{"q": "...", "a": "..."}])
# Tag / promote versions
client.tag_version("support-qa", new_version, "production", exclusive=True)
client.untag_version("support-qa", new_version, "production")
# resolve_dataset_id maps a slug to its Convex _id (used by upload/publish)
internal_id = client.resolve_dataset_id("support-qa")
Conversations:
# list_conversations -> ConversationListResponse (.conversations, .total, .next_cursor)
page = client.list_conversations(mode="qa", feedback="good", limit=50)
# download_conversations -> httpx.Response (JSONL with full messages)
resp = client.download_conversations(min_score=4, limit=5000)
Result types¶
| Type | Returned by | Key fields |
|---|---|---|
DatasetSummary |
list_datasets |
name, display_name, type, latest_version, record_count, updated_at, description |
ConversationListResponse |
list_conversations |
conversations, total, next_cursor |
ConversationSummary |
conversation metadata (used in eval report summaries) | id, thread_id, mode, started_at, model_used, and message_count / score / status properties |
download_dataset, download_dataset_by_tag, and download_conversations return raw httpx.Response objects — read .content (or .text) yourself.
Authentication¶
Every surface resolves the same two settings, in priority order:
| Setting | Env var | latent.toml |
Default |
|---|---|---|---|
| Base URL | AGENT_STUDIO_URL |
[agent_studio].url |
http://localhost:8000 |
| API key | AGENT_STUDIO_API_KEY |
[agent_studio].api_key |
(none) |
When set, the API key is sent as Authorization: Bearer <key>. A 401/403 raises a PermissionError prompting you to set the key. Every entry point also accepts per-call overrides: base_url / api_key on StudioClient(...), or base_url / api_key keys on agent_studio.Dataset and agent_studio.Report catalog entries.