Eval runs

Start durable Sapient eval runs and poll their progress from the CLI.

Eval runs start Sapient evaluation work from the public API. Use them when you want to trigger prompt evals or API Performance evals from automation, then poll progress by eval run ID.

Commands

Command	Description
sapient eval-runs create	Start a durable eval run.
sapient eval-runs retrieve --eval-run-id <eval_run_id>	Retrieve status and progress for a durable eval run.

Start a prompt eval run

Run selected Visibility prompts against configured prompt targets:

sapient eval-runs create \
  --request '{"type":"prompts","prompt_ids":["prompt_123"],"force":true}'

If you omit prompt_ids, Sapient runs all active prompts. If you omit targets, Sapient uses your configured scheduled prompt targets.

Start an API Performance eval run

Run configured API Performance evals for a source:

sapient eval-runs create \
  --request '{"type":"api_performance","source_type":"cli","integration_id":"int_cli"}'

You can override configured targets for a one-off run:

sapient eval-runs create \
  --request '{"type":"api_performance","source_type":"cli","targets":[{"target_type":"agent","execution_platform":"codex","model_id":"gpt-5","enabled":true}]}'

Useful API Performance fields:

Field	Description
source_type	Source to run. Use rest_api, sdk, cli, or custom_evals.
integration_id	Source integration ID for REST API, SDK, or CLI evals.
targets	One-off text or coding-agent targets. Omit to use configured targets.
models	Legacy flat model selection. Prefer targets.
endpoint_ids	Optional endpoint filters.
endpoint_paths	Optional METHOD:path filters, such as GET:/v1/items.
env_profile_ids	Environment profiles to use for this run.

Start a custom eval run

Run one or more custom evals:

sapient eval-runs create \
  --request '{"type":"api_performance","source_type":"custom_evals","custom_eval_ids":["custom_eval_123"]}'

If you omit custom_eval_ids, Sapient uses the custom evals configured for your organization.

Poll progress

The create response returns an id and job_id. Pass the id to retrieve progress:

sapient eval-runs retrieve --eval-run-id eval_run_123

Example response:

{
  "id": "eval_run_123",
  "job_id": "job_123",
  "type": "api_performance",
  "source_type": "custom_evals",
  "status": "running",
  "stage": "dispatching",
  "progress": {
    "total": 4,
    "queued": 1,
    "running": 2,
    "completed": 1,
    "failed": 0,
    "cancelled": 0,
    "interrupted": 0,
    "percent": 25
  }
}

Statuses include pending, running, summarizing, completed, failed, cancelled, and interrupted.