> ## Documentation Index
> Fetch the complete documentation index at: https://lightsage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval runs

> Start saved or direct eval runs and inspect their lifecycle state from the Lightsage CLI.

`lightsage evals run` either runs a saved [eval](https://lightsage.com/docs/cli-reference/evals.md) or creates
and starts one directly. A run fans out across every prompt and agent, then
repeats each pair according to `--runs`.

## Commands

| Command | Description |
| - | - |
| `lightsage evals run` | Start a saved or direct eval run. |
| `lightsage evals monitor` | Stream live progress for active runs. |
| `lightsage eval-runs list` | List eval runs. |
| `lightsage eval-runs retrieve --run-id <id>` | Retrieve or poll one run. |
| `lightsage evals analysis --run-id <id>` | Retrieve persisted judge analysis. |

## Create and run directly

```bash theme={null}
lightsage evals run \
  --prompt "Use the CLI to create an eval." \
  --prompt "Use the MCP server to retrieve an eval." \
  --judge "The requested workflow succeeds." \
  --agent codex:gpt-5-4 \
  --agent claude-code:claude-opus-4-8 \
  --cli "brew install lightsagehq/tools/lightsage" \
  --mcp @lightsage-mcp.json \
  --setup-command "npm install" \
  --cleanup-command "npm run db:cleanup" \
  --env LIGHTSAGE_API_KEY="$LIGHTSAGE_API_KEY" \
  --runs 2
```

Repeat `--prompt`, `--judge`, `--agent`, `--cli`, `--mcp`, `--setup-command`,
`--cleanup-command`, `--env`, `--skill`, and `--tag` once per value. The
response includes the created `eval_id` and the parent run `id`.

Setup commands run sequentially before agent execution and fail the attempt
fast if one exits unsuccessfully. Cleanup commands run best-effort after result
capture and before sandbox deletion, including when agent execution fails.

## Run a saved eval

Run the saved defaults:

```bash theme={null}
lightsage evals run --eval-id eval_123
```

Add a configuration when the run needs a saved repository, environment
profile, skill set, or legacy tool selection:

```bash theme={null}
lightsage evals run \
  --eval-id eval_123 \
  --configuration-id config_123
```

You can replace saved prompts, persona, agents, CLIs, MCP servers, setup
commands, cleanup commands, or repeat count for one launch. An explicitly empty
`clis`, `mcps`, `setup_commands`, or `cleanup_commands` array in a JSON body
disables the saved collection for that launch.

## Monitor and inspect a run

```bash theme={null}
lightsage evals monitor --run-id run_123
lightsage eval-runs retrieve --run-id run_123 --output-format json
lightsage evals analysis --run-id run_123 --output-format json
```

The parent run reports its agents, persona, prompt count, runs per matrix cell,
attempt count, status, and progress. Attempt-level analysis and trace records
include the exact prompt, prompt index, agent, and repeat index.

Inline environment values are write-only. Lightsage keeps them in private job
configuration and does not return them in run, trace, result, or analysis
responses.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.