# Evals & observability

> Use deterministic regressions, live scenarios, and optional metadata spans to inspect Kin behavior.

kin verifies itself with two eval tracks: a **deterministic regression
sentinel** that runs at commit time (`task verify`, no model, no network —
one leg of [`task ship`](/docs/kin/internals/testing/#promotion-proof),
the local promotion gate) and a **live golden-set gate** that drives the real
app against a real endpoint before a promote, run by hand rather than as part
of `task ship`. Alongside them sits an opt-in **JSONL span sink** that
records OTel-GenAI-shaped spans for every turn, model call, and tool
invocation. See [Testing](/docs/kin/internals/testing/) for how both fit into the
wider pytest / live-script layout.

<!-- SOURCE: tests/test_evals.py, scripts/live_drive_vllm.py, src/kin/harness/spans.py, src/kin/harness/session/, src/kin/harness/settings/, snapshots/evals_baseline.json, Taskfile.yml -->

## The regression sentinel (`task verify-evals`)

`tests/test_evals.py` runs a pinned set of scenarios on the scripted
`FakeBackend` — the same scenario discipline the live driver uses, minus the
wire — and compares each scenario's **deterministic fingerprint** against the
checked-in baseline at `snapshots/evals_baseline.json`:

- the collapsed event-type sequence (streaming runs collapse to one token, so
  chunk-size changes don't fire it)
- the ordered tool-call names, tool errors, and approval requests
- the terminal shape (`done:stop` / `done:error` / `done:loop_detected` — the
  [doom-loop guard](/docs/kin/guide/headless/#fleet-safety--the-per-run-token-budget)'s
  no-progress stop), error counts, and the retry (`redrive`) flag
- parallel-batch stamping (`batch_id` on concurrent tool calls)
- the span names the [span sink](#the-span-sink) derives for the scenario

Any drift is a named **REGRESSION** with a field-level diff. Only
deterministic code graders gate here — stochastic signals (pass^k, model
judgment) are deliberately kept out of the commit path and live in the
[live gate](#the-live-gate-task-live-drive-all) instead, where they are
reported, never blocking.

```bash
task verify-evals                       # part of `task verify`
uv run pytest tests/test_evals.py --update-baseline   # re-pin after a REVIEWED contract change
git diff snapshots/evals_baseline.json            # the diff IS the review
```

Changing a scenario script, an event shape, or a loop contract is a baseline
change: re-pin in the same commit and review the diff, exactly like the
[SVG snapshot discipline](https://github.com/kinra-ai/kin/blob/dev/snapshots/README.md).

## The live gate (`task live-drive-all`)

[`scripts/live_drive_vllm.py`](https://github.com/kinra-ai/kin/blob/dev/scripts/live_drive_vllm.py)
drives every scenario through the real `KinApp` + a real endpoint (the
pre-promote check). Two eval features layer on top of the per-scenario
assertions:

**Pinned baseline.** After an `--all` run the driver reports
`PASS`/`REGRESSION` against `snapshots/live_baseline.json` — a checked-in
record of which scenarios passed on the last known-good run (plus its model,
git sha, and informational wall times). A scenario that was pinned green and
now fails is called out as `REGRESSED`; one that was already failing at pin
time reports as `known-fail` so a pre-existing breakage isn't mistaken for a
new one. Re-pin from a known-good run:

```bash
task live-drive-all -- --update-baseline
```

**pass^k.** Agents are stochastic — a single-shot pass/fail lies (τ-bench's
lesson). `--passk K` re-runs each scenario K times in fresh workdirs and
reports pass^k (all K attempts pass) per scenario, both in the summary table
and in the JSON aggregate:

```bash
task live-drive-all -- --passk 3
```

pass^k runs are exploratory by design: the baseline compare is skipped and
the result is a **reported** pre-promote signal, never a commit gate (the
canonical gate stays K=1).

## The span sink

Set `KIN_SPANS=1` (or `spans_enabled = true` in `settings.toml`) and every
session appends OpenTelemetry-GenAI-shaped spans — one JSON object per line —
next to its journal:

```
<session dir>/<project>/<session_id>.spans.jsonl
```

Three span kinds, derived at the emit seam by
[`harness/spans.py`](https://github.com/kinra-ai/kin/blob/dev/src/kin/harness/spans.py):

| Span | One per | Named attributes |
|---|---|---|
| `invoke_agent kin` | user turn (the root of its own trace) | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.agent.name`, `gen_ai.conversation.id` (the session id — the cross-trace link), `gen_ai.request.model`, `gen_ai.response.finish_reasons`, `error.type` |
| `chat {model}` | model round inside the turn | `gen_ai.request.model`, `gen_ai.response.model` (the discovered id), `gen_ai.usage.input_tokens`, `kin.usage.cached_tokens` |
| `execute_tool {tool}` | tool / MCP invocation (subagent tool calls nest under the parent `task` span) | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`, `error.type` |

Attribute names follow the OTel GenAI semantic conventions (the
`semantic-conventions-genai` repository, Development status). There is **no
OTel SDK and no OTLP exporter** — the records are hand-emitted JSONL you can
`jq` directly, feed to a collector later, or ignore.

Two properties are contractual:

- **Metadata only.** No prompt text, no completion text, no tool arguments or
  results — timing, names, and token counts. The semconv's content-capture
  opt-in stays off, so the file is as safe to keep as the journal beside it.
- **Never in the way.** The tap wraps the emit callable and swallows every
  internal error; an unwritable sink disables itself for the session. Spans
  can go missing — a turn can't break because of them.

Off by default. The toggle is project-safe (a local metadata file, no
network / egress / secret) and human-only — the model can't flip it via
`propose_settings`. See [Environment variables](/docs/kin/reference/environment-variables/)
and [settings.toml keys](/docs/kin/reference/settings-toml/).

_Source authority: Kinra Site (src/content/docs/kin/guide/evals.md)._