# Testing
> The kin test surface is split into two homes:
The kin test surface is split into two homes:
- **`tests/`** — the pytest suite. Headless, no model needed, no real
network (most files). Runs under `task verify`. This is where you add
tests for harness internals, headless flows, and the like.
- **`scripts/`** — live / smoke / probe scripts. Real model, remote service,
network, or subprocess. Runs under `task live-drive*`,
`task live-git-tools`, `task smoke*`, and `task probe-mcp*`. Not pytest;
they stay as CLI scripts because their assertion surface doesn't fit a
unit-test frame.
The split matches the project character: most of what we test is harness
internals (pytest is the right tool); a small but important slice is
real-wire behavior (CLI scripts are the right tool).
The structured Git/GitHub boundary has one deliberately separate publication
gate:
```bash
KIN_LIVE_GIT_PUBLISH=1 task live-git-tools
```
It refuses to start without that exact environment opt-in or unless local
`HEAD` matches the remote base. It then creates a disposable worktree/branch,
drives the structured local Git operations, and fetches/pushes through Kin's
real gh-backed HTTPS credential route even when the saved remote uses SSH. An
independent temporary repository advances the disposable branch while the
first worktree has one unpublished commit; `git.sync` must replay it. The gate
then publishes that result, makes a test-only local amend, proves an ordinary
push rejects the non-fast-forward, and proves the structured exact
force-with-lease succeeds. Finally it opens/retries/reads a draft PR and closes
the PR plus exact remote/local branch. It is **not** part of `task ship`:
remote publication and cleanup require an explicit operator decision, while
the promotion gate remains hermetic.
## Pytest layout
```
tests/
├── __init__.py # env-before-import gate (sets KIN_HOME etc.)
├── conftest.py # shared fixtures + helpers
├── test_settings.py # headless settings
├── test_banner.py # headless banner
├── test_integration.py # end-to-end with FakeBackend + KinApp
├── test_workflow.py # the 17 closed-namespace / AST-filter tests
├── test_search.py # FTS5 workspace search
├── test_memory.py # the memory_20250818 command set
├── test_provider_presets.py # the first-party preset system
├── test_evals.py # the regression sentinel (--update-baseline)
├── test_web.py # web_fetch SSRF + Brave formatting
├── test_office_security.py # docx/xlsx read_file: zip-bomb cap + entity-expansion guard
├── test_cli.py # `kin --version` + `kin doctor` (install/config/network chunks)
├── test_ship_gate.py # parallel gate + agent-output contract
├── test_docs_audit.py # static guide-screenshot ownership graph
├── test_harness/ # the split harness subpackage
├── test_app/ # the split UI subpackage
├── test_outpost/ # the outpost dashboard (requires --extra outpost)
├── test_pty.py # the out-of-process PTY smoke
├── test_snapshot.py # the SVG baseline guard
├── test_worktrees.py # worktree isolation
├── test_perf.py # synthetic journal perf benchmark
└── test_live_drive.py # thin wrapper over scripts/_live_drive_scenarios
```
Most top-level `test_.py` files correspond 1:1 to a historical
`verify_.py` script they migrated from (Phases 1-5, 2026-07-02) —
see the per-task table below. `test_office_security.py` and `test_cli.py`
are newer, added directly as pytest with no `verify_*.py` ancestor. The
split subpackages (`test_harness/`, `test_app/`, `test_outpost/`)
replaced the monolithic `verify_harness.py` / `verify_app.py` /
`verify_outpost.py` files (~12K + ~4K + ~9K lines each) with focused
submodules.
Adding a tool, a permission kind, or a subagent profile all need matching
coverage under `tests/test_harness/` — see
[Extending kin](/docs/kin/internals/extending-tools/) for where each of those lands.
## The local promotion gate
`task ship` is the contract before a `dev → main` merge — see
[`AGENTS.md`](https://github.com/kinra-ai/kin/blob/dev/AGENTS.md)'s
Commands table for the day-to-day command surface and
[Multi-machine dev setup](/docs/kin/internals/multi-machine-setup/) for running it on a
second machine (`task kloud:verify`). No CI server, no GitHub Actions —
Blake + AI peers run it as the explicit verification step. The shape:
```bash
task ship # exhaustive promotion proof; compact on green, failure-first on red
```
The gate has two waves:
1. `check`, `verify`, `docs-build`, `docs-audit`, `memory-check`, and
`code-audit` run concurrently. Verification itself has two concurrent
lanes: every non-Outpost file is grouped across at most eight xdist workers,
while `tests/test_outpost/` remains serial because its process-lifecycle and
journal fixtures are not yet safe under blanket distribution. `docs-audit`
hard-fails decision conventions, bounded `STATUS.md` sections, and the
deterministic ownership graph
joining both guide capture manifests, committed SVG/PNG assets, and
non-empty Markdown references; broader `SOURCE:` and deleted-document
findings remain warnings.
2. `live-drive-dry` runs only after the independent first wave is green.
On the 2026-07-17 reference machine, the pre-change serial pytest selection
took 102.9s and the complete gate took 111.3s. The safe split runs the same
selection in about 22–23s; complete-gate wall time is the longest first-wave
stage plus the roughly 5s dry drive, not the sum of every row. Reference runs
of the new topology are green in about 28s.
If any sub-command fails, the merge is blocked. All independent first-wave
stages finish so one peer turn reports every failure family; ordered stages are
shown as not run. The contract remains **"I ran it, here's the output"** — not
"looks right."
`task ship` is one Taskfile target rather than a separate workflow, so the
promotion path and the ad-hoc local command cannot drift apart.
### Agent-facing output
Passing stage output is captured rather than streamed. A green run reports the
commit, aggregate pytest counts, wall time, per-stage timings, and the slowest
test:
```text
SHIP PASS 27.8s | sha abc1234+dirty | 2509 passed, 5 skipped
stages: check 0.4s · verify 22.9s · docs-build 1.7s · ... · live-drive-dry 4.9s
verify lanes: verify-main 15.2s (1736 passed, 1 skipped) · verify-outpost 22.9s (773 passed, 4 skipped)
slowest: 7.12s tests/test_outpost/test_transcript.py::test_name
```
A red run reports every failed stage together, retains useful head-and-tail
evidence within Kin's shell-output cap, prints exact pytest node reruns when
available, and preserves the complete logs under `/tmp/kin-ship-*`. Successful
runs remove their temporary logs. The SHA carries `+dirty` when the proof ran
against uncommitted or untracked changes, so an agent cannot mistake the base
commit for the complete verified source state. The per-lane counts also make a
base install's clean Outpost package skip visible instead of hiding it inside
the aggregate green result.
Pytest's global configuration deliberately keeps test-case progress quiet while
retaining high assertion verbosity and short tracebacks. The hermetic lanes
also report slow tests and apply a 60-second per-test backstop; slow/live tests
keep their specialized bounds outside this gate.
### AI-peer repair loop
Use the narrowest relevant node while changing code:
```bash
task verify-one -- 'tests/test_harness/test_tools.py::test_name' -vv
uv run pytest --lf -x
task check
```
Then run `task ship` exactly once as the exhaustive completion proof. Last-
failed, stepwise, and affected-test techniques are repair-loop accelerators,
not substitutes for the promotion gate. Set `KIN_TEST_WORKERS` to reduce the
default eight-worker cap on a constrained machine:
```bash
KIN_TEST_WORKERS=4 task verify
```
## Markers
Two markers route tests out of the default `task verify` pass:
| Marker | Meaning | Run with |
|---|---|---|
| `slow` | machine / perf / Textual-version sensitive | `task test-slow` |
| `live` | needs real network / real endpoint | `task test-live` |
The default `task verify` runs everything that ISN'T marked — i.e. the
hermetic headless suite. The slow and live suites run on demand via their
dedicated `task test-*` entry points (or by directly passing `-m slow` /
`-m live` to pytest).
The fifteen rendered TUI comparisons in `tests/test_snapshot.py` are `slow`
because Rich/Textual SVG bytes depend on the locked rendering environment and
machine layout. `task verify-snapshot` compares all fifteen baselines;
`task docs-screenshots-tui` regenerates the ten `test_docs_` scenes published
directly under `docs/assets/screenshots/`. `task docs-screenshots-web`
regenerates the five local-only Outpost PNGs at their fixed desktop/phone
viewports, and `task docs-screenshots` runs both capture paths. The static
dual-manifest ownership checks in `tests/test_docs_audit.py` are ordinary
hermetic tests and remain in `task verify` and `task ship`; rendering and
Playwright do not.
Outpost tests are hermetic, not a separate marker class. They are collected by
`task verify` when the `outpost` extra is installed and skipped as a package by
`tests/test_outpost/conftest.py` (via its `aiohttp` check) on a base install. To
require that full package rather than permit the skip, run:
```bash
uv run --extra outpost pytest tests/test_outpost/
```
## Conftest surface — fixtures + helpers
`tests/conftest.py` is the shared surface pytest applies across the collection.
The headline wins:
### Registered CLI options
- `--update-baseline` → `update_baseline` for the deterministic eval
fingerprint.
- `--update` → `update_snapshots` for SVG regeneration. Registering this in
conftest is required because pytest finalizes options before it imports
`tests/test_snapshot.py`; `task verify-snapshot -- --update` would otherwise
fail during argument parsing.
### Fixtures (function-scoped unless noted)
- `temp_kin_home(tmp_path, monkeypatch)` — per-test KIN_HOME override with
`settings.reset_cache()` so cached lookups from prior tests don't bleed
through. Use this in any test that mutates KIN_HOME.
- `temp_session_dir(tmp_path, monkeypatch)` — per-test KIN_SESSION_DIR
override. Rare; mainly for tests that exercise persistence paths.
- `fake_backend` — a fresh `FakeBackend` with an empty responder. Pair
with the `make_responder(...)` helper to script turns.
- `session` — a real `Session` wired with a captured-events list + the
FakeBackend. Tests assert on the event stream via `session._captured_events`.
- `pilot_app` — fresh `KinApp` per scenario, wired with the FakeBackend
via `app.run_test()`. Yields `(app, pilot, sess, captured_events)`.
**CRITICAL gotcha:** if you do `monkeypatch.delenv("KIN_HOME",
raising=False)` to clear env keys, it UNDOES `temp_kin_home`'s
`monkeypatch.setenv("KIN_HOME", tmp_path)` and the test falls back to
`~/.kin` (the user's real home). Skip KIN_HOME in any env-clear loop.
This bug bit 3 of the original 8 migrations in Phase 2.
- `env_snapshot(monkeypatch)` — capture-and-restore helper for tests
that mutate env keys.
- `_reset_env_between_tests` (**autouse**) — snapshots `os.environ` at
the start of every test, restores it on teardown, AND calls
`kin.harness.settings.reset_cache()` so cached `kin_home()` lookups
from earlier tests don't bleed through. **No opt-out by design** —
the cost of a surprise env leak is much higher than the cost of an
opt-out. Tests that legitimately need to mutate env use
`monkeypatch` (auto-restoring) or `env_snapshot` (manual restore).
Cost: ~1ms per test.
### Helpers (module-level functions, used inside test bodies)
- `import_tool_module(name)` — `importlib.import_module(...)` alias.
Used to bypass `kin.harness.tools.__init__`'s tool-instance shadow when
monkeypatching tool internals. Modules using it: `web_fetch`,
`web_context`, `_media`, `workflow`, etc.
- `async settle(pilot, predicate, *, tries=80, sleep=0.05)` — poll a
predicate until truthy. Consolidates the historical integration and
live-drive settle helpers.
- `async wait_for_event(events, predicate, *, timeout=2.0)` — poll a
captured-events list for an event matching the predicate.
- `clean_terminal(text)` / `clean_interrupted(text)` — strip ANSI /
control sequences from a Textual screen dump.
- `_git_short_sha(cwd=None)` — current git short SHA, or "unknown".
Consolidates the historical eval and live-drive helpers.
- `assert_eventually(condition, *, timeout, interval, message)` —
sync version of `settle`.
- `make_responder(*scripted_turns)` — build a FakeBackend responder from
a list of `(text | tool_call | (text, [tool_calls...]))` tuples.
## Env-before-import architecture
`tests/__init__.py` (NOT conftest.py) sets the KIN_HOME / KIN_SESSION_DIR
/ KIN_SPANS env vars at MODULE TOP. This is critical because:
1. Several verify scripts (now test modules) do top-level
`from kin.harness import presets, settings`.
2. `settings.kin_home()` is read at first import, not lazily.
3. Pytest imports `tests/__init__.py` before any test module — guaranteeing
the env is correct for every test module's top-level kin imports.
`setdefault` (not `=`) is used so a developer's explicit env wins for
local debugging. The conftest session fixture does the teardown
(`rmtree` of the tmpdir).
**Do NOT** add `from kin.harness...` at the top of `tests/__init__.py`
itself; that would defeat the ordering guarantee.
## The `importlib` shadow-bypass idiom
`kin.harness.tools.__init__.py` re-exports tool INSTANCES, which shadow
the submodule names. `from kin.harness.tools import web_fetch` gives you
the tool INSTANCE, not the module. Tests that monkeypatch (e.g.
`wf._validate_ip`, `wc.brave_get`) need the REAL module object:
```python
# DON'T:
from kin.harness.tools import web_fetch # returns the Tool INSTANCE
wf = web_fetch
wf._validate_ip = lambda ip: None # mutates the tool instance, not the module
# DO:
wf = import_tool_module("kin.harness.tools.web_fetch")
wf._validate_ip = lambda ip: None # mutates the real module
```
The `import_tool_module(name)` helper consolidates this idiom in one
place. It also covers `kin.harness.tools._media` (the media submodule
that `tests/test_web.py` needs) and `kin.harness.tools.workflow` (the
workflow executor module that `tests/test_workflow.py` needs).
## Coverage interpretation
Lazy SDKs (`openai`, `anthropic`, `mcp`, `playwright`, `tomlkit`, etc.)
are imported inside the code that needs them. Headless tests using
`FakeBackend` never trigger `OpenAIBackend` / `AnthropicBackend` imports,
so the SDK modules show up as "no statements" in the coverage report —
**NOT "missing" or "untested"**.
Don't interpret low coverage on a lazy-imported module as a regression.
Use `pytest --cov-context=test` for per-test context if a specific module
needs diagnosing.
Run `task test-cov` to see the coverage report:
```bash
task test-cov # = uv run pytest tests/ --cov=src --cov-report=term-missing \
# -m "not slow and not live"
```
There's NO `--cov-fail-under` threshold — coverage is visibility, not
enforcement. The lazy-SDK interaction makes a meaningful threshold
premature (most of `kin.harness.backends.*` would show 0% under headless
tests using FakeBackend).
## Adding a new test
1. **Pick the right home.** Headless? → `tests/`. Live / smoke / network?
→ `scripts/`.
2. **Use the shared fixtures** (`temp_kin_home`, `fake_backend`, `session`,
`pilot_app`) instead of hand-rolled env mutation.
3. **Use `assert`** instead of `check()` — pytest captures failures
natively and the test function name becomes the test ID.
4. **Preserve the assertion semantics.** Same checks, same expected
values — just pytest-shaped.
5. **Add an `if __name__ == "__main__":` guard** at the bottom if you
want ad-hoc `python tests/test_.py` debugging.
6. **If your test is slow or live,** mark it so the default `task verify`
skips it. Outpost tests belong under `tests/test_outpost/`; that package's
conftest owns the optional-dependency skip.
7. **Don't edit `tests/conftest.py`** if you're working in a parallel
branch / migration; add the fixture locally in your test file. The
shared conftest is the only one that needs the KIN_HOME / settings
reset dance, so adding fixtures there is rarely necessary.
## Isolation guarantees
The autouse `_reset_env_between_tests` fixture (see Conftest surface
above) closes the 4 known isolation failures from the migration — they
now pass in the full suite, not just in isolation. The fixture is the
contract: every test starts with a clean `os.environ` and a fresh
`kin.harness.settings._cache`. **No opt-out by design** — the cost of
a surprise env leak is much higher than the cost of an opt-out.
Tests that need to mutate env use `monkeypatch` (auto-restoring) or
`env_snapshot` (manual restore). **Module-top `os.environ[...] = ...`
mutations are the original sin:** pytest collects modules before
running anything, so module-top mutations persist for the entire
session. The pytest migration removed the ones it found in
`test_evals.py`; the autouse fixture is the backstop for any that slip
in later.
## Per-task pytest invocations
| Old `verify_*.py` | New pytest task |
|---|---|
| `verify_settings.py` | `task verify-settings` (`uv run pytest tests/test_settings.py`) |
| `verify_banner.py` | `task verify-banner` (`uv run pytest tests/test_banner.py`) |
| `verify_integration.py` | `task verify-integration` (`uv run pytest tests/test_integration.py`) |
| `verify_workflow.py` | `task verify-workflow` (`uv run pytest tests/test_workflow.py`) |
| `verify_search.py` | `task verify-search` (`uv run pytest tests/test_search.py`) |
| `verify_memory.py` | `task verify-memory` (`uv run pytest tests/test_memory.py`) |
| `verify_provider_presets.py` | `task verify-provider-presets` (`uv run pytest tests/test_provider_presets.py`) |
| `verify_evals.py` | `task verify-evals` (`uv run pytest tests/test_evals.py`; add `-- --update-baseline` only when intentionally re-pinning) |
| `verify_harness.py` | `task verify-harness` (`uv run pytest tests/test_harness/`) |
| `verify_app.py` | `task verify-app` (`uv run pytest tests/test_app/`) |
| `verify_outpost.py` | `uv run --extra outpost pytest tests/test_outpost/` (`task verify` also collects it when the extra is already installed) |
| `verify_outpost_live.py` | retired — `tests/test_outpost_live.py` + `task verify-outpost-live` tested the ttyd/tmux browser-terminal surface, deleted wholesale in the middle-way plan's Step 3 ttyd cut (2026-07-06, DR 0033) |
| `verify_pty.py` | `task verify-pty` (`uv run --group pty pytest tests/test_pty.py -m slow`) |
| `verify_snapshot.py` | `task verify-snapshot` (`uv run pytest tests/test_snapshot.py -m slow`); `task docs-screenshots-tui` regenerates the ten guide-owned `test_docs_` targets |
| `verify_worktrees.py` | `task verify-worktrees` (`uv run pytest tests/test_worktrees.py -m live`) |
| `verify_perf.py` | `task verify-perf` (`uv run pytest tests/test_perf.py -m slow`) |
| `verify_web.py` | `task verify-web` (`uv run pytest tests/test_web.py`) |
| `live_drive_vllm.py` | stays a CLI script; thin pytest wrapper at `tests/test_live_drive.py` |
| `live_git_tools.py` | stays a CLI script; `KIN_LIVE_GIT_PUBLISH=1 task live-git-tools` is an explicitly armed remote publication proof, never part of `task ship` |
`tests/test_docs_audit.py` has no historical `verify_*.py` ancestor. It pins
the bounded `STATUS.md` contract and combined TUI/browser screenshot graph used
by `task docs-audit`; unlike rendering either asset type, those read-only checks
are deterministic and belong in the file-grouped hermetic lane. `task
status-trim` removes whole recently-landed records by age and count; it never
leaves continuation lines behind.
`verify_harness.py` was kept as a backward-compat shell through the
migration (Phases 1-5) so existing cron / muscle-memory invocations kept
working while the pytest files landed; it's now removed (the pytest
collection is the canonical entry point).