Testing

The kin test surface is split into two homes:

• tests/ — the pytest suite. Headless, no model needed, no real network (most files). Runs under task verify. This is where you add tests for harness internals, headless flows, and the like.
• scripts/ — live / smoke / probe scripts. Real model, remote service, network, or subprocess. Runs under task live-drive*, task live-git-tools, task smoke*, and task probe-mcp*. Not pytest; they stay as CLI scripts because their assertion surface doesn’t fit a unit-test frame.

The split matches the project character: most of what we test is harness internals (pytest is the right tool); a small but important slice is real-wire behavior (CLI scripts are the right tool).

The structured Git/GitHub boundary has one deliberately separate publication gate:

KIN_LIVE_GIT_PUBLISH=1 task live-git-tools

It refuses to start without that exact environment opt-in or unless local HEAD matches the remote base. It then creates a disposable worktree/branch, drives the structured local Git operations, and fetches/pushes through Kin’s real gh-backed HTTPS credential route even when the saved remote uses SSH. An independent temporary repository advances the disposable branch while the first worktree has one unpublished commit; git.sync must replay it. The gate then publishes that result, makes a test-only local amend, proves an ordinary push rejects the non-fast-forward, and proves the structured exact force-with-lease succeeds. Finally it opens/retries/reads a draft PR and closes the PR plus exact remote/local branch. It is not part of task ship: remote publication and cleanup require an explicit operator decision, while the promotion gate remains hermetic.

Pytest layout

tests/
├── __init__.py                 # env-before-import gate (sets KIN_HOME etc.)
├── conftest.py                 # shared fixtures + helpers
├── test_settings.py            # headless settings
├── test_banner.py              # headless banner
├── test_integration.py         # end-to-end with FakeBackend + KinApp
├── test_workflow.py            # the 17 closed-namespace / AST-filter tests
├── test_search.py              # FTS5 workspace search
├── test_memory.py              # the memory_20250818 command set
├── test_provider_presets.py    # the first-party preset system
├── test_evals.py               # the regression sentinel (--update-baseline)
├── test_web.py                 # web_fetch SSRF + Brave formatting
├── test_office_security.py     # docx/xlsx read_file: zip-bomb cap + entity-expansion guard
├── test_cli.py                 # `kin --version` + `kin doctor` (install/config/network chunks)
├── test_ship_gate.py           # parallel gate + agent-output contract
├── test_docs_audit.py          # static guide-screenshot ownership graph
├── test_harness/               # the split harness subpackage
├── test_app/                   # the split UI subpackage
├── test_outpost/               # the outpost dashboard (requires --extra outpost)
├── test_pty.py                 # the out-of-process PTY smoke
├── test_snapshot.py            # the SVG baseline guard
├── test_worktrees.py           # worktree isolation
├── test_perf.py                # synthetic journal perf benchmark
└── test_live_drive.py          # thin wrapper over scripts/_live_drive_scenarios

Most top-level test_<area>.py files correspond 1:1 to a historical verify_<area>.py script they migrated from (Phases 1-5, 2026-07-02) — see the per-task table below. test_office_security.py and test_cli.py are newer, added directly as pytest with no verify_*.py ancestor. The split subpackages (test_harness/, test_app/, test_outpost/) replaced the monolithic verify_harness.py / verify_app.py / verify_outpost.py files (~12K + ~4K + ~9K lines each) with focused submodules.

Adding a tool, a permission kind, or a subagent profile all need matching coverage under tests/test_harness/ — see Extending kin for where each of those lands.

The local promotion gate

task ship is the contract before a dev → main merge — see AGENTS.md’s Commands table for the day-to-day command surface and Multi-machine dev setup for running it on a second machine (task kloud:verify). No CI server, no GitHub Actions — Blake + AI peers run it as the explicit verification step. The shape:

task ship       # exhaustive promotion proof; compact on green, failure-first on red

The gate has two waves:

1. check, verify, docs-build, docs-audit, memory-check, and code-audit run concurrently. Verification itself has two concurrent lanes: every non-Outpost file is grouped across at most eight xdist workers, while tests/test_outpost/ remains serial because its process-lifecycle and journal fixtures are not yet safe under blanket distribution. docs-audit hard-fails decision conventions, bounded STATUS.md sections, and the deterministic ownership graph joining both guide capture manifests, committed SVG/PNG assets, and non-empty Markdown references; broader SOURCE: and deleted-document findings remain warnings.
2. live-drive-dry runs only after the independent first wave is green.

On the 2026-07-17 reference machine, the pre-change serial pytest selection took 102.9s and the complete gate took 111.3s. The safe split runs the same selection in about 22–23s; complete-gate wall time is the longest first-wave stage plus the roughly 5s dry drive, not the sum of every row. Reference runs of the new topology are green in about 28s.

If any sub-command fails, the merge is blocked. All independent first-wave stages finish so one peer turn reports every failure family; ordered stages are shown as not run. The contract remains “I ran it, here’s the output” — not “looks right.”

task ship is one Taskfile target rather than a separate workflow, so the promotion path and the ad-hoc local command cannot drift apart.

Agent-facing output

Passing stage output is captured rather than streamed. A green run reports the commit, aggregate pytest counts, wall time, per-stage timings, and the slowest test:

SHIP PASS 27.8s | sha abc1234+dirty | 2509 passed, 5 skipped
stages: check 0.4s · verify 22.9s · docs-build 1.7s · ... · live-drive-dry 4.9s
verify lanes: verify-main 15.2s (1736 passed, 1 skipped) · verify-outpost 22.9s (773 passed, 4 skipped)
slowest: 7.12s tests/test_outpost/test_transcript.py::test_name

A red run reports every failed stage together, retains useful head-and-tail evidence within Kin’s shell-output cap, prints exact pytest node reruns when available, and preserves the complete logs under /tmp/kin-ship-*. Successful runs remove their temporary logs. The SHA carries +dirty when the proof ran against uncommitted or untracked changes, so an agent cannot mistake the base commit for the complete verified source state. The per-lane counts also make a base install’s clean Outpost package skip visible instead of hiding it inside the aggregate green result.

Pytest’s global configuration deliberately keeps test-case progress quiet while retaining high assertion verbosity and short tracebacks. The hermetic lanes also report slow tests and apply a 60-second per-test backstop; slow/live tests keep their specialized bounds outside this gate.

AI-peer repair loop

Use the narrowest relevant node while changing code:

task verify-one -- 'tests/test_harness/test_tools.py::test_name' -vv
uv run pytest --lf -x
task check

Then run task ship exactly once as the exhaustive completion proof. Last- failed, stepwise, and affected-test techniques are repair-loop accelerators, not substitutes for the promotion gate. Set KIN_TEST_WORKERS to reduce the default eight-worker cap on a constrained machine:

KIN_TEST_WORKERS=4 task verify

Markers

Two markers route tests out of the default task verify pass:

Marker	Meaning	Run with
slow	machine / perf / Textual-version sensitive	task test-slow
live	needs real network / real endpoint	task test-live

The default task verify runs everything that ISN’T marked — i.e. the hermetic headless suite. The slow and live suites run on demand via their dedicated task test-* entry points (or by directly passing -m slow / -m live to pytest).

The fifteen rendered TUI comparisons in tests/test_snapshot.py are slow because Rich/Textual SVG bytes depend on the locked rendering environment and machine layout. task verify-snapshot compares all fifteen baselines; task docs-screenshots-tui regenerates the ten test_docs_ scenes published directly under docs/assets/screenshots/. task docs-screenshots-web regenerates the five local-only Outpost PNGs at their fixed desktop/phone viewports, and task docs-screenshots runs both capture paths. The static dual-manifest ownership checks in tests/test_docs_audit.py are ordinary hermetic tests and remain in task verify and task ship; rendering and Playwright do not.

Outpost tests are hermetic, not a separate marker class. They are collected by task verify when the outpost extra is installed and skipped as a package by tests/test_outpost/conftest.py (via its aiohttp check) on a base install. To require that full package rather than permit the skip, run:

uv run --extra outpost pytest tests/test_outpost/

Conftest surface — fixtures + helpers

tests/conftest.py is the shared surface pytest applies across the collection. The headline wins:

Registered CLI options

• --update-baseline → update_baseline for the deterministic eval fingerprint.
• --update → update_snapshots for SVG regeneration. Registering this in conftest is required because pytest finalizes options before it imports tests/test_snapshot.py; task verify-snapshot -- --update would otherwise fail during argument parsing.

Fixtures (function-scoped unless noted)

• temp_kin_home(tmp_path, monkeypatch) — per-test KIN_HOME override with settings.reset_cache() so cached lookups from prior tests don’t bleed through. Use this in any test that mutates KIN_HOME.
• temp_session_dir(tmp_path, monkeypatch) — per-test KIN_SESSION_DIR override. Rare; mainly for tests that exercise persistence paths.
• fake_backend — a fresh FakeBackend with an empty responder. Pair with the make_responder(...) helper to script turns.
• session — a real Session wired with a captured-events list + the FakeBackend. Tests assert on the event stream via session._captured_events.
• pilot_app — fresh KinApp per scenario, wired with the FakeBackend via app.run_test(). Yields (app, pilot, sess, captured_events). CRITICAL gotcha: if you do monkeypatch.delenv("KIN_HOME", raising=False) to clear env keys, it UNDOES temp_kin_home’s monkeypatch.setenv("KIN_HOME", tmp_path) and the test falls back to ~/.kin (the user’s real home). Skip KIN_HOME in any env-clear loop. This bug bit 3 of the original 8 migrations in Phase 2.
• env_snapshot(monkeypatch) — capture-and-restore helper for tests that mutate env keys.
• _reset_env_between_tests (autouse) — snapshots os.environ at the start of every test, restores it on teardown, AND calls kin.harness.settings.reset_cache() so cached kin_home() lookups from earlier tests don’t bleed through. No opt-out by design — the cost of a surprise env leak is much higher than the cost of an opt-out. Tests that legitimately need to mutate env use monkeypatch (auto-restoring) or env_snapshot (manual restore). Cost: ~1ms per test.

Helpers (module-level functions, used inside test bodies)

• import_tool_module(name) — importlib.import_module(...) alias. Used to bypass kin.harness.tools.__init__’s tool-instance shadow when monkeypatching tool internals. Modules using it: web_fetch, web_context, _media, workflow, etc.
• async settle(pilot, predicate, *, tries=80, sleep=0.05) — poll a predicate until truthy. Consolidates the historical integration and live-drive settle helpers.
• async wait_for_event(events, predicate, *, timeout=2.0) — poll a captured-events list for an event matching the predicate.
• clean_terminal(text) / clean_interrupted(text) — strip ANSI / control sequences from a Textual screen dump.
• _git_short_sha(cwd=None) — current git short SHA, or “unknown”. Consolidates the historical eval and live-drive helpers.
• assert_eventually(condition, *, timeout, interval, message) — sync version of settle.
• make_responder(*scripted_turns) — build a FakeBackend responder from a list of (text | tool_call | (text, [tool_calls...])) tuples.

Env-before-import architecture

tests/__init__.py (NOT conftest.py) sets the KIN_HOME / KIN_SESSION_DIR / KIN_SPANS env vars at MODULE TOP. This is critical because:

1. Several verify scripts (now test modules) do top-level from kin.harness import presets, settings.
2. settings.kin_home() is read at first import, not lazily.
3. Pytest imports tests/__init__.py before any test module — guaranteeing the env is correct for every test module’s top-level kin imports.

setdefault (not =) is used so a developer’s explicit env wins for local debugging. The conftest session fixture does the teardown (rmtree of the tmpdir).

Do NOT add from kin.harness... at the top of tests/__init__.py itself; that would defeat the ordering guarantee.

The importlib shadow-bypass idiom

kin.harness.tools.__init__.py re-exports tool INSTANCES, which shadow the submodule names. from kin.harness.tools import web_fetch gives you the tool INSTANCE, not the module. Tests that monkeypatch (e.g. wf._validate_ip, wc.brave_get) need the REAL module object:

# DON'T:
from kin.harness.tools import web_fetch   # returns the Tool INSTANCE
wf = web_fetch
wf._validate_ip = lambda ip: None        # mutates the tool instance, not the module

# DO:
wf = import_tool_module("kin.harness.tools.web_fetch")
wf._validate_ip = lambda ip: None        # mutates the real module

The import_tool_module(name) helper consolidates this idiom in one place. It also covers kin.harness.tools._media (the media submodule that tests/test_web.py needs) and kin.harness.tools.workflow (the workflow executor module that tests/test_workflow.py needs).

Coverage interpretation

Lazy SDKs (openai, anthropic, mcp, playwright, tomlkit, etc.) are imported inside the code that needs them. Headless tests using FakeBackend never trigger OpenAIBackend / AnthropicBackend imports, so the SDK modules show up as “no statements” in the coverage report — NOT “missing” or “untested”.

Don’t interpret low coverage on a lazy-imported module as a regression. Use pytest --cov-context=test for per-test context if a specific module needs diagnosing.

Run task test-cov to see the coverage report:

task test-cov    # = uv run pytest tests/ --cov=src --cov-report=term-missing \
                 #       -m "not slow and not live"

There’s NO --cov-fail-under threshold — coverage is visibility, not enforcement. The lazy-SDK interaction makes a meaningful threshold premature (most of kin.harness.backends.* would show 0% under headless tests using FakeBackend).

Adding a new test

1. Pick the right home. Headless? → tests/. Live / smoke / network? → scripts/.
2. Use the shared fixtures (temp_kin_home, fake_backend, session, pilot_app) instead of hand-rolled env mutation.
3. Use assert instead of check() — pytest captures failures natively and the test function name becomes the test ID.
4. Preserve the assertion semantics. Same checks, same expected values — just pytest-shaped.
5. Add an if __name__ == "__main__": guard at the bottom if you want ad-hoc python tests/test_<area>.py debugging.
6. If your test is slow or live, mark it so the default task verify skips it. Outpost tests belong under tests/test_outpost/; that package’s conftest owns the optional-dependency skip.
7. Don’t edit tests/conftest.py if you’re working in a parallel branch / migration; add the fixture locally in your test file. The shared conftest is the only one that needs the KIN_HOME / settings reset dance, so adding fixtures there is rarely necessary.

Isolation guarantees

The autouse _reset_env_between_tests fixture (see Conftest surface above) closes the 4 known isolation failures from the migration — they now pass in the full suite, not just in isolation. The fixture is the contract: every test starts with a clean os.environ and a fresh kin.harness.settings._cache. No opt-out by design — the cost of a surprise env leak is much higher than the cost of an opt-out.

Tests that need to mutate env use monkeypatch (auto-restoring) or env_snapshot (manual restore). Module-top os.environ[...] = ... mutations are the original sin: pytest collects modules before running anything, so module-top mutations persist for the entire session. The pytest migration removed the ones it found in test_evals.py; the autouse fixture is the backstop for any that slip in later.

Per-task pytest invocations

Old verify_*.py	New pytest task
verify_settings.py	task verify-settings (uv run pytest tests/test_settings.py)
verify_banner.py	task verify-banner (uv run pytest tests/test_banner.py)
verify_integration.py	task verify-integration (uv run pytest tests/test_integration.py)
verify_workflow.py	task verify-workflow (uv run pytest tests/test_workflow.py)
verify_search.py	task verify-search (uv run pytest tests/test_search.py)
verify_memory.py	task verify-memory (uv run pytest tests/test_memory.py)
verify_provider_presets.py	task verify-provider-presets (uv run pytest tests/test_provider_presets.py)
verify_evals.py	task verify-evals (uv run pytest tests/test_evals.py; add -- --update-baseline only when intentionally re-pinning)
verify_harness.py	task verify-harness (uv run pytest tests/test_harness/)
verify_app.py	task verify-app (uv run pytest tests/test_app/)
verify_outpost.py	uv run --extra outpost pytest tests/test_outpost/ (task verify also collects it when the extra is already installed)
verify_outpost_live.py	retired — tests/test_outpost_live.py + task verify-outpost-live tested the ttyd/tmux browser-terminal surface, deleted wholesale in the middle-way plan’s Step 3 ttyd cut (2026-07-06, DR 0033)
verify_pty.py	task verify-pty (uv run --group pty pytest tests/test_pty.py -m slow)
verify_snapshot.py	task verify-snapshot (uv run pytest tests/test_snapshot.py -m slow); task docs-screenshots-tui regenerates the ten guide-owned test_docs_ targets
verify_worktrees.py	task verify-worktrees (uv run pytest tests/test_worktrees.py -m live)
verify_perf.py	task verify-perf (uv run pytest tests/test_perf.py -m slow)
verify_web.py	task verify-web (uv run pytest tests/test_web.py)
live_drive_vllm.py	stays a CLI script; thin pytest wrapper at tests/test_live_drive.py
live_git_tools.py	stays a CLI script; KIN_LIVE_GIT_PUBLISH=1 task live-git-tools is an explicitly armed remote publication proof, never part of task ship

tests/test_docs_audit.py has no historical verify_*.py ancestor. It pins the bounded STATUS.md contract and combined TUI/browser screenshot graph used by task docs-audit; unlike rendering either asset type, those read-only checks are deterministic and belong in the file-grouped hermetic lane. task status-trim removes whole recently-landed records by age and count; it never leaves continuation lines behind.

verify_harness.py was kept as a backward-compat shell through the migration (Phases 1-5) so existing cron / muscle-memory invocations kept working while the pytest files landed; it’s now removed (the pytest collection is the canonical entry point).