# xpt xpt is an inline expect-test library for Python 3.11 and later. You write `expect(value)` in a test, run it, and xpt writes the value it saw into the test source when you accept it. It is built so coding agents can work with failing tests cheaply and safely, and so the people reviewing their work can see what changed. ```python from xpt import expect def test_parser(): expect(parse("x + y")) ``` xpt has no runtime dependencies. It ships an `expect` function, a pytest plugin with a short report, and an `xpt` command for reviewing and accepting what a run recorded. ## Install ```sh uv add --dev xpt ``` `pip install xpt` works too. The pytest plugin registers itself when xpt is installed. ## What a run looks like A test with a wrong expectation and a test with none: ```python def test_order(): expect(make_order(), Order(7, ["apple"], 12.5)) def test_new(): expect({"status": "ok", "count": 3}) ``` `pytest` prints one short block per failure and a footer with the next command: ```text FAILED tests/test_demo.py::test_order tests/test_demo.py:18 > expect(make_order(), Order(7, ["apple"], 12.5)) b211 mismatch value.items[1]: added "pear" 1 changed, 3 unchanged FAILED tests/test_demo.py::test_new tests/test_demo.py:22 > expect({"status": "ok", "count": 3}) ee17 missing actual: {"status": "ok", "count": 3} 2 failed in 0.08s. Next: xpt accept b211 ee17 ``` `xpt accept b211 ee17` rewrites the two calls in place: ```python def test_order(): expect(make_order(), Order(id=7, items=["apple", "pear"], total=12.5)) def test_new(): expect({"status": "ok", "count": 3}, {"status": "ok", "count": 3}) ``` The next run passes. ## Where to go next - [Quickstart](quickstart.md) walks through the loop from install to a passing run. - [expect()](expect.md) covers what values xpt can record and compare. - [Reviewing failures](review.md) covers `failures`, `show`, `accept`, `reject`, and `--expect-create`. - [Coding agents](agents.md) has a snippet to paste into your `AGENTS.md`. - [Design](design.md) explains why xpt works the way it does. The site is also published as plain Markdown for language models. [llms.txt](llms.txt) indexes every page, and [llms-full.txt](llms-full.txt) holds all of them in one file. Each page is also available at its own path with a `.md` extension. # Quickstart This walks through one loop: write a test with no expectation, let xpt fill it, change the code, and review the diff. ## 1. Install ```sh uv add --dev xpt ``` ## 2. Write a test without an expected value ```python title="tests/test_demo.py" from xpt import expect def summarize(items): return {"status": "ok", "count": len(items)} def test_summary(): expect(summarize(["a", "b", "c"])) ``` ## 3. Fill it ```sh pytest --expect-create ``` ```text FAILED tests/test_demo.py::test_summary tests/test_demo.py:9 > expect(summarize(["a", "b", "c"])) ee17 missing actual: {"status": "ok", "count": 3} 1 failed in 0.07s. Filled 1 missing expectation: tests/test_demo.py:9. Next: run the tests. ``` The run still fails, because the expectation was empty when the test ran. The file now reads: ```python def test_summary(): expect(summarize(["a", "b", "c"]), {"status": "ok", "count": 3}) ``` Run `pytest` again and it passes. Read the new value before you commit it. `--expect-create` writes what the code returned, which is only right if the code is. ## 4. Change the code and review the diff Say `summarize` starts counting one too many. The next `pytest` run shows only what changed: ```text FAILED tests/test_demo.py::test_summary tests/test_demo.py:9 > expect(summarize(["a", "b", "c"]), {"status": "ok", "count": 3}) 0eae mismatch value["count"]: 3 -> 4 1 changed, 1 unchanged 1 failed in 0.04s. Next: xpt accept 0eae ``` Now decide. - The new value is right: `xpt accept 0eae` writes it into the test. - The new value is a bug: fix the code, or run `xpt reject 0eae` so xpt stops offering it for acceptance while you work on the fix. `xpt accept` refuses if the expected argument changed since the run, so it never writes over an edit you made after the run. ## 5. Keep `.xpt/` out of Git xpt records runs under `.xpt/` and writes `.xpt/.gitignore` the first time, so nothing there is committed. `xpt clean` deletes the directory. # expect() ```python from xpt import expect expect(actual) # missing: fails and records actual expect(actual, expected) # passes when actual equals expected, fails with a diff otherwise ``` `expect` compares `actual` with `expected`. When they differ, or when `expected` is left out, the test fails with `ExpectationFailed` and xpt records the actual value under `.xpt/` so `xpt accept` can write it into the source. Pass `expected` as the second positional argument or as `expected=`. When accepting, xpt replaces the argument where it is. ## Values xpt can record | Type | Notes | |---|---| | `str`, `int`, `float`, `bool`, `None` | `True` and `1` are different values. `nan`, `inf`, and `-0.0` round-trip. | | `list`, `tuple`, `dict` | Order is kept. Dict keys can be any supported value. | | `set`, `frozenset` | Members are sorted when written, so the source does not depend on hash order. | | Dataclasses | Must compare by value (`eq=True`, the default) and have every field in `__init__`. | Anything else fails with a message that names where the value is and its type: ```text value["handler"] is a builtins:builtin_function_or_method; xpt cannot record it. Register an adapter or expect a field of it instead. ``` Adapters for other types are on the roadmap and not yet available. Until then, expect a field or a plain conversion of the value, for example `expect(str(path))` or `expect(dataclasses.asdict(obj))`. Subclasses of builtins, such as an `IntEnum` member, are refused rather than recorded as their base type, because writing `1` where the code returns `Color.RED` would hide a real difference. A value that contains itself is refused and the message names the path to the cycle. ## Comparison A call passes when `actual == expected`. When it fails, the diff and the grouping of repeated calls use the same structural equality, so `1` and `1.0` are equal. A dataclass is compared field by field, and its class must be the same. A record over 1 MB keeps its status but drops both values. Expect a smaller part of the value instead. ## Writing a dataclass xpt writes a dataclass by its class name, for example `Order(id=7, items=["apple"], total=12.5)`, but only when the test file already binds that name at module level by an import, a class statement, or an assignment. Otherwise the accept is skipped and the reason names the missing import. ## Helpers A helper that calls `expect` for its caller passes `_depth=1`. The failure then points at the caller's line, and accept writes the expected value into the caller's call: ```python def check(actual, expected=None, *, label=""): expect(actual * 2, expected, _depth=1) def test_it(): check(21, label="half") ``` `xpt accept` turns the test's line into `check(21, 42, label="half")` and leaves the helper alone. ## Calls xpt can't rewrite xpt finds a call by its function, its position among the `expect` calls in that function, and a hash of the expected argument's text. A failure is reported as `unlocatable`, and never accepted, when: - the code has no source file, as with `exec` - the file no longer parses - the file changed after it was imported - one line holds several calls and Python ran with `-X no_debug_ranges` The comparison still runs in each case, so the test still fails or passes correctly. ## Without pytest `expect` works in plain `unittest` and in scripts. Without the pytest plugin, xpt records under the current directory and writes the run when the process exits. `XPT_NO_RECORD=1` turns recording off. # Reviewing failures A pytest run records every failed `expect` call under `.xpt/`. The `xpt` command reads that record, so you can review and accept without running the tests again. ## Call sites and ids Failures are grouped by call site, the `expect` call in the source. A call inside a loop or a parametrized test is one call site that ran several times. Each call site has a short id such as `b211`. The id is a hash of the file, the function, and the call's position among the `expect` calls in that function, so it stays the same across runs when lines move. It gets longer, like a git short hash, when two ids collide. ## Statuses | Status | Meaning | Acceptable | |---|---|---| | `missing` | `expect(actual)` with no expected value | yes | | `mismatch` | The values differ | yes | | `conflict` | The call ran more than once with different values, or the file changed during the run | no | | `unsupported` | The value has a type xpt can't record, or is over 1 MB | no | | `unlocatable` | xpt can't find the call in the source | no | A conflict usually means the call needs one `expect` per case. A call site that you rejected also shows `rejected` after its status. ## `xpt failures` Lists every failed call site of the latest run with its diff, and ends with the next command. ```text b211 tests/test_demo.py:18 mismatch 1 test value.items[1]: added "pear" 1 changed, 3 unchanged ee17 tests/test_demo.py:22 missing 1 test actual: {"status": "ok", "count": 3} 2 failures in run 20261119T150648-5031 (1 mismatch, 1 missing), 2 acceptable. Next: xpt accept b211 ee17 ``` Plain `assert` failures appear in the pytest report but not here, because xpt can't write them. ## `xpt show ID` Shows one call site with its function and the tests that hit it. `--full` lifts the limits on how many changes and how much of each value it prints, and adds both whole values. ```text b211 tests/test_demo.py:18 mismatch 1 test function: test_order test: tests/test_demo.py::test_order value.items[1]: added "pear" 1 changed, 3 unchanged Next: xpt accept b211 ``` ## `xpt accept` Writes each call site's actual value into the source as its expected value. ```sh xpt accept b211 ee17 # these call sites xpt accept --missing # only empty expectations xpt accept --all # every acceptable call site that was not rejected xpt accept --dry-run b211 # report what would change, write nothing ``` `xpt accept` with no ids and no flag does nothing and says so. Accept is careful about what it writes. - It checks that the expected argument's text still matches the run, and refuses with `stale` if you edited it since. Run the tests again to record a fresh value. - Lines added or removed elsewhere in the file don't matter. - A second accept of the same value is a no-op and reports `already_accepted`. - It rewrites each file once, through a temporary file, under `.xpt/accept.lock`. The new text must compile before it is written. - It reprints the call the way `ruff format` would at line width 100, and leaves the rest of the file byte for byte. A call with a comment inside it keeps the comment, and only the expected argument is replaced. - It refuses files that aren't UTF-8. ## `xpt reject` ```sh xpt reject b211 ``` Use it when the diff shows a bug rather than a new expectation. It records the call site and a hash of the actual value in `.xpt/decisions.jsonl`. From then on, `xpt failures` and the pytest footer mark that call site `rejected` and stop suggesting `xpt accept` for it. `--all` and `--missing` skip it. A rejection stops matching as soon as a run records a different value at that call site. The test keeps failing until the code is fixed. Accepting a rejected call site by its id still works. ## `pytest --expect-create` Runs the tests, then fills every missing expectation the way `xpt accept --missing` does. It writes after the session ends, never during it, and never replaces an existing expected value. The run still fails, because the expectations were empty when their tests ran. Run again to see them pass. ## Other commands - `xpt runs` lists the recorded runs, newest first. xpt keeps the last five. - `xpt clean` deletes `.xpt/` and every recorded value. - `--run ID` on `failures`, `show`, `accept`, and `reject` reads an older run. See the [CLI reference](cli.md) for every option and exit code. # pytest report When xpt is installed, its pytest plugin replaces pytest's terminal output with a shorter report. Agents read every byte of a test run, and pytest's header, progress dots, rules, and full tracebacks cost tokens without saying more. ```text FAILED tests/test_parser.py::test_parser tests/test_parser.py:14 > expect(parse(source), tree) a81f mismatch value.operator: "addition" -> "subtraction" 1 changed, 42 unchanged FAILED tests/test_api.py::test_totals tests/test_api.py:30 > assert totals(order) == {"total": 42, "items": 3, "tax": 0, "due": 42} value["total"]: left 41, right 42 1 changed, 3 unchanged FAILED tests/test_io.py::test_load tests/test_io.py:8 > src/app/io.py:12 read_config > return json.loads(text) json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) caused by: FileNotFoundError: [Errno 2] No such file or directory: 'app.toml' 6 failed, 2 passed in 0.31s. Next: xpt accept a81f ``` ## What each block says - The header names the test and the deepest frame in a project file, outside `site-packages` and pytest. When the error comes from a helper, it shows the chain of project frames from the test down. - The next line is the source line at that frame, after `> `. - An `expect` failure shows its id, its status, and the diff by path. - A plain `assert left == right` on values xpt can record gets the same diff, labeled `left` and `right`. - Any other exception is one line, `Type: message`, cut to 200 characters, plus a `caused by:` line for a chained exception. - Parametrized cases that fail the same way fold into one block with an `also fails:` line per other case. - A failing test prints the last 5 lines of its captured stdout, stderr, and log. - Setup and teardown errors say `ERROR` and name the phase. The footer is pytest's counts and duration, then the next command: `Next: xpt accept ` when there are call sites to accept, `Next: xpt failures` when there are expect failures but none can be accepted. ## Verbosity | Option | Effect | |---|---| | default | 10 changes per expectation, 80 characters per value | | `-v` | 50 changes, 400 characters per value, 20 output lines, pytest's line per test | | `-vv` | no limits, plus pytest's traceback | | `-q` | the same compact report | | `--no-summary` | the footer only | ## Options | Option | Environment | Effect | |---|---|---| | `--expect-create` | | After the session, fill every missing expectation. See [Reviewing failures](review.md#pytest-expect-create). | | `--expect-no-record` | `XPT_NO_RECORD=1` | Don't record failing `expect` calls under `.xpt/`. | | `--xpt-report=pytest` | `XPT_REPORT=pytest` | Keep pytest's own output. The default is `xpt`. | | `--xpt-json=PATH` | | Also write the full report, with no limits, as [JSON](json.md#pytest-xpt-json). | ## When pytest's report is kept xpt replaces only pytest's header, progress, failure sections, default short summary, and final line. Everything else still prints: warnings, plugin output, `--durations`, `-x` stops, `--lf` messages, live logs, the debugger, and xdist status lines. The exit code is always pytest's. These options ask for pytest's own failure output, so the run keeps pytest's report: - `--tb` with anything but `auto`, `-l`, and `--full-trace` - runs that don't run tests: `--co`, `--fixtures`, `--fixtures-per-test`, `--markers`, `--setup-only`, `--setup-plan`, `--help`, `--version` `--xpt-json` with one of these is a usage error, because there is no xpt report to write. `-r` with anything but the default keeps pytest's short summary beside the xpt report. pytest-sugar, when it is on, keeps its own report. If the report can't install or render, or pytest is older than 8.0, the run prints pytest's report and one line starting with `xpt:` that says why. ## Compatibility | | Supported | |---|---| | Python | 3.11 and later | | pytest | 8.0 to 9.2, tested on every minor release | | pytest-xdist | 3.5 to 3.8. The controller prints one report. | | unittest | `expect` works without the plugin | Known gaps: pytest 9.2's terminal progress indicator stays at 0%, and line rewriting on a real terminal is tested only through the code paths a pipe uses. # Coding agents xpt is built for agents that run tests, read the output, and change code. Three things matter for that. - **The report is short.** One block per failure with the diff by path, and a footer that names the next command. An agent spends its tokens on what changed. - **Accepting is explicit.** A run never writes to the source. `xpt accept` writes only the call sites it is given, refuses when the expected argument changed since the run, and `xpt reject` stops a bug from being offered again. - **Everything has JSON.** Every command takes `--json`, and `pytest --xpt-json=PATH` writes the full report. The risk with any snapshot tool is an agent that accepts a wrong value to make a test pass. xpt can't decide whether a value is right. It makes the decision visible: every diff is on screen before the footer suggests an accept, and a rejected value stays rejected. ## Paste into AGENTS.md ```markdown ## Tests with xpt This project uses xpt inline expect tests (https://harville-labs.github.io/xpt/). - Write new expectations as `expect(value)` with no expected argument, run `pytest --expect-create`, then read the filled value in the diff before committing it. - When a test fails, read the diff in the pytest output or `xpt failures`. Decide whether the new value is correct. - Correct, because the code's behavior was meant to change: `xpt accept `. - A bug: fix the code. If you can't fix it now, run `xpt reject ` so it isn't accepted by accident. - Never run `xpt accept --all` without reading every diff `xpt failures` prints first. - Don't hand-edit an expected value that xpt can write. Accept it, so the value is exactly what the code returned. - `xpt show --full` prints whole values when the diff is cut. - Use `--json` on any xpt command when parsing its output. ``` ## Reading the footer | Footer | Meaning | |---|---| | `Next: xpt accept b211 ee17` | These call sites can be accepted. Read their diffs first. | | `Next: xpt failures` | There are expect failures, but none can be accepted. `xpt failures` says why. | | `Next: run the tests.` | `--expect-create` filled every missing expectation. | | no `Next:` | Every failure is a plain assert or exception. Fix the code. | ## LLM-readable docs This site is published as Markdown for models. [llms.txt](llms.txt) is an index, and [llms-full.txt](llms-full.txt) holds every page in one file. # CLI reference ```text xpt {failures,show,accept,reject,runs,clean} [options] ``` `python -m xpt` runs the same command. ## Common options | Option | Commands | Meaning | |---|---|---| | `--json` | all | Print one JSON object on stdout. See [JSON output](json.md). | | `--root ROOT` | all | The project root. Default `.`. | | `--run RUN` | `failures`, `show`, `accept`, `reject` | A run id from `xpt runs`. Default the latest run. | ## Commands ### `xpt failures` List the failed call sites of a run, with their diffs. ### `xpt show ID` Show one call site with its function and tests. | Option | Meaning | |---|---| | `--full` | Lift the change and length limits and print both whole values. | ### `xpt accept [ID ...]` Write actual values into the test source. | Option | Meaning | |---|---| | `--all` | Every acceptable call site that was not rejected. | | `--missing` | Only empty expectations. | | `--dry-run` | Report what would change and write nothing. | ### `xpt reject ID [ID ...]` Record actual values as bugs, so xpt stops offering them for acceptance. ### `xpt runs` List recorded runs, newest first. ### `xpt clean` Delete `.xpt/` and every recorded value. ## Exit codes | Code | Meaning | |---|---| | `0` | The command did what it was asked, or there was nothing to do. | | `1` | `failures` found failures, or `accept` or `reject` left a requested call site stale or skipped. | | `2` | A usage error, an unknown id or run, no recorded run, a malformed record, or a held `.xpt/accept.lock`. The error is one line on stderr that says what to do next, and nothing is written. | ## Files ```text .xpt/ .gitignore "*", written on first use runs//.jsonl latest the id of the newest complete run decisions.jsonl rejections, one per line accept.lock held while accept or reject writes ``` Each pytest process, the controller and each xdist worker, writes its own file, so nothing is shared during a run. The last five runs are kept. # JSON output Every `xpt` command takes `--json` and prints one JSON object on stdout. `pytest --xpt-json=PATH` writes the whole pytest report the same way. Each object has `"schema_version": 1`. ## Values Values use xpt's recorded form. It is plain JSON where JSON can say the type, and a one-key object where it can't: | Python | JSON | |---|---| | `{"a": [1, 2.5, True, None]}` | `{"dict": {"a": [1, 2.5, true, null]}}` | | `(1,)` | `{"tuple": [1]}` | | `{1: "x"}` | `{"dict": [[1, "x"]]}` | | `float("nan")` | `{"float": "nan"}` | | `Point(1, "a")` from module `values` | `{"values:Point": {"x": 1, "y": "a"}}` | `expected` is left out when there is none, so a recorded `None` stays distinct from a missing expectation. ## Group `xpt failures --json` returns the run and its groups. Each group is one call site: ```json { "schema_version": 1, "command": "failures", "run": "20261119T150800-b910", "groups": [ { "id": "0eae", "site": {"path": "tests/test_demo.py", "line": 5, "qualname": "test_count"}, "status": "mismatch", "acceptable": true, "rejected": false, "tests": ["tests/test_demo.py::test_count"], "detail": "", "actuals": [{"dict": {"status": "ok", "count": 4}}], "expected": {"dict": {"status": "ok", "count": 3}}, "changes": [ {"path": "value[\"count\"]", "op": "replace", "expected": 3, "actual": 4} ], "unchanged": 1 } ] } ``` - `status` is one of `missing`, `mismatch`, `conflict`, `unsupported`, or `unlocatable`. - `acceptable` is true for `missing` and `mismatch`. - `detail` says why a group can't be accepted. - `actuals` holds the distinct actual values, one when every call agreed. - `changes` and `unchanged` appear on a mismatch. `op` is `replace`, `add`, or `remove`. `xpt show ID --json` returns one group in the same form. ## Accept and reject ```json { "schema_version": 1, "command": "accept", "run": "20261119T150800-b910", "dry_run": true, "outcomes": [ {"id": "0eae", "site": {"path": "tests/test_demo.py", "line": 5, "qualname": "test_count"}, "status": "accepted", "reason": ""} ], "changed": ["tests/test_demo.py"] } ``` An accept outcome's `status` is `accepted`, `already_accepted`, `stale`, or `skipped`, and `reason` says why it wasn't written. `xpt reject --json` has the same `outcomes`, with `rejected`, `already_rejected`, or `skipped`, and no `changed`. ## Runs ```json { "schema_version": 1, "command": "runs", "runs": [ {"id": "20261119T150800-b910", "latest": true, "failures": 1} ] } ``` ## pytest `--xpt-json` ```json { "schema_version": 1, "exit_code": 1, "summary": "1 failed in 0.04s. Next: xpt accept 0eae", "counts": ["1 failed"], "duration": 0.04, "run": "20261119T150800-b910", "created": null, "failures": [ { "test": "tests/test_demo.py::test_count", "when": "call", "path": "tests/test_demo.py", "line": 5, "function": null, "callers": [], "source": "expect({\"status\": \"ok\", \"count\": 4}, {\"status\": \"ok\", \"count\": 3})", "cause": null, "detail": {"kind": "expect", "observation": {"...": "..."}}, "output": {}, "traceback": "...", "group": {"id": "0eae", "...": "..."} } ] } ``` - Nothing is truncated. The JSON has every change and the full traceback. - `created` is null without `--expect-create`. With it, `created` holds `outcomes` and `changed` in the same form as `xpt accept --json`. - An expect failure adds `group` in the same form as `xpt failures --json`. - A compared `assert` adds `changes` and `unchanged`, with sides named `left` and `right`. # Python API `xpt` exports one name, `expect`. Tools that want the CLI's behavior without parsing its output import `xpt.api`. The CLI is a thin wrapper over these functions. ```python from pathlib import Path from xpt import api root = Path(".") run = api.latest_run(root) groups = api.mark(api.group(api.load_run(root, run)), api.load_rejections(root)) result = api.accept(root, [g for g in groups if g.offered], dry_run=True) for outcome in result.outcomes: print(outcome.group.id, outcome.status, outcome.reason) ``` ## Reading a run | Function | Returns | |---|---| | `run_ids(root)` | Recorded run ids, oldest first. | | `latest_run(root)` | The id of the newest complete run. Raises `RecordError` when there is none. | | `load_run(root, run=None)` | The run's `Observation`s. `None` reads the latest run. Raises `RecordError` on a malformed record and names the file and line. | | `group(observations)` | A tuple of `Group`, one per call site, sorted by path, line, and position. Pure. | ## Deciding | Function | Effect | |---|---| | `accept(root, groups, *, dry_run=False)` | Writes each acceptable group's actual value into its source. Returns `AcceptResult(outcomes, changed)`. Raises `AcceptError` when `.xpt/accept.lock` is held. | | `reject(root, run, groups)` | Records each group's actual value as a bug. Returns a tuple of `RejectOutcome`. | | `load_rejections(root)` | The recorded rejections, as a frozenset of `Rejection`. | | `mark(groups, rejections)` | The groups with `rejected` set. Pure. | ## Comparing | Function | Returns | |---|---| | `diff(expected, actual)` | `Diff(changes, unchanged)`, where each `Change` has `path`, `op`, `expected`, and `actual`. | | `format_path(path, root="value")` | A change path as text, such as `value.items[1]`. | | `to_json(value)`, `from_json(data)` | A `Value` to and from its [JSON form](json.md#values). | ## Types | Type | Fields | |---|---| | `Observation` | `run`, `test`, `site`, `status`, `expected`, `actual`, `detail` | | `Status` | `"missing"`, `"mismatch"`, `"unsupported"` | | `Group` | `id`, `site`, `status`, `actuals`, `expected`, `tests`, `detail`, `rejected`, and the properties `acceptable` and `offered` | | `GroupStatus` | `Status` plus `"conflict"` and `"unlocatable"` | | `CallSite` | `path`, `qualname`, `callee`, `ordinal`, `arg_sha`, `line` | | `Unlocatable` | `reason`, `path`, `line` | | `Outcome` | `group`, `status`, `reason` | | `OutcomeStatus` | `"accepted"`, `"already_accepted"`, `"stale"`, `"skipped"` | | `RejectOutcome` | `group`, `status`, `reason` | | `RejectStatus` | `"rejected"`, `"already_rejected"`, `"skipped"` | | `Value` | The recorded value tree. | Every type is a frozen dataclass or a `Literal`. `offered` is true when a group is acceptable and not rejected, which is what `xpt accept --all` selects. The API is pre-1.0. Names may change between minor versions. # xpt design Status: draft for review. Package and CLI name `xpt`. `expect`, `expectly`, `pyexpect`, and `expt` are taken on PyPI. xpt is an inline expect-testing library for Python. A test calls `expect(actual)` or `expect(actual, expected)`. When the value is missing or different, the run records what it saw. A separate command shows the difference and writes an accepted value into the test source, without running the tests again. It is built for coding agents and must stay just as usable by people. It needs no model, network service, or agent framework. The guiding question for every decision: does this make writing, running, understanding, or maintaining tests cheaper for a coding agent, without costing correctness or usability for a human? ## Prior art and where we differ We read the source of all four projects (default branch, November 2026). File references are in `docs/research.md`. | Capability | expecttest | inline-snapshot | syrupy | ApprovalTests | xpt | |---|---|---|---|---|---| | Inline expectations in test source | Strings only | Yes, any value | No | Docstrings only | Yes, any supported value | | Native values, not strings | No | Yes | Yes (external) | JSON or text | Yes | | Create missing values | Accept mode | `create` | `--snapshot-update` | Approve file | `accept --missing` | | Accept later without a rerun | No | No | No | Yes, by moving files | Yes, per change | | Review per change | No | No, one prompt per category | No | Per file, in a diff tool | Yes | | Persisted proposed changes | No | No | No | `.received` files | `.xpt/runs/` | | Stale-source detection | No | No | No | No | Yes | | Structured path diff | No | No | No | No | Yes | | JSON output | No | No | No | No | Yes | | pytest-xdist | Edits from each worker | Disabled under xdist | Locks files | Not handled | Workers record, no edits | | unittest | Yes | No | No | Yes | Yes | | Runtime dependencies | None | 5 (executing, asttokens, rich, pytest, typing-extensions) | pytest | pytest plus 2, and extras installed by default | None | | Source rewriting | Regex over a line range | Offset edits through asttokens, then optional black | n/a | Rewrites the whole file | Offset edits through stdlib `ast` | What is not new, and we should say so: - Inline values that are real Python values, rewritten in place with formatting kept. inline-snapshot does this well. - Finding the running call precisely. inline-snapshot keys on `(code object, f_lasti)` and uses `executing`. We use the same idea through stdlib `co_positions`. - Approving later without a rerun. ApprovalTests leaves `.received` files on disk and writes an approval script. - Pluggable serializers, sorted output, cycle limits, and path matchers. syrupy has them. - Detecting conflicting values from one call site. expecttest does it within one process. What we found in none of them, which is where xpt differs: 1. One observation per assertion, persisted across processes, with list, show, accept, and reject for each change. 2. Stale-source detection before any write. 3. Path-based structured diffs with counts of unchanged elements. 4. A JSON interface with a schema version for every operation. 5. xdist that works, because workers never write source. 6. No runtime dependencies. ### Why not contribute to inline-snapshot instead We considered this seriously, because inline-snapshot is mature and well designed. Two of its foundations work against our core feature: - It builds new source in `pytest_sessionfinish` from live Python objects, with access to the test's local and global variables (`customize(value, builder, local_vars, global_vars, ...)`). Accepting later in another process needs a serialized value, and its code generation is built around the live object. - It is pytest-only and turns itself off under xdist by design. Changing either would be a rewrite of its core, not a pull request. We should still open a discussion upstream before M2, describing the record-then-accept model. If the maintainer wants it, contributing is better for users than a new library. ## Decisions that change the original brief The brief was a good starting point. These parts of it change. 1. **Recording is on by default. There is no `--expect-record` flag.** The point is to avoid a second test run. If an agent must rerun with `--expect-record` after a plain `pytest` fails, the flag costs the very run we are trying to save. Every run writes its failed and missing observations to `.xpt/`, which is gitignored. `--expect-no-record` turns this off. The cost is a few kilobytes per failing assertion. The privacy cost is no worse than pytest's own failure output, which already prints the same values. `xpt clean` deletes the directory. 2. **`pytest --expect-create` does not edit files during the run.** It runs the tests, then applies the recorded `missing` observations through the same code path as `xpt accept --missing`, after the session ends. One rewrite path means one set of safety checks. Test execution and source edits stay separate, as the brief asks. A mismatch is never accepted by this flag. Under xdist only the controller writes, after every worker has finished. The exit status is still pytest's, so a run that fills expectations fails, because those tests failed when they ran. `--expect-create` with `--expect-no-record` or `XPT_NO_RECORD` is a usage error, because there would be nothing to fill. 3. **No LibCST.** The prototype in `docs/prototypes/` shows that the standard library is enough. `code.co_positions()` (Python 3.11+) gives the exact span of the running call. `ast` gives the span of each argument. A byte-offset splice replaces only the expected argument, so comments, formatting, and every other line stay untouched. LibCST would add a native wheel and a YAML dependency to rewrite one argument we can already locate exactly. We still parse the result with `compile()` before writing. 4. **No runtime dependencies.** pytest is optional. `executing` and `asttokens`, which inline-snapshot uses, are not needed on Python 3.11+. 5. **Python 3.11 is the floor.** 3.10 reached end of life in October 2026, and 3.11 brings `co_positions` and `co_qualname`. 6. **JSON Lines files, not SQLite.** Each process writes its own file, so there is no shared write target and no lock (see Persistence). 7. **The CLI is `xpt`, with `python -m xpt` as a fallback.** A command named `expect` would collide with the Tcl `expect` tool that many systems install. 8. **Benchmarking agent efficiency is its own project phase, not an MVP requirement.** It is the claim the project lives or dies on, so it gets a real harness (milestone M5) instead of a spreadsheet of guesses. ## Data shapes Everything else follows from these records. ```python CallSite # where an expect() call lives in source path: str # relative to the project root qualname: str # "TestParser.test_add", from co_qualname callee: str # the called function's source text: "expect", or a helper's name ordinal: int # index of this call among calls to the same callee in that function arg_sha: str # sha256 of the expected-argument source text, or "" when absent line: int # for display only, never used to locate the call Unlocatable # a call xpt cannot rewrite (no source file, file changed since import) reason: str path: str line: int Value # a serialized, typed value tree. Never code, never pickle. "a" | 1 | 1.5 | true | null | [Value, ...] # plain JSON where JSON keeps the type {"tuple" | "set" | "frozenset": [Value, ...]} {"float": "nan" | "inf" | "-inf"} {"dict": {str: Value, ...}} # every key is a str {"dict": [[Value, Value], ...]} # any other keys {"module:Qualname": {field: Value, ...}} # a dataclass {"adapter": {"name": "numpy.ndarray", "data": Value}} Observation # one failed call of expect() in one run run: str # run id test: str # pytest node id, or the expect() caller's qualname outside pytest site: CallSite | Unlocatable # an unlocatable site is shown but never accepted status: "missing" | "mismatch" | "unsupported" expected: Value | None # None when missing or unsupported actual: Value | None # None when unsupported or over the size cap detail: str # "" normally; the UnsupportedValue message, or "too large: N bytes, sha256 " Group # every failed observation of one call site in a run, built at read time id: str | None # short display id, None for an unlocatable site site: CallSite | Unlocatable status: "missing" | "mismatch" | "unsupported" | "unlocatable" | "conflict" actuals: [Value] # the distinct actual values, one when the calls agree expected: Value | None # None when missing, or when the calls disagree on it tests: [str] # every test that reached the site detail: str # why the group cannot be accepted, "" when it can rejected: bool # its one actual value matches a Rejection in decisions.jsonl acceptable # derived: status is missing or mismatch offered # derived: acceptable and not rejected; what footers suggest Outcome # what accept did with one group group: Group status: "accepted" | "already_accepted" | "stale" | "skipped" reason: str # why it was not written, "" when accepted or already accepted AcceptResult outcomes: [Outcome] # one per group, in the order given changed: [str] # project-relative paths that were rewritten Rejection # a call site's actual value rejected as a bug, one line of decisions.jsonl site: (path, qualname, callee, ordinal) # the CallSite identity, as for the display id actual_sha: str # sha256 of the actual value's canonical JSON RejectOutcome # what reject did with one group group: Group status: "rejected" | "already_rejected" | "skipped" reason: str # why it was not recorded, "" otherwise Change path: [str | int] # ["expression", "operator"] op: "replace" | "add" | "remove" expected: Value | None actual: Value | None ``` An observation stores no diff and no id. Readers derive both: `diff(expected, actual)` gives the changes, and `site_id(site)` gives the id. One copy of each rule means a stored diff can never disagree with the values it came from. `conflict` is not recorded either. It is the status of a group of observations whose values differ (see Conflicting observations). A passing assertion writes nothing. Run state lives on disk as records. In memory there is only the open file handle. ## Identity and stale detection An expectation's identity is its `CallSite` minus `arg_sha` and `line`: `(path, qualname, callee, ordinal)`. The callee is part of it because a helper called with `_depth=1` and a direct `expect()` can share an ordinal in one function. The display id is the first 4 hex characters of `sha256(path, qualname, callee, ordinal)`, lengthened like a git short hash when two ids collide. The id stays the same across runs as long as the call stays in the same function at the same position among its sibling `expect()` calls. Line numbers can change freely. `arg_sha` is the fingerprint. At accept time we: 1. Parse the file as it is now. 2. Find the function by `qualname` and the call by `callee` and `ordinal`. 3. Hash the expected argument's current source text and compare it to `arg_sha`. 4. Refuse if the function, the call, or the hash does not match. The group is `stale`, and its reason names the file and the fix: `tests/test_parser.py:12 changed since the run. Run the tests again.` Before the hash check, accept renders the call it would write. If the file already holds exactly that text, the group is `already_accepted`. This is what makes a second accept a no-op instead of a stale refusal. This refuses exactly when it should. Editing an unrelated test, adding a line above, or reformatting a different call does not invalidate the record. Editing the expectation itself, or adding an `expect()` call above it in the same function, does. The second case is a false refusal we accept, because it is cheap to fix and a false accept is not. We do not hash the whole file. A whole-file hash would refuse every accept after any edit, and agents edit tests constantly. ## Conflicting observations One call site can run many times: in a loop, in a parametrized test, from a helper. The records for one site are grouped at read time. - If every observation has the same value, it is a single change. - If the values differ, the status is `conflict`. Accept refuses and lists the distinct values with their test ids. The fix is a separate `expect()` per case, or `expect(value, expected, key=param)` in a later milestone. - Values are equal when `norm` says so, the same equality `diff` uses, so `1` and `1.0` agree. - Calls that disagree on `arg_sha` are a conflict too, because the file changed during the run. So are calls with equal actual values but different expected values, which happens when the expected argument depends on a parameter. - If any call recorded no actual value (unsupported, or over the size cap), the group is `unsupported` and cannot be accepted. - Unlocatable observations group by path, line, and reason. They are shown with status `unlocatable` and never accepted. `xpt.group.group(observations)` does this. It is pure, and its result is sorted by path, line, and ordinal, so the input order does not matter. expecttest detects the same case within one process. We detect it across processes and across xdist workers, because the grouping happens after recording. ## Lifecycle ``` pytest run expect(actual, expected?) compare -> pass: nothing written -> fail: Observation appended to .xpt/runs//.jsonl session end: with --expect-create, accept the run's missing groups (as xpt accept --missing) one block per failure, a one-line footer with the next command, pytest's exit code xpt failures every group of the latest run, each with its diff xpt show one group with its function and tests, paths only, with limits xpt show --full no limits, plus both whole values xpt accept ... [--dry-run] xpt accept --missing fill every missing expectation, nothing else xpt accept --all every group not rejected; those that cannot be accepted are reported with their reason xpt runs the recorded runs, newest first xpt clean delete .xpt/ xpt reject ... record each group's actual value as a bug, never to be offered ``` Every command takes `--json`, and `--root PATH` for a project other than the current directory. `failures`, `show`, `accept`, and `reject` take `--run ID` to read an older run instead of the latest. `python -m xpt` runs the same command. Exit codes: - `0`: the command did what it was asked, or there is nothing to do. - `1`: `failures` found failures, or `accept` or `reject` left a requested group stale or skipped. - `2`: a usage error, an unknown id or run, no `.xpt/latest`, a malformed record or decision, or a held `.xpt/accept.lock`. The error is one line on stderr that says what to do next, and nothing is written. Accept is a state machine per observation group: `recorded -> accepted | rejected | stale`. A group moves to `stale` when its fingerprint check fails. It moves to `rejected` when `xpt reject` records its actual value as a bug. ### Rejection `xpt reject ...` is for an agent or a person who has read a diff and decided the actual value is a regression, not a new expectation. Without it, every later `xpt failures` and pytest footer suggests `xpt accept` for the same bug again, and an agent that follows the footer accepts it. A rejection is one line in `.xpt/decisions.jsonl`: ```json {"schema_version": 1, "decision": "reject", "run": "20260101T120000-ab12", "site": {"path": "tests/test_parser.py", "qualname": "test_parser", "callee": "expect", "ordinal": 0}, "actual_sha": "<64 hex>"} ``` - It names the call site by its identity, the same four fields the display id hashes, so it survives line moves like the id does. `run` records where the decision came from and is not used to match. - It names the value by `actual_sha`, the sha256 of its canonical JSON (sorted keys, no spaces). It stores no value, so the file holds no test data and is never pruned. - A group is `rejected` when it is acceptable, has one actual value, and some line matches its site and that value's hash. Conflicts, unsupported values, and unlocatable sites have no single value to reject, so `reject` reports them `skipped` with the group's detail. - A rejection does not expire by time. It stops matching as soon as a run records a different actual value at that site, and that group is offered again. If the same wrong value comes back later, it is still marked rejected. `xpt clean` deletes the file with the rest of `.xpt/`. - Rejecting a value twice writes nothing the second time and reports `already_rejected`, so `reject` is idempotent. - `reject` rewrites the file whole to a temp file and moves it into place with `os.replace`, under the same `.xpt/accept.lock` as accept. A reader never sees a partial line, and two writers cannot drop each other's lines. A second writer fails on the held lock with exit code 2. `xpt failures` marks rejected groups instead of hiding them. The test still fails, and pytest still reports it, so a group missing from `failures` would leave an agent hunting for it. The heading reads `a81f tests/test_parser.py:14 mismatch rejected 1 test`, the JSON group carries `"rejected": true`, and the footer counts them (`1 acceptable, 1 rejected`) and leaves them out of `Next: xpt accept`. `acceptable` keeps its meaning, that the status allows a write. `xpt show` says the group was rejected and how to accept it anyway. The pytest report adds `rejected` after the status and leaves the group out of its footer. `accept --all` and `accept --missing` leave rejected groups out of their selection, as `--missing` leaves out mismatches. Accepting one by its id still works, because naming an id is a deliberate decision that overrides the earlier one. Accept is idempotent. Running it twice leaves the file as it was after the first run, because the second run finds the fingerprint already matches the accepted value and has nothing to do. `accept` without arguments does nothing and says so. Replacing existing expectations needs ids or `--all`. `--missing` fills only empty expectations, because creating one does not overwrite a reviewed value. The design first named the bulk flag `--all-mismatches` so that typing it would be deliberate. `--all` is just as explicit, also fills missing values, and is the flag agents reach for. `xpt failures` prints every diff before the footer suggests an accept, so the change is on screen when it is accepted. ## Persistence ``` .xpt/ .gitignore # "*", written on first use runs//.jsonl latest # the run id of the newest complete run decisions.jsonl # reject decisions, one per line (see Rejection) ``` - Each pytest process, controller or xdist worker, appends to its own file. Nothing is shared, so nothing is locked. This follows "separate before serializing shared state." - The controller writes `latest` at session end with `os.replace`, so a crashed run never becomes the latest. - Old runs are pruned to the last 5 by default. - Each line is one JSON object with `"schema_version": 1`. A value that is absent (`expected` when missing) is left out of the object, so a recorded `None` stays distinct from no value. Reading a run validates every line and names the file and line of a malformed one. - A process with no driver, such as plain unittest or a script, records under the current directory and writes `latest` from an `atexit` hook. Under xdist, the controller hands its run id to each worker. - A record over 1 MB keeps its status and site but drops both values. Its detail holds the record's size and the sha256 of the value. - `XPT_NO_RECORD` set to any non-empty value turns recording off, like `--expect-no-record`. - `XPT_REPORT` set to `pytest` keeps pytest's terminal output, like `--xpt-report=pytest`. - If a record cannot be written, the test still fails with its own message, plus one line naming the I/O error. - Source writes go to a temp file in the same directory, then `os.replace`, keeping the file's mode. Every accepted change to one file is applied in a single parse and write, last offset first. The new text must compile before it is written. ## Writing the expected value - A missing expectation becomes the second positional argument. A mismatch replaces the expected argument where it is, positional or `expected=`. Other arguments, such as `_depth=1`, stay where they are. - The whole call is reprinted the way `ruff format` would at line width 100. It stays on one line if it fits, then splits to one indented line of arguments, then one argument per line with a trailing comma. A magic trailing comma or a multi-line string forces the last form. - A call with a comment, `*args`, or `**kwargs` inside it is not reprinted. Only the expected argument is written, so the comment survives. The result is valid Python but may need a `ruff format`. - The rest of the file keeps its bytes. Offsets are UTF-8 byte offsets, so non-ASCII text before a call does not shift the edit. The file keeps its newline style and its byte order mark. - Files that are not UTF-8, by declaration or by content, are refused. - A dataclass value is written by its class name only when the file binds that name at module level, by import, class, or assignment. Otherwise the group is skipped and the reason says which import is missing. - Groups that are not acceptable are skipped with their `detail`, without reading the file. SQLite would buy queries we do not need. It would also make concurrent writers our problem. JSON Lines is readable with `jq` and easy to delete. ## Serialization, comparison, and source generation These are separate functions over the `Value` tree: - `serialize(obj) -> Value` turns a Python object into a tree. It raises `UnsupportedValue` with the path and type for anything it cannot represent: `result["handler"] is a function; xpt cannot record it. Register an adapter or expect a field of it instead.` It detects cycles by `id()`. - `diff(expected, actual) -> [Change]` walks two trees. Dicts compare by key, sequences by index, sets as sorted members. The walk is pure and easy to test exhaustively. - `to_source(Value, indent, width) -> str` prints a tree as a Python literal. Dict order is kept as observed. Set members are sorted. Floats use `repr`, which round-trips. It fits on one line when possible, and otherwise uses one element per line with a trailing comma, so the output already matches ruff's format. Comparison is Python `==` on the reconstructed expected value, with `diff` used only to explain a failure. That keeps xpt's results identical to what a human writing `assert actual == expected` would get. A dataclass prints as `Point(x=1, y=2)`. At record time we check that `Point` resolves to the same class in the test module's globals. If it does not, we mark the value unsupported instead of writing code that will raise `NameError`. Adapters are a small registry: ```python @register(numpy.ndarray) class NDArray: def serialize(self, value) -> Value: ... def to_source(self, value: Value, ctx) -> str: ... def diff(self, expected, actual, *, atol, rtol) -> list[Change]: ... ``` There is no `supports()` method and no `compare()`. The registry looks up types through the MRO. Comparison stays `==`, unless the adapter defines tolerance-aware `diff`. The adapters for NumPy, pandas, PyTorch, and Pydantic import lazily and live behind extras. Acceptance never runs code from the record. It builds source text from the `Value` tree. It never calls `eval`, `pickle`, or the observed object's `__repr__`. ## Reporting When xpt is installed, it replaces pytest's terminal output. Agents read every byte of a test run, and pytest's header, progress dots, rules, and full tracebacks spend tokens without saying more. The report stays readable for people, so it is the default. `--xpt-report=pytest`, or `XPT_REPORT=pytest`, keeps pytest's own output. ``` FAILED tests/test_parser.py::test_parser tests/test_parser.py:14 > expect(parse(source), tree) a81f mismatch value.operator: "addition" -> "subtraction" 1 changed, 42 unchanged FAILED tests/test_api.py::test_totals tests/test_api.py:30 > assert totals(order) == {"total": 42, "items": 3, "tax": 0, "due": 42} value["total"]: left 41, right 42 1 changed, 3 unchanged FAILED tests/test_io.py::test_load tests/test_io.py:8 > src/app/io.py:12 read_config > return json.loads(text) json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) caused by: FileNotFoundError: [Errno 2] No such file or directory: 'app.toml' FAILED tests/test_math.py::test_double[0] tests/test_math.py:22 > assert double(x) == x + 2 AssertionError: assert (0 * 2) == (0 + 2) stdout: debug line 0 also fails: tests/test_math.py::test_double[1] assert (1 * 2) == (1 + 2) also fails: tests/test_math.py::test_double[3] assert (3 * 2) == (3 + 2) 6 failed, 2 passed in 0.31s. Next: xpt accept a81f ``` - Each failure is one block. The header names the test and the deepest frame in a project file, outside `site-packages` and pytest. ` in ` follows when that frame is not the test function. When the error is raised in a helper, the header shows the chain of project frames from the test down instead, as `path:line > path:line function`, keeping the first frame and the three nearest the error. The test's own name is left out because the node id has it. The next line is the source line at the deepest frame, after `> `. Setup and teardown errors say `ERROR` and name the phase. Collection errors say `ERROR (collect)` and keep pytest's `E` lines. - An `expect()` failure shows its short id, its status, and the path diff. - A plain `assert left == right` on values xpt can serialize gets the same path diff, from `pytest_assertrepr_compare`, with each side labeled: `left 41, right 42`. It has no arrow, because `expected -> actual` would read backwards for `assert actual == expected`. Scalars, unsupported values, and other operators keep pytest's explanation. - Any other exception is one line, `Type: message`, cut to 200 characters. An exception raised from another, with `raise X from Y` or inside an `except` block, adds `caused by: Type: message`. `raise X from None` adds nothing. - Parametrized cases that fail at the same line with the same kind of error fold into the first case. Each other case gets an `also fails:` line with its full node id, which can be pasted into a pytest command, and the first detail line where it differs from the first case, without the shared exception type. The default limit of 10 applies, and the rest are counted. - A failing test prints the last 5 lines of its captured stdout, stderr, and log. - The footer is pytest's counts, warnings included, and duration, plus `Next: xpt accept ` when recorded expect failures can be accepted and were not rejected, or `Next: xpt failures` when none can. With `--expect-create`, it first says `Filled N missing expectations: path:line ...`, and `Next:` covers only what was not written. When every expect failure was written, it says `Next: run the tests.` With `--xpt-report=pytest`, the same `Filled` line prints after pytest's summary. - `-v` raises the limits to 50 changes, 400 characters per value, 20 output lines, and 1000 characters per message. `-vv` lifts them and adds pytest's traceback. `-q` prints the same compact report. - `-v` and `-vv` also keep pytest's line per test. `-r` with anything but the default `fE` keeps pytest's short summary beside the report, and `-rP`/`-rA` keep its PASSES section. `--no-summary` prints only the footer. `--no-header` and `-q` drop the plugin header lines. - Everything else pytest prints still prints: the warnings summary, plugin header lines and terminal summaries, `--durations`, `-x` stops, keyboard interrupts, interrupted collection, `--lf` messages, `--setup-show`, live logs, the debugger, and xdist's worker status lines. Exit codes are pytest's. Under xdist the controller prints one report. - `--tb` other than `auto`, `-l`, and `--full-trace` ask for pytest's failure sections, so those runs keep pytest's report. So do runs that don't run tests: `--co`, `--fixtures`, `--fixtures-per-test`, `--markers`, `--setup-only`, `--setup-plan`, `--help`, and `--version`. `--xpt-json` with one of these is a usage error, because there is no xpt report to write. - `--xpt-json PATH` also writes the report as one JSON object with `"schema_version": 1`, `exit_code`, `summary`, `counts`, `duration`, `run`, `created`, and `failures`. `created` is null without `--expect-create`. With it, `created` holds `outcomes` and `changed` in the same form as `xpt accept --json`. A held `.xpt/accept.lock` skips every missing group with the lock's message instead of failing the session. Each failure has its test, phase, location, `callers`, `source`, `cause`, `detail`, captured `output`, and full `traceback`, with no limits. An expect failure adds its `group` in the same form as `xpt failures --json`, and a compared assert adds `changes` and `unchanged`. A compared assert's changes name their sides `left` and `right`, as the text does. `op` reads from left to right, so `add` is a value only the right side has. The expect limits are 10 changes per expectation and 80 characters per value, cut with a length marker. `xpt show --full` lifts both. `xpt failures` prints the same block per group, a one-line footer with the counts, and the next command: ``` a81f tests/test_parser.py:14 mismatch 2 tests value.operator: "addition" -> "subtraction" 1 changed, 42 unchanged 1 failure in run 20260101T120000-ab12 (1 mismatch), 1 acceptable. Next: xpt accept a81f ``` There is no color and no banner. `--json` prints one JSON object on stdout with `"schema_version": 1` and `"command"`. Values use the recorded JSON form, and `expected` is left out when there is none, so a recorded `None` stays distinct. A mismatch adds `changes` (path, op, expected, actual) and `unchanged`. Accept adds `outcomes` and `changed`. Reject adds `outcomes`. Each group carries `rejected`. The CLI handlers are thin wrappers over `xpt.api`, which re-exports `load_run`, `group`, `accept`, `reject`, `load_rejections`, `mark`, `diff`, and their data types, so an MCP server later is another thin wrapper and not a redesign. ### How the report fits into pytest The report has to work on every pytest a project might pin, and it can't take away anything pytest prints. Three designs were compared on pytest 8.0 through 9.2: - **A separate reporter of our own**, registered as `terminalreporter` in place of pytest's. pytest's reporter implements about 16 hooks, and the set changes between releases: 8.0 has `pytest_runtest_logfinish`, 9.2 drops it and adds a `pytest_runtestloop` wrapper. Exit messages, `--max-warnings`, `--lf` messages, keyboard interrupts, and the warnings summary all live in those hooks. Plugins also read the reporter's attributes: `_tw` (the debugger), `stats`, `write_sep`, `section`, `write_line`, `rewrite`, `ensure_newline`, `hasmarkup` and `isatty` (xdist), `showfspath`, `startpath`, and `reported_progress`, and `config.get_terminal_writer()` returns `_tw`. A reporter of our own would have to reimplement all of that and follow every release. - **A wrapper that delegates to pytest's reporter.** It inherits nothing, so it needs the same list, kept by hand. - **pytest's reporter, subclassed in place** (chosen). At configure time xpt changes the registered reporter's class to `AgentReporter`, a subclass, and registers the same object again so pluggy sees the new hooks. Objects that already hold the reporter, such as xdist's `DSession.terminal`, keep it. The subclass overrides only methods pytest names publicly: `summary_failures`, `summary_errors`, `short_test_summary`, `summary_stats`, `report_collect`, `pytest_collection`, the per-test log hooks, `showheader`, and a `pytest_report_teststatus` wrapper that blanks the progress letter while keeping the category, so `stats` stay pytest's. Every other hook and attribute is pytest's own code for that release, so there is no list of pytest's surface to maintain. The subclass depends only on those method names, which are the same across the tested range. `pytest.TerminalReporter` is public from 8.4, and older releases import it from `_pytest.terminal`. 8.0 is the floor because the report uses `Config.get_verbosity`. On an older pytest the plugin prints pytest's report and a one-line notice. If the swap raises, xpt puts pytest's class back, registers it again if needed, and prints a notice. If rendering raises, `summary_stats` prints pytest's failure sections, short summary, and final line, then the notice. Neither path changes the exit code. pytest-sugar replaces the reporter with its own class before xpt's `trylast` configure runs, so xpt sees a reporter that isn't pytest's and stands aside. `tests/test_report_compat.py` holds this to account. It runs a suite with every outcome, a plugin that writes through the reporter, and 32 option sets, once with `--xpt-report=pytest` and once with the xpt report, and asserts equal exit codes, equal stderr, and that every stock line outside the replaced sections also appears in the xpt run. CI runs the whole test suite on each pytest minor release from 8.0 and on the oldest and newest supported pytest-xdist, and the compatibility tests with pytest-sugar installed. Known gaps: pytest 9.2's OSC 9;4 terminal progress indicator stays at 0%, because the report doesn't advance `reported_progress`. Behavior on a real terminal, where pytest rewrites lines, is tested only through the same code paths a pipe uses. ## Concurrency - xdist: workers record and never write source. The controller sees every worker's file at read time. No IPC is needed. `--expect-create` runs in the controller's `pytest_sessionfinish`, after every worker has finished. - Two pytest sessions at once in one project each get their own run directory. `latest` names whichever finished last. That is acceptable, and `--run ` selects another. - Accept while pytest runs: accept creates `.xpt/accept.lock` with `O_CREAT | O_EXCL`, which behaves the same on POSIX and Windows, and deletes it when done. A second accept fails with a message naming the lock file. A crash can leave the file behind, so the message says to delete it if no accept is running. A test run racing an accept is caught by the fingerprint check on the next accept. `reject` and `clean` take the same lock, so the error says another xpt command is writing to `.xpt/`. ## Risks - **Rubber-stamping.** The tool makes accepting cheap, so agents will accept regressions. Mitigations: no bulk accept by default, the diff shown at accept time, and `--dry-run`. The benchmark (M5) measures how often wrong accepts happen. We do not claim to prevent them. - **Dynamic call sites.** `expect` called through a helper reports the helper's frame. We support `expect(..., _depth=1)` the way expecttest supports `skip`. A call created through `functools.partial` or `exec` has no source to rewrite. We record it as unsupported for rewriting, and the comparison still runs. - **Generated or non-file sources** (doctests, notebooks, `python -c`): same as above. - **Floats and nondeterminism.** Dict order, set order, timestamps, ids, and memory addresses make snapshots flaky. Sorting covers sets. `ignore=` and adapters cover the rest. Default `repr` of arbitrary objects is never used, because it embeds addresses. - **Large values.** Records are capped at 1 MB per observation by default. Larger values are recorded as a hash plus a summary and cannot be inline-accepted. - **Sensitive data.** `.xpt/` holds real values from test runs. It is gitignored on creation, pruned to 5 runs, and removed by `xpt clean`. The README says so plainly. - **Python 3.11 floor** excludes 3.10 users. 3.10 is end of life, so this is acceptable. - **`co_positions` can be disabled** with `-X no_debug_ranges`. We detect `None` positions and fall back to line-only matching, refusing when a line holds more than one call. ## Minimum public API ```python from xpt import expect expect(actual) # missing: fails, records expect(actual, expected) # compares with ==, records on mismatch ``` Later, as keyword-only arguments: `ignore=`, `unordered=`, `atol=`, `rtol=`, `key=`. These stay in the source, so a reviewer can see them. pytest plugin options: `--expect-create`, `--expect-no-record`, `--xpt-report=xpt|pytest`, and `--xpt-json PATH`. unittest works with no plugin, because `expect` raises `AssertionError` and records through the same process-level recorder. Its test id falls back to the qualname. ## Roadmap Each milestone ships on its own and is tracked as GitHub issues. - **M0. Scaffold.** uv, ruff, ty, pytest, CI on 3.11 to 3.14, MIT license. - **M1. Core assertion.** `Value`, `serialize`, `diff`, `to_source` for builtins and dataclasses. `expect()` compares and raises with the compact message. No recording yet. - **M2. Record and accept.** Call-site identity, JSON Lines recorder, `failures` / `show` / `accept` / `reject` / `clean`, fingerprint checks, atomic multi-edit writes, `--expect-create`. This is the differentiator and the MVP line. - **M3. Reporting polish and comparison options.** pytest longrepr replacement, limits, `ignore=`, `unordered=`, `key=` for parametrized tests. - **M4. Adapters and xdist.** NumPy, pandas, PyTorch, Pydantic extras with tolerances. An xdist test matrix. - **M5. Agent benchmark.** Fixture repos with seeded regressions. Claude Code and Codex run headless against plain asserts, expecttest, inline-snapshot, and xpt. We measure tokens, tool calls, test runs, wall time, wrong accepts, and final correctness. - **M6. Release.** Name check, docs, PyPI. # Prior art notes Read from source on 2026-11-19. Paths are relative to each repository root. | Project | Commit | License | |---|---|---| | pytorch/expecttest | 65d0976 | MIT | | 15r10nk/inline-snapshot | 4542e42 | MIT | | tophat/syrupy | 098dd11 | MIT | | approvals/ApprovalTests.Python | 32ccbd1 | Apache-2.0 | ## expecttest - One 465-line module, `expecttest/__init__.py`, with no dependencies. Requires Python 3.8.1 or later. - `assert_expected_inline(actual: str, expect: str, skip=0)` compares strings and prints a unified diff on failure. - Setting `EXPECTTEST_ACCEPT=1` rewrites the file during the run. It finds the call through `traceback.extract_stack`, which gives a line number only. It then uses an `ast.walk` to widen that line to the full statement, and a regex to replace the first triple-quoted literal in the range (`RE_EXPECT`, `replace_string_literal`). - An `EditHistory` adjusts line numbers after each in-process edit. It also catches one line accepted twice with different values ("Are you running a parametrized test?"). - It writes `.bak` before the first edit to a file. - Limitations: - Values must be strings. - The expectation must be a triple-quoted literal. - Two calls on one line are ambiguous. - Source is edited during the run. - Nothing persists after the process exits. ## inline-snapshot - API: `snapshot()`, `external`, `outsource`, `Is`, `HasRepr`, `customize` hooks (`src/inline_snapshot/__init__.py:23-45`). It supports `==`, `<=`, `>=`, `in`, and `snapshot()[key]` (`docs/categories.md:31-39`). - Finding the call: frame inspection plus `executing`, keyed on `(id(frame.f_code), frame.f_lasti)` (`_core/_inline_snapshot.py:33-50`, `_core/_types.py:18-19`). - Rewriting: `Replacement(range, text)` edits applied through `asttokens` offsets, with a check that edits don't overlap (`_core/_rewrite_code.py:49-53,146-154,176-188`). - The whole file is reformatted only when it was already black-clean, or when `format-command` is set (`_rewrite_code.py:166-191`). - Black is an optional extra. - Code generation: - A pluggy `customize(value, builder, local_vars, global_vars, snapshot_value)` hook (`_core/plugin/_spec.py:28-36`). - Many built-in types are handled (`_core/plugin/_default_plugin.py:33-290`). - Anything else falls back to `repr()`, wrapped in `HasRepr` when the repr does not parse (`_core/_customize/_builder.py:69-74`). - Categories: create, fix, update, trim, and fix-assert (insider only). - Defaults are `create,review` in a terminal and `report` otherwise. - In CI it acts as `disable` (`pytest_plugin.py:175-177`, `_snapshot_session.py:50-73`). - Review is one `Confirm.ask` per category, not per change (`_snapshot_session.py:103-118`). - Changes are applied in `pytest_sessionfinish` of the same process (`pytest_plugin.py:200-241`). Proposed changes are not persisted. - No stale-source detection was found. Offsets computed from the earlier parse are applied to the file re-read at write time (`_rewrite_code.py:156-188`). - Failure output is the normal pytest assert diff, plus a unified source diff in rich panels. No JSON output was found. - xdist: inline-snapshot turns itself off (`docs/limitations.md:6-16`). - Runtime dependencies: asttokens, executing, rich, pytest, typing-extensions, and tomli before 3.11. Python 3.9 or later. Updates work on CPython only. No unittest support. - Loops that produce different values fail even under `create`. The workaround is `Is()` or `snapshot()[key]` (`docs/eq_snapshot.md:266-288`). ## syrupy - Snapshots are external `.ambr` files under `__snapshots__/`, sorted by name and written atomically (`extensions/amber/serializer.py:116-120,187-217`). "Syrupy does not support inline snapshots" (`README.md:587`). - Extensions: - Whole-format extensions through `use_extension`. - Per-type serializer plugins (`serializer.py:81-95,366-370`). - Dict keys, sets, and dataclass fields are sorted. Cycles and a depth limit of 99 print as `...` (`serializer.py:113,341-345`). - Matchers: - `path_type`, `path_value`, and `compose_matchers` (`matchers.py:38-116`). - Filters `paths`, `paths_include`, and `props` (`filters.py`). - No numeric tolerance was found. - `--snapshot-update` rewrites snapshots during the run and deletes unused ones unless told not to (`session.py:463-511`). `--snapshot-update-new-only` exists. There is no review step and no deferred accept. - Failure output is a line text diff with collapsed context. No JSON. - xdist: `O_EXCL` lock files with a 60-second timeout. Worker reports are merged through `workeroutput` (`utils.py:124,222`, `session.py:301-344`). - Depends on pytest only. Python 3.11 or later. ## ApprovalTests.Python - `.received.` is written next to `.approved.`. A matching received file is deleted. A differing one stays (`file_approver.py:112-122`). - Inline mode keeps the expected value in the test's docstring (`namer/inline_comparator.py:17-36`). On a mismatch it writes a full copy of the test file with the new docstring (`namer/inline_python_reporter.py:85-129`). - Approving later without a rerun is possible. Failures are logged to `.approval_tests_temp/.failed_comparison.log`. `ReporterThatCreatesAnApprovalScript` appends `mv received approved` lines to a script (`reporters/reporter_that_creates_an_approval_script.py:22-41`). - There is no stale detection. - There are no per-change list, show, or reject commands. - It downloads helper scripts from GitHub at runtime unless `APPROVALTESTS_DISABLE_SCRIPT_DOWNLOADS=1` is set (`log_commons.py:11-33`). - Scrubbers are regex replacements to text (`scrubbers/scrubbers.py:12-76`). There are no path ignores and no tolerances. - No xdist handling. No JSON output. ## Prototype: call-site location without dependencies `docs/prototypes/call_site.py` and `docs/prototypes/test_call_site_demo.py` were run on CPython 3.12 under pytest, with assertion rewriting on: ```bash cd docs/prototypes && PYTHONPATH=. uv run --python 3.12 --with pytest pytest -p no:cacheprovider test_call_site_demo.py ``` Two tests fail on purpose, because they contain mismatches. Observations go to `observations.jsonl`. Results: - `code.co_positions()[frame.f_lasti // 2]` matched exactly one `ast.Call` for every call. This held with two calls on one line, a multiline call with comments, a call after non-ASCII text on the same line, and a call inside a method in a loop. - Splicing at UTF-8 byte offsets changed only the expected argument. Comments and the rest of each line were unchanged, and the result passed `compile()`. - The loop recorded two observations at the same position with values `0` and `1`, so conflicts can be found by grouping records. - Not yet covered: `-X no_debug_ranges`, decorators, `async def`, nested functions, lambdas, and calls through helpers.