Aontu

Explanation: how and why Aontu works

Rendered from docs/explanation.md in the engine repository — where a correction belongs, and where the test suite executes every example on this page.

This document is discursive. It explains the ideas behind Aontu and the shape of the implementation, and argues some of the trade-offs. It is the place to build a mental model; for precise rules use the Language reference, and for recipes the How-to guides.

The core idea: one operation, three jobs

Most configuration stacks use three different mechanisms: a schema language to say what is allowed, a defaults mechanism to fill gaps, and a merge/override step to layer environments. Aontu — following CUE — collapses all three into a single operation, unification, by making types, defaults, and data the same kind of thing: values in a lattice.

  • A schema like port: integer is just a value (the set of all integers).
  • A default like port: *8080 is just a value (a preference annotation on 8080).
  • Data like port: 9090 is just a value.

Combining any of them is the same act: take the greatest lower bound of the two values — the most general value that is at least as specific as both. When that bound exists you get a result that honours every input at once; when it does not, you get a precise error. There is no precedence to remember and no order dependence: a & b always equals b & a, and re-applying a fact changes nothing (a & a == a).

This is why the same document can be schema, defaults, and data simultaneously, and why merging two configurations can never silently pick a winner — it either narrows to a consistent answer or fails loudly.

The lattice

Values are ordered from general to specific:

top  ⊐  string ⊐ "ada"
     ⊐  number ⊐ integer ⊐ 1
     ⊐  boolean ⊐ true
                         ⊐ … ⊐  nil (⊥)
  • top sits above everything and is the unit element: x & top == x. An unconstrained field is top.
  • nil sits below everything; it is what you get when two values have no common lower bound. nil carries a message and poisons generation.
  • Unification walks down the lattice. number & integer → integer (more specific); integer & 1 → 1; 1 & 2 → nil (no value is both).

Disjunction (|) builds a small lattice of alternatives; unifying a concrete value against it keeps the branches that survive and discards the rest. Preference (*) is a tie-breaker annotation that says “if the choice is otherwise unforced, pick me”.

The pipeline

Both implementations share the same three-stage pipeline:

source text ──parse──▶ Val AST ──unify (fixpoint)──▶ unified Val ──generate──▶ native value
                                                              ├──canon──▶ source-like text
                                                              └──ask────▶ vet · get · why · subsume · hash

  1. Parse. @tabnas/jsonic plus the expr, path, multisource, and directive plugins turn relaxed JSON-with-operators into a tree of Val nodes — MapVal, ConjunctVal, RefVal, ScalarKindVal, and so on. Operators like &, |, *, ., + are configured as parser expression operators with explicit binding powers, which is how precedence is defined (see ts/src/lang.ts).

  2. Unify. A fixpoint loop repeatedly unifies the tree with top until it stops changing or an error appears. Each Val subclass knows how to unify itself with a peer.

  3. Generate / canon. A converged tree is either emitted as a native value (gen) or rendered as canonical source (canon).

The third arrow is not a fourth stage but a fan-out. vet, get, why, subsume and hash all interrogate the same converged tree that generate renders, which is why they agree with one another by construction rather than by care, and why none of them can be a partial evaluation. What they are for is argued below, in Why there is a verb surface.

Why a fixpoint, not a single pass

References make a single pass insufficient. Consider:

a: { v: $.b.v }
b: { v: $.c.v }
c: { v: 99 }

On the first pass $.b.v resolves to another reference ($.c.v), which only resolves to 99 once c has settled. So unification runs in rounds: each pass refreshes the root and re-resolves references against the latest tree, and the loop ends when every node reports “done” (dc == DONE) or an error is collected. The loop is bounded (a small maximum pass count) so a pathological model terminates rather than spinning.

The dispatch ladder

The heart of the engine is a binary unite(a, b) function (see ts/src/unify.ts and go/unify.go). It is a careful ladder of cases:

  • degenerate/top cases first (unit element);
  • nil short-circuits (bottom is absorbing);
  • complex values that know how to “absorb” a peer — conjunction, disjunction, reference, preference, function — drive their own unify;
  • otherwise the two concrete values are matched directly (equal scalars collapse; mismatches become nil).

Each Val type implements only the cases it understands and defers the rest by unifying with top. This keeps the type-specific logic local: MapVal.unify knows about keys and spreads, DisjunctVal.unify knows about trying alternatives, ScalarKindVal.unify knows that number subsumes integer, and none of them needs to know about the others.

Distribution and trials

Conjunction distributes over disjunction: x & (a | b) tries x against each alternative independently and keeps the survivors. The implementation runs each alternative as a trial with its own throwaway error bucket — if the trial collects an error, that branch is dropped. The TypeScript version optimises this hot path with a shared “trial nil” sentinel and save/restore of the error array instead of cloning a context per alternative, because schemas with many disjunctions (GET | PUT | POST | …) make this the busiest path in the engine.

Immutability

A foundational rule, stated right on the base class: unify must not mutate its operands. Unification returns a new value; the inputs are left intact. This is what makes order-independence and the fixpoint loop sound — a value can be unified many times, against many peers, across many passes, and shared structurally between branches, without one unification corrupting another. Cloning carries a value to a new path (references resolve relative to where a value is), but the original is never altered in place.

Marks: separating schema from data

Two boolean marks ride along with every value: type and hide. They do not change what a value unifies to; they change whether it is emitted. A type-marked field is schema and a hide-marked field is a working value — both are omitted when their enclosing map is generated, yet both still constrain unification. This is how a single document can carry its own schema inline without that schema leaking into the output, and why copy() (which clears the marks) is the way to turn a schema node back into emittable data.

Why there is a verb surface

A definition that can only be evaluated answers exactly one question: what is the value? That is enough when a person reads the answer and a program consumes it. It stops being enough when the thing on the other end is writing the document as well as reading it, because then the interesting questions are all the other ones. Does this data hold against that definition, and where does it not? What does it say at this one path? Why does it say that — who wrote the value that made it so? Has its meaning changed since the pin I recorded? Can I change it without breaking somebody downstream?

Every one of those is already answerable from the converged tree. The source sites are on the nodes, the contributions met at known paths, the canon is a deterministic rendering. Not exposing them does not make the questions go away; it makes every consumer re-derive them from generated JSON, badly and out of band. A definition that can validate, be queried, explain itself and be diffed is not really a document any more — it is ground truth that something else can act on, and the verbs are what turn one into the other. The roster is in the API reference; what follows is why it has the shape it has.

The emit → validate → repair loop

One loop explains most of the surface. Something writes a document, vet says what does not hold and where, the author — human or otherwise — repairs it and goes round again. Taking that loop seriously decides several things that would otherwise be arbitrary.

Exit codes are verdict classes, not a pass/fail bit, because the three ways to fail call for three different next moves. A contradiction (invalid) means repair what you emitted. An unsatisfied truth (incomplete) means keep writing — nothing is wrong yet, the document is merely unfinished. An unusable schema (error) means stop: the fault is not the data’s, and another round of repair cannot reach it. Collapsing those into “failed” discards exactly the bit the repairing end needs in order to choose. For the same reason a finding labels its two sites by provenance instead of by source order, and puts the data’s site first: that is the one you are meant to edit.

why is the positive twin of a failure report. An error explains what did not unify; why explains what did, listing the values that met at a path with the site each was written at, in source order. It exists because “the value is 3” is not actionable and “a spread offered *1|integer here, and this line pinned 3” is. The same record is what the language server can append to a hover and what the MCP tool of that name returns: one answer, three ways in.

get buys the size of the answer, not the cost of it. Unification has no partial mode — the whole document converges or none of it does — so a query verb cannot be an optimisation, and it would be dishonest to present it as one. What it is instead is a way for a reader with a small question to receive a small answer, which for a consumer paying by the token is not a small thing. Its shape views are held to a stronger claim than “a summary”: each is itself a valid document that subsumes the truth, so a projection may generalise but may never mislead.

set appends by default, and that follows from the lattice rather than from a gap in the tooling. Because unification is order-independent, a change written into a second file is the same value as the same change written into the first — so an overlay needs no format-preserving rewriter, and cannot damage the document it is changing.

What appending cannot do is override a value that is already pinned: the lattice refuses, because unification only narrows. That was a real hole rather than a principled refusal, and an embarrassing one — the commonest validation failure of all is “the data says the wrong thing”, and the verb built for repair could only fill holes.

--in-place closes it, and the interesting part is what it turned out not to need. The rewriter was deferred for a long time behind a comment-preserving CST, on the reasonable-sounding grounds that you cannot put a document back together without one. But a CST is what you need to re-serialise a document, and replacing one value serialises nothing: it writes new text over a known span and leaves every other byte alone — every comment, every blank line, every alignment space — because it never reads them. The prerequisite was for a different job than the one that had to be done.

What it does need is the ability to say which bytes, and to be right. That is the second thing this cost more than expected. A site names the token it points at, not the value it belongs to, and for anything compound those are different: min(1) reports min, 1+2 reports 1, {b:1} reports {. Writing over those spans produces a: 5(1) and a: 5+2 — a corruption with the same shape as the one this whole line of work started from, arrived at by a different route.

The fix is worth stating because it generalises past this verb: rather than enumerate which shapes are safe — a list is a thing to be incomplete about — the candidate text is parsed on its own and required to mean what the contribution meant. The same unifier that produced the value decides whether the text is the whole of it, so the answer cannot drift from the engine, and the awkward case falls out without being named: 0x1F canons to 31, which is not its own spelling but is the contribution’s canon, so a hex literal is editable while min is not.

One case shows why that check is not the whole of the answer. Verification asks whether the text at the span is what the site claims, and there is a shape where it says yes and is still wrong: an included document holding a: 42 at row 1 column 4, and the overlay holding x: 42 at row 1 column 4. Same position, same text, so the check passes — and the splice rewrites x while reporting a replacement of $.a. What fails there is not the verification but the premise underneath it, that a position identifies a place in this file. So the evaluation that decides what to edit refuses to load at all, and the ambiguity never arises: what resolves is what the overlay says by itself. Denying the shape is worth more than detecting the collision, because a detector is a list of shapes, and the argument here is about what happens when the list turns out to be incomplete.

Rewriting is opt-in for the reason appending was preferred in the first place: appending is reversible in a way that overwriting is not, and the two failure modes are not symmetric. So the flag is asked for, never inferred; where the span cannot be established the assignment appends exactly as it would have without it, and says why. Refusing to edit is always available, and always safe. Editing the wrong bytes is neither.

Meaning, not bytes

hash answers “has this changed?” over what a document means rather than over the bytes it is stored in. Reformat it, reorder its keys, split it into three files pulled back together by @"…" includes, and the pin holds; flip one default and it moves. That is only possible because canon renders the converged tree rather than the source text.

The trade is real and taken deliberately. Canon is deterministic syntax, not a unique normal form, so two documents that denote the same set of values can still hash differently — number|integer and number are the standing example. The failure direction is the safe one: a spurious “changed” costs somebody a second look, whereas a spurious “unchanged” would ship the break — and the extra spellings the hash form carries over ordinary canon are there precisely to keep that second failure out of reach. Trading cheap false alarms for an assurance that cannot fail in the dangerous direction is a good bargain, and it is the same bargain subsume and breaking make one level up, where the question is not “did the meaning change?” but “did it change in a direction that hurts anyone downstream?” A check that answers undecided, and fails the gate for saying so, is honest; one that guesses “compatible” ships the break it was installed to catch.

The same answers, through other transports

The MCP server, the language server, the published grammar and the agent skill are not four more features; they are the argument above carried to callers that do not run a shell. The MCP tools return the identical JSON contract the CLI prints, and a tool that refuses — an invalid document, a path naming nothing — answers with its own report rather than a protocol error, because the report is the answer. The grammar published for constrained decoding accepts less than the parser does and never more, and leaves out @"…" includes entirely, on the view that a generated document should describe values rather than reach for files. Served evaluation is confined for the same reason rather than as a deployment option: a tool that has to remember to restrict itself is one that will eventually forget, silently. And the skill’s example documents are executed by the test suite, on the same principle that governs the rest of this repository — a teaching pack that taught something the engine no longer did would fail the build rather than mislead a reader quietly.

Closed-world validation is a dial

Schema languages usually take a global stance on unknown keys: JSON Schema is open until you write additionalProperties: false, protobuf is closed and you work around it. Aontu cannot take a global stance, because the same tree is schema and data at once and at different stages of completion. A half-written definition has to be allowed to be incomplete while the finished one beside it is allowed to be strict, and no single default serves both.

So closedness is a property of a node. close() seals one map or one list and open() lifts that seal, and the seal covers the node it was written on rather than everything beneath it: a map nested inside a closed map is still open, and so is a list. That is deliberate, and the alternative — a mark that travels further than it was written — is worse in a language where a subtree is routinely a template someone else will extend. aontu vet --closed is the same dial at the command line: it closes the anchor being validated, not the whole document, so “no keys I did not declare here” is a question you can ask without sealing everything else.

The cost is that closedness has to be written rather than assumed. An author who never reaches for close() is never told about a typo in a key, and the sealing has to be repeated at each tier that wants it. That is the bill for letting one notation carry both a finished definition and the half-written one beside it.

Two implementations, one behaviour

TypeScript is canonical; Go is a port kept in lock-step. Parity is not maintained by reading code side by side but by a shared, data-driven contract: the test/spec/*.tsv files. Each row is a name / mode / src / expect tuple, and both ts/test/spec.test.ts and go/spec_test.go load the same files and assert the same results — canonical form, generated JSON, or error substring.

Two things make this work in practice:

  1. The same parser stack. Both sides use the @tabnas family (jsonic + expr + path + multisource + directive), TypeScript natively and Go via the matching Go ports. Surface syntax therefore parses identically, so the spec can exercise real syntax rather than a lowest common denominator.

  2. A single source of truth for behaviour. A new behaviour is added to the canonical TypeScript implementation, captured as a spec row, then made to pass in Go. A row is only committed once both pass, so the spec always describes agreed, shared behaviour. Language-specific tests (the rich TypeScript *.test.ts suites, the Go-native sanity tests) live alongside the shared spec but never define cross-language behaviour on their own.

The Go port deliberately implements the subset that the spec covers (which is, today, the full surface language) and mirrors the TypeScript architecture closely — the same Val interface, the same unite ladder, the same fixpoint loop — so that a change on one side has an obvious counterpart on the other.

What the arrangement costs is worth naming, because it is paid on every change. A language feature is written twice, and a row is only committed once both engines produce it, so the cheapest possible change to behaviour is still two ports and a spec row in one commit; features are designed knowing that. What it buys is a claim no single implementation can make. Two independently written engines agreeing byte for byte across the whole shared suite is evidence about the specification, not about one codebase’s tests — and the suite becomes an unusually good detector of accidents, because an optimisation that quietly reorders a fold shows up as failing rows on whichever side moved. That friction is the mechanism working.

Where the meaning is ours

Parity is enforceable only while everything either port does is ours to fix. re() broke that assumption. A pattern is handed to a host subsystem — JavaScript’s RegExp on one side, RE2 on the other — and those are not two implementations of one specification; they are different languages, in different complexity classes, over different alphabets. \A is an anchor in one and a literal A in the other; \s is Unicode whitespace in one and ASCII in the other; one matches UTF-16 code units where the other matches code points. None of it can be fixed from this repository.

The first attempt was a blacklist — enumerate the constructs known to differ, refuse those, pass the rest through — and it leaked three times in a day. The instructive part is not that the list was short. It is that a blacklist’s correctness is a claim about the author’s knowledge of two large external systems: it decays silently as those systems evolve, and no test can falsify it. Two of the three leaks were found by reading and one while writing documentation. None by the suite.

So ADR-003 inverts it: where a host subsystem supplies semantics, Aontu defines the meaning and rewrites the input, and the host is given only constructs it cannot read two ways. \d is [0-9] because the ADR says so, not because the hosts happen to agree. What that costs is paid at the point of use — \s no longer means what a regex habit expects in either language, and an author has one more small thing to learn. The gain is that the guarantee stops depending on what the implementer happens to know, and becomes checkable instead: a committed corpus pins both normalisers, so a drift fails in whichever port drifted.

The rule is stated generally on purpose, because a date parser, a collation order or a number formatter would each inherit it. It also admits what it cannot close. Complexity is not a property of the pattern language — backtracking makes some patterns exponential where an automaton is linear — so no rewriting reaches it, and that axis is held by a syntactic restriction instead. The principled end state is to own the matcher, at which point there is no host subsystem left to normalise; that is recorded as the direction, not as a plan.

Performance shape

Unification is pointer-chasing over many small immutable nodes, run for several passes, so most of the engineering effort goes into not allocating. The TypeScript implementation in particular carries a number of deliberate optimisations, documented inline:

  • Type discriminators live on the prototype. Every Val answers isMap, isRef, … via prototype defaults; a subclass overrides only its own flags. This removes dozens of property writes (and the hidden class transitions they cause) from every node construction.
  • Lazy site and err. Source positions and error arrays are allocated on first use; the common node never pays for either, and all error-free nodes share one frozen empty array.
  • A path trie in the context. Cycle detection and reference resolution need a stable index per path; the context memoises (parent, key) → index and reuses the materialised path array across passes instead of re-concatenating it.
  • A per-parent descend cache. Visiting the same (parent, key) child context across fixpoint passes returns a cached context rather than a fresh Object.create.

None of these change behaviour — the shared spec guards that — but they are why the engine stays usable on realistically large models. The Go port keeps the same overall structure but, lacking references-with-cycles in its hottest paths, uses a simpler depth guard in place of the TypeScript seen-map.

Two rules that surprise readers

Both of the rules below are specified, pinned by shared rows, and — judging by how often they are written wrong — surprising. They belong here rather than in a bug list, because in each case the rule is defensible and the surprise is real, which is the shape of a trade-off rather than of a defect.

A preference is gated by kind, not by family

Overriding a scalar default is judged by the preferred value’s own kind: a concrete peer replaces it only where the peer is the same kind of thing. *8080 meeting 9090 yields 9090; *8080 meeting 3.5 is a conflict, and so is *2.2 meeting 3. A peer that merely constrains the default rather than replacing it — *1 & integer, *1 & number — leaves the preference standing, because it said nothing the preferred value did not already satisfy. The rule is stated plainly in the language reference, which also records the case it does not cover: a preferred map or list has no kind yardstick (superior() is top), so any peer overrides one. The gate is a scalar gate, and the reference says so rather than letting the general phrasing stand for it.

The surprise is that the two numeric leaves do not mix around a preference, even though they share a supertype:

a: *2.2 & 3         → refused: [aontu/no_scalar_unify] at $.a
a: *1.5 & integer   → refused: [aontu/scalar-type] at $.a

This is a choice, and the other one shipped for a while. A gate that widens to the numeric family lets *2.2 & 3 through, which reads as a kindness — an author should not have to know which numeric leaf they happened to write their default in. But that is one rule seen in one direction, and the other direction is the idiom that looks most like “a typed default”:

port: *8080 | integer   with a later  port: 1.5

Under a family gate that generates {"port":1.5}. The preference widens its branch to the base kind — number — so the integer the author believes they wrote is not the constraint that survives, and every key written the way the agent skill teaches it is quietly a number key. No kind-based gate can keep the convenience and refuse that; the leaf gate refuses both.

So the idiom now means what it looks like:

port: *8080 | integer
    alone                          → generates {"port":8080}
    with a later  port: 9090       → generates {"port":9090}
    with a later  port: 1.5        → refused: [aontu/|:empty] at $.port

and an author who wants the whole family writes it in the branch, where a reader can see it:

port: *8080 | number    with a later  port: 1.5  → generates {"port":1.5}

port: *8080 & integer is still not the way to write a default, tempting though it reads: a conjunction is not a choice, so the value is pinned at 8080 and 9090 is refused along with 1.5.

What the tightening costs is named rather than hidden: mixing the numeric leaves around a preference is now an error instead of a silent widening. *1.5 & integer used to answer integer and discard a default that could never apply; it now says so, and names the line that has to change. Only the numeric leaves were ever affected, because only they sit under a common supertype — *"us-east" | string meeting a later 42 was always an empty disjunction.

The rule is written down in test/spec/pref.tsv and test/spec/number-tower.tsv, and the suite pins both of its directions: the pref-kind-gate-* rows pin the cross-kind REFUSAL (*1 against a map, a string, a boolean, a list), pref-override-within-kind-gens pins the same-kind override that is the same rule from the other side, and the tower’s pref-idiom-refuses-other-leaf / pref-idiom-number-still-admits pair pins the numeric case that used to be the exception. Both ports agree on it byte for byte.

A list literal is positional

tags: [string] reads as “a list of strings” and is not one. A list literal is positional with an open tail: it constrains element 0 and says nothing about anything after it.

tags: [string]     with a later  tags: [core, 7]  → generates {"tags":["core",7]}
tags: [&: string]  with a later  tags: [core, 7]  → refused: [aontu/no_scalar_unify] at $.tags.1

The homogeneous form is the spread, [&: string], which applies its template to every element — the same &: that templates a map. That consistency is the defence: a list is a value like any other, so [a, b] meeting [c, d] element-wise is the reading that keeps & meaning one thing everywhere, and a bracket that silently meant “and all the rest, too” would be the exception. Closing the enclosing map does not close the tail either, because closedness is a mark on the node the author closed; close([string]) does close it, and refuses element 1 outright.

The surprise is nonetheless worth naming, because the two spellings differ by three characters and fail in opposite directions: the positional one accepts what it looks like it should refuse.

Termination is part of the offer

The surface argued above is meant to be driven by something that is not watching: a CI gate, a tool call, a repair loop several steps from anyone’s attention. For that caller, “usually finishes” is not a weaker version of “finishes” — it is a different product. A definition language that can loop is one you have to supervise, and a definition you have to supervise is not ground truth; it is a program you are running on faith.

That is the reason for the refusals listed below, rather than the other way round: they are not concessions to implementation difficulty, they are what the guarantee is made of. The trust contract (trust.md) states the guarantee formally and is candid about the clauses that are conditional today.

Limitations and trade-offs

  • No user-defined functions. The function set is fixed (28 built-ins today — the original twelve plus the constraint atoms, deprecate, id/refer and the generator combinators). This keeps the language total and analysable. Of the IDEAS.md sketches, piping (|>) and the placeholder _ ARE now implemented — as fixed syntax, via the G8 design — while custom functions were considered there and refused: recursion would trade away the termination guarantee.
  • The fixpoint is bounded. Extremely self-referential models hit the pass/cycle limits and surface a cycle error rather than diverging — correct, but it means some legal-looking models are rejected for practical termination reasons.

Further reading