Understanding %gnat

01The idea

A gnat application is a declarative, typed, addressable namespace. You write a shape, and you annotate the shape with how data flows through it. Three moves make up the whole system.

State has a schema, and the schema knows two extra things. A format is a data definition whose syntax is also Hoon type syntax. Every format constructor additionally defines how a path addresses into it, and what a change to it looks like. Types that carry their own address space and their own diff algebra.

Effects are state. There is a place in the namespace called /act. You write a poke into it; the runtime sends it, fills in the result, and deletes the entry. In the other direction, arriving pokes appear in /rat; your logic consumes them; the runtime deletes them. Input jacks fill and drain, output motors fill and drain, and the middle is pure.

Dataflow is declared. Notes — %hear, %coup, %rend, %mask, %read, %sift — hang off nodes of the shape and say what depends on what. Because the dependency graph is written down, the runtime can cache, invalidate, push updates to subscribers, filter subscriptions at their source, and migrate old data forward.

The compression

Event dispatch, subscription bookkeeping, cache invalidation, and state migration are the same four problems in every application. gnat's bet is that all four are derivable from a sufficiently rich type. The format is that type.

02The model

An app is a tree of formats with annotations, mounted at thirteen well-known roots. Some roots the runtime writes and you read. Some you write and the runtime reads. Some are pure and yours. Some hold no data at all and are computed when asked.

Everything the runtime does for you — caching, invalidation, sync, migration, routing, deletion — it does because the shape says enough for it to be done. Declaring the shape is the work.

You write Hoon, and plenty of it, at the leaves of that declared graph. The Hoon in a %hear answers: given this change at this path, what changes here? The Hoon in a %rend answers: given this path and this context, what is the value? Making things happen belongs to the propeller, and the propeller runs on data you leave in it.

03Three organs

Antenna senses. Computer thinks. Propeller acts. The middle organ is pure; the outer two are where I/O lives, and there I/O is spelled as data appearing and disappearing.

   the                                                                           the
  world                                                                        world
    |                                                                              ^
    v                                                                              |
+------------------------+   +------------------------+   +------------------------+
|        ANTENNA         |   |        COMPUTER        |   |       PROPELLER        |
|   input · jacks fill   |-->|     pure · no I/O      |-->|    output · motors     |
|       and drain        |   |      reaches here      |   |         drain          |
+------------------------+   +------------------------+   +------------------------+
| /rat  operator · pokes |   | /nut  brain  · truth   |   | /act  pokes out        |
| /get  browser  · HTTP  |   | /dex  index  · eager   |   | /ply  HTTP PUT/POST    |
| /mir  mirror   · state |   | /vue  view   · lazy    |   | /exp  export to peers  |
| /syn  synth    · fake  |   |                        |   | /web  serve open web   |
| /tim  timer    · dates |   |  %hear reacts          |   | /pro  processes        |
|                        |   |  %coup couples         |   |                        |
|  system fills,         |   |  %rend renders         |   |  you write,            |
|  you read,             |   |  %mask scopes          |   |  system reads,         |
|  system deletes.       |   |                        |   |  system deletes.       |
+------------------------+   +------------------------+   +------------------------+
          SENSE                         PLAN                         ACT
                                          ^                            |
                                          +----------------------------+
                    transaction results flow back into the brain

Thirteen lobes, three organs.

Jacks and motors

An input jack is a container in the antenna that the runtime fills. A motor is a container in the propeller that the runtime drains. Both are ordinary formats, usually a log or a map, and both hold request–response pairs: the entry says what was asked and, once available, what came back.

In /get and /tim you write the request half as well — there the request is output and the response is the input. /syn is the true exception: you write the input itself. Deletion is the runtime's in all of them.

Deletion is load-bearing

Once the brain has learned that a thing happened, the antenna is free of it. That is what keeps persistent state bounded and meaningful. The antenna is a buffer; history is a decision the brain makes, in the brain's format.

The computer

Three lobes, drawing a line most systems leave blurry: where does a derived value live, and when is it computed?

/nut — brain

Source of truth, physically stored, changed by %hear reactions to antenna input and to propeller results. State machines live here, routed by what they concern.

/dex — index

Physically stored, and defined as a pure function of the brain via %coup. The system maintains it incrementally, so lookups are cheap.

/vue — view

Computed on demand. Routed and typed like stored data, with %rend (compute this) or %alas (this is that other path) at the leaves. Cached, with eager invalidation.

The gradient

%hear is imperative derivation: a change arrives and you decide what to write. %coup is equational and eager: you assert that a node equals a function of a path, and the system keeps it true. %rend is equational and lazy: the same assertion, evaluated when asked. Push work from %hear toward %coup and %rend; each step hands the runtime more freedom.

04Formats

A format is a Hoon type restricted so that two extra operations always exist. The same syntax that parses to a format parses to a Hoon type, and in the right context means the same thing.

The catalogue of constructors is fixed and small, and each one carries two things beyond its shape:

• Path syntax — how a path descends into this node.
• Change kinds — the finite vocabulary of edits to this node.

That restriction is the whole game. gnat admits the shapes for which addressing and diffing are obvious, and every constructor pays for its place by supplying both.

The catalogue

Eleven shapes. Everything an app holds is spelled in these.

Lists

Because list indexes are unstable, paths cannot flow through them. You may address a list element. A durable path stops there rather than passing into the subtree below it: insert at the head and every index shifts, so a subscription to /msgs/3/text would quietly begin pointing at a different message.

Stable identity beats positional identity. For durable addressing use a %map or a %log, where keys are stable by construction. A list is for ordered bulk you consume positionally.

The change-kind ladder for lists is worth copying wherever you find yourself diffing a sequence:

::  the runtime tries the cheap analyses first
%stack  push/pop at the front         O(m) in depth of change
%queue  push front / drop back        O(n)
%edit   general diff/patch            O(n log n)

A stack and a queue are the two access patterns that account for most list mutation, and both admit a cheap witness. Only when neither fits does the system pay for a full diff. An arbitrary patch remains expressible as a degenerate sequence of stack or queue operations, so a listener written against the cheap kinds stays correct, and is at worst occasionally inefficient.

Change kinds become arms

We turn kinds of change into core arms. The arm is +$name-$kind.

A listener is a core whose arms are named after the change kinds it cares about. A listener on a log might define +msgs-add and +msgs-mod and nothing more. The runtime reads the core, sees which arms exist, and dispatches to them.

The asymmetry is deliberate: for any edit that involves deleting data, the absence of an arm means we don't care. When data is added, we have to ignore it specifically.

Read that twice

Deletions may be ignored silently; additions must be ignored explicitly. Missing new data loses information. Missing removed data leaves a stale copy, which is the smaller mistake.

05Paths

A path is an address, a pattern with variables, and a query, all at once.

::  a path is a list of segments, plus an optional trailing query
[path=(list path-segment) query=(unit hoon)]

::  and a segment is one of
[%wild name=@tas]              ::  *who        — only in listeners
[%hoon =hoon]                  ::  $(expr)     — computed from subject
[%link =aura @]                ::  ~zod, 42    — literal, typed
[%comp p=segment q=segment]    ::  compound key

One grammar addresses state, matches events, names subscriptions, routes HTTP requests, and specifies migrations. A wildcard learned in one of those places works in all five.

Wildcards bind

All variables in the path are exported into the subject, which also includes: your ship, the date, and the app name.

A listener on /rat/*who/orders/*id matches and binds. who and id are then in lexical scope for the listener's Hoon and for any path that Hoon constructs, so a reaction reads /nut/customers/[who] and writes /act/[who]/billing/… with nothing parsed along the way.

The cost

A %hear with a wildcard works as if there had been a hear with each matching value. This can result in a lot of processing. A wildcard listener is a join. An implementation needs an index from path prefix to interested listeners — a trie of subscriptions — and a cost model to go with it.

Selectors

A path may end in :hoon, a selector, allowed in renderers. Here a path becomes a query: give me the part of this node matching this. Selectors pair with %sift, %peck, and %pick.

Compound keys

Map keys may be tuples: [a b c], [a [d e] c]. Rendered externally — in a URL — they use an underscore code: a_b_c, with nesting doubling the separator, a_d__e_c.

A key must therefore be an atom with a @ta rendering, and something narrower than raw @ta or @t, so that the encoding stays unambiguous in the presence of free text.

Get this under test first

Doubling as an escape needs care at depth: three levels wants ___, and the parser has to count runs. Property-test decode(encode(k)) == k over every key shape on day one, or choose a bracketed or length-prefixed encoding and accept the different ugliness.

06Notes

A %note wraps a format and gives it behaviour. These twenty-one annotations are the API surface of gnat. Learn them and you know the system.

%mask

The rest are legible from their names. %mask does two jobs at once.

[%mask (list (pair name=@tas =path))]

Capability. A mask is a chroot(). No state data is visible until it is pulled into a mask. All state data is relative to the mask. Code below a mask reaches exactly what the mask named. Least authority, applied to reads — which makes a renderer's dependency set declared and enforced rather than inferred.

Cache key. Because the mask is the complete input to everything below it, the mask is the memoization key:

This creates a dependency pattern of alternating %make and %mask where we refine a query from intermediate results. If the intermediate result matches a cache, we have a cache hit; different intermediate data may still create the same final mask, creating another cache hit.

That last clause is early cutoff — the property that makes an incremental system fast as well as correct. Different inputs that converge on the same mask converge on the same answer, and the work below stops there.

One-way doors

%core (once you wrap code around a subject, there is no way to remove it) and %mask (which replaces the context outright) are both irreversible on the way down. That is what makes the subject at any node predictable from the path alone, which in turn is what makes caching and dependency tracking tractable.

Filters

A publisher marks a node %sift: a selection may be made here, and here is the function from a custom filter to a literal one. A subscriber supplies one of two things:

• %peck — an objective filter: structural, evaluated by the system. A boolean for units, a range tree for lists, a key list for maps and logs.
• %pick — a subjective filter: a value whose meaning is the publisher's business. The empty unit means everything.

That is the line every query engine draws, between a predicate the planner can reason about and a function it can only call.

Selections are live. All selections are reactive and dynamic… There is no "one-time" mirroring. Listeners will see delete events when entries drop out of the selection. A filtered mirror is a standing query, and keeping "which entries currently match" up to date incrementally is where the expense sits.

07Input

The antenna's five lobes obey one rule: data arrives, is learned, and is forgotten. Everything in the antenna is in flight.

The gnat runtime removes any inputs that won't have an effect again. That means except for the mirror, everything has to have its effect in the event when it is received. If you can't do that you're in synthetic territory.

The runtime performs liveness analysis on the antenna and reaps whatever cannot fire again — which, for an entry already dispatched to its listeners, is immediately. Input is single-event: your %hear runs, and by the end of that event the entry is gone. What survives is what the brain wrote down.

/mir is the exception, and it is exceptional for a principled reason. A mirror is a standing invariant: %read says this local subtree equals that remote subtree, and that keeps having effect every time anything reads it. It never goes dead, so the reaper never touches it. The exception falls out of the rule.

The jacks

/rat — operator

Incoming gnat commands. Pokes land in routed logs. Pokes fill in their own routes, creating map entries where needed. The top level of the operator is always a map of ships.

The sender's identity is structural: it is the first path segment. Authorization becomes a path pattern rather than a guard clause, and who is how you got here.

Pokes create their own containers, so per-ship state needs no initialization — a poke from ~zod materializes /rat/~zod/…. That is what makes %tool self-initializing.

A poke that cannot be handled nacks its caller. The failure travels back to the sender, and what happens next is the sender's decision.

/get — browser

Outgoing HTTP GETs, modelled as input, because the point of a GET is the response. A browser log entry is a request-response pair. The browser will see and send the requests and fill in the responses, then delete the entry after the response is processed.

The request half waits as long as it takes: the request is output. The input is the response, and it is absorbed the moment it appears.

/mir — mirror

%read takes an absolute path — /ship/app/version/… — and clones it locally, reactively, indefinitely.

Because the runtime knows the format, it can compute the difference between what you hold and what the publisher holds. Resubscription and resynchronization after a partition are therefore the runtime's work. Declaring the invariant is yours; keeping it true is the platform's.

/tim — timer

Marked %wake. You append a date; the fire comes back as input. Like /get this is a request–response pair, so the single-event rule governs the fire rather than the waiting.

A pending timer is ordinary typed state in your own namespace: %add sets one, %mod reschedules it, %drop cancels it. You can read your own schedule, route timers by what they concern, and migrate them with a shim like anything else.

A timer is a way of being woken.

The synthesizer

/syn is marked %fake. It works like the brain — routed, formatted, able to hold state — and what it holds is input:

%fake: mark a synthetic input injection site. In the synthesizer, %gnat does not feed in the input. You do. %gnat still deletes used input, though.

You own the write side. The delete side stays with the runtime, which is what keeps /syn an input jack.

You need state

You're making keypress events out of key up and key down. You need state.

A device sends key-down, and later key-up. What the app reacts to is a keypress, and the keypress is the pair. At key-down, one has not happened yet. At key-up, you can report one only if you still know which key went down.

So key-down cannot have its effect in the event that delivered it. That is the rule tripping. What you do is remember the key-down, and when key-up arrives, combine the two and write a keypress into /syn, where it becomes an input event that %hears react to as though the device had sent it.

"You need state" is the test:

It is the job of turning typed characters into words. No single character is a word, and the word ends when the space arrives, so something holds the letters in between.

This is why the synthesizer works like the brain. Synthesis is stateful, so the synthesizer needs brain tissue: somewhere has to hold the pending key-down. /syn is the part of the antenna allowed to remember, because remembering is its job — and %fake's shape follows from that. "You fill it, gnat drains it" is the form a lobe takes when its entries are computed by you from earlier input. The drain half still applies: a synthesized keypress is consumed and reaped like any other input.

Two neighbours

Scheduling belongs to /tim: wanting something to happen later is not the same as needing to remember anything. Retry belongs to the sender, who gets the nack. Both involve later, which is what makes them easy to mistake for synthesis. Synthesis is a stateful transformation of an input stream.

Fixtures, replay, and load

Because you control what appears and when, /syn also carries recorded transcripts for replaying a bug, generated traffic for learning what a wildcard %hear costs, and plausible activity for a demo. Unlike the other components, these can be mixed in with actual data — fake ships chat alongside real ones.

Synthetic input reaches the same listeners by the same paths, dispatches to the same arm names, and is reaped by the same rule. The only difference is who wrote the entry, and that difference is visible in the state.

08Output

Just as deleting input after it is received is fundamental to the task of the antenna, deleting output after it is sent is fundamental to the task of the propeller.

You write down that an effect should happen. The entry sits in your state — durably, transactionally, beside the state change that caused it — and a component you did not write picks it up, performs it, writes the result back, and removes the entry.

The reward is that a state change and the intent to act on it are always atomic. The record of the order and the sending of it are one event, and the result returns to the brain the same way any other input does.

The motors

/act — action

Outgoing gnat pokes, routed like /rat: ships, then app names. Motors marked %poke, the target protocol marked %send version.

/ply — apply

Outgoing HTTP PUT and POST. Routing is yours; motors marked %post; request and response formats are standard.

/exp — export

Virtual. What peers may read — an API surface, shaped for its audience.

/web — serve

Virtual. Routes on URL prefix, with the rest of the request in the subject. Match path segments against an inbound routing format, then %rend or %alas at a leaf.

/pro — process

Multi-step output logic, able to do what the other lobes do. Motors marked %work; the response should be (each success fail).

Virtual lobes

/exp and /web are computed on demand, which puts them in the same family as /vue. One mechanism — render with dependency tracking — serves three audiences: yourself, your peers, and the open web.

That makes cache invalidation a network protocol concern, which the spec states plainly:

Renders are saved and cached efficiently. Cache invalidation is eager. If another urbit is mirroring a render we export, and that render is invalidated, they have to get the update. Very effective dependency tracking is necessary.

This is the project's largest technical risk, and %mask is the mechanism that makes it approachable: dependencies are explicit and bounded by construction. How tightly bounded is an empirical question, and an early prototype answers it more cheaply than a late one.

The shape of the whole thing is a namespace as the API. You make something happen by writing, you learn something by reading, and the files are typed, diffable, and remotely reactive.

09Versions

An app is a list of packs: append-only, each pack a complete statement of the app's code, formats, and migrations at one version.

+$  pack
  $:  version=@ud                          ::  must match index in the list
      libraries=(list [publisher name version])
      kernel=hoon                          ::  your code, after libraries
      formats=(map @tas (pair up=(list glue) down=(list glue)))
  ==

Three consequences follow from the list being append-only:

1. The history is the definition. Version 3 stays as it was; version 4 is a new entry.
2. Every version is reachable. Migrations are stored per format per pack, so version 1 data walks forward to version 7 by composing shims.
3. Both directions. Each format carries up glue and down glue.

      up:  migrate my old data forward  ------------------------------->

  +----------+  up  +----------+  up  +----------+  up  +----------+
  |  pack 0  |----->|  pack 1  |----->|  pack 2  |----->|  pack 3  |
  |          |      |          |      |          |      |          |
  |  %boot   |<-----|  %move   |<-----|  %omit   |<-----|  %molt   |
  |          | down |  %drop   | down |          | down | (last    |
  |          |      |          |      |          |      |  resort) |
  +----------+      +----------+      +----------+      +----------+

   <------------  down:  speak to a peer who hasn't upgraded

   %send targets a version; composing down-glue lets a new app talk to an old fleet

Down glue is what makes version skew survivable, and surviving version skew is what lets a network upgrade gradually. Good practice to send to the previous application version, allowing smooth fleet upgrades.

Shims, cheapest first

Structural operations are analyzable: the system can invert a rename, compose two renames, check that a fleet-wide migration is safe, and generate down glue on its own. A %molt is opaque, and once one enters the chain those properties end there.

Two kelvins

Arvo is always forward compatible but not always backward compatible: new tools can be added. A kelvin at zero is a permanent promise. gnat counts the other way, and says so.

10Further reading

Each piece of gnat has a literature. These are the shortest routes into them.

• Cambria (Ink & Switch) — bidirectional schema lenses: rename, add, remove, wrap, hoist. Nearly one-to-one with the shim vocabulary, and the best available argument that up-and-down glue works.
• Salsa and Adapton — demand-driven recomputation with early cutoff. The nearest neighbours of /vue, %mask, and invalidation.
• Differential dataflow — incremental view maintenance from deltas. The nearest neighbour of %coup and /dex.
• re-frame — effects and coeffects as data, subscriptions as a dependency graph, in a codebase you can run today.
• Nix and Bazel — declare the inputs, sandbox so that undeclared reads are impossible, reuse the output. %mask's bet, applied to builds.
• Avro — reader and writer each hold a schema and the runtime reconciles them. The deployed form of migration as schema resolution.
• Datalog — variables unify across a rule head and body, as wildcards do across a path.
• Plan 9 — the namespace is the whole API, and effects are writes.

Inside Urbit, %clay is the template: a typed, versioned, diffable namespace. gnat is those semantics for live application state.

Three, if you read three

Cambria for the shim model, Salsa's design notes for invalidation, re-frame for effects as data.

11Open questions

The sketch is a sketch, and some of its breakage hides decisions that have not been made. These are the ones an implementation runs into.

In the sketch

Mechanical, though worth confirming the intent. +$ note is written as a $: around all twenty notes, which makes a twenty-tuple where a choice is wanted; it should be $%. The $?s over tagged cells in +$ form and +$ path-segment want $%, for head-tag dispatch. +$ format and +$ path are written as a $% around +$ arms, and want a core with =<. %row and %fork use the constant %tas where the aura @tas is meant, and %fork's (list %tas form) wants (list (pair @tas form)). +$ key's $- makes it a gate. In +$ gnat, code=(list pack) lands inside meta, though its own comment makes it a sibling.

Two things are named and never defined:

• glue. formats=(map @tas (pair up=(list glue) down=(list glue))) carries the entire migration mechanism, and its element type is absent. If glue is shim, why the second name? If it is more, what does it add — a source format, a target format, a guard?
• The reserved formats. file, text, json, xml, httq, httr are declared to live in the dictionary. Two of them are noun-shaped, with path syntax and change kinds still to be decided. What does a %hear watching inside a json see?

Smaller: %hear and %tick take path then hoon while %coup takes hoon then path — pick one. %mime p=(list @ta) is presumably the permitted mime types. %peck's range tree for lists is named but unspecified.

In the design

12Build order

Everything rests on one thing: a format that can be walked by a path and diffed into change kinds. Build that first, prove it, and the rest is assembly.

Steps 1–3 are the whole project. Steps 4–6 prove it. Steps 7–11 are where it becomes valuable. Step 12 is where other people can use it.

/syn sits at 6½ deliberately: once the loop exists, driving it from a recorded or generated stream is what makes every later step testable.

Two pieces of advice

Write the diff and patch property tests before the code. Correctness reduces to patch(a, diff(a,b)) == b and to path descent agreeing with the format. Once those hold for every constructor, most of what remains is performance.

Build one real app alongside the runtime. Something with a map, a log, a list, a mirror, and a render — a chat, or a feed reader. The catalogue's expressiveness ceiling is found by hitting it.

13Glossary

One sentence to carry into the implementation:

Anything the runtime is expected to do for you — cache, invalidate, sync, migrate, route, delete — must be visible in the shape.