Grubbery

What Grubbery Is

Grubbery is a tree-shaped manager for stateful, long-running processes on Urbit. It turns a single Gall agent into a small operating system whose entire state is a filesystem, and whose every file has a pulse.

In ordinary Gall, an agent is one blob of state with a fixed set of handlers. If you want ten independent concurrent processes — a block-height poller, a counter, an HTTP session, a peer gateway — you hand-weave them all through one on-poke/on-arvo/on-agent and carry their bookkeeping in one state noun. Concurrency, isolation, and lifecycle are all your problem.

Grubbery inverts this. It takes over one agent and exposes a tree. Directories carry behavior. Files carry content and a running process. You add a process by writing a file; you stop it by deleting the file; you compose systems by nesting directories. The agent underneath is an interpreter.

The project's own README names three faces of the same idea:

Everything in a live grubbery — the tree, the running processes, the sandbox rules, the version history, the compiled code — lives in one agent state noun and its content-addressed satellites. Because processes are rebuilt from their files on every reload, the running system is always a pure function of the tree.

What It's Good At

Grubbery is opinionated. It pays a real complexity cost to buy a specific set of powers. Reach for it when you want:

Many independent long-running processes in one agent

Each grub is its own coroutine with private state, its own suspension points, and its own crash domain. A block poller can loop forever, a hundred HTTP request handlers can spin up and tear down, and a counter can tick every second — all in one agent, none aware of the others, none able to corrupt the others' state.

Complex async workflows written as flat, linear code

This is the headline capability, and why gwbtc exists. Polling a Bitcoin node every two seconds, walking blocks one at a time, firing dozens of dependent getrawtransaction RPCs, then building and broadcasting commit/reveal transaction chains — all of it reads as straight-line ;< x bind:m (call) code with ordinary ?~/?: control flow and sleep/retry, instead of the callback-fragmented on-arvo/wire-dispatch soup a Gall agent would require.

Processes that survive reloads by construction

Fibers are never persisted as closures. Only their state — content-addressed in a shared store — survives. On every reload, every process is respawned from its file's nexus and handed its old state. The running system is always derivable from the tree, so hot code updates never strand a process.

Capability-based sandboxing of untrusted code and remote ships

A weir on a directory restricts what its children can reach. Foreign ships enter the tree as ordinary sandboxed grubs — a directory, a permission set computed from usergroups, and no ability to make raw system calls. You can host other people's code and other people's ships inside your agent and bound exactly what they touch.

A live, reactive filesystem — with a built-in HTTP API

A grub can subscribe to a directory and get a notification whenever anything under it changes, initial state delivered on subscribe. The browser gets the same thing over one URL scheme: GET /grubbery/api/keep/<path> opens a Server-Sent-Events stream of new/upd/del events. Reactive dashboards fall out directly — the same skeleton scales from a toy counter to a Bitcoin consensus walker.

The Seven Words

Grubbery's whole design fits in seven nouns. Learn these and the code reads itself.

The Tarball & Blots

Everything begins with the tree, defined in lib/tarball.hoon. The develop branch reworked the content model around two ideas: blots (marks reimagined as tree paths) and content addressing (files stored by hash, deduplicated, versioned).

Blots: marks as paths

A blot is a mark identity — but instead of a flat @tas like clay's %json, it is a rail (a path plus a name), so marks are hierarchical. Content is always paired with its blot:

+$  neck   rail              :: a nexus identity (directory-level mark)
+$  blot   rail              :: a mark identity, hierarchical  e.g. [/ %json]
+$  bars   [a=blot b=blot]   :: a blot pair — keys a conversion tube
+$  sage   (pair blot vase)  :: grubbery "cage": blot + typed vase
+$  reus   (each vase boom)  :: a typed vase, OR a crash (boom = [tang noun])
+$  sang   (pair blot reus)  :: grubbery "content": blot + (vase or error)
+$  bask   (pair blot noun)  :: grubbery "page": blot + raw noun

So a file at rest is a sang (its blot, and either a validated vase or a stored crash). A file in flight — what you pass to a poke or a make — is a bask (blot + noun) or a sage (blot + vase). Converting a file "to JSON" means finding the tube keyed by the bars pair [source-blot /json] and running it.

Directory nodes: lump and ball

The tree is an axal whose nodes are lumps. A lump is one directory: its nexus id, its weir, a "gain" pin flag, an optional stored crash, and its files.

+$  lump
  $:  neck=(unit neck)         :: which nexus governs this dir
      weir=(unit weir)         :: this dir's sandbox
      gain=?                   :: pin flag (retain history)
      bang=(unit tang)         :: a stored crash, if this node failed
      contents=(map @ta [=sang gain=? bang=(unit tang)])  :: the files
  ==
+$  ball   (axal lump)         :: THE filesystem
+$  pulp   ...                 :: a content-only mirror of lump (bask, no vases)
+$  bole   (axal pulp)         :: the value-only mirror, used by loaders/diffs

Two shapes worth keeping straight: ball is the live, fully-typed tree; bole is a lighter value-only mirror (raw nouns, no vases) that on-load returns and that tree-diffing walks. You author against ball; the loader hands back a bole.

Addressing: rails, folds, roads

+$  rail  [=path name=@ta]     :: address of a FILE (dir path + filename)
+$  fold  path                :: address of a DIRECTORY
+$  lane  (each rail fold)     :: [%& rail] a file | [%| fold] a dir
+$  bend  (pair @ud lane)      :: RELATIVE: N steps up, then a lane
+$  road  (each lane bend)     :: [%& lane] absolute | [%| bend] relative

A road names a destination. Absolute roads name an exact lane; relative bends say "go up N directories, then descend." Fibers usually write roads as strings — (cord-to-road:tarball '../../counters/') parses to "two up, into counters/."

Content addressing & version history

The agent does not store the ball directly. It stores the tree's history and a content-addressed blob store, and materializes the ball on demand. History is a per-file ordered map of pace entries:

+$  pace                                  :: one version's disposition
  $%  [%firm tags=(set @t) p=(unit lobe:clay)]  :: permanent until tombed
      [%temp tags=(set @t) p=(unit lobe:clay)]  :: replaced on next write
      [%tomb ~]                                 :: tombstone (deleted)
  ==
+$  hist  ((mop cass:clay pace) cor)      :: version -> disposition, ordered
+$  born  (axal [fold=hist file=(map @ta hist)])   :: history, mirrors the tree
+$  silo  [nouns=(map lobe [refs noun]) jects=(map lobe [refs ject])]

Each version points at a lobe (a content hash). The silo is a ref-counted store keyed by that hash, holding both raw nouns and structured tree/leaf "jects." Identical content is stored once; deleting a version decrements a refcount. Reading a file "as of" revision N or date D walks the hist map to a lobe, then fetches from the silo. %firm pins a version permanently; %temp is overwritten on the next write; tags label versions.

The Fiber Monad

A fiber is the beating heart of a grub. It is a computation that can pause itself, emit effects, and be resumed with an answer — Urbit's strand pattern, rebuilt for the tree.

If you know strands (strandio), fibers will feel familiar: straight-line code with ;< binds, each firing an effect and blocking until its response arrives. The novelties are the effect/event vocabulary (darts and intakes), reload-durability, and a cold/hot split in how events are queued.

The shape of a fiber

A fiber is a gate from an input (its private state plus the event it is ingesting) to an output (darts to emit, a new state, and a control verb saying what to do next).

+$  input   [state=vase in=(unit intake)]   :: my state + event (~ = start)

++  output-raw
  |*  value=mold
  $:  darts=(list dart)          :: effects to emit now
      state=*                    :: my new state
      $=  next
      $%  [%wait ~]              :: consume this intake, await the next
          [%skip ~]              :: defer this intake, await the next
          [%cont self=(form-raw value)]  :: continue computing now
          [%fail err=tang]       :: abort with an error
          [%done =value]         :: finish with a result
      ==
  ==

+$  process  _*form:(fiber ,~)   :: a fiber that returns nil
+$  spool    $-(prod process)    :: what on-file returns — takes prod, returns a process
+$  prod     (unit tang)         :: ~ = clean start, [~ tang] = restart after crash

+$  proc                         :: a live grub's fiber + its queues
  $:  process=(each process tang)  :: the running fiber OR a stored crash
      next=(qeu take)              :: held inputs to process
      skip=(qeu take)              :: deferred inputs
  ==

It's a monad — state plus continuation

pure is return; bind is sequencing. bind runs the first form, threads its darts and state through, and on %done feeds the result into the continuation. So a fiber is a state monad (the vase threaded through state) fused with a continuation monad (the next verb as suspension points). You never write the plumbing; you write ;< x bind:m (io-call ...) and it composes.

Cold intakes, hot state

An event queued for a fiber is stored not as a live intake but as a pend — a "cold" event carrying content hashes (lobes and code keys), no vases. Only when the fiber is about to consume it does the engine hydrate the pend into a real intake: resolving the lobe from the silo and re-validating it through the compiled mark. This keeps the work queue small and content-addressed, and is why validation happens lazily at consumption.

fiberio — the author's toolbox

lib/fiberio.hoon is to fibers what strandio is to strands: a library of ready-made darts and intake-waiters. Alias it (io=fiberio) and call its arms.

Darts & Intakes

A fiber's entire relationship with the world is two closed unions. It emits darts; it receives intakes. Everything a grub can do is one; everything that happens to it is the other.

Darts — down to three heads

The develop branch shrank the dart union from six heads to three. Nearly everything is now a %node dart carrying a destination and a load operation:

+$  dart
  $%  [%node =wire road=road:tarball =load]   :: an operation on the tree
      [%here =wire]                           :: ask for my own location
      [%kept =wire]                           :: inspect my own subscriptions
  ==

The workhorse is load — the operation carried by a %node dart:

+$  load
  $%  [%poke =bask:tarball]        :: poke a grub's process
      [%make force=? =make]        :: create a grub or directory
      [%cull ~]                    :: delete a grub or directory
      [%sand weir=(unit weir)]     :: set a directory's sandbox
      [%load ~]                    :: re-run a nexus's on-load
      [%peek blot=(unit blot) case=(unit case) deep=?]  :: read (optionally re-marked / versioned / recursive)
      [%keep blot=(unit blot)]     :: subscribe to a destination
      [%drop ~]                    :: unsubscribe
      [%lose =lose]                :: prune history
      [%gain flag=?]               :: set the pin flag (recursive on dirs)
      [%firm ~]                    :: promote current version to permanent
      [%seek =lobe:clay]           :: reverse lookup: rails holding this hash
      [%peep =find]                :: query history entries
      [%code ~]                    :: look up compiled artifacts at a dest
      [%font ~]                    :: find which nexus governs a node
  ==

Intakes — the events back

Most intakes are responses to darts; some are unsolicited inputs; some are lifecycle. The full union grew to match the expanded load:

Process start and restart are not intakes. They arrive as the prod passed to the fiber's initializer: ~ for a clean start, [~ tang] for a restart carrying the crash. That is what rise-wait inspects.

The Nexus

If the fiber is what a grub does, the nexus is what a directory is. A nexus is the behavior definition for a directory: how its files are laid out, and what process each file runs. Nexuses are the unit of code you write, and on develop they shrank to two arms:

+$  nexus
  $_  ^?
  |%
  ++  on-load                    :: declare / migrate this directory's file layout
    |~  ball:tarball
    *bole:tarball
  ++  on-file                    :: return the process for a file at this rail
    |~  [rail:tarball blot:tarball]
    *spool:fiber
  --

on-load — declare the shape of the directory

Runs on nexus creation and every reload. In practice you read a schema version and call spin:loader with a list of rows describing the files and subdirectories that should exist. Anything not listed is dropped; %fall means "create if absent, keep if present"; %over always overwrites. This is where a directory seeds its files and where a parent assigns nexuses to its children by giving a child directory a neck.

on-file — manufacture the process

Given a file's rail and blot, return a spool — an initializer that, given the prod lifecycle signal, produces the running process. This is the arm you spend the most time in: a big ?+ rail switch, one branch per kind of file, each a fiber program. Because on-file is re-run on every reload to rebuild each process, the process must be reconstructable from its stored state alone — the central discipline grubbery asks of you.

Weirs, /sys & Remote Ships

Three things share one mechanism here: sandboxing, system calls, and talking to other ships. All of them are just darts travelling through the tree, gated by weirs.

Weirs — capability by location

A weir is three allowlists — one per dart category — each a set of destination roads:

+$  weir
  $:  make=(set road)   :: allowed dests for %make, %cull, %sand, %load, %gain…
      poke=(set road)   :: allowed dests for %poke
      peek=(set road)   :: allowed dests for %peek, %keep
  ==

Darts travel up from the emitting grub to the nearest common ancestor with their destination, then down. Downward is always legal; every upward step is checked against that directory's weir. The engine computes the nearest governor (the strict common ancestor, a neutral authority), walks up to it, and applies each weir's filter for the dart's category. A blocked dart never executes — the fiber gets a %veto intake instead.

/sys — the runtime as a set of grubs

The most striking idea on develop: the Arvo vanes are materialized as grubs under /sys/, and a fiber interacts with them by poking files. The agent intercepts pokes whose destination is under /sys/ and turns them into real Arvo cards, replying with an ack.

The io helpers wrap all this: sleep pokes /sys/behn; fetch pokes /sys/iris then waits for the response; get-our pokes /sys/bowl. You never see the cards.

Remote ships — content-addressed peek

External ships are turned into ordinary sandboxed grubs. Addressing another ship means targeting a road under /sys/ames/ships/~ship/root/…; the engine detects that prefix and stages a cross-ship operation. The read protocol is a four-step content-addressed negotiation so a large state is never re-sent when the requester already has most of it:

1. peek — ship A asks ship B for a path.
2. snap — B replies with a snapshot manifest: the version and the set of content hashes that make it up (pinning them briefly with an expiry timer).
3. want — A diffs that manifest against its own silo and asks only for the hashes it is missing.
4. data — B ships just those blobs; A hash-verifies each one before merging, then the staged %peek intake finally fires.

Inbound remote operations are treated as a dart originating from /sys/ames/ships/~src/ship.sig, so the same weir machinery sandboxes them. Your own ship gets full access; a foreign ship gets a weir computed from usergroups (/sys/ames/usergroups/…, with a public group applying to everyone). On the read side, peek-remote:io sugars the road rewrite; remote writes are done by hand-building the ames road and using ordinary make/poke/keep.

The HTTP & Blot API

Grubbery ships a generic HTTP surface that exposes the whole tree over one URL scheme. You rarely write a custom endpoint — you read and write files, and the browser gets live updates for free.

The endpoints

Everything lives under /grubbery/api/<endpoint>/<tree-path>, dispatched in lib/ball-api.hoon:

blot — the serialization knob

The ?blot= query parameter is the HTTP-level version of a mark conversion. ?blot=/json asks grubbery to run the stored file through the conversion tube keyed by [its-blot /json] before responding. It works on GET /file, on the SSE event bodies, and on POST bodies (interpreting an incoming JSON body as some richer mark). One knob, one meaning everywhere: "give me / take this as blot X."

How a live subscription works

A GET /api/keep/<path> is served by a fiber that:

1. sends SSE headers, then keep:ios the target road — getting the initial wave;
2. emits an old event per file in the initial wavefront (each body serialized per ?blot);
3. arms a 30-second keep-alive timer and loops on take-news-or-wake:io;
4. on a timer %wake, sends a keep-alive and re-arms; on a %news, diffs the old wave against the new and emits per-file new / upd / del events, each carrying the file's version as its SSE id.

The Engine

Under all of this sits one Gall agent, app/grubbery.hoon (now ~5,400 lines), whose only job is to interpret the tree. On develop its state is state-2:

+$  state-2
  $:  %2
      =born:nexus     :: version history (mirrors the tree)
      =silo:nexus     :: content-addressed blob store (nouns + jects)
      =subs:nexus     :: subscription indices (who watches what)
      =pool:nexus     :: the running fibers, per directory per file
      =code:nexus     :: compiled namespaces, per /code scope
      =bins:nexus     :: content-addressed compiled artifacts (by ckey)
      =vale:nexus     :: validation cache (content × mark → ok/err)
      =remo:nexus     :: cross-ship peek/snap negotiation state
      =upki:nexus     :: the rail that backs jael PKI subscriptions
      last=[now=@da eny=@uvJ]   :: virtual-bowl snapshot
  ==

One event, a cascade of fibers

Every Gall event becomes a take pushed onto a work queue. The driver, abet, drains it: pop a take, hydrate its cold pend into a live intake, run the target fiber one step inside mule (crash-safe), and loop. Stepping a fiber produces darts, and darts that expect answers enqueue more takes — so a single incoming poke can cascade through many fibers before the agent returns. Each step's returned state vase is re-validated against its mark before being committed.

Dispatching a dart runs the weir check first, then interprets the load: normal tree ops mutate born/silo; a destination under /sys/ is bridged to a vane; a destination under /sys/ames is staged as a remote op; %code/%font read the compiler indices. Wire tags carry the grub's rail and a life number so that a response to a since-restarted process is discarded rather than delivered to the wrong incarnation.

Reload — rebuild the world from the tree

On on-load the agent migrates the state version, then runs a cold-start sequence: bootstrap the foundational marks, pull the code desk into the tree, re-run every nexus's on-load top-down, recompile changed /code namespaces, respawn a fresh fiber for every file (handed its old state, a clean ~ prod), and re-sync every /sys service. Fibers are respawned, not resumed — which is exactly why a fiber must find its place from state alone.

The in-tree compiler

Grubbery compiles its own Hoon. Marks are marc cores built from source; nexuses are compiled from /nex files; a Ford-style engine in lib/build.hoon parses imports, topologically sorts, and compiles a whole ball, caching every artifact content-addressed by key in bins. The %code and %font darts let a running fiber introspect what's compiled and which nexus governs any node. Editing code and reloading swaps behavior live.

Writing Grubs

Theory lands when you see the code. Three examples, in rising complexity: a counter, a one-shot LLM call, and a Bitcoin block walker — all the same skeleton.

The counter — the canonical shape

nex/counter.hoon is a directory of auto-incrementing counters, a live HTML view of them, and an HTTP endpoint. Its on-load seeds the layout; on-file dispatches per file. Here is the ticking counter — the minimal fiber loop:

++  on-file
  |=  [=rail:tarball =blot:tarball]
  ^-  spool:fiber:nexus
  |=  =prod:fiber:nexus
  =/  m  (fiber:fiber:nexus ,~)
  ^-  process:fiber:nexus
  ?+    rail  stay:m
      [[%counters ~] @]                    :: one process per counter file
    ;<  ~  bind:m  (rise-wait:io prod "%counter: process failed")
    |-
    ;<  count=@ud  bind:m  (get-state-as:io ,@ud)
    ;<  ~          bind:m  (sleep:io ~s1)
    ;<  ~          bind:m  (replace:io +(count))
    $
  ==

A complete, independent, reload-durable process in six lines. Its state is one @ud; on restart it reads that @ud and keeps counting. The wildcard rail [[%counters ~] @] means one process template is instantiated per file — the counter set is data, each element a self-owned grub.

The reactive view — subscribe and re-render

The HTML page grub keeps the counters directory, then loops on take-news, re-peeking and re-rendering into its own file:

  [[%ui %views ~] %'page.html']
;<  ~  bind:m  (rise-wait:io prod "…failed")
;<  init=wave:nexus  bind:m
  (keep:io /ctrs (cord-to-road:tarball '../../counters/') ~)
|-
;<  upd=wave:nexus  bind:m  (take-news:io /ctrs)
;<  =seen:nexus     bind:m  (peek:io (cord-to-road:tarball '../../counters/') ~)
?.  ?=([%& %ball *] seen)  $
=/  page=manx  (counter-page …)          :: fold the counters into HTML
;<  ~  bind:m  (replace:io (crip (en-xml:html page)))
$

The one-shot — a fiber that runs once and dies

lib/oneshot.hoon wraps a single LLM or web-search call as a self-contained fiber. No persistent state; fire the request, block for the one response, resolve, terminate:

++  call
  |=  =spec
  =/  m  (fiber:fiber:nexus ,result)
  ^-  form:m
  ;<  got=(each @t @t)  bind:m  (request spec)   :: HTTP to the model
  ?.  ?=(%& -.got)
    (pure:m [%| %| p.got])
  (constrain p.got output.spec)                 :: coerce the reply to a target mark

The nice trick is output constraining: every call names a target mark, the model's text is tubed to that mark, and on a parse failure the crash is fed back into the prompt for a retry (call-retry). Compose several with ;< and you have a research briefing pipeline — generate queries, run searches, synthesize — as linear code.

The walker — a Bitcoin consensus machine as one fiber

nex/groundwire.hoon is the payoff. Its walker grub owns urb-state.urb-state — the cursor is a field inside that state, and the file is the published PKI read-model. It keeps the tip poller, blocks when caught up, fetches the next block when behind, runs it through the crypto pipeline, and fans results out to other files:

;<  urb-state=state:urb  bind:m  (get-state-as:io ,state:urb)
=/  processed=@ud  num.block-id.urb-state       :: cursor lives in the state
;<  *  bind:m  (keep:io /t height-road ~)        :: subscribe to the tip poller
|-
?:  (lte tip processed)                          :: caught up — block on the poller
  ;<  *  bind:m  (take-news:io /t)
  …read new tip, loop…
…else fetch block via send-request:io / take-client-response:io…
=/  [fx new-state]  (handle-block:(abed:urb-core urb-state) ublk precommits)
;<  ~  bind:m  (write-point-effects fx new-state)  :: fan out to /points/<ship>.json
;<  ~  bind:m  (over:io latest-road [[/ %json] latest-jon])
=.  urb-state  new-state
=.  processed  +(processed)
;<  ~  bind:m  (replace:io urb-state)             :: state IS cursor AND read-model
$

Alongside it: a tip poller grub (getblockcount every two seconds, replace-ing a bare height.ud), an SSE stats grub (watches two files, re-renders one HTML fragment), and a reg-tester grub that builds Taproot commit/reveal transaction chains on demand. Each is one fiber. The pure parts — the crypto in lib/groundwire, the scheduling math in lib/cron — stay fiber-agnostic and testable; the nexus supplies only I/O and self-ownership.

Mental Models

Six ways to hold grubbery in your head. Reach for whichever fits the problem in front of you.

1. The filesystem is a process supervisor

Grubbery is systemd where the unit file is the process and the directory tree is the supervision hierarchy. To start a service, write a file. To stop it, delete it. To restart-on-crash, do nothing — that's the default. The nexus is the service template for a directory of like services.

2. Grubs are actors; darts and intakes are messages

Each grub is an isolated actor with private state and a mailbox. It cannot reach into another grub's state; it can only send darts and receive intakes. The tree gives actors addresses (rails and roads) and capabilities (position under weirs). If you know the actor model, you know the runtime — the novelty is that the address space is a filesystem.

3. Everything is one dart through the tree

Sandboxing, system calls, and networking are not three subsystems. A weir blocks a dart; a /sys path bridges it to a vane; an /sys/ames path bridges it to another ship. Learn the up-then-down routing rule once and you understand security, syscalls, and the network all at once.

4. State is a pure function of the tree

Fibers are respawned from files on every reload; no hidden runtime state escapes the content-addressed store. The tree plus each file's saved state fully determines the running system. This makes hot updates safe — and is the reason for the one discipline grubbery imposes: your fiber must reconstruct its position from state alone. Write processes that read their state and resume, never processes that assume they've been running.

5. Content is addressed by hash, everywhere

Files, versions, compiled code, and cross-ship transfers are all keyed by content hash in ref-counted stores. That single choice buys deduplication, free version history, cheap validation caching, and a network protocol that only ever ships the bytes the other side is missing. When something feels magical (instant history, minimal-diff sync), it's the silo.

6. Async is linear because suspension is a value

The reason a two-second poll loop, a multi-RPC block walk, and a commit/reveal transaction chain all read as flat ;< code is that a fiber can freeze itself mid-computation and be resumed by an intake. Callbacks, wires, and state machines collapse into ordinary sequential control flow. This is the "Groundwire" face — and the whole reason a Bitcoin wallet was tractable to write here.

Type Reference

The load-bearing molds, for quick lookup. Files: lib/tarball.hoon, lib/nexus.hoon, lib/fiberio.hoon, lib/ball-api.hoon, lib/migrations.hoon, app/grubbery.hoon.