UnREST by example
From “keep my REST” to “worth rewriting for.”
The same substrate meets you at three depths. Point it at the REST you already wrote and it gets faster on the wire with no code change. Write a little for it and coordination problems that used to need a server just… go away. Commit to it fully and you can rebuild something as hard as live video from the ground up. This page walks all three — with every listing drawn from the build manual, Magnum Croakus, and linked to the section it lives in.
Your existing HTTP, JSON, XML, CSV — intercepted, compressed, and fanned out for free.
Coordinate through a shared tuple space instead of messages. The server category disappears.
Live video, captured and handled a new way — possible only because of FrogNet and SotF.
What it is: a private network layer that lives alongside the general Internet and has full access to it. Your local devices join the FrogNet; the FrogNet reaches the Internet — directly, or through another FrogNet acting as the gateway — so users have both at once, all the time: FrogNet services and ordinary public sites, side by side. The switch is automatic and needs no thought. Traffic addressed to the FrogNet — the private 10.x network — rides the adaptive FrogNet node, with all the semantic compression and self-healing this page describes; everything else takes the normal path straight out. The address decides, the two coexist on the same machine, and ordinary Internet traffic is not taxed by FrogNet sitting in the path — non-10.x requests route out at normal speed. And it stays out of your way on your own turf: if you already run machines on a private 10.x network of your own, FrogNet coexists without interfering — FrogNet nodes need unique addresses, but your other 10.x hosts keep working exactly as they did.
What it is not: a way to optimize and customize general Internet traffic the way it does your own. Those gains — the semantic compression, the adaptive wire, the custom handlers — need a cooperating handler on both ends, a FrogNet proxy in front of the caller and another in front of the origin, because the whole benefit comes from two sides sharing a learned reference. Talk to an ordinary public web server with no FrogNet on it and there is nothing to share a template with; you reach it fine, but as plain HTTP, with none of the FrogNet advantages. And it does not scale without bound: the size of a FrogNet is limited by the hardware of the machines on it — most of all the elected database host, which holds the shared memory every node reads and writes. A Raspberry Pi as database host bounds a small pond; a capable box carries a larger one. Those are real ceilings, and we would rather state them than let you find them in production.
Don't send what the other side already remembers.
The web re-states the world on every exchange: a client ships a whole request, a server rebuilds context it already had and ships a whole reply. UnREST borrows Gelernter's tuple spaces instead — the network is a shared memory. You write a value where you compute it and read it where you need it; only the difference ever travels. Everything below is that one idea, applied at three depths.
The accelerant: BLDC-1. The memory model only works because “send the difference” is nearly free. Three states run over a template both sides learn once — after that, only what changed moves, and an unchanged value collapses to a heartbeat.
It is all one object: the handler. Every exchange on a FrogNet is processed by a handler conforming to one interface with a few virtual slots. UnREST Core is simply the default behavior of that interface — which is why the leap from Stage 1 to Stage 3 is not three systems, but one interface overridden more and more deeply.
class MyPathway(UnRESTHandler):
ROLE_NAME = None # a content handler, no role
def learn_request_template(self, req): ... # learn the shape
def extract_request_dynamic(self, req): ... # pull the values
def rebuild_reply(self, tpl, vals): ... # rebuild far side
# register in the one format registry -> dispatch is uniform.
# then you do not call it — it is triggered.Keep the REST you already wrote. Get this for free.
HTTP, JSON, XML, and CSV are a family of content handlers — each one a parser that knows its own format well enough to turn it into SAME / DIFF / FULL. You install nothing and change no application code. The app keeps making ordinary calls; the family sits underneath and does three things to every exchange.
Every request the app makes hits the local proxy on :80, not the network directly. The proxy sniffs the body, asks the one format registry which handler matches its shape — JSON, XML, CSV, a game move on _game:1, media on _sotf:1 — and vectors the exchange to it. No branching on type; a new format is one new subclass, not a change to the dispatch path.
Instead of a connection per request, many requests fan in on one persistent socket, each tagged with a sequence number, and a pool of daemon workers runs the codec in parallel. Independent channel sets keep a bulk file sync from ever head-of-line-blocking a latency-critical stream — each set has its own socket and seq space.
A worker answers whichever request finishes first. The daemon holds a pending map and wakes each caller by sequence as its reply returns — seq 42 can come home before seq 41. The wire never sits idle waiting on a slow reply the way serial request-reply does.
What the fanout looks like on the wire. Three callers, one established socket, frames tagged and interleaved — then matched home as each completes, in whatever order it completes.
# three callers, sent back-to-back: writer -> seq 41 REQ_FULL (dashboard poll) writer -> seq 42 REQ_REPEAT (sensor read) writer -> seq 43 REQ_FULL (file list) # replies wake their callers as they land — any order: reader <- seq 42 RESP_SAME -> sensor caller (first back) reader <- seq 43 RESP_DIFF -> file-list caller reader <- seq 41 RESP_DIFF -> dashboard caller # pending now empty. out-of-order completion, no stalls.
The same call the app already makes. A plain HTTP request returns plain JSON. The app is unchanged; underneath, it rode a length-prefixed FNWP-1 frame on the shared socket and came back a diff.
GET http://10.80.80.1/getHosts.php
[
{ "ip":"10.80.80.1", "name":"HomeBase" },
{ "ip":"10.84.84.1", "name":"ShopBox" },
{ "ip":"10.90.90.1", "name":"MomBox" }
]
# nothing above the line changed. everything below it did.Request then reply, serial. A new connection per request, or pipelining that head-of-line-blocks behind a slow reply. Half the wire sits idle.
Many requests in flight on one socket, fanned in, completed out of order, matched home by seq. Independent channel sets isolate streams.
Each parser learns a template from its own format
That is the only per-format knowledge a content handler carries: how to read its structure well enough to say what is fixed and what varies. Everything after — the fanout, the socket, the async completion — is shared.
FULL { "sensor":"barn.temp", "unit":"F", "value":54.2, "ts":1719… } ~480 B DIFF value -> 54.6 ts -> … ~12 B SAME (nothing changed) ~2 B
FULL <reading sensor="barn.temp" unit="F"> <value>54.2</value><ts>1719…</ts> </reading> learn the tree DIFF value -> 54.6 ts -> … changed nodes SAME (structure + values unchanged) heartbeat
FULL fqdn,eth0IP,wlan0IP,wlan1IP # learn header HomeBase.frognet,10.80.80.1,, # the values DIFF wlan0IP -> 10.80.80.9 # a link came up SAME (unchanged since last echo) # sub-ms, cached
That CSV echo — one line answering “who are you?” — runs in well under a millisecond from cache. The whole discovery walk is built on it. Content handlers fill only the codec slot; election and lifecycle stay inert, inherited as do-nothing defaults.
And it's already measurable — with zero code changed
A real workload run against a stock Apache endpoint, then the identical requests through the FrogNet proxy on port 80. Same app, same responses. BLDC-1's SAME/DIFF and the FNWP-1 socket do the rest.
Stage 1 asked nothing of you. You changed no code — and got 42% off your latency, concurrent coalescing, and a wire that carries only diffs. That is the floor.
Coordinate through shared memory, and the server disappears.
A multiplayer game is the smallest honest test of coordination: several machines, one truth, cheating to prevent. Do it with messages and you rebuild a request/response server with extra steps. Do it the FrogNet way and there is no server — the board is a value in the tuple space, and playing is just reading and writing shared memory.
The board is one tuple. It lives at databasehost.frognet — the shared memory every player reads. Nobody polls a server for “the current state”; they read a value that is always current, and an unchanged read is nearly free.
Two rules make it safe, and they are the whole model. Hold both and coordination is trivial; break either and the message-passing world sneaks back in.
A participant writes only its own object — it states intent in a value that is its own. It never writes the board, never writes another player's object.
The governor is the one writer of record and the only unforgeable authority. The engine ignores any value a participant isn't allowed to write — you cannot fake a roll or seat yourself.
Tic-Tac-Toe: intent in, board out. A player writes a move into its own object. The governor reads the changed intent, checks turn and legality, writes the board back, and keeps no private state — what it acted on lives in the shared board itself.
POST <game>.frognet/api.php?entity=move&action=upsert_by_name
{ "_game":1, "game":"tictactoe", "table":"kitchen-1",
"op":"move", "seat":"X", "cell":8, "intent_seq":4 }
# governor: intent_seq 4 > consumed{X:3} -> new
# X's turn? cell 8 empty? -> legal
# mark cell, check line, turn: O
# write the BOARD, record consumed{X:4}board = get("tictactoe", "board", table="kitchen-1") # tuple space
render(board) # always current
if board["turn"] == my_seat:
put("move", my_scope, {"op":"move", "cell":chosen, "intent_seq":n})
# a WATCHER is the same first two lines — and never the third.Backgammon earns a real bug — and an oracle catches it. A second player tries to JOIN a hosted table. The old engine has no join op, so the seat stays open. Because coordination is shared memory, the test is just: write the intent, then assert the board.
tbl = engine.host("backgammon", seat="X", who="Ada")
engine.apply({"op":"join", "table": tbl, "who":"Bao"})
board = engine.board(tbl)
assert board["seats"]["O"] == "Bao" # Bao is seated
assert board["seats"]["X"] == "Ada" # Ada untouched
# old -> no "join"; seat O open RED
# new -> Bao seated at O GREENThe fix: govern the join like any move. Bao writes a JOIN intent into Bao's own object; the governor decides whether seat O is really open and writes the board. Same two rules, new op — and the split-and-merge case is handled the same way, because the memory itself re-converges after a partition.
# Bao writes a JOIN into Bao's OWN object (not the board) POST <game>.frognet/api.php?entity=move&action=upsert_by_name { "_game":1, "game":"backgammon", "table":"kitchen-1", "op":"join", "who":"Bao", "intent_seq":1 } # governor: table real? seat O open? -> else ignored # seat Bao at O, write board, record consumed # the oracle above is now GREEN.
Because a participant only ever reads the board and writes its own intent, a watcher is safe by construction: it receives, and never injects. No seat is consumed, no anti-cheat is touched. Remember that shape — receive, never inject — because it is exactly what lets someone silently join a live video call in Stage 3.
Stage 2 asked for a little: write for the tuple space instead of against a server. In return, the entire category of session, retry, and reconnect code stopped existing — and so did the REST calls themselves.
There are no REST endpoints left to define, write, and debug — and in practice that is the single biggest time-sink in network programming. You don't hand-roll request and response shapes, wire up serializers, version an API surface, or chase the bug where client and server disagree about a field. You read and write shared memory; the substrate carries it. The work that used to be the project mostly disappears.
Live video, captured and handled a completely new way.
The Communicator is not a call app bolted onto a network. It is a ground-up rearchitecture of how video is captured and moved — one that only becomes possible once shared memory and semantic compression are the substrate. Same handler interface as a board game, taken to the hardest payload there is: a live call over a link a phone would have dropped.
One handler owns two planes — and treats them as opposites. This is the rearchitecture. Conventional stacks push every frame through one pipeline and salvage the wreckage when the link degrades. SotF splits the problem in two and answers each the right way: it converges the state, and never converges the stream.
The session envelope — id, codec, sample rate, layout, current ladder level — converges like any tuple: FULL once, then DIFF against the per-peer reference, collapsing to a 25-byte REQ_REPEAT when nothing changed.
The live media. A frame never equals the one before it, so diffing is pure cost. Frames ship raw, fire-and-forget, content-blind — each just a sequence number, a level, and an opaque payload the transport never opens.
# CONTROL — the session envelope, converged FULL { "_sotf":1, "session":"call-7", "codec":"opus+vp8", "rate":48000, "layout":"hd", "level":6 } # learn it DIFF level -> 3 # starved: HD -> low DIFF level -> 6 # recovered: climbs SAME REQ_REPEAT # 25 B — the floor # DATA — the live media, never converged frame seq 9001 level 6 <opaque> frame seq 9002 level 6 <opaque> frame seq 9003 level 3 <opaque> # degrade, don't stall
The ladder steps from HD to a heartbeat and back with no redial because the quality level is a value in shared memory (Stage 2), not a renegotiated session — and it costs almost nothing to carry because the envelope rides BLDC-1 (Stage 1). Capture adapts to the level the far side wrote. That is why it is a rearchitecture, not a feature: the capture pipeline itself is driven by shared state.
A watcher joins a live call the same way a spectator opens a game board: it reads the session and receives frames, and never injects. Receive, never inject — the anti-cheat shape from the games is the safe-spectator model in the Communicator. Nothing new had to be invented for it; it was already true of the substrate.
Stage 1 made your existing REST leaner for free; Stage 2 deleted the server by coordinating through shared memory; Stage 3 rebuilt capture itself around that memory. Same handler interface throughout — overridden a little, then a lot.
This is the payoff that makes a rewrite worth it: once quality is a value in shared memory, the conversation never stops — it thins and recovers on its own, because there was never a session to fall off.