The semantic engine
It doesn't compress your data. It stops sending the parts that haven't changed.
When nothing has changed, a request and its answer cost twenty-one bytes each way — in place of a full HTTP request and a full JSON response. Nothing is being squeezed to get there. The payload simply isn't sent, because both ends already agree on what it is and can prove that agreement cheaply.
Semantic Compression — the byte accounting on camera: BLDC-1 against FNWP-1.
On camera: the Proxy and Daemon video shows a 21-byte SAME and a 31-byte reconstructed DIFF in journalctl, with the byte accounting live — this page's whole argument. Watch the series →
Twenty-one bytes, and here is every one of them.
A steady-state exchange on the FrogNet plane is this and nothing more. It is worth spelling out because the obvious reading — "clever entropy coding" — is wrong.
Protocol magic. Names the frame as FNWP-1 so nothing else can be mistaken for it.
The opcode. Which of the wire's verbs this is — repeat, diff, full, same.
The identifier. Binds this exchange to a request and an answer both ends already hold.
21 bytes out. 21 bytes back. For what would otherwise be a full HTTP request and a full JSON response. The saving is structural, not statistical — which is why there is no ratio to quote and no benchmark to lose.
Compression ratios are the wrong frame entirely. The right frame is that most network traffic is repetition — status polls, sensor readings, health checks, dashboards refreshing — and repetition can be named instead of transmitted.
Four moves, and the last one is the one people get wrong.
-
The node becomes the network
The semantic engine takes the well-known HTTP port on every FrogNet node. The web server moves to a private port behind it and becomes an origin only the local machinery talks to.
The effect: every HTTP client on the box gets this without knowing anything exists. A script, a browser, a sensor, an application you shipped years ago. No library to link, no SDK to adopt, no code to change. This is the difference between a fabric and a library, and it's the single most important architectural fact about the system.
-
It learns the shape of your traffic
The first time a given kind of request crosses a link it goes uncompressed, and both ends learn a template from it — which parts identify the request, and which parts vary from call to call. That template is stored and shared. Every subsequent request of that kind is described as a difference against it.
-
It sends only what changed
Each end keeps a reference: the last set of values it successfully exchanged with that peer for that kind of request. Three outcomes follow. Nothing sent before — send it once, in full. Some fields changed — send only those, by index. Nothing changed at all — send the identifier and nothing else.
-
The answer is proven, not assumed
When a repeated request arrives, the far end re-executes it against the real origin. It compares the new answer to the previous one, and only if they are genuinely identical does it reply "same" — with a token cryptographically bound to both the request and the response content, so it cannot be forged or guessed. If the data changed, the change comes back as a difference.
This is the part most people expect to be a cache
It is not a cache. A cache serves an old answer and hopes it is still true. This re-runs the work every time and only says "unchanged" once it has confirmed that from the origin itself.
Stale answers are not possible by construction — because no answer is ever served without being recomputed. That is a deliberate and more expensive choice than caching, and it is the reason the system can be trusted with data that matters. Correctness before compression, in that order.
Two mechanisms stop the mesh asking the same question twice.
They are keyed differently and they remove different costs, so neither substitutes for the other.
Protecting the link
When several parts of one node ask the same question at the same moment, one request goes on the wire and everyone shares the answer. The link is the scarce resource on a slow bearer.
Protecting the origin
When several different nodes ask the same question at once, the receiving node executes it once and serves them all from that single execution. The origin is the scarce resource on a small computer.
When something changes, word spreads — with three brakes on it.
A node tells its immediate neighbours, each of them tells theirs, and the mesh converges. Left alone that is a broadcast storm, so it is bounded three ways.
Brake one
Seen it, dropped it
Each wave carries an identity, and a node that has already seen that identity drops it. This is what makes an epidemic terminate instead of ringing forever.
Brake two
Neighbours, not the fleet
Each node notifies only its direct neighbours rather than every node it knows of — turning fan-out from fleet-sized to neighbour-sized, while re-propagation still covers everything.
Brake three
A burst is one event
Arrivals are debounced, so a flurry of notifications collapses into a single reconciliation instead of a queue of redundant ones.
And the notifications are themselves ordinary HTTP requests — so they ride the engine like everything else. Only the event identifier differs between them, so after the first one to a peer, every subsequent notification carries a single changed field. The gossip layer is nearly free because it rides the compression layer.
For the guild · open problem
The thresholds are a decision, not a discovery.
The boundaries between SAME, DIFF, and FULL were set by judgement and have held up in practice — but nobody has measured where they should sit for a given traffic shape. That is a bounded, self-contained piece of work with a clear way to prove you were right. Ask for a licence → — and if you want to work on this, say so and say which. The questions are optional and the licence does not depend on them.
What this actually changes.
Repetition, not entropy
Conventional compression asks how few bits can represent this message. FrogNet asks whether the message needs to be sent at all. Those are different questions, and the second has far better answers on the traffic real networks actually carry.
No adoption cost
Because the node takes the port, existing software gets the benefit unmodified. Most efficiency technologies require you to rewrite something to use them. This one requires you to install it.
And the honest boundary
This is the bridge, not the destination. Semantic exchange is what makes a network you own affordable to run — but the reason to want one is UnREST and network shared memory: exchanging memory instead of messages, so the plumbing stops existing rather than getting cheaper. The engine is how you afford the prize; it isn't the prize.
Language detection, and languages we do not have
We need: compression people, format and parsing people, anyone who has built a schema inference engine or a domain-specific encoder.
The seam is language detection and language handling. Today the classifier is deliberately empirical — sniff_body_mode never trusts Content-Type, treating it as a weak hint and letting the body decide. It resolves to one of five modes: json, xml, html, text, raw. Each has a handler, and each handler learns structure from traffic and diffs against what the far end already holds.
Better detection. The classifier is a set of _looks_like_* predicates over a decoded body. That is honest, and it is also the crudest part of the pipeline. Misclassification costs a template: a body that falls through to raw learns nothing and diffs against nothing, so every send is a FULL.
More languages, and custom ones. Five modes is what the general case needed. A protobuf handler, a CBOR handler, a CSV handler, a handler for a specific industrial or medical format — each is self-contained and touches nothing else. A domain-tuned codec can beat the general one badly, because it knows what the fields mean rather than only what shape they are. Someone who owns a data format professionally is the right person to write its handler, and they never have to read the election code.
How you would know it worked. A corpus of real bodies in the target language, with measured FULL/SAME/DIFF ratios before and after. The compression win is the proof, and it is directly measurable.
None of these is hidden behind a fallback. Each one fails visibly, says what it is, and is written down — which is the only honest way to ship something unfinished. The codec seam → · Take the free licence →
This is an initial implementation, and I don't claim it is the best.
What I claim is that it appears to work, and work correctly. That is a smaller claim than it sounds, and it is deliberately falsifiable.
BLDC-1 is one codec, arrived at by one person. The template learning, the SAME/DIFF boundary, the decision to run stock LZ4 over an already-reduced payload rather than tune a compressor — every one of those is somewhere a better answer could be demonstrated. The codec is a working group in the guild, open to anyone who wants to look at it, and the simulator is how a proposal gets settled: run it, show the numbers, and the argument is over.
Contributions do not have to be a replacement algorithm. A payload the template learner handles badly is useful. So is a case where SAME is returned and should not be. Somebody will eventually take responsibility for this subsystem, and it will be whoever has done the work rather than whoever asked for the title.
I welcome contributions that can prove a better algorithm. A messiah has followers; a journeyman has a guild.
One megabyte of JSON, six dynamic fields: 92 KB moved, not 31 MB.
The payload was what made it slow, and the payload was almost entirely structure both ends already had. Six fields moved. That is one fact, not two — and it is the row that cannot be dismissed as a small-message trick.
| Workload | Direct to Apache | Through FrogNet | Bytes on the wire |
|---|---|---|---|
| 1 MB JSON, 6 dynamic fields | 0.9 req/s | 4.4 req/s | 92 KB, not 31 MB |
| 1 MB, burst 300 | — | 4.4 req/s | 38 KB, not 347 MB |
| echo ×300 | 4.0 req/s | 4.7 req/s | 1 KB, not 39 KB |
| echo, burst 300 | — | 65.3 req/s | 12.3 KB, not 478 KB |
| chat ×25 | 2.7 req/s | 4.6 req/s | 6.4 KB, not 414 KB |
| chat, burst 300 | — | 20.4 req/s | 28.8 KB, not ~4 MB |
On the 300-way burst, 279 of 300 succeeded; 21 hit a client-side deadline. A failure to wait is not a failure to serve, and saying so is why the rest is believable. Other published sessions (this site's earlier run, the book's Seattle-to-New-York session) give different absolute rates on different hardware — read the ratios, not the targets; these numbers are from the 17 August session.