Computer Broscience - Joey Innes

The question that broke Dave

Mon, 27 Jul 2026 00:00:00 +0000

I wrote about Dave here in May 2024 — a peer-to-peer network that disseminates packets of data, without accounts, without a token, and without anyone in charge. That article carries an edit note admitting it went out of date within months. This is what happened in those months, and why I’ve just thrown most of it away.

Underneath every version of it there’s one question, and it’s the reason I keep coming back to this instead of using something that exists:

Can you build a network that behaves like a shared pool of memory — anyone writes, anyone reads, the data stays around — with no payment system, no token, and no accounting of who is storing what? Just an incentive structure simple enough that nothing has to be coordinated at all?

The constraint is what makes it interesting. Filecoin, Arweave and Storj all answer the durability half with money and proofs of storage, and they work. Take money off the table and something else has to make a node want to keep your bytes, or the pool is only ever as good as the goodwill sitting in it.

Two years on, Dave has produced three different answers to that question. Two of them are in this post because I built them. The third I only recognised while writing it, and it had been sitting in the code since 2024.

The question in this article’s title is a much smaller one, and much more boring. It’s the one that broke it.

One thing to be clear about before any of the criticism, because it changes how you should read it. This was months of full-time work, and the decisions in it were deliberate — argued through, mostly written down, often revised more than once. What follows isn’t a list of things I hadn’t considered. It’s largely the opposite: mechanisms I designed carefully, which fail for reasons that only surface when you push a formula to its limit, or think like an attacker for five minutes, or run the thing at a scale I never reached. That’s a more useful kind of mistake to read about than carelessness, and a far more common one.

Where the code is now

I’ve reset godave’s main branch back to commit 4bc87d5, from 19 November 2024, and tagged it pre-signed-dats. Everything after it — seventy-nine commits, three more weeks — lives on a branch called signed-dats. Nothing is lost, but the default branch is now the version I actually want people to read.

That version is 1,074 lines. It builds, and its tests pass. The version I abandoned in December is 3,573 lines across nine packages, and I could no longer hold it in my head.

The whole divergence is one commit, and you can see it in a single diff of the protobuf spec:

 enum Op {
-    DAT = 2;                    a thing that exists
+    PUT = 2;                    an operation on a named slot
 }
 message M {
-    bytes v = 3;   // value
-    bytes t = 4;   // time
-    bytes s = 5;   // salt
-    bytes w = 6;   // work
+    bytes datKey = 3;           addressable
+    bytes val = 4;
+    bytes time = 5;
+    bytes salt = 6;
+    bytes work = 7;
+    bytes pubKey = 8;           ownership
+    bytes sig = 9;              asymmetric crypto
 }

Addressability, ownership and signatures all arrive together. Before that commit a dat was a value, a time, a salt and a proof of work — content-addressed by its own hash, with no key and no owner. Nobody could update it, because there was no it to update; there was just the data, and the data was its own name.

The rename tells the story on its own. DAT is a thing that exists. PUT is an operation on a named slot. I changed one into the other in an afternoon and spent the next three weeks discovering what that cost.

What it did

Each node keeps a table of peers and a table of dats. Every epoch it picks one dat at random and sends it to one peer at random. That’s the entire dissemination mechanism, and it’s the part I still like.

Storage was governed by something I called mass:

func Mass(work []byte, t time.Time) float64 {
	return float64(nzerobit(work)) * (1 / float64(time.Since(t).Milliseconds()))
}

Difficulty divided by age. The idea was that a dat with a harder proof of work should persist longer, and that everything should eventually decay unless somebody keeps re-publishing it. Trust worked the same way: when a peer sent you a dat you didn’t already have, its trust score went up by the mass of that dat, and trusted peers got picked more often for gossip.

That is the first answer to the question at the top: measure what a participant contributes, and pay them in the only currencies the network has — a share of its disk, and a share of its bandwidth. No token, no ledger, and a pleasing symmetry between the two halves.

The part that worked

Before the things that didn’t, the part I’d write again unchanged — and which survives into the rewrite more or less intact.

Peer discovery and liveness are the same pair of messages. You send GETPEER; a well-behaved node replies PEER with a couple of addresses. If a peer hasn’t been heard from in DROP, it’s deleted. There’s no separate heartbeat, no separate bootstrap protocol, and no membership list — the packet that discovers peers is the packet that proves they’re alive.

That doubling-up turns out to do a third job as well, and it’s the one I’m most pleased with.

Nothing in the code mentions NAT. The traversal falls out of pinging everything you hear about, on a tick short enough that both sides do it at once.

Most nodes are behind a home router, and two of them can’t exchange UDP without help. The usual answer is a rendezvous server that tells both sides to transmit at the same moment, so each one’s outbound packet opens a return path through its own router. Dave has no such server and no code that mentions NAT at all. It gets there anyway, out of three unremarkable decisions:

// a packet from a stranger adds the stranger
_, ok := peers[pkfp]
if !ok { peers[pkfp] = &peer{pd: pkpd, fp: pkfp, added: time.Now(), seen: time.Now()} }

// gossiped addresses are added unconditionally
case dave.Op_PEER:
    for _, mpd := range pk.msg.Pds { if !ok { peers[mpdfp] = &peer{...} } }

// and the tick pings every peer in the table, verified or not
case <-pingTick.C:
    for pid, p := range peers {
        if !p.edge && time.Since(p.seen) > DROP { delete(peers, pid) }
        else if time.Since(p.peermsg) > PING { pktout <- &pkt{&dave.M{Op: dave.Op_GETPEER}, addr} }
    }

A newly gossiped peer has never sent a PEER message, so time.Since(p.peermsg) is enormous and it gets a GETPEER on the very next tick. Which means: once an edge node has told A about B and B about A, both sides start pinging each other within eight seconds, independently. Each one’s outbound GETPEER opens its own router for the other. Whichever packet arrives once both mappings exist gets through, and the receiver adds the sender on sight.

That’s a simultaneous open with nobody coordinating it. It works through full-cone and address- or port-restricted routers, which is most domestic kit; symmetric NAT and carrier-grade NAT still defeat it, because there the external port differs per destination and the address the edge saw is no use to anyone else. Good enough that I ran Dave across EC2 instances and Raspberry Pis at home without ever writing traversal code.

There’s a defence in there too, and it’s the detail I’d most like to have back if I’d lost it. Look at the two constants:

DROP        = 16 * time.Second // Time until silent peers are dropped from the peer table.
SHARE_DELAY = 20 * time.Second // Time until new peers are shared. Must be greater than DROP.

When you reply to a GETPEER, you only offer peers you’ve known for at least SHARE_DELAY. A peer that never answers is deleted at DROP. Sixteen is less than twenty, so an address that doesn’t respond is dropped before it ever becomes eligible to be passed on. You can inject a bogus address into one node’s table — anyone can, it’s a UDP packet — but it dies there, and no amount of injecting makes it spread. That’s peer-table poisoning closed off by an inequality between two constants rather than by any code that knows what poisoning is.

The same section has a rate limit on PEER messages: one accepted per PING interval, so a malicious peer can’t flood you with addresses to drown out the honest ones. I found that hole while writing the 2024 post and fixed it in the same afternoon, which is the only time writing an article has directly found me a bug.

Two honest limits on that, before I sound too pleased with myself. It stops addresses that are dead or invented; it does nothing about a live malicious node, which responds and therefore gets shared like anyone else. And discv5’s endpoint proof — the grown-up version of this — has exactly the same limitation, so at least the ceiling is in the right place.

I should be calibrated about the rest of it too. Combining discovery with liveness is what Kademlia’s FIND_NODE does. Refusing to gossip reputation is a call several deployed networks got wrong, but it isn’t new. Implicit hole punching happens incidentally in most DHTs — what’s unusual here is only that it was designed for rather than stumbled into. The delay inequality is the one I’d defend as properly neat, and even there the goal is standard; it’s getting it from an ordering between two timeouts rather than a round trip and some cryptography that I haven’t seen elsewhere.

So: not novel, mostly. I’ve stopped thinking that matters much. Landing independently on the same answers as the systems that got it right is the outcome you want from a first attempt at something, and it’s a better one than a genuinely new idea that doesn’t work.

None of this is what went wrong. The dissemination and storage side is where it comes apart, and it does so in three separate ways.

One packet is enough

Look at what happens to mass as age approaches zero.

The cheapest legal dat is eight leading zero bits, about 256 hashes to find. At any age below 0.32 ms it is worth more than the entire trust cap.

Trust accumulates like this:

if novel {
    trust := peers[pkfp].trust
    if trust < MAXTRUST {
        trust += Mass(pk.msg.W, Btt(pk.msg.T))

MAXTRUST is 25. The cheapest dat the network accepts has eight leading zero bits, which costs about 256 hashes to find — call it free. Mass for that dat is 8 / age_in_ms, so it exceeds 25 at any age below 0.32 milliseconds.

An attacker mints a dat, hands it straight to the socket, and it arrives at a peer sub-millisecond on a LAN or from a colocated machine. It is novel by construction, because they invented it. One packet takes them from nothing to the trust ceiling.

And if the timestamp on the dat is the current millisecond, time.Since(t).Milliseconds() is zero, 1 / 0 in float64 is +Inf, and mass is infinite. The cap saves the arithmetic from doing any damage, but only by discarding whatever the number was supposed to mean.

The mistake underneath the arithmetic is the interesting one. I was rewarding peers for supplying novel data with valid proof of work, and I’d convinced myself that was evidence of useful contribution. It isn’t. It’s evidence of having spent CPU, and the attacker has CPU. A reputation metric only works if the thing it rewards is scarce to an adversary and abundant to an honest node, and “bytes nobody has seen before” is the opposite: trivially abundant to whoever is willing to make some up.

MAXTRUST exists precisely because I’d thought about farming — an uncapped score is obviously gameable, and the cap bounds what any peer can accumulate no matter how much they send. What defeats it is the interaction between the cap and the divergence, and that only shows up if you take the formula to its limit and ask what mass is worth at an age of zero. A cap you can reach in one packet isn’t a reputation system; it’s a single bit, and the bit says “has sent me something”, which every attacker can also say. The defence was real. It was aimed at the wrong axis.

Mass was recency wearing a costume

Dats are stored in shards, and the shard is chosen by difficulty:

func keys(work []byte) (uint8, uint64) {
	return min(MAXWORK, nzerobit(work)) - MINWORK, xxhash.Sum64(work)
}

Every dat in shard k has exactly k + 8 leading zero bits. When the pruner keeps the most massive dats in a shard, the difficulty term in mass is the same constant for every dat it’s comparing — so the comparison collapses to one over age. Within a shard, mass is recency. The difficulty half of the formula does no work at all.

It isn’t quite doing nothing globally, and what it is doing is the best idea in the codebase — which I only worked out properly while writing this.

Difficulty picks your shard, and each shard has its own independent capacity. So a harder proof doesn’t buy you priority directly; it moves you into a less contended queue. And because every node prunes its own shards independently, the effect compounds across the network into a self-clearing price for permanence. Want a dat to survive? Do more work and land somewhere emptier. As more people do sixteen bits, that tier fills and the price drifts to twenty. Nobody sets the price, nobody verifies anything, no payment changes hands, and there is no ledger, no contract and no challenge-response anywhere in it. It falls out of local pruning decisions alone.

That’s worth sitting with, given where the project went. Every other way of buying durability in a decentralised store is heavy: Filecoin and Arweave use money and proofs of storage; Freenet uses popularity. The version of Dave I abandoned used storage challenges — the mechanism that dragged in replica tracking and reassignment and eventually ate the whole design. I already had a mechanism that gets a weaker version of the same property for nothing, and I walked past it to build the expensive one.

I can’t claim it’s unprecedented; I know the well-known systems rather than every paper. But I haven’t seen proof-of-work difficulty used as a storage tier with per-tier caps anywhere else, and I’d have expected to.

What defeats it in practice is mundane. Almost nobody does more work than the minimum, because nothing in the interface suggests a reason to, so everything piles into the first shard and fights over the same slots while fifty-five tiers above it sit empty. A market with one participant doesn’t clear. Combined with the mass formula collapsing to recency, the effective policy was “keep the most recent few thousand dats”, which is not what the documentation said. The interaction is subtle enough that I wrote both halves and still didn’t see it: partitioning by difficulty and then ranking by a formula containing difficulty looks like belt and braces, right up until you notice the second term is identical across everything it’s being asked to compare.

The idea deserves a better implementation than I gave it, and it’s separable from all of this — see below.

The anonymity claim was too strong

From the README on that pinned commit:

Propagating dats in this way ensures that an adversary is unable to create a timing attack to discern the source of a dat, even with a broad view of network traffic.

The mechanism is sound and I still like it. Because every node emits one random dat to one random peer every epoch, at a constant rate, there’s no correlation between when a node received something and when it passes it on. A forwarder is genuinely indistinguishable from another forwarder.

But that’s forwarding. Origination is a different act, and the code does it differently:

case dave.Op_DAT:
	put(dats, &Dat{m.V, m.S, m.W, Btt(m.T)})
	for _, p := range rndpeers(peerList, trustSum, FANOUT, 0, 0) {
		pktout <- &pkt{m, addr}
	}

When an application writes a dat, it goes out to five peers immediately, and then repeats on a timer for however many rounds the caller asked for. The originator is the first node to transmit it and the most frequent. Random push hides everyone downstream of the source. It doesn’t hide the source, which is precisely what the sentence claims it does.

I don’t think I was being dishonest when I wrote that. I think I’d designed one mechanism carefully, and then let the claim it supported creep one step wider than the evidence.

The one I never solved

Those three are things I got wrong. This one is different — it’s the problem I worked hardest on, never cracked, and still think is the most interesting thing in the project.

Every epoch a node sends one dat to one peer. Which dat?

v1’s answer is: pick one at random. That works beautifully at the scale I ran it — data moved around exactly as it should across a handful of machines. But the selection probability for any given dat is one in N, so as the store grows a newly-arrived dat waits proportionally longer for its turn. At a thousand dats it’s fine. At a million, a new dat is essentially never chosen, and propagation stops working at all.

The left panel is what v1 does. The right is what the problem actually wants — two mechanisms, because the two jobs have opposite requirements.

I tried a lot of things. The one still in the pinned code is a ring buffer: every novel dat is written to a thousand-entry ring, and each push epoch sends one dat from the ring as well as one from the store at large. That genuinely fixes the acute half — new data propagates immediately, however big the store gets.

What it doesn’t fix is the tail. A dat that falls out of the ring is back to one in N, so anything a node missed while offline, or anything that existed before it joined, only ever reaches it by luck. I also tried cuckoo filters: track what each peer has already seen, and don’t waste a push sending them something they’ve got. That work is on a branch called filter and it isn’t in the pinned version, which tells you how it went.

None of it worked because I was asking one mechanism to do two jobs with opposite requirements. Getting new data out fast wants a small working set, aggressive sending, and a short life. Repairing the long tail wants completeness, an unbounded working set, and can afford to be slow. A single random pick from a single pool is a compromise that is bad at both.

The literature has known this since 1987. Demers et al., Epidemic Algorithms for Replicated Database Maintenance, splits it exactly along that line: rumour mongering for new items — push them hard, and each time you push to someone who already has it, probabilistically stop, so the hot set limits itself and stays small — and anti-entropy in the background, periodically reconciling state with a random peer. You run both. My ring buffer is rumour mongering with its stopping rule missing; I had the instinct without the feedback that makes it terminate.

And the cuckoo filter was simply the wrong primitive, which is the part I’d most want to tell 2024 me. A membership filter answers “do you have this one?”, an item at a time. The question you actually need answered is “what are you missing?” — in one round trip, in bytes proportional to the size of the difference rather than the size of the set. That’s set reconciliation, and it’s a different tool entirely: Merkle trees over sorted ranges, as Cassandra and Dynamo use for repair; IBLTs, as Graphene uses for blocks; or PinSketch, which is what Bitcoin’s Erlay uses — having hit precisely this problem with transaction relay and solved it by pairing low-fanout flooding with periodic reconciliation.

The rewrite changes the shape of the question rather than answering it head-on. Flooding doesn’t select: every cell goes to every neighbour exactly once, deduplicated by the seen-set, so for new data there’s no choice to make and the problem doesn’t arise. What’s left is the tail — a node that was offline, or has just joined — and that is now a separate mechanism with its own name, the time-range sync. It’s the same split Demers describes, arrived at from the other direction.

Which puts the interesting work in exactly one place. That sync is currently naive and re-sends things you already have. Replacing it with real range-based reconciliation is the piece I most want to build, and it’ll be the first time in this project that the hard problem in front of me is one with a well-understood answer.

The question

Here’s the one that actually killed it.

A dat is addressed by its own work hash. To retrieve one, you send GET with that hash, and any peer that has it replies. So: how do you get the hash?

You don’t. Not from the network. There’s no index, no enumeration, no namespace, nothing to browse. You can only ask for something you were already told about through some other channel — and if that channel exists, a lot of the reason for the network existing has quietly moved into it.

It’s worse than that, because GET doesn’t route. A read checks your own store first, and on a hit it returns without sending a single packet:

case dave.Op_GET:
	shardKey, datKey := keys(m.W)
	dat, ok := dats[shardKey][datKey]
	if ok {
		found = true
		apprecv <- &dat
	}
	if !found {
		for _, p := range rndpeers(peerList, trustSum, FANOUT, 0, 0) {
			pktout <- &pkt{&dave.M{Op: dave.Op_GET, W: m.W}, addr}
		}
	}

Left: what godave did on a local miss. Right: what the rewrite does instead. The second panel isn't a better lookup — it's the absence of one.

On a miss it goes to five peers picked at random and repeats on a timer. No key space, no distance metric, nothing pointing the query towards whoever holds the data.

That local check is worth dwelling on, because it looks like the answer and isn’t. It’s a cache lookup, and a cache is only as good as the thing it misses to. What sits in a node’s store is a random sample of recent traffic — you receive one random dat per peer per epoch, and the pruner discards everything past the shard capacity every minute. The chance that an arbitrary hash happens to be in there is low, and it falls as the network holds more data. The fallback is doing the real work, and the fallback is the broken part.

Which is also why I didn’t see this for months. On a dozen nodes on my desk, with capacity far exceeding the data, the local hit rate is close to one and the network path almost never fires — and when it does, five out of twelve peers is most of the network. Dave behaved correctly at every scale I ever ran it at. The lookup only fails once the data in the network exceeds what a single node can hold, which is precisely the point at which you would want it to work. I called Dave a distributed hash table in that first article. It was a distributed hash table with no table, and a small enough network to hide it.

So I gave dats a key, and an owner, and a signature — the commit at the top of this piece. Now data had a name you could form without being told it, and a namespace per public key you could enumerate. That is a genuine fix for a genuine problem, and I’d make the same call again with the information I had.

What followed was mechanical. If dats have keys, the sensible place to put a dat is near its key, so I added XOR distance. If placement is deterministic, some node is supposed to hold each dat, so I added replica tracking. If replicas are tracked, peers leaving means reassigning them, so I added that. If replicas can be reassigned, I need to know whether a peer is actually storing what it claims, so I added storage challenges. Values then outgrew a UDP datagram, so I added a TCP fallback — a second transport, with its own code path for the same operation.

Every one of those follows from the one above it. Not one of them was a bad decision on its own terms. Together they deleted the property the whole thing was for: once placement is deterministic, an observer knows exactly which nodes should hold what, and “random push for anonymity” no longer hides anything at all.

By December I was sketching a third system inside the second — direct messaging, layered on the keyed store, with recipient-keyed namespaces. I never committed it. That’s where I stopped.

Deleting the question

I’m writing it again, in Zig, and the whole design rests on one rule I wrote down before any code:

A node never needs to know anything about the network beyond its immediate neighbours.

The point of the rule isn’t what it permits. It’s what it refuses. GET(key) means routing a query towards a node you can’t see, so there is no lookup — and with no lookup there is no placement, no replica set, no reassignment, and no storage challenge. The entire collapse is rejected by one sentence, and it’s rejected automatically, before I have a chance to talk myself into each step.

So how do you read anything, with no lookup?

You already have it. Everything floods to everyone, and reading is a query against your own disk. Following someone is a filter over your own store by their key. There’s no relay to ask, no query to route, and nothing to place, because nothing was ever anywhere in particular.

That is the same first move v1 made, with a completely different guarantee behind it. v1 checked a cache and fell back to a broken query; this checks a replica, and there is no fallback because none is needed. The difference isn’t the lookup — both start by looking in the same place. The difference is whether the design is working for or against that lookup succeeding. Random push and mass pruning actively hold a v1 node’s store down to a small sample; flooding and a retention window deliberately fill it.

That sounds like it can’t possibly scale, and it scales further than I expected. At a million users each posting ten times a day, a node carries about 0.6 Mbit/s and 2% of one core. Ingress doesn’t bind until several hundred million users on a normal home connection. Storage is the real cost, and it’s each node’s own choice of how far back to keep — a phone keeps a week, a machine with a spare disk keeps everything and becomes an archive.

That last part is deliberately not a protocol feature. There’s no mechanism to declare an archive, verify one, reward one or challenge one, and there is never going to be, because that mechanism is exactly what “storage challenges” were. Data stays available for as long as somebody chooses to keep it, which is how Usenet worked and how Nostr works. Any protocol promising more than that is either lying or has an accounting system that will eventually eat it, and I’ve now written the accounting system that ate mine.

That is the second answer, and it works by refusing the question. Nobody is doing anyone a favour, so there is nothing to pay for: you receive everything regardless, and you keep the part you want to read. The simplest possible incentive structure is no incentive structure — just self-interest, and no coordination because none is required.

It’s an honest answer and I think it’s the right one to build on. It also has a hole in it, which took me a while to see. Self-interest keeps the data somebody wants. It does nothing at all for data that nobody in particular wants yet — the unpopular, the archival, the thing you publish today that matters in five years. Under this scheme that data quietly evaporates, and there is no way to pay to keep it, because paying is the thing we removed.

Everything on the wire is one fixed size:

Every packet is exactly this shape, so size and structure leak nothing. A message that doesn't fit is split across several cells, each sealed independently.

There’s no recipient field. Every node tries to open every cell against its own key, and the authentication tag says whether it worked — so nobody, including the node holding a cell, can tell who it’s for. On my machine that costs 31 microseconds a cell, essentially all of it the X25519 scalar multiplication.

The peer layer comes across almost unchanged: discovery and liveness in one message pair, gossip only in replies, sharing gated behind a delay longer than the drop timeout, and reputation kept strictly local and never gossiped. The one correction is that identity becomes the public key rather than the address, because a table keyed by ip:port treats a node that moves as a stranger — and on IP, moving is the normal case, not the exception.

The traversal needs writing down this time, which it never was. Nothing in v1 records that the ping interval is load-bearing for anything but liveness, and the obvious optimisation — back off the tick, skip unverified peers — is correct for liveness and quietly fatal for hole punching. I nearly made exactly that change while reviewing this code, having read the same three lines and concluded the traversal was accidental. It wasn’t; it was the reason the discovery design looks the way it does. The mechanism survived, and the note explaining it didn’t.

Why Zig

Partly because I want to, which is a sufficient reason for a hobby project. But there’s a real argument too, and it’s about where this code can run.

The protocol has no business needing an allocator on the receive path: a cell is a fixed 224 bytes, and opening one is arithmetic on a fixed-size buffer. Zig lets me write that once and compile it for a server or for a microcontroller with no operating system underneath it. I checked rather than assumed — the whole receive path, X25519 and BLAKE3 and ChaCha20-Poly1305 and the framing, cross-compiles to a Cortex-M4 in an eighteen-kilobyte object, with no allocator and no OS.

I’m not building the radio version now; I scoped this rewrite to IP so that it might actually get finished. But it means the door stays open, and it’s the one thing this project can do that the mature alternatives can’t.

Zig 0.16 is also recent enough that most of what’s written about the language online no longer compiles. main takes a std.process.Init, randomness comes through std.Io, std.time.Timer is gone. I’ve been finding these by compiling rather than by reading.

A loose end worth picking up

The third answer is the one I didn’t recognise until I sat down to write this, and it had been in the code since 2024. It’s also the piece that fills the hole in the second: a way to buy permanence for data nobody else wants, paid for in work rather than money, with nothing keeping accounts.

I want to write it down properly, because it doesn’t need a peer-to-peer network to be interesting and I’d like to come back to it.

The mechanism. Every stored item carries a proof of work of difficulty d, in leading zero bits. The store is partitioned into tiers by d, and each tier has its own capacity. When a tier is full, it evicts within itself — oldest first is fine. A writer chooses d before publishing, so a writer is choosing which queue to stand in.

Why it should clear. Treat each tier as a queue with capacity C and arrival rate λ. In steady state, an item’s expected residency is T = C / λ — Little’s law, and for evict-oldest it’s exact. So the writer faces a straight trade: pay 2^d hashes, receive C_d / λ_d seconds of retention.

That’s a market, and it clears without anyone running it. If a tier gets popular its arrival rate climbs, residency falls, and the marginal writer moves up a bit. Cost doubles per bit while capacity stays put, so demand spreads itself across the tiers geometrically. No payment, no ledger, no proof of storage, no contract — just local eviction, on every node independently, adding up to a price.

What was missing, and it’s the whole failure. A market needs four things. A cost function: proof of work, present. Differentiated goods: the tiers, present. Independent clearing per good: per-tier caps, present. And a price signal — which was entirely absent. Nothing in Dave ever told a writer that tier 0 was full and tier 12 was empty. There was no way to observe occupancy, so there was no way to act on it, so every writer rationally paid the minimum. The market had a price and no way to display it, and a market with one participant doesn’t clear.

The fix is small: have nodes report how full each of their tiers is, and let a writer sample a few and pick a difficulty from how long they want the thing to live. Occupancy is aggregate information about your own disk, so publishing it leaks very little.

How to test it with no network at all. This is why it’s worth doing separately. Simulate a population of writers with some distribution of how much they care about persistence, and one node’s store with tiers and caps. Feed arrivals in, evict, and measure whether residency time ends up proportional to cost. That’s a discrete-event simulation of a caching policy in a couple of hundred lines, and it answers the question completely — no sockets, no peers, no protocol.

Four things I’d want it to tell me:

How to divide capacity across tiers. v1 gave every tier the same cap, which wastes most of the disk when demand is concentrated at the bottom. Allocating in proportion to observed demand is self-defeating, because the tier you escape to grows to meet you. There’s a right answer here and I don’t know what it is.
Whether it oscillates. Publish occupancy and everyone can see the same empty tier. They all move there, it fills, they all move again. That’s textbook herding, and it probably needs damping or hysteresis to settle.
What lying buys you. A node that reports tier 20 as empty makes writers burn work for nothing. Sampling several nodes and taking a median helps, and a writer can check empirically whether the item actually survived, but this needs thinking about.
Whether the absolute numbers work at all. Dave’s minimum was eight bits — 256 hashes, which is nothing. Twenty bits is about a million, still only seconds of CPU, so the “expensive” tiers can be squatted cheaply. The scheme only prices anything if the top tiers are genuinely costly, which means calibrating against current hardware rather than against a number I picked in 2024.

If it survives that, it’s a retention policy any store could use, and it never needed to be part of a network in the first place.

What I don’t know

Whether the question at the top has a good answer at all. Three attempts so far. Reward measured contribution — which failed, because anything I could measure locally was something an attacker could manufacture. Refuse the premise and rely on self-interest — which works, and leaves anything unpopular to rot. Price permanence in work and let local eviction clear a market nobody runs — which is untested, and is the only one of the three that’s an incentive structure in the sense I originally meant. One is built, one is a cautionary tale, and one is a weekend of simulation away from being either interesting or dead.

Whether spam is survivable. Flooding means anyone can inject, everyone must process, and now that data is retained, spam persists as well as propagates. My current answer is to ration rather than reward — per-neighbour rate limits, and a single shared budget that all unknown authors compete for, so ten thousand fake identities get one stranger’s worth of throughput between them. That’s an argument, not a result. So was the trust metric.

Whether anyone will use it. Almost certainly not many, and that’s fine. I want it to be good, and used by a handful of people I know, which is a different target from the one I was implicitly aiming at in 2024 and a much more reachable one.

What I do know is the thing I’d tell 2024 me. The first version had no rule that could reject a feature. Every addition answered “is this good?”, and each one honestly was. None of them ever had to answer “is this ours?” — and by the time that question would have been useful, there was no longer a clear enough idea of what “ours” meant to answer it with.

The code is at github.com/intob/godave, pinned where it was still one idea.

Writing a panel-method solver in Zig

Sun, 26 Jul 2026 00:00:00 +0000

I’ve been flying soft wings for years without really understanding the sections they’re built from. Paragliders, speedwings, kites — they all live or die on the shape of a slice through the wing, and I’ve mostly taken that shape on trust. So I’ve started building something to fix that: a simulation backend called Camber, which takes a wing section, an angle, and an airspeed, and tells you what the air does about it.

This is the first of what I expect to be quite a few posts. It covers the part that works: the panel method. My maths is not strong, so I’ve had to build the intuition first and let the equations follow, and that’s how I’m going to explain it. If you can picture air moving, you can follow all of this.

Two stories about air

Air behaves in two completely different ways depending on how close to the wing you are, and almost every practical aerodynamics tool is built on that split.

Nearly everywhere, air is so slippery that friction may as well not exist. It just has to flow around the shape rather than through it. Solve that, and you get the pressure everywhere on the surface — and pressure is where lift comes from.

Then there’s a film glued to the surface, about one percent of the chord thick, where friction is the only thing that matters. That film is where drag comes from, and where stall comes from.

This post is entirely about the first story. It’s the easier one, it runs in about a millisecond, and it gets you lift and pressure. The second one is much harder, I’m still fighting it, and it’ll get its own post.

A wing is a list of points

The first job is turning “NACA 4412” into something a computer can chew on. You walk all the way around the section and write down coordinates, then join consecutive points with straight lines. Those straight segments are the panels.

Chord is always normalised to 1, so the nose is at x=0 and the tail at x=1, whatever the real wing measures.

Two details in that picture do real work.

The points aren’t evenly spaced. They’re bunched at the nose and the tail using a cosine distribution. Over the middle of a section the flow does almost nothing interesting — the pressure changes gently and slowly. Around the nose it goes berserk: air comes to a dead stop, then accelerates to well above freestream speed within a couple of percent of chord. Even spacing would spend most of your panels describing the boring part and smear the most important feature on the whole surface.

And the direction you walk matters. I go clockwise — tail, along the bottom, round the nose, back along the top. Every panel computes its outward normal as “the direction I’m travelling, rotated ninety degrees”. Walk the other way and every normal points into the wing, and the sign of the lift flips. I have this written in capital letters in my notes.

The trick: throw the wing away

Here’s where it gets clever, and where I had to stop and think for a while.

You’d assume you solve the flow around a solid object. You don’t. You delete the object entirely and replace its surface with a vortex sheet — imagine laying a carpet of tiny spinning rollers along where the skin used to be. Each roller drags the air around with it.

The rollers have no physical existence. They're a device for producing a flow that happens to follow the shape you care about.

Now ask one question: how hard does each roller have to spin so that the wind plus all the rollers together never pushes air through the line where the skin used to be?

Answer that, and you’ve solved the flow — because a flow that doesn’t cross the skin is indistinguishable from a flow around a solid wing.

Why spinning rollers, specifically? Because spin is what makes lift. You could lay down a carpet of little sprinklers instead, blowing air outwards, and they’d make the shape appear fatter — but they’d generate exactly zero lift no matter how you tuned them. Net circulation around the section is lift. Vorticity is the only ingredient that can produce it.

Chop the surface into N panels, let the spin vary linearly along each one, and the unknowns become the values at the N+1 corners. Then write one equation per panel: at this panel’s midpoint, the flow perpendicular to the surface is zero.

That’s N equations and N+1 unknowns. One short.

The bit where nature has to choose

That missing equation isn’t a bookkeeping mistake. It’s the most interesting thing in the whole method.

Frictionless flow theory genuinely does not have a unique answer. There’s an entire family of valid solutions around any wing section — one for every possible amount of circulation — and every single one satisfies “no air through the skin” perfectly. In one of them the air whips around the sharp trailing edge from below to above. In another it goes the other way. In exactly one, the flow leaves the trailing edge smoothly, both streams merging and departing together.

Both are mathematically valid. Only the right-hand one is ever observed.

Nature picks the smooth one. It picks it because real air has viscosity, and viscosity flatly refuses to let flow turn a sharp corner at infinite speed. The first attempt to do so sheds a vortex downstream — you can watch this happen in a water tank — and that departing vortex leaves behind precisely the circulation needed to make the trailing edge smooth ever after.

So the missing equation is a fact about sticky air, smuggled into a model that has no friction in it anywhere. In my code it is one line:

// Kutta condition: gamma_0 + gamma_N = 0 (closes the (m x m) system).
a[n * m + 0] = 1.0;
a[n * m + n] = 1.0;

The spin strength just under the tail, plus the spin strength just above it, equals zero. Since the sheet strength turns out to be the local surface speed, that says the speeds on the two sides of the trailing edge match: no jump, no infinite turn.

This is where the lift comes from. Everything else in the file is bookkeeping. When I had a sign error early on and my sections produced negative lift, this was where the bug was.

It’s all one linear system

With that last equation in place you have a square system: a matrix, a vector of unknown spin strengths, and a right-hand side.

N rows of “no air through this panel”, plus one row of physics.

Each entry of the matrix answers a very concrete question: if roller j were spinning at unit strength, how much air would it push through panel i? That’s pure geometry — there’s a closed-form expression involving a logarithm and an arctangent, and it’s the same on every panel method ever written.

The bit worth noticing is what isn’t in the matrix. The angle of attack appears nowhere in it. It only touches the right-hand side. So the expensive half — factorising the matrix, which is the cubic-time part — gets done once per shape, and each new angle is a cheap back-substitution. Running a whole polar sweep across twenty angles costs roughly the same as solving one.

Getting the answer back out

Once you’ve solved for the spin strengths, you might expect to have to sum up every roller’s contribution at every point to find the surface speed. You don’t, and this is my favourite part.

A vortex sheet is defined as a jump in tangential velocity across it. The solution was constructed so the air inside the imaginary body is perfectly still. So the jump across the sheet is simply the speed outside it. The strength you solved for is the speed of the air sliding over the skin.

Then Bernoulli turns speed into pressure — fast air means low pressure — and the whole thing collapses to one line:

const gamma_mid = 0.5 * (gamma[i] + gamma[i + 1]);
cp_arr[i] = .{ .x = geom.xc[i], .cp = 1.0 - gamma_mid * gamma_mid };

Cp is the pressure coefficient: local pressure minus ambient, divided by the pressure you’d get by bringing the freestream to a dead stop. Dividing through like that means it doesn’t depend on how fast you’re going or how thick the air is — the same curve holds at 10 m/s at sea level and 30 m/s at three thousand metres. It’s the universal currency of this field.

Here’s what actually comes out of the binary:

echo '{"naca":"4412","aoa_deg":6,"n_points":200,"re":1000000}' \
  | ./solver/zig-out/bin/camber-solver

Straight out of the solver: 198 panels, Cl = 1.2386. Everyone plots Cp upside down, so that the area between the curves reads as lift.

Three things to read off it. Cp = +1, bottom left, is air brought completely to rest — the stagnation point, and the highest value physically possible. The big negative spike just behind the nose is the suction peak, where air is moving far faster than freestream and pressure has dropped well below ambient; most of the lift lives there. And the long climb from the peak back towards zero at the tail is the pressure recovery — air being asked to move into rising pressure. That climb is the whole ballgame for the second story, because it’s what eventually pushes the boundary layer into separating.

The drag that isn’t drag

The solver reports a drag number. It’s a lie, and I’ve had to be quite disciplined about saying so.

In genuinely frictionless flow, a closed body experiences no drag at all. The pressure pushing back on the front is exactly cancelled by pressure pushing forward on the rear. This is d’Alembert’s paradox, and it bothered people for about a century before boundary layers were understood.

So the drag my panel solver computes is not physical. It’s the residue of chopping a smooth curve into flat segments — a measure of how good my panelling is, nothing more. On the run above it’s 0.000172, which is a couple of orders of magnitude smaller than the real drag of that section. If it ever gets large, that tells me my resolution is bad, not that I’ve designed a draggy wing.

I have a rule written into the project: never report this number as drag. It would be very easy to ship something that looks like a working aerodynamics tool and is quietly nonsense.

Two checks that keep me honest

The pressure integration gives lift. But there’s a completely independent route to the same number — Kutta–Joukowski, which says lift is just twice the total circulation. Two different calculations from the same solution:

cl      = 1.2386
cl_circ = 1.2387

If those ever diverge by more than a percent, the solver attaches a warning to its own output. It’s a cheap and brutal test: the two agree only if the solution is actually right.

The second check is that every fast path has a slow twin. The influence matrix is built with SIMD, four panels at a time, and there’s a plain scalar version of the same code that a test requires to agree to floating-point round-off. The optimisation is only safe to keep because the twin exists.

One result that looks like a bug and isn’t: my lift slope comes out at 6.86 per radian, against the textbook thin-aerofoil value of 2π ≈ 6.28. That’s 9.2% high. It’s meant to be. Thin-aerofoil theory linearises the thickness away, and thickness genuinely adds lift. Every thick-section panel method overshoots the textbook line by about this much. It took me an embarrassingly long time to stop trying to “fix” it.

Why Zig

I wrote about first impressions of Zig a while back, and this project has mostly deepened what I liked.

Arena allocators suit this problem perfectly. A solve allocates a lot of scratch — matrices, geometry arrays, per-panel state — and every bit of it dies at the same moment. So you allocate freely from an arena and throw the whole thing away at the end, with no per-object bookkeeping at all.

@Vector(4, f64) gives you SIMD without intrinsics or a library, and it stays readable.

And the discipline that’s paid off most isn’t a language feature but a rule I set early: no global state, everything passed as a parameter. That felt slightly pedantic while writing it. Then I wanted to run six independent validation cases at once, and it turned out there was nothing to synchronise, because there was nothing shared. It just worked, on the first attempt, for a 3.6× speedup.

The thing I still find genuinely hard is the build system. It’s powerful and it’s clean, but 0.16 moved to a module-based build.zig and most of what you’ll find online is written against something older.

What it can’t do

This is worth being blunt about, because a plausible-looking wrong answer is worse than no answer.

There is no viscosity in any of this. So: no real drag, no stall, no separation, no transition from laminar to turbulent flow, no compressibility. You cannot get a glide ratio out of it, because you cannot get drag out of it. You cannot find the stall angle, because nothing in the model knows how to stall — push the angle up and the lift just keeps rising happily, forever, which real wings famously do not.

What it does give you, and gives you fast and accurately, is lift and the pressure distribution for attached flow. That turns out to be exactly the input the second story needs.

The boundary layer. That’s where drag and stall live, and it’s a much nastier problem — the thin film and the outer flow each need the other’s answer, so you end up iterating between two solvers until they stop arguing. I have that working, mostly. I also have a validation gate that it currently fails on two criteria out of five, which I’ll write about honestly, because the failures are more interesting than the passes.

The diagrams in this post were generated from the solver itself rather than drawn by hand, which felt like the right way to do it. If any of it is wrong, I’d genuinely like to know.

How we built Soaring Spirit

Sun, 26 Jul 2026 00:00:00 +0000

Soaring Spirit is my first company. It’s a Swiss GmbH, it runs speedflying and paragliding trips and coaching, and the people in it are pilots rather than programmers — Greg cooks and does the osteopathy, Jamie has more speedwing hours than almost anyone alive, Oli drives the bus and keeps the books, Nika gets us into the parts of Georgia nobody else can reach. So when I say “we built”, I mean the company is a we. The software is mine, and this post is about the software.

First commit was 2025-03-13. Sixteen months and 706 commits later it books trips, takes deposits and balances through Stripe, issues invoices, runs a coaching-availability calendar, hosts a flying syllabus, keeps a database of speedflying incidents scraped from the Swiss and French fatality reports, and logs in every user without ever having stored a password.

All of it is one Go binary, cross-compiled to arm64 and copied onto a Raspberry Pi in my house.

I want to write about the constraints rather than the features, because the constraints are the interesting part. Nearly every decision here was some version of “what is the smallest thing that can possibly work”, and a few of them were wrong in ways that took months to surface.

One binary, one machine

The deploy script is worth quoting more or less in full, because it is the whole operations story:

GOOS=linux GOARCH=arm64 go build -ldflags "..." -o ss_linux_arm64 .
scp ss_linux_arm64 "$HOST:~/ss.new"
ssh "$HOST" "mv ~/ss.new ~/ss && sudo systemctl restart ss"

That’s it. Build, copy, restart. The Pi sits behind a Cloudflare Tunnel, so there is no public IP, no inbound port forwarding, no load balancer and no bill. Postgres runs on the same machine. pgbackrest ships backups to S3 on a schedule.

The frontend is deployed by rsync, and then the Cloudflare cache is purged. There is no build step for it at all — I’ll come back to that.

The obvious objection is that this is one machine and one power cut. That’s true, and it’s a real risk I’ve accepted rather than solved. The mitigation is entirely on the data side: the backups are off-site and verified, and the binary can be rebuilt and redeployed onto anything with an ARM chip in about ninety seconds. What I’d lose in a hardware failure is availability, not the business. For a company that sells six trips a year to people who mostly book after talking to us in person, that’s the right trade. It would be a bad trade for something with real-time revenue.

The less obvious cost is that one process serving both the API and the site means a deploy takes the whole site down for the second it takes systemd to restart. I’ve never minded this and I doubt anyone has noticed, but it’s the kind of thing that’s fine until it isn’t.

Ten dependencies

The go.mod has ten direct requirements: a Postgres driver, Stripe, WebAuthn, an SMTP pool, three image libraries, and three golang.org/x packages. No web framework, no ORM, no dependency injection container, no code generation.

The router is 154 lines. Here is essentially all of it:

func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	path := req.URL.Path
	for _, route := range r.routes {
		params, ok := matchPath(path, route.Segments)
		if ok && route.Method == req.Method {
			// ...inject params into the context, wrap in middleware, serve
		}
	}
	r.notFoundHandler.ServeHTTP(w, req)
}

A linear scan over about 110 routes, splitting the path on / and comparing segments. It is not clever and it does not need to be — this is a Raspberry Pi answering maybe a few thousand requests a day, and the linear scan is nowhere near the top of any profile I’ve taken. I wrote it before Go 1.22 gave the standard library pattern matching, and I’ve left it because it works and because the middleware ordering is explicit in a way I like.

I’m not going to pretend this is universally the right call. I wrote a router, a session store, an image pipeline and a mail queue that all exist as libraries, and each of them cost me a weekend I could have spent on the actual product. What I got back is that when something breaks at eleven at night before a trip, the stack trace goes through code I wrote, and I can read all of it. That has paid for itself several times.

The honest counterweight is the test suite: about 11,000 lines of Go tests against 19,000 lines of everything else, across 31 integration test files that run against a real Postgres in Docker Compose. Writing your own primitives is only defensible if you also write the tests that a library would have shipped with. I didn’t understand that at the start and the early code shows it.

No frontend build step

The frontend is about 20,000 lines of hand-written JavaScript. Lit is vendored as a single file. There is no bundler, no transpiler, no npm run build, no node_modules in the deploy. Files are served from disk, so an edit is live on reload.

Routes are plain objects with a dynamic import:

{
	path: "trip/:id",
	component: "ss-trip-detail",
	action: async () => await import("./components/app/trip/trip-detail.js")
}

The browser does the code splitting, because ES modules already are code splitting. Every component is a LitElement with its styles in a tagged template literal next to its markup.

What this buys is that the edit-to-live loop is a file save, and that there is no build to break. What it costs is that node --check is the entire compile step — nothing catches a typo in a property name until it renders wrong. Over sixteen months, that has cost me less than I expected. Shadow DOM turns out to do most of what a build step’s CSS scoping would.

It has one wart I still haven’t fixed properly. The server pre-renders the SPA shell’s <head>, so it keeps its own table of route titles and descriptions in app/prerender.go, mirroring the one in site/index.js. Two hand-kept copies of the same data. It’s documented in three places and I’ve still got it wrong once.

Money is where the discipline goes

Everything above is “how little can I get away with”. Bookings are where that stops.

A booking moves through ten states — DRAFT, IN_REVIEW, AWAITING_DEPOSIT, DEPOSIT_PAID, DEPOSIT_TO_REFUND, DEPOSIT_REFUNDED, AWAITING_BALANCE, FINALISED, VOID, REJECTED — and the legal transitions are an explicit switch, not a convention:

switch current {
case BookingStateInReview:
	if next == BookingStateAwaitingDeposit || next == BookingStateRejected || next == BookingStateVoid {
		return nil
	}
case BookingStateDepositPaid:
	if next == BookingStateDepositToRefund || next == BookingStateAwaitingBalance {
		return nil
	}
// ...
}
return errInvalidTransition

Every transition goes through one function, which opens a transaction, takes SELECT state FROM app.bookings WHERE id = $1 FOR UPDATE, validates, and writes with the old state in the WHERE clause. The row lock is there because the check-then-write without it is a textbook TOCTOU: two concurrent callers both read AWAITING_DEPOSIT, both decide the transition is legal, both proceed.

There’s one flag on it that took me a while to get right. A transition to the state you’re already in is treated as a no-op rather than an error, because Stripe redelivers webhooks and a retried handler re-observing its own applied state is normal. But a caller that’s about to issue an actual refund on the strength of that transition cannot accept a no-op — two callers racing there means refunding twice. So RequireStateChange turns the no-op back into a conflict for exactly those callers:

// Set this for callers about to perform an external side effect (e.g. a
// Stripe refund) on the strength of this transition, so a second,
// concurrent caller that raced past the same check doesn't repeat the
// side effect.

The retry loop

The bug I’m most glad to have found is not in that machine, it’s in what surrounds it.

Stripe redelivers any webhook that doesn’t return 2xx, backing off over several days. That’s correct for a transient failure — database down, network blip — where a later attempt might work. It is actively harmful for a deterministic rejection.

During the host merge in July, a past invoice.paid event got resent for a booking that had already reached FINALISED. The handler read it as a deposit payment and attempted FINALISED → DEPOSIT_PAID. The state machine rejected it, correctly. The handler returned 500. Stripe retried. The state machine rejected it again, identically, because the state machine is deterministic — that’s the entire point of it.

So it retried for days.

The fix is a distinction rather than a patch: an invalid-transition error is now a sentinel that the webhook handler acknowledges with a 2xx, because “received, nothing to do” is the honest answer when the booking is already in its terminal state. Only genuinely transient errors get a 5xx. It’s obvious in hindsight and it was not obvious at all in advance, and it’s the clearest lesson the project has taught me — retry semantics are a property of the error, not of the endpoint.

Let the database say no

The application checks that two coaching sessions don’t overlap. That check is a race, always, no matter how carefully it’s written, because two requests can pass it concurrently before either commits.

btree_gist is needed because the constraint mixes two kinds of comparison: plain equality on contact_id and overlap on the date range. A stock gist index handles the range; the extension is what lets an ordinary scalar sit beside it in the same index.

Postgres will just do this for you:

CREATE EXTENSION IF NOT EXISTS btree_gist;

ALTER TABLE app.coaching_sessions
    ADD CONSTRAINT excl_coaching_sessions_overlap
    EXCLUDE USING gist (
        contact_id WITH =,
        daterange(window_start, window_end, '[]') WITH &&
    ) WHERE (status IN ('PENDING', 'CONFIRMED'));

The application check stays, because it produces a friendly error message at the right moment in the UI. The constraint is the thing that’s actually true. I’ve come round to thinking this is the correct shape for most validation: the readable error in the app, the authority in the schema.

The same instinct runs through the rest of the schema — seventy-odd numbered migration files, currently up to 0193. Images are content-addressed by SHA-256 and stored as bytea, so uploading the same photo twice is a no-op; they’re resized and encoded to both JPEG and AVIF in-process, on the Pi, which is slower than I’d like and has never been slow enough to matter.

My favourite migration is 0130_remove_passwords.sql, which drops six columns and is the entire content of the file. Login is passkeys, or a six-digit code by email. Nothing to leak.

Two ways to get email wrong

Until three weeks ago, sending mail went straight to SMTP inline, and it managed to be wrong in two opposite directions at once.

Most call sites sent from a detached goroutine and logged the error. A momentary SMTP outage meant a user silently never learned their booking was approved — no record, no retry, no way to answer “did they actually get told?”.

The Stripe webhook did the opposite: it returned the send error up, so a failed email failed the whole webhook, and Stripe retried the hook and re-ran every side effect around it. A bounced email could void an invoice twice.

Both are the same bug — treating delivery as part of the transaction that caused it. The fix is a transactional outbox: write the intent to a table, drain it from a worker with backoff and a max attempt count, and key rows on an idempotency key so a redelivered webhook produces one receipt rather than two. Delivered rows are kept for ninety days rather than deleted, because “did that user get told?” should be a query.

The thing that stopped running

For eleven days in July, every page on the site told Google it was a duplicate of the homepage.

The SPA ships one generic <title>, description and canonical in index.html, and rewrites them in JavaScript once the router resolves. That’s invisible to anything that doesn’t execute JavaScript, so to fix it for crawlers there was a Cloudflare Pages Function — site/functions/_middleware.js — which filled the head in server-side. It worked. It had worked since the day I wrote it.

Added 2026-07-01 with the rest of the SEO work — sitemap, robots.txt, per-route meta. All of that survived the move. The middleware was the only piece that depended on where the site was served from.

Then on 15 July I moved site hosting off Pages and onto the Pi, so the site would deploy by rsync and sit behind the same cache rules as everything else. That was a good change and I’d make it again. It also switched the middleware off, because a Pages Function is a thing Pages does, not a file that runs wherever it happens to sit.

Nothing errored. Nothing logged. site/functions/_middleware.js stayed in the repo looking exactly as maintained as it had the week before, and every page quietly went back to shipping <link rel="canonical" href="https://soaringspirit.ch/"> — each one declaring itself a duplicate of the homepage.

Eleven days is about as cheap as this class of bug gets, and I only caught it because I was planning the host merge and went to check what the middleware would do afterwards. What bothers me isn’t the eleven days, it’s that nothing could have caught it. There’s no test for “this file is still in a position to execute”. The deploy got faster and simpler, and what broke was a capability rather than a line of code.

The fix folded into a merge I wanted anyway. api.soaringspirit.ch and soaringspirit.ch were two processes from the same binary on the same machine behind the same tunnel, and the split existed for no reason beyond history — while costing a CORS middleware, a preflight round-trip on every non-simple request, and a sitemap on the wrong host. Moving the API under /api on the site origin deleted the CORS code entirely and let the head for /trip/:id be rendered from a direct database query instead of an HTTP call to ourselves.

Sixteen route collisions had to be resolved to do it, and http.StripPrefix meant all 110 route registrations stayed exactly as written. Everyone gets the same correct head now — no user-agent sniffing, because varying HTML by user agent would fragment the edge cache and there’s no upside.

What I’d tell myself at the first commit

Write the integration tests before the third feature, not after the thirtieth. They’re the only reason the no-framework approach is defensible, and I built up months of debt before I understood that.

Push correctness into the schema earlier. Every exclusion constraint and foreign key I added late replaced application code I’d been quietly trusting.

And when you move a platform, inventory what the old one was doing for you. Migration notes list what you’re carrying across, and I wrote good ones. What they don’t list is the work the old platform did for free, because nobody had to think about it while it worked — that’s exactly the part that leaves no diff, throws no error, and shows up weeks later in a search result. A file can keep being correct long after it has stopped being reachable, and those two things fail completely independently.

The company is sixteen months old and the software has never lost a booking. For a first attempt at both, I’ll take it.

Progress on the Peregrine HTTP server

Thu, 09 Jan 2025 00:00:00 +0000

I’m implementing a HTTP server from scratch in Zig, as a fun way to learn the Zig systems programming language, and also to get more comfortable and familiar with some modern kernel features that make highly performant network IO possible.

I started this project on the 24th December 2024, and within a couple of days, I had a bare-bones TCP server that could parse and respond to HTTP requests. Initially, I followed Karl Seguin’s excellent guide to creating a TCP server in Zig. If you’re interested in learning Zig, after solving some Ziglings, I suggest that you look at Karl’s blog posts on Zig.

After some light optimisation work, I wrote some example applications because I know that the best way to test the usability of your library is to use it.

Initially, the server was initialised with an on_request(request, response) void handler function. I found that this approach made it difficult to create a stateful request handler. After trying a VTable-based interface, I settled on the more idiomatic Zig generic pattern, leveraging Zig’s comptime metaprogramming. Basically, you just call Server(MyHandler).init(config), where Server is a function that returns a type of Server that has your handler built in at compile time. You then simply define your handler as the implementation of an interface, containing at least the required methods. If the user messes anything up, the program will simply fail to compile, and some useful error will be output.

Anyway, with the nicer API, I implemented a simple counter server where the handler increments a counter on each request, and responds with the current value. I then wanted to see my server doing something useful, so I wrote a directory server, and served this website locally. To my delight, the server was working great, and the implementation of the DirServer helper required no significant changes to the server. It was fun to see the server working well, serving binaries and text files that were rendered correctly by the browser.

At this point, the server was slightly more performant than Go’s stdlib HTTP server. Go’s HTTP server is a good benchmark, because it’s well written and used extensively in production. Also, it supports HTTP/2 and TLS out of the box, which is really great. Unfortunately, Peregrine does not yet support HTTP/2 or TLS. However, Peregrine does beat Go’s HTTP server in one regard, and that is built in support for WebSockets. Peregrine features full support for WebSockets, and adding WebSocket support to your application is as easy as handling the connection upgrade by calling the upgrade helper function, and implementing the WS handler methods in your handler. It really could not be easier, and I’m quite happy with how that turned out.

Slightly more performant than Go’s HTTP server? There is no way that we can be happy with that… after all, many HTTP servers outperform Go’s.

From my own tests, NGINX only marginally outperforms Go’s, even with some tuning such as enabling Kqueue/Epoll and setting the thread count equal to the CPU core count. That surprised me, because as far as I know, Go relies on its runtime scheduler, and does not use Kqueue/Epoll for socket readiness notifications, which should eliminate wasted CPU time spent polling the sockets.

At the front of the pack is Zap, built on Facil.io. Zap is basically Zig bindings for the HTTP server part of Facil.io. Facil.io is a marvellous networking library written in C. The more I study and benchmark Facil.io, the more I’ve come to appreciate its stability and performance. On my M2 Pro, Zap will process in excess of 170K static GET requests per second with 1000 connections. I’ve not yet found a HTTP server that outperforms this with 1000 connections. Please let me know if there is one.

As a baseline, on 28th December, 4 days after the start of the project, Peregrine was doing around 86K static GET requests per second with 1000 connections. Similar to Go’s stdlib HTTP server and also similar to (tuned) NGINX. At this point, Zap/Facil.io does more than twice the throughput. We have a lot of work to do…

Step 1: Improve the handler

As I mentioned earlier, the nature of the library’s on_request hook made implementing a stateful handler inconvenient at best. This was at the top of my list. I implemented a VTable-based interface, but I didn’t like that the user needed to declare the VTable in their handler:

// commit 0beb0019df043abb43e924d4aa8234f479eba607
// ./example/basic.zig
const MyHandler = struct {
    const vtable: pereg.HandlerVTable = .{
        .handle = handle,
    };

    pub fn handle(ptr: *anyopaque, req: *pereg.Request, resp: *pereg.Response) void {
        const self = @as(*@This(), @alignCast(@ptrCast(ptr)));
        self.handleWithError(req, resp) catch |err| {
            std.debug.print("error handling request: {any}\n", .{err});
        };
    }

    inline fn handleWithError(_: *@This(), _: *pereg.Request, resp: *pereg.Response) !void {
        _ = try resp.setBody("Kawww\n");
        const len_header = try pereg.Header.init(.{ .key = "Content-Length", .value = "6" });
        try resp.headers.append(len_header);
    }
};

This version actually performed quite well, doing a little over 160K requests per second. That’s nearly a 2x improvement from our baseline! In retrospect, I’m surprised that this version performs so well, because the indirection of the VTable lookup should cost at least a few nanoseconds, not to mention that this indirection should increase CPU cache misses for calls to handler methods. It seems that function pointers are even more costly than VTables. Anyway, this performs well, but it’s quite ugly, and does not make use of what Zig has to offer. We need a better way to ‘inject’ our handler into the server.

// commit 2d13a14cd5ed77de040bea003ad37a55020fb54d
// ./example/basic.zig
const std = @import("std");
const pereg = @import("peregrine");

const MyHandler = struct {
    pub fn init(allocator: std.mem.Allocator) !*@This() {
        return try allocator.create(@This());
    }

    pub fn deinit(_: *@This()) void {}

    pub fn handle(self: *@This(), req: *pereg.Request, resp: *pereg.Response) void {
        self.handleWithError(req, resp) catch |err| {
            std.debug.print("error handling request: {any}\n", .{err});
        };
    }

    fn handleWithError(_: *@This(), _: *pereg.Request, resp: *pereg.Response) !void {
        _ = try resp.setBody("Kawww\n");
        const len_header = try pereg.Header.init(.{ .key = "Content-Length", .value = "6" });
        try resp.headers.append(len_header);
    }
};

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    defer _ = gpa.deinit();
    const srv = try pereg.Server(MyHandler).init(allocator, 3000, .{});
    std.debug.print("listening on 0.0.0.0:3000\n", .{});
    try srv.start(); // Blocks if there is no error
}

By the way, sorry for @This() in the method signatures, this was pasted as committed. I’ve since started declaring const Self = @This(); to clean up method signatures. This looks much better. Our handler simply implements the 3 required methods; init, deinit, and handle. Our handler is then given to the Server by making Server a function that returns a type. Our handler is passed to the ‘constructor’ as a parameter. Under the hood:

// commit 2d13a14cd5ed77de040bea003ad37a55020fb54d
// ./src/server.zig
pub fn Server(comptime Handler: type) type {
    return struct {
        comptime {
            if (!@hasDecl(Handler, "init")) {
                @compileError("Handler must implement init(std.mem.Allocator) !*@This()");
            }
            if (!@hasDecl(Handler, "deinit")) {
                @compileError("Handler must implement deinit(*@This()) void");
            }
            if (!@hasDecl(Handler, "handle")) {
                @compileError("Handle must implement handle(*@This(), *Request, *Response) void");
            }
        }

        handler: *Handler,
        // ...
    };
}

Pretty simple, right? As it should be. As you can see, we will get a compile error if our handler is missing any of the 3 required methods.

Step 2: optimise the request parser

Being a text protocol, efficient parsing of a HTTP request comes down to well-implemented buffered reading from the socket, and optimised parsing operations. The parts of a HTTP request are delimited by CRLF (sometimes clients may use bare LF line endings). The first line contains the request method, path, and HTTP version. The following lines contain headers, until an empty line. If present, the request body would follow. The length would be specified in a Content-Length header. For example (without body):

GET /path/to/resource HTTP/1.1\r\n
Host: localhost\r\n
User-Agent: Microsoft Internet Explorer 6\r\n
Accept: */*\r\n
Connection: keep-alive\r\n
\r\n

As a side note, the Connection header is the client’s request for the server to either close or keep-alive the TCP connection. HTTP/1.1 introduced the keep-alive mechanism, allowing connections to be reused. This greatly improves HTTP’s efficiency, especially as modern websites typically consist of many resources that are requested individually. In HTTP/1.0, clients had to create a new connection for each request. Our server must handle both HTTP/1.0 and HTTP/1.1 as a bare minimum. HTTP/2 and HTTP/3 would be great to have, but these are binary protocols, and must be upgraded to from a HTTP/1.1 connection. Additionally, HTTP/3 is not over TCP, but QUIC, which is over UDP. Support for HTTP/2 and 3 may come later.

So, how to parse this request? Logically, we should read it line-by-line. So far, I’ve found that efficient request parsing is about minimising search and copy operations.

If you’re interested, check out ./src/reader.zig. I’ll spare you a complicated code-dump.

Method, path and version

In essence, the method and version are parsed first, at the beginning and the end of the first line. Parsing the method was originally done using a compile-time generated lookup table of 8-byte masks. SIMD operations were then used to compare the first 8 bytes of the request with the lookup table. In practice, I was not happy with how some methods were parsed fairly quickly (9ns), but other methods further down the lookup table took considerably longer (over 30ns) due them requiring more comparisons.

While a few nanoseconds may seem negligible, I’m trying to keep request parsing under 2000 CPU cycles. That’s around 600ns on my M2 Pro.

I also tried a switch of the method length before byte comparisons, but this necessitated finding the first ’ ‘, which cost more than the saving of narrowing down the number of possible methods.

Today, I settled on the following:

pub const Method = enum(u4) {
    GET,
    PUT,
    HEAD,
    POST,
    PATCH,
    TRACE,
    DELETE,
    CONNECT,
    OPTIONS,

    pub fn parse(bytes: []const u8) !Method {
        if (bytes.len < 3) return error.UnsupportedMethod;
        return switch (bytes[0]) {
            'G' => .GET,
            'P' => switch (bytes[1]) {
                'O' => .POST,
                'U' => .PUT,
                'A' => .PATCH,
                else => error.UnsupportedMethod,
            },
            'H' => .HEAD,
            'T' => .TRACE,
            'D' => .DELETE,
            'C' => .CONNECT,
            'O' => .OPTIONS,
            else => error.UnsupportedMethod,
        };
    }

    pub fn toLength(self: Method) usize {
        return switch (self) {
            .GET => 3,
            .PUT => 3,
            .HEAD => 4,
            .POST => 4,
            .PATCH => 5,
            .TRACE => 5,
            .DELETE => 6,
            .CONNECT => 7,
            .OPTIONS => 7,
        };
    }
};

As beautiful as the comptime-generated lookup table was, I found that the overhead of initialising SIMD vectors was too expensive for this use-case. The above solution is much faster, with methods parsed between 6ns and 9ns. We simply give it at least 3 bytes, and in the fewest operations possible, it will return the method.

We then parse the HTTP version. Not worthy of a code-dump, we simply take the last 8 bytes of the request line, and check that the 6th byte is ‘1’ (HTTP/1), the 7th byte is ‘.’, and switch over the 8th byte to return the version enum 1.0 or 1.1. You can see it in ./src/version.zig.

Next is extracting the path. You probably noticed the toLength method in the Method enum. We know that everything between the method and the version (minus the spaces) is the path and query. We also know that the version is 8 bytes in length. Therefore, by calling toLength() on the method, we have the position of the beginning of the path without a single scalar search for the delimiting spaces (’ ‘).

But what about separating the query from the path? We could simply defer this until (if ever) the query is needed, omitting the search for ‘?’, but I found that this hurt the usability of the library. I may change this again because searching for ‘?’ is relatively expensive, and not always required. For now, the path and query are separated, but the query is never parsed, and instead stored in its raw URL-encoded form. Calling request.parseQuery() will populate a hash map in the request.

Parsing headers

Here, we simply read the headers line-by-line, parsing them using a scalar search for ‘:’ which is the delimiter for the key-value pair. The header is simply a struct with key and value fields. If you’re looking at ./src/header.zig and wondering why Header contains fixed-sized arrays and length fields… Requests are pre-allocated with a fixed-sized array of 32 Headers. This means that we don’t need to allocate headers for each request.

Some of the most important optimisations can manifest as quite ugly ‘hacky’ tricks. Such is the case for the Connection header. The Connection header is a little special, because it isn’t really for an application/library user to think about, but a client request for the server to handle the connection in a specific way. The value is normally either “keep-alive” or “close”. The server may still choose to close the connection, for example, if the limit of connection requests has been reached. Therefore, when we respond to a HTTP/1.1 request, we most often need to evaluate the Connection header. Initially, I was searching for the Connection header while generating the response, but I noticed that we could save a few nanoseconds by moving this into the header parsing step, as we’re already iterating over the headers. This was as simple as adding a boolean flag to the request keep_alive. Looking at the implementation at time of writing, it’s so ugly that I don’t want to dump the code here! If you want, check it out in the parseHeaders method in ./src/reader.zig. Basically, we do a quick process of elimination by raw length, before checking if the value starts with ‘cl’ for ‘close’. Notice the conn_header_found flag in the method to prevent wasted operations after we’ve found it.

SIMD for `reader.readLine`

How could I forget? reader.readLine(fd: posix.socket_t) is right in the critical path. See ./src/reader.zig.

One important optimisation here is the use of SIMD (vectors/chunks) when searching for line-endings. Rather than check byte-by-byte if we have a new-line character, we can leverage the SIMD (single-instruction-multiple-data) support of modern CPUs. This allows us to fill a mask with new-line characters, and check whether each 16-byte chunk contains the new-line. If it does, we simply call the built-in @ctz (count trailing zeros) to calculate its offset (position) in the chunk.

For any remaining bytes (if the line-length is not divisible by 16), we simply iterate over them byte-by-byte.

The use of SIMD is often integral to efficient data processing, and you will see it a lot.

Step 3: Write iovecs

Finally for this article, when the ./src/worker.zig responds to a request, making a syscall to write each part of the response, or concatenating the response status line, headers, and body buffers, we can make use of posix.writev to write a slice of iovecs (IO vectors) with a single syscall. This is much more efficient. Thanks to Karl Seguin for his guide to writing iovecs in Zig. I’d like to reiterate just how helpful I’ve found his guides, so thank you Karl.

Wrap up

I hope you found some of this interesting. I quite enjoy writing these, as it’s a good way for me to reflect on some of the decisions that I’ve made while. While writing this, I even found a couple of little things that I can improve.

Thanks to Bo (author of Facil.io), for his advice for optimising the server implementation. Facil.io still outperforms Peregrine by around 10%, and shows better stability of response times under load. I think that this is due to Bo’s superior solution for distributing and scheduling the computation.

Today, I implemented a fairly performant ring buffer queue (50ns for one read and one write). Maybe I can still improve this by a factor of 2, as a hardware lock should only take a single cycle.

const std = @import("std");

pub fn RingQueue(comptime T: type) type {
    return struct {
        const Self = @This();

        allocator: std.mem.Allocator,
        buffer: []T,
        read_index: usize = 0,
        write_index: usize = 0,
        mutex: std.Thread.Mutex = .{},
        not_empty: std.Thread.Condition = .{},
        not_full: std.Thread.Condition = .{},

        pub fn init(allocator: std.mem.Allocator, capacity: usize) !*Self {
            const self = try allocator.create(Self);
            self.* = .{
                .allocator = allocator,
                .buffer = try allocator.alloc(T, capacity),
            };
            return self;
        }

        pub fn deinit(self: *Self) void {
            self.allocator.free(self.buffer);
            self.allocator.destroy(self);
        }

        fn isFull(self: *Self) bool {
            return (self.write_index + 1) % self.buffer.len == self.read_index;
        }

        fn isEmpty(self: *Self) bool {
            return self.write_index == self.read_index;
        }

        pub fn write(self: *Self, data: T) void {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.isFull()) {
                self.not_full.wait(&self.mutex);
            }
            self.buffer[self.write_index] = data;
            self.write_index = (self.write_index + 1) % self.buffer.len;

            self.not_empty.signal();
        }

        pub fn read(self: *Self) T {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.isEmpty()) {
                self.not_empty.wait(&self.mutex);
            }
            const data = self.buffer[self.read_index];
            self.read_index = (self.read_index + 1) % self.buffer.len;
            self.not_full.signal();
            return data;
        }
    };
}

This queue will make it possible to distribute work (requests to handle) to a thread pool on a FIFO basis. This would prevent long-running handlers from blocking an entire worker, which could block multiple connections.

I also learned that Zig supports async/await as this could be an even better way for each worker in the pool to handle multiple requests concurrently.

Peace & love! J

The beginning of project Peregrine - a bleeding fast HTTP server

Thu, 26 Dec 2024 00:00:00 +0000

If you’re very familiar with Zig, then you’ll know that the standard library HTTP server is not written to be highly performant.

Zap is an alternative. Zap is built on the fantastic Facil.io networking library written in C.

Now, while there are plenty of fast HTTP servers, at time of writing, I couldn’t find a production-ready high-performance HTTP server written purely in Zig. There is http.zig, but according to Karl, the developer, it must sit behind a reverse-proxy because it’s not resilient to DoS attack.

As I need a real project to get deeper into the language, and having never written a HTTP server from scratch before, I felt that this would be a great opportunity to have fun learning about Zig, async IO, vectored IO, memory alignment, zero-alloc, zero-copy and a bunch of other systems stuff. I’ve also found that working with constraints (such as following a spec) is great for forcing one to find a solution, rather than skirting around an issue. Also, writing a library is a particularly good way to learn a language because it forces you to write reusable (and usable) software.

Peregrine - a bleeding fast HTTP server

My goal with this project is to write a robust, performant and simple HTTP server. In doing so, I will learn much of the Zig language, and a bunch about systems programming.

I started this project on 2024-12-24, two days ago at time of writing. So how far have I got in two days?

Currently, we have the main components of the server implemented. The server is written in a library form, and exposes a clean API to create a server and handle requests.

There is a worker pool. Each worker runs on its own thread. For now, connecting sockets are assigned to a worker using round-robin distribution. This means that we don’t need a scheduler.

Kqueue is the FreeBSD/MacOS solution for async IO. It’s a kernel event system. Basically, you register interest in a type of event. You then process the events. We have one of these event buses for our server’s main thread, where we accept connections. Each connecting socket is then assigned to a worker by registering interest in the client socket’s events with the worker’s event bus. This means that when clients write to the socket, the appropriate worker is notified and will process the event. In this case, the worker processing the event means reading from the socket.

For the Linux kernel, there is a different solution, called epoll. In some ways Kqueue is superior to epoll because Kqueue allows registering interest in multiple events with a single system call. Conversely, with epoll, each interest must be registered with separate sys calls. In our case, this won’t make any difference because we make a sys call every time we accept a new connection. If we didn’t, clients would have to wait until their socket was registered. Otherwise, epoll is very similar to kqueue, as far as I know. In the next days, I will implement epoll to support Linux.

When each worker starts, it does all of its heap allocations that it needs for normal operation. Ideally, no further heap allocations will be made per-request. This zero-alloc approach greatly improves performance because we don’t need to wait for memory to be allocated on the heap before progressing with processing each request.

Each worker can handle one request at a time. This constraint allows us to reuse the buffers and data structures that it allocates. The only thing we have to ensure is that they are properly reset before reuse.

Getting deeper into the guts of the server, each worker creates a request reader. This request reader reads from a given socket, and maintains a buffer of read-but-unprocessed data. When we call readRequest, unprocessed data in the buffer will be read before compacting the buffer and making another syscall to get more data. This approach means that fewer syscalls are used to read each request, while not needing a particularly large buffer.

One optimisation that I’ve made already is that buffers are properly aligned. This means that the data’s memory address is a multiple of its size. Memory alignment is crucial for high-performance software because the CPU accesses memory most efficiently when data is naturally aligned. This allows the processor to fetch memory in a single operation, whereas misaligned data may require multiple accesses. Additionally, aligned data structures help to optimise cache efficiency. When data crosses cache line boundaries, it can result in additional cache misses, resulting in performance degradation.

Below is the critical path of the worker’s event loop. Each kernel event is processed as follows.

// ./src/worker.zig
fn handleKevent(self: *Worker, socket: posix.socket_t, reader: *request.RequestReader) !void {
    defer posix.close(socket);
    self.req.reset();
    try reader.readRequest(socket, self.req);
    self.resp.reset();
    self.on_request(self.req, self.resp);
    if (!self.resp.hijacked) try self.respond(socket);
}

fn respond(self: *Worker, socket: posix.socket_t) !void {
    const headers_len = try self.resp.serialiseHeaders(&self.resp_buf);
    if (self.resp.body_len > 0) {
        var iovecs = [_]posix.iovec_const{
            .{ .base = @ptrCast(self.resp_buf[0..headers_len]), .len = headers_len },
            .{ .base = @ptrCast(self.resp.body[0..self.resp.body_len]), .len = self.resp.body_len },
        };
        const written = try posix.writev(socket, &iovecs);
        if (written != headers_len + self.resp.body_len) {
            return error.WriteError;
        }
        return;
    }
    try writeAll(socket, self.resp_buf[0..headers_len]);
}

One optimisation you can see above is in the respond method. This is called after the user’s on_request handler is called, providing that they did not hijack the response. More on that later. If the response has a body, vectored IO is used to write the headers and body in a single syscall, without needing to concatenate the buffers. If there is no body, the response is written by repeatedly writing to the socket until all bytes are written. Without a body, this is likely to be achieved in a single syscall.

Making use of x86’s SIMD and ARM’s NEON capabilities allow us to speed up a variety of operations. One example that you will see often is chunking. This is where slices are processed in chunks, rather than operating on single bytes. This is important because operating on single bytes does not make good use of the cache. A good example of this is in Zig’s stdlib: std.mem.indexOfScalarPos.

This searches a slice for a specific value. Normally this would be a slice of bytes, and we’d be searching for a byte. If we were to search the slice byte-by-byte, the CPU would have to individually load each byte into a register for comparison. In std.mem.indexOfScalarPos, you can see how a bit mask is used to efficiently test if the target byte is somewhere in the chunk. The naive and much slower implementation would be as follows:

pub fn indexOfPos(comptime T: type, slice []const T, start_index: usize, value T) ?usize
    if (start_index >= slice.len) return null;
    var i: usize = start_index;
    while (i < slice.len) : (i += 1) {
        if (slice[i] == value) return i;
    }
    return null;

Often, adding some room for complexity allows us to solve problems more efficiently. Good engineering is partly about balancing performance gains with simplicity. A good engineer knows what to optimise, and what not to. Tight loops in critical paths tend to offer the greatest reward for optimisation.

Anyway, hopefully that was interesting. After a couple of days, we have the bare bones of a high-performance HTTP server, written purely in Zig. Considering that I’m new to systems programming and Zig, I feel that I’m doing alright. It surely is a very fun and expressive language. I’m enjoying the build speed and the quality of the compiler’s errors.

Stay tuned for my progress, as I will be working on this to make it production-ready. I’m not sure how long it will take, but I am sure that I’ll continue learning a lot.

Peace and love! J

First thoughts on Zig

Mon, 23 Dec 2024 00:00:00 +0000

I’ve done very little systems programming. I wrote Conway’s Game of Life in C as a teenager. I also did a bit of assembly at GE Aviation Systems. A few years ago, I completed the Rust programming book, but struggled with the language beyond that, and never grew to love it.

I’m currently diving quite deep into systems programming in Zig. While there are many things I love about the language, namely its elegant approach to generics and meta-programming, I do feel that the language is lacking in some ways.

Simplicity

What I like most about the language is its simplicity. Similar to Go in this regard, it didn’t take me long to become comfortable with the syntax.

Errors lack context

One of the most obvious flaws with the language is its error type. It’s impossible to add context or details to errors. In contrast, an error in Go for example, is just a value. This allows you to add context as the error is passed up the call stack.

I’m not sure if Zig will offer a solution for this.

Despite this, errors in Zig are otherwise great, and very easy to handle or throw up the stack.

Powerful build system

Zig exposes a very powerful yet usable build API. Linking C libraries is quite effortless, and I’m a noob to this.

While I’m not yet familiar with much of the build system, I have the feeling that it is probably the best build system that I’ve seen yet. For me, it was also the hardest part of Zig to become comfortable with. I’m still not really comfortable with it. In addition, as the language is so young and volatile, much of the 3rd party documentation for the build system is out of date.

Fortunately, the API is very clean, and so it’s not so difficult to figure things out on your own.

Lack of interfaces

Zig’s philosophy is to provide low-level building blocks, rather than high-level abstractions. As such, the Zig team felt that interfaces are an unnecessary abstraction.

While I understand this view, I feel that the language’s lack of built-in support for interfaces can make interface implementations feel quite clunky. This became particularly apparent to me when I began using the standard library’s filesystem types. I’m sure that this could be greatly improved as the language server matures.

Expressive language

I am often surprised by Zig because it tends to work exactly as you’d hope. This is particularly noticeable when implementing polymorphic types and platform-dependent behaviour. This makes writing cross-platform software remarkably easy. For me, Zig really shines in this regard.

Generics are beautiful

Thanks to Zig’s expressive compile-time metaprogramming, generics are very easy to implement, and are perfectly type-safe.

The following is a generic thread-safe message queue:

const std = @import("std");

pub fn Queue(comptime T: type) type {
    return struct {
        const Node = struct { data: T, next: ?*Node };
        mutex: std.Thread.Mutex,
        head: ?*Node,
        tail: ?*Node,
        allocator: std.mem.Allocator,

        pub fn init(allocator: std.mem.Allocator) !*Queue(T) {
            const queue = try allocator.create(Queue(T));
            queue.* = .{
                .mutex = std.Thread.Mutex{},
                .head = null,
                .tail = null,
                .allocator = allocator,
            };
            return queue;
        }

        pub fn deinit(self: *Queue(T)) void {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.head) |node| {
                const next = node.next;
                self.allocator.destroy(node);
                self.head = next;
            }
            self.tail = null;
        }

        pub fn write(self: *Queue(T), data: T) !void {
            const new_node = try self.allocator.create(Node);
            new_node.* = .{
                .data = data,
                .next = null,
            };
            self.mutex.lock();
            defer self.mutex.unlock();
            if (self.tail) |tail| {
                tail.next = new_node;
                self.tail = new_node;
            } else {
                self.head = new_node;
                self.tail = new_node;
            }
        }

        pub fn read(self: *Queue(T)) ?T {
            self.mutex.lock();
            defer self.mutex.unlock();
            if (self.head) |node| {
                const data = node.data;
                self.head = node.next;
                if (self.head == null) {
                    self.tail = null;
                }
                self.allocator.destroy(node);
                return data;
            }
            return null;
        }
    };
}

test "basic usage" {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    const queue = try Queue(u8).init(allocator);
    try queue.write(1);
    try queue.write(2);
    try queue.write(3);
    try std.testing.expect(queue.read() == 1);
    try std.testing.expect(queue.read() == 2);
    try std.testing.expect(queue.read() == 3);
    try std.testing.expect(queue.read() == null);
}

In Go, I very rarely found myself using generics. Generics in Go feel like an after-thought, and that’s probably because they are.

Conversely, in Zig, I’ve used this comptime struct pattern many times to write reusable components.

Standard library is still rough around the edges

Some packages in the standard library are not entirely production-ready. For example, the Blake3 implementation uses the reference rather than the optimised implementation.

There are faster and more robust HTTP servers than the one in the standard library, namely Zap built on facil.io (written in C).

As Zig is still young, it seems only natural that the standard library is not yet polished. Obviously, the language should mature and stabilise before polishing the standard library can become a priority.

Memory management made easy

When I learned Rust (or tried to), I never became comfortable with it, and I was constantly fighting the compiler.

Possibly, I’m simply a better programmer these days, but I find it more likely that Zig’s approach to memory management is just simpler and more explicit.

Zig is not a fancy language, it really does feel close to C. I feel that this inherently makes it easier to learn than Rust.

Mostly writing Go recently, I expected that Zig’s memory management would be a really big hurdle. While it has taken me a few days to become comfortable allocating memory on the heap, I was surprised how easy I found it.

I’m still not fully familiar with each of the standard library’s allocators, and I mostly use the GeneralPurposeAllocator.

Conclusion

I’m only a couple of weeks into my journey with Zig.

Would I use Zig for work? If the project would benefit from optimal performance and reliability, absolutely. For bits such as tooling for managing infrastructure or automated key rotation, where performance is not a concern, I’d likely opt for Go. At time of writing, Zig’s ecosystem is still very small compared to Go’s.

My reason for beginning with Zig was to get into systems programming, and I’m really enjoying it. I remember faffing with header files in C as a teenager, and I know that building C projects can be a real pain. So far I’m quite happy with Zig, and I’m optimistic that a lot of the kinks will be ironed out as the language matures.

There will never be a perfect language for everyone for all use-cases. As I’ve been learning Zig, I’ve come to appreciate how hard it is to design a language that is fit for decades of mission-critical software engineering. It’s incredible how good C is, despite its (lack of) build system. I’ve also come to realise how much of Go they got right. Writing concurrent software in Zig has left me with an appreciation for Go’s channels. The only thing that I would change about Go is the verbose error handling.

I feel that learning a new programming language allows us to open our minds to alternative paradigms. Questioning what we already know is fundamental to making progress.

Comparing the performance of a de Bruijn sequence against a simple lookup table

Fri, 17 May 2024 00:00:00 +0000

While optimising the naive proof-of-work algorithm in my dave protocol implementation, I compared the efficiency of some methods of counting the number of leading zero bits in a byte slice.

Approaches

1. Naive approach

func nzerobit_1(key []byte) int {
    var n int
    for _, b := range key {
        for i := 0; i < 8; i++ {
            if b&(1<<i) == 0 {
                n++
            } else {
                return n
            }
        }
    }
    return n
}

Iterating over each byte in the given key, we use the bitwise AND operator & to check if the i-th bit of the byte is zero. The expression 1<<i creates a bitmask with only the i-th bit set to 1, and the & operator performs a bitwise AND operation between the byte and the bitmask. If the result is zero, it means that the i-th bit is zero, and we increment n. If the bit is set, we return the current value of n.

2. Right shift

Instead of using b&(1<<i) == 0, we can use (b>>i)&1 == 0, avoiding the expensive left shift operation. This should yield a small performance improvement.

func nzerobit_2(key []byte) int {
    var n int
    for _, b := range key {
        for i := 0; i < 8; i++ {
            if (b>>i)&1 == 0 {
                n++
            } else {
                return n
            }
        }
    }
    return n
}

3. Unroll loop

Loops are expensive. Loop unrolling can help reduce overhead, and enable better instruction-level parallelism.

func nzerobit_3(key []byte) int {
    var n int
    for _, b := range key {
        if (b>>0)&1 == 0 { n++ } else { return n }
        if (b>>1)&1 == 0 { n++ } else { return n }
        if (b>>2)&1 == 0 { n++ } else { return n }
        if (b>>3)&1 == 0 { n++ } else { return n }
        if (b>>4)&1 == 0 { n++ } else { return n }
        if (b>>5)&1 == 0 { n++ } else { return n }
        if (b>>6)&1 == 0 { n++ } else { return n }
        if (b>>7)&1 == 0 { n++ } else { return n }
    }
    return n
}

4. Simple lookup table

We can create a lookup table storing the number of leading zero bits for every possible byte value.

var zeros = [256]int{
    8, 7, 6, 6, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4,
    3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
}

func nzerobit_4(key []byte) int {
    var n int
    for _, b := range key {
        n += zeros[b]
        if b != 0 {
            return n
        }
    }
    return n
}

While simple, this approach can be significantly faster for small input sizes. As my use case is counting the number of leading zero bits in a 256 bit hash, this method could yield great results.

5. De Bruijn sequence

var debruijn64 = [64]byte{
    0, 1, 2, 53, 3, 7, 54, 27, 4, 38, 41, 8, 34,
    55, 48, 28, 62, 5, 39, 46, 44, 42, 22, 9, 24,
    35, 59, 56, 49, 18, 29, 11, 63, 52, 6, 26,
    37, 40, 33, 47, 61, 45, 43, 21, 23, 58, 17,
    10, 51, 25, 36, 32, 60, 20, 57, 16, 50, 31,
    19, 15, 30, 14, 13, 12,
}

func nzerobit_5(key []byte) int {
    for i, b := range key {
        if b != 0 {
            x := uint64(b&-b)*0x03f79d71b4ca8b09>>58
            return i*8 + int(debruijn64[x])
        }
    }
    return len(key) * 8
}

The debruijn64 array is a precomputed lookup table that maps the position of the least significant set bit in a byte to the number of trailing zeros.

i*8 gives the number of zero bits in the previous bytes. b&-b isolates the least significant set bit in the current byte. uint64(b&-b)*0x03f79d71b4ca8b09 multiplies the isolated bit with a magic constant to generate a unique index into the table. >>58 shifts the result to the right by 58 bits to bring the relevant bits into the least significant position. The looked-up value represents the number of trailing zeros in the current byte, which is added to i*8 to get the total number of leading zero bits.

The de Bruijn sequence lookup table provides a constant-time method to determine the number of trailing zeros in a byte. By combining this with the byte index, the function efficiently calculates the total number of leading zero bits in the byte slice.

Benchmarks

The following results are from a 12-core Apple M2 Pro.

The simple lookup and de Bruijn sequence are the clear winners, regardless of byte length and number of leading zero bytes. As there is no clear winner between these two, I suggest running tests for your specific use case.

16 bytes, 4 leading zero bytes

Benchmark_1-12    	53595952        22.68 ns/op
Benchmark_2-12    	54691712        22.68 ns/op
Benchmark_3-12    	76803686        15.74 ns/op
Benchmark_4-12    	421736720       2.861 ns/op
Benchmark_5-12    	423180349       2.864 ns/op

32 bytes, 2 leading zero bytes

Benchmark_1-12    	80881609        14.65 ns/op
Benchmark_2-12    	86697114        13.87 ns/op
Benchmark_3-12    	100000000       11.22 ns/op
Benchmark_4-12    	577513431       2.087 ns/op
Benchmark_5-12    	570226458       2.076 ns/op

32 bytes, 4 leading zero bytes

Benchmark_1-12    	49990105        23.07 ns/op
Benchmark_2-12    	54375429        21.99 ns/op
Benchmark_3-12    	76524105        15.73 ns/op
Benchmark_4-12    	413261740       2.862 ns/op
Benchmark_5-12    	403608087       2.912 ns/op

32 bytes, 8 leading zero bytes

Benchmark_1-12    	34218296        35.23 ns/op
Benchmark_2-12    	35122378        34.62 ns/op
Benchmark_3-12    	62954121        19.30 ns/op
Benchmark_4-12    	315074923       3.810 ns/op
Benchmark_5-12    	315627609       3.805 ns/op

64 bytes, 4 leading zero bytes

Benchmark_1-12    	50546887        23.29 ns/op
Benchmark_2-12    	51958827        22.18 ns/op
Benchmark_3-12    	73563591        15.65 ns/op
Benchmark_4-12    	401217024       3.052 ns/op
Benchmark_5-12    	386922139       3.040 ns/op

128 bytes, 4 leading zero bytes

Benchmark_1-12    	53089023        22.98 ns/op
Benchmark_2-12    	53183925        22.37 ns/op
Benchmark_3-12    	74646594        16.12 ns/op
Benchmark_4-12    	142048627       8.512 ns/op
Benchmark_5-12    	142594008       8.442 ns/op

256 bytes, 4 leading zero bytes

Benchmark_1-12    	51616419        23.62 ns/op
Benchmark_2-12    	53396121        22.98 ns/op
Benchmark_3-12    	57543146        18.51 ns/op
Benchmark_4-12    	100000000       10.83 ns/op
Benchmark_5-12    	100000000       10.89 ns/op

256 bytes, 64 leading zero bytes

Benchmark_1-12    	5252018         231.0 ns/op
Benchmark_2-12    	5165072         231.0 ns/op
Benchmark_3-12    	12656263        97.03 ns/op
Benchmark_4-12    	57885492        20.43 ns/op
Benchmark_5-12    	58068614        20.46 ns/op

Repository

https://github.com/intob/zerobitperf

Dave

Wed, 01 May 2024 00:00:00 +0000

EDIT: In September & October I made significant changes to the DHT, and so much of this article is out of date. I have not yet found time to update this article, and I’m not currently working on the project.

So… what am I doing? When I started this project (first commit 2024-04-05), I didn’t know exactly, beyond designing a peer-to-peer network application that disseminates information. Now my vision is finally becoming clear, and I’m able to describe it.

I’m writing a peer-to-peer application that allows anyone, or any device, to read and write information without a value transaction of any kind. A writer pays in CPU cycles and electricity, while a reader pays in network participation.

Simplicity, anonymity, efficiency, performance, and high availability are key aspects that I’m aiming for.

Simplicity will come at the cost of pushing much application-specific complexity up the stack. We must be free to re-imagine a broken game, if it turns out to be broken… At this time, I feel that this approach ensures a modular and upgradeable stack of protocols, while maximising the probability of success of the dave protocol.

I’m aware that such projects have been attempted and failed. I’m also aware that such a challenging project may not be an ideal start for a budding p2p network researcher. With that in mind, I do need a worthy challenge and vision to hold my interest.

Dave - distributed KV store

Dave is a protocol for continuously disseminating information in a decentralised, anonymous, and censorship-resistant manner, without the need for a native token or value transaction.

New values are available nearly instantly, with sub-second lookup time. Old values have a significantly faster lookup time, because more nodes will likely have the data. Real-world speed depends on the bandwidth provisioned by the nodes. Most likely, the response time is better than average network ping, even with a naive implementation.

Use-cases: Decentralised social media applications, serverless forms, communication layer for any decentralised application.

Design

The protocol is designed around a single message format with an enumerated operation code. There are 4 operation codes, as follows; GETPEER, PEER, PUT, GET.

Execution is of a cyclic mode, with the minimum period defined by constant EPOCH. The program sends a uniform stream of packets, minimising packet loss and maximising efficiency. This allows the protocol to be adjusted safely, and in a way that could preserve interoperability with a network running variations for different bandwidth ideals & constraints.

GETPEER & PEER Messages

These are the first two op-codes that I defined, and initially the only operations that the network performed. These two messages allow nodes on the network to discover peers, and to verify their availability.

For every PING EPOCH, iterate over the peer table. If a peer is found that has not been seen in the last PING EPOCH, and has not been pinged within the last PING EPOCH, send to the peer a message with op-code GETPEER.

A protocol-following peer will reply with op-code PEER, a message containing NPEER addresses for other peers.

I often refer to these addresses as peer descriptors, as in future, they may not necessarily be IP addresses. I would like the possibility to cleanly implement interoperable transports. This would allow nodes to serve as bridges. As software-defined radio is a field of interest to me, I imagine that the dave protocol could used to create alternative low-energy networks that could bridge our existing infrastructure built on IP. Imagine a free and open internet that does not require an internet service provider, or any knowledge of its users. Know that I have not yet adjusted the protobuf specification to support this, as I prefer to focus on a building working proof over UDP for now.

If a peer does not respond with any valid message, after DROP * EPOCH has elapsed, the peer is deleted from the peer table. Unresponsive peers are no-longer advertised after OPEN * EPOCH has elapsed without message, so as to ensure that unresponsive peers are not re-added from latent gossip.

PUT Message

A message with op-code PUT is a packet of data containing a Key, Value, Time, Salt, Work, Sig and PubKey. Salt and Work are outputs of the cost function, into which Key, Value and Time are passed.

Every EPOCH, each node sends one randomly selected dat to one randomly selected peer.

This ensures that information propagates and re-propagates through the network reliably, until it is eventually no-longer stored by any node. I describe the selection algorithm later.

This provides a good level of anonymity for original senders, because it is virtually impossible to discern the origin of a dat, even with a broad view of network traffic.

Anonymity is achieved by ensuring no correlation between the timing of a dat being received for the first time, and its eventual propagation to other nodes. This happens at random, but at a constant interval.

GET Message

A GET message is a packet with op-code GET, containing a Key. If the remote has the dat, the remote should reply with a message with op-code PUT, containing the data. A node may request the same dat from many peers simultaneously. This is up to the application to optimise, although the reference implementation should be pretty good.

DAT Selection by Mass

Proof-of-work

A Salt is found, that when combined with the Key, Value & Time fields using a hash function, the output begins with some desired number of leading zero bits. The number of leading zeros (or some other constraint) probabilistically accurately reflects the energetic cost equivalent of computing the proof. This is commonly known as proof-of-work, and is well known for its use in Bitcoin mining.

The number of leading zero bits is referred to as the “difficulty” throughout the remainder of this document.

We compute a salty hash from an initial hash of key, value and time. For large values, this performs better than a single hash.

work = blake3(salt, blake3(key, value, time))

Mass Calculation

As each machine has limited resources, we need a mechanism by which the software can select which dats should be stored, at the expense of others being dropped. In a peer-to-peer application without any central coordination or authority, each node must be able to decide which dats to keep on its own.

So how can we give priority to some dats over others?

As the cryptographic proof contains the value, and time, neither may be modified without invalidating the proof. As such, we can use the time, and the difficulty in the calculation of a score, referred to here as mass.

mass = difficulty * (1 / millisecondsSinceAdded)

The mass tends to zero over time. Dats with a harder proof of work persist longer in the network. In addition, difficulty scales exponentially (each added zero byte increases the difficulty by 256 times.

Peer Trust Mechanism

A decentralised network depends on a trust system that incentivises fair play. It’s much more challenging than operating in a private network, as most web applications do today. So why do it? It allows us to build more powerful software that cannot be controlled by any single entity.

The goal of the trust mechanism is to ensure that energy is not lost to malicious or protocol-deviating peers.

Earning Trust

Each time a packet is received containing a dat not already stored, the remote peer’s trust value is incremented by the mass of the dat.

If a peer is dropped, and then re-joins the network, they will begin with a trust score of zero.

Trust scores are not gossiped, as this implies additional attack surface and complexity.

Use of Trust

The trust score is weighed in to the random peer selection. A random threshold between the maximum trust score and zero is chosen. A peer with a trust score greater than the random threshold is selected at random.

Therefore, peers with a higher trust score are more likely to be selected for gossip messages. This in turn increases the chance for the peer to learn of new dats earlier, reinforcing the pattern.

In essence, the longer a peer (ip-port) remains in an other’s peer table, the higher the trust score will likely be, and therefore more bandwidth is allocated to that peer.

Resilience to Attack

Sybil Attack

A Sybil attack is one where an attacker creates many peers that join the network. Failing to consider this would result in the network losing energy to these numerous byzantine nodes that are inexpensive to spawn.

Eclipse Attack

An Eclipse attack is one where an attacker attempts to poison the peer tables of peers that it discovers on the network. This poisoning may be either byzantine nodes controlled by the attacker, or simply bogus peer descriptors.

Denial of Service

This attack is the simplest to orchestrate, and yet one of the hardest to deal with. A denial of service attack is one where an attacker sends packets that induce the maximum amount of computation on the remote. HTTP servers are commonly the recipients of such attacks, which is why we use CDNs where possible to shield our origin servers. This is obviously not possible for stateful APIs and databases. That’s why we put databases in private networks, and employ efficient filtering of API requests, often using an IP-based rate limiter. Obviously we can’t run our peer-to-peer application in a private network unless we only want to use it in a datacenter.

Unfortunately, if an attacker has more bandwidth than any given node, the node can simply be denied service by exhausting their bandwidth. If the attacker has more bandwidth than the entire network combined, the whole network is denied service. So if you advocate for a given peer-to-peer network, the best way that you can support the network is by giving your bandwidth. Run a node.

Attack Mitigations

Dat Delay

A node must wait some time before receiving dats. They must participate in the gossip of peers without receiving any dats, until the defined period has elapsed. They will then start to receive dats very slowly, based on the PROBE constant.

Trust

A node normally selects only trusted peers to receive dats, but occasionally an untrusted peer is probed. Over time, a node is able to build trust with peers, therefore receiving dats more frequently. As a peer earns trust, they are more likely to be selected, thereby having more bandwidth allocated to them.

NPEER Limit

We only accept PEER messages containing up to NPEER peer descriptors. I recommend that this value is either 2 or 3. I err on the side of caution with 2, because there is little benefit to speeding up the peer discovery in this way. It would be more secure to ask for peers more often.

Funny! While writing this (2024-05-29), I discovered a vulnerability to eclipse attack. A byzantine peer may send many PEER messages, limited only by the packet filter. This made address poisoning trivial. This was resolved by recording the last time a PEER message was received, and only accepting a PEER message after the expected duration has elapsed since the last.

Storing Large Files

As mentioned earlier, all application-specific complexity is pushed up the stack. Dave is purely a packet-sharing protocol, with no built-in features for storing large files. That said, I have considered the need for storing large files in the network.

Linked List

The simplest approach to storing large files, and the earliest proof that I implemented, is a linked-list. Simply prepend or append the hash of the previous dat in next dat. This simple approach has the disadvantage that reading is slow because GETs cannot be made in parallel.

Merkle Tree

An optimised approach is to write the large file as a sequence of dats, and concurrently build a merkle tree, where additional pointer dats are used to reference dats containing the file’s content. The hash of the root of the tree may then be used as the reference to the file. A reader may then traverse the tree, collecting all of the dats in parallel, and re-assemble the file. Reads of this nature are significantly faster than the linked-list approach.

Some Repositories

godave is the protocol implementation in library form, written in Go.

Protocol https://github.com/intob/godave/

daved is a program that executes the protocol, as any other application that may join the network.

A tiny cli https://github.com/intob/daved/

garry is a HTTP gateway.

Anything can work with dave. https://github.com/intob/garry/

State of Operations

I’m no-longer running public edge (bootstrap) nodes myself. I was using AWS, with a bunch of t4g.nano arm64 VMs running Debian 12. I used scripts & programs to control groups of additional machines as I needed. I view logs by grepping the linux system journal. Logs begin with a short path prefix /fn/proc/action, allowing me to efficiently grep logs without need for typing quotes around the query.

Thank you for reading. I value advice and ideas. If you have any, please do reach me.

Run as Node

Executing with no command (just flags) puts the program in its default mode of operation, participating in the network.

daved # listen on all network interfaces, port 1618
daved -v # verbose logging, use grep to filter logs
daved -v | grep /d/pr # verbose logging, with only the daemon's prune procedure logs output
daved -l :2024 # listen on all network interfaces, port 2024. Same as -l [::]:2024
daved -e :1969 # bootstrap to peer on local machine, port 1969
daved -e 12.34.56.78:1234 # bootstrap to peer at address

Commands

daved set hello_dave # write hello_dave to the network with default difficulty of 16 bits
daved -d 32 set hello_world # write hello_world to the network with difficulty of 32 bits
daved get $HASH # get the dat with the given work hash from the network
daved setf myfile.txt # write a very small file to the network

logd

Fri, 15 Mar 2024 00:00:00 +0000

Started ~2023-12 Real-time logging (tail & query) for virtually unlimited logs. A map of in-memory ring buffers. Built on UDP & homegrown ephemeral HMAC. Go std lib + Protobuf. Image is built from scratch (Linux kernel + app executable).

2024-05: Looking back, this was an enjoyable project to rediscover writing software. It’s amazing to return to something after a change of mind. A new mind has more fun, and is more productive. The same thing happened when I re-imagined speedflying after my spinal-cord injury. Some years after my recovery, I rediscovered a lost joy with a new mind.

RE quality of this program: Looking back on past work is always funny, I leave this here as it still serves as a journal. Now, I would just run a process that reads the linux journal. Useless abstractions such as loggers pollute the code. This project is a useless abstraction, like most other software.

2024-12: Looking back at this project, I find it amusing to think that I struggled to solve the replay attack vulnerability.

As the protocol is over UDP, each packet must be authenticated. I had tried using a ring-buffer of recent packet hashes, A/B cuckoo filters, and probably other things too. I should have just gone with a sequence counter, incremented and included in each packet. The server can then immediately reject packets that are far behind the sequence counter. A small ring-buffer could then be used to check packets that are slightly behind the sequence counter. This prevents replays, and is far more efficient than a large ring-buffer or A/B cuckoo filters.

I learned of this sequence counter approach when I read through the WireGuard spec.

Repo

https://github.com/intob/logd

arpload

Sat, 09 Mar 2024 00:00:00 +0000

Arweave is a decentralised storage network that offers a unique approach to data storage.

The network is designed to provide permanent information storage, where users pay a one-time fee to store data for at least 200 years, with the potential for it to last much longer, possibly forever.

This is based on the assumption that the cost of storage will continue to decrease over time, and the initial fee paid by the uploader is invested in an endowment that accrues value, similar to interest in a bank account.

The endowment is a critical component of Arweave’s economic model. It is designed to ensure that the rewards for storing data remain higher than the cost of storage over time. The network uses a portion of the storage fees to pay miners in the future, which should theoretically allow the model to sustain itself indefinitely.

This approach is expected to provide a solution to the problem of data impermanence on the internet.

Arweave’s network is often compared to Bitcoin but for data, creating a permanent and decentralised web within an open ledger.

The network uses a blockchain-like structure called the blockweave, which allows for the storage of large datasets and incentivises miners to store a larger amount of data.

The Arweave protocol is also designed to be accessible through traditional web browsers, making it user-friendly and widely adoptable.

The AR token is the native cryptocurrency of the Arweave network and is used to pay for transactions and storage fees. The supply of AR tokens is capped, which contributes to the economic model of the network.

In short, the network aims to ensure data permanence by leveraging the decreasing cost of storage and the economic incentives provided to miners to maintain the data over centuries.

Finally, I got around to somewhat finishing a simple Arweave file uploader.

The story goes like this; I wanted to upload a 4k video to Arweave network.

Web wallet? Nope

I first tried to use the Arweave web wallet. This would have worked, but the upload hung for so long without visible feedback (I forgot to open dev tools). I incorrectly assumed that the upload was failing. I quickly realised that I paid for the failed transactions, so I should stop wasting AR.

ArDrive? Nope

I tried ArDrive, because I just wanted a nice UI for safe & easy upload of a large file. The web interface, is indeed user-friendly. However, ArDrive uses bundling, and I want a standalone transaction.

Ok… let’s build

Finally, I concluded that I should write an application to do this. Thankfully, there is a great library from everFinance, goar. This made it trivial to write the uploader.

Today, I finally got around to somewhat finishing it by implementing the functionality to resume an interrupted upload. It can now serve as an example, or maybe even help you to upload something.

Demo

Tx: TK60GE39cvuTh_DlLK5ZCnhcoOkN42YFQjsJfzwvrk4

Repo

https://github.com/intob/arpload

Computer Broscience - Joey Innes

The question that broke Dave

Where the code is now

What it did

The part that worked

One packet is enough

Mass was recency wearing a costume

The anonymity claim was too strong

The one I never solved

The question

Deleting the question

Why Zig

A loose end worth picking up

What I don’t know

Writing a panel-method solver in Zig

Two stories about air

A wing is a list of points

The trick: throw the wing away

The bit where nature has to choose

It’s all one linear system

Getting the answer back out

The drag that isn’t drag

Two checks that keep me honest

Why Zig

What it can’t do

Next

How we built Soaring Spirit

One binary, one machine

Ten dependencies

No frontend build step

Money is where the discipline goes

The retry loop

Let the database say no

Two ways to get email wrong

The thing that stopped running

What I’d tell myself at the first commit

Progress on the Peregrine HTTP server

Step 1: Improve the handler

Step 2: optimise the request parser

Method, path and version

Parsing headers

SIMD for reader.readLine

Step 3: Write iovecs

Wrap up

The beginning of project Peregrine - a bleeding fast HTTP server

First thoughts on Zig

Simplicity

Errors lack context

Powerful build system

Lack of interfaces

Expressive language

Generics are beautiful

Standard library is still rough around the edges

Memory management made easy

Conclusion

Comparing the performance of a de Bruijn sequence against a simple lookup table

Approaches

1. Naive approach

2. Right shift

3. Unroll loop

4. Simple lookup table

5. De Bruijn sequence

Benchmarks

16 bytes, 4 leading zero bytes

32 bytes, 2 leading zero bytes

32 bytes, 4 leading zero bytes

32 bytes, 8 leading zero bytes

64 bytes, 4 leading zero bytes

128 bytes, 4 leading zero bytes

256 bytes, 4 leading zero bytes

256 bytes, 64 leading zero bytes

Repository

Dave

Dave - distributed KV store

Design

GETPEER & PEER Messages

PUT Message

GET Message

DAT Selection by Mass

Proof-of-work

SIMD for `reader.readLine`