Stateful is why you're still on the cloud

Before Laneful, we built SendGrid. In the early years our state lived on dedicated servers: machines we rented in someone else's data center, racked by people we never met. We couldn't walk to them. When a disk died, the fix was a support ticket.

RAID 10 was the safe choice, and here's what the safe choice looked like from the wrong end of a ticket queue. A disk dies. Fine, that's what mirrors are for. You file the ticket, remote hands pull the drive, the rebuild starts. Now every surviving disk is being read end to end at full speed for hours, which is exactly the workload that kills the dead disk's mirror partner. The drives shipped in the same batch, so they aged like siblings. When a rebuild lost that race (or the controller died, or a filesystem came back wrong after a power event), we no longer had a storage layer. We had a recovery project: days in rescue mode, imaging half-dead drives over the network, coaxing arrays back to life to rescue mail that existed nowhere else on earth, while the queues backed up behind us.

So the company started moving to the cloud, and the reason was never elastic compute or global reach. We wanted to stop fighting for our own data, and we wanted the application to get simpler. Every workload that landed on S3 or RDS was a fight we never had again. During that transition we evaluated Ceph and ran it as a stopgap, a bridge on the way to the cloud, never the destination. Even as a bridge, it was a system you staffed, not a system you ran. I left in 2018 with the migration still in progress. The direction was set, and the direction was right.

Eight years later, I think about that direction differently. Compute is a commodity now: rent it anywhere, or rack it yourself in an afternoon. Stateless services move freely. What doesn't move is state. The hard part of running on your own hardware was never CPUs. It's the data that can't be lost, and everything you have to get right about disks, networks, and failure before you trust yourself with it.

That's what this post, the first in a series, is about: the questions state forces you to answer, what the cloud actually sold us, and an honest look at what running state on your own hardware looks like in 2026.

The interrogation

Every stateful system puts its owner through the same interrogation, and the order matters, because each answer constrains the next. Here's the list I carried for years, pretty much as I used to write it:

1. Can I rebuild the state? If so, focus on keeping as much of it in memory.
2. If I can't, how do I replicate it?
3. What consistency? Sync vs async?
4. Do I need a leader to synchronize writes?
5. What do I use for leader election? Do I do things manually, since leaders don't fail often?
6. What are the performance profiles of the different options?
7. What kind of hardware do I need?
8. Is there an existing technology that can help me achieve this?

It's a decent list. It's also a list that starts in the middle. It jumps straight to mechanism (rebuild, replicate, elect) before asking what's actually at stake. It treats consistency as a coin flip. And it never mentions backups, blast radius, or what happens to the old leader once you've picked a new one. The list I'd write today starts two questions earlier.

How much data can you afford to lose, and how long can you afford to be down? These two have industry names, RPO and RTO (recovery point objective and recovery time objective), but they're just two numbers, both measured in time. RPO: if the system died this instant, how much recent history are you willing to never get back? Zero seconds? Five minutes? A day? RTO: once it dies, how long until you're serving again? Email makes the pair concrete. Until your mail server says 250 OK, the sender's retry queue is your safety net: a crash costs you nothing they won't resend. The moment you say 250 OK, your acceptable loss for that message is zero, because the sender's queue forgets it the instant you accept it. Your RTO, meanwhile, has slack that a checkout page doesn't: senders retry for days. Every other answer on this list is derived from these two numbers, and most teams have never written them down. "Zero and zero" is a legitimate answer. It's also the most expensive item on the menu, and you should price it before claiming it.

Is this data the source of truth, or derived from one? That was question one on my old list, and I used it to justify keeping things in memory. Wrong axis. Caches, indexes, and materialized views earn relaxed durability because the truth lives elsewhere and a rebuild is a cost, not a catastrophe. A queue holding mail that exists nowhere else earns nothing. Memory versus disk is an implementation detail. Derivation is the property.

What does replication actually promise you? Not "sync vs async". That's a false binary. Async replication is a data loss window you've chosen. Fully synchronous is a latency and availability tax you've chosen. The workhorse in between is quorum: commit when a majority acknowledges. Raft, Aurora, Kafka's ISR, Postgres's quorum commit. The systems that survive in production almost all live at this middle point. Where the replication happens (which layer of the stack does the copying) is a big enough question that it gets the next post to itself.

Who is allowed to write, and when that changes, how do you stop the previous writer? Picking a leader is a solved shopping problem now: etcd leases, Kubernetes, Patroni, an embedded Raft library. My old list asked whether to just handle it manually, since machines rarely fail. That answer works until the night it doesn't. And the hard part was never election anyway. It's fencing. Somewhere there's a deposed leader with a warm page cache and a working network card that hasn't heard the news. Split brain isn't caused by failing to pick a leader. It's caused by failing to un-pick one.

How do you get yesterday back? Replication is not backup. Replication faithfully propagates your DROP TABLE at the speed of light. Backups, point-in-time recovery, and a tested restore (the step everyone skips) are a separate discipline, answering a different question. A backup you've never restored is a hypothesis.

What's the blast radius? Disk, host, rack, room, region. Each rung is a different failure you either survive or don't. Two replicas in one rack share a top-of-rack switch and a sprinkler system. Synchronous replication inside one building is high availability, not disaster recovery. Decide which rungs you're paying to survive, out loud.

What does your answer assume about the hardware? Every replication design silently assumes disks that tell the truth about fsync and networks with a certain latency. Those assumptions are purchasable, and cheap to get wrong. More on this near the end.

Build, adopt, or buy? The last question and the honest one: of everything above, how much do you actually want to own?

What the cloud actually sells you

The story I believed (and wrote into an early draft of this post) is that cloud providers solved all this by replicating volumes underneath you. It's worth being precise about why that's not quite true, because the truth explains the moat better.

EBS replicates within a single availability zone. AWS's own feature page says it plainly: a volume lives in one zone and is replicated there. There's no multi-AZ EBS volume. Google will synchronously mirror a persistent disk across exactly two zones. Azure offers zone-redundant disks spanning three zones, with sharp edges like not covering their newest disk types. Exceptions, not the norm. And volume replication only ever buys crash consistency: the database on top still keeps a write-ahead log, still fears torn pages, and something still has to fail over the compute attached to that volume.

Where multi-AZ durability actually comes from is the service layer. RDS Multi-AZ runs a synchronous standby in another zone and manages the failover for you. Aurora is the tell: when Amazon rebuilt its flagship database, it abandoned block replication entirely and shipped the log instead, six copies across three zones, committing on four of six, because copying whole pages over a network is mostly waste. Read the SIGMOD '17 paper, or Werner Vogels' summary as a gentler entry point. Their own name for the design was blunt: "the log is the database" is a section title in the paper. Hold that thought. This series will come back to it.

So the cloud never really sold us replicated disks. It sold answers to the interrogation, one product per question: RPO and RTO as a dropdown on RDS, fencing hidden inside a managed failover, backup as a retention slider, blast radius as a region menu. We were the customer that catalog was built for. We didn't start SendGrid's migration to rent compute. We started it to stop answering these questions ourselves.

Compute never kept anyone on the cloud. The catalog of answered questions is the moat.

On-prem in 2026: an honest look

I left the hardware world in 2018, so everything in this section is researched rather than lived. It's the homework I did to update an eight-year-old mental picture, with links so you can check me. The picture turned out to be badly out of date, in a good way.

Start with DRBD. In my 2018 picture, DRBD was a mirrored disk with no story for who's in charge: it copied blocks between two machines and left leadership, failover, and split brain to you and a Pacemaker configuration only one person in the building understood. That picture is dead. DRBD 9 grew quorum, with a cheap diskless third node as tiebreaker, and LINBIT now ships drbd-reactor, whose promoter plugin does the automation that used to require the Pacemaker priesthood. If a node has quorum and the resource isn't active elsewhere, it promotes it and starts the services. Lose quorum and it stops the services and steps aside so another node can take over. The fair criticisms today are different ones: it's a one-vendor ecosystem, quorum means three nodes minimum, and a decade of Pacemaker-era horror stories still colors its reputation.

Ceph is the one I have personal history with, so let me be careful. The Ceph we bridged onto around 2017 was a system you staffed: hand-run deployment, upgrades as events, tuning as a discipline. By every account I can find, today's Ceph is operationally a different animal. cephadm turned deployment and upgrades into orchestrated operations, Rook did the same inside Kubernetes, and Proxmox ships Ceph as an integrated feature you enable from a UI, which is how a generation of shops fleeing VMware's new pricing ended up running it almost by accident. What hasn't changed: it wants real NVMe, real network bandwidth, and enough nodes for rebuild headroom, and it remains a poor fit below four or five nodes. But the system that consumed us as a stopgap is now a reasonable default for a small cluster. That sentence would've sounded like science fiction to me in 2018.

Between those two, a middle tier grew up that didn't exist when I left, most of it born inside Kubernetes. Longhorn optimizes for simplicity. OpenEBS Mayastor chases performance over NVMe-over-fabrics. Piraeus wraps the LINBIT stack for Kubernetes. Vitastor is the small fast outsider. The pattern across benchmarks and operator reports is consistent: the simple ones cost you performance, the fast ones cost you maturity, and none of them is turnkey. Meanwhile GlusterFS, for years the easy answer, is effectively gone. Red Hat wound down its Gluster product and development has largely stopped.

So the claim I'd have made in 2018 (your options are DRBD or Ceph, and both will hurt you) is out of date. The 2026 version: the options are many, young, and none of them is turnkey. That's real progress. It's also not the whole story.

The fallback everyone actually uses

Because in practice, most teams running state on their own hardware do neither. They run the database they already have and let it do the replicating. This used to feel like an admission of defeat. It's now the mainstream pattern, with real tooling behind it. Patroni and CloudNativePG turn Postgres failover (election, fencing, the whole interrogation) into configuration. Strimzi does it for Kafka. NATS ships it in the box. These are the cloud's managed services rebuilt as self-hosted operators, usually running on local NVMe, where they routinely outperform any distributed block layer underneath them. A database replicating its own log semantically beats a storage layer blindly copying every page it scribbles.

The honest residue: this leaves you operating several separate replication stories, one per system, each with its own failover logic, its own backup story, and its own personality at 3am. Hold that tension. It's exactly where the next post picks up.

Before you buy hardware

One paragraph worth stealing. The single most consequential line on a storage spec sheet is power loss protection. Workloads that fsync constantly (database write-ahead logs, etcd, anything consensus-shaped) collapse to a few hundred operations per second on consumer SSDs, because a drive that can't guarantee its cache survives a power cut has to flush the slow way or lie to you. Enterprise NVMe with power loss protection can safely acknowledge synced writes from cache, and the difference isn't a percentage. It's orders of magnitude. Half the "distributed storage is slow" complaints on the internet are actually "I bought consumer drives." Beyond that: 25 gigabit networking is cheap now, used enterprise gear is cheaper, and the round trip you pay for replication is purchasable down to microseconds.

One humility note before you write any of this yourself. Postgres (yes, Postgres) mishandled fsync errors for roughly twenty years before fsyncgate surfaced it in 2018. If the flagship open source database got fsync wrong for two decades, the lesson is to buy tested replication, not write it.

Why now

If this were just my scar tissue, it wouldn't be worth a series. But the ground moved. 37signals walked out of AWS and published the receipts: roughly two million dollars a year saved on compute after replacing it with about $700,000 of Dell servers, then the storage exit: eighteen petabytes of Pure Storage arrays, bought for about one year of their S3 bill (around $1.5 million a year) and costing under $200,000 a year to run, with AWS waiving $250,000 in egress fees on the way out. Last summer they deleted the AWS account. Broadcom's VMware pricing pushed a generation of shops onto Proxmox, where Ceph is a checkbox. Oxide will sell you a rack with distributed block storage built in. European sovereignty rules are pushing state onto owned hardware whether teams want it there or not.

Repatriation stopped being contrarian. But everyone who repatriates hits the same wall, and it's not compute. It's the interrogation.

Where this goes

I've now been on every side of this. I fought RAID arrays through a ticket queue and lost both sleep and mail doing it. I watched a company point itself at the cloud because owning state was breaking us, and that was the right call. Building Laneful put me back in front of the interrogation, question by question, this time by choice.

This series is what we've learned. Next post: the first and biggest fork in the road. When you replicate, which layer of the stack should do the copying? The disk, the database, or something else entirely? That one choice organizes everything downstream.