Git at any scale

From the article:

Over the years, companies that tried hosting Git repositories at scale noticed that this packfile-based design was a major limitation on both availability and scalability. Packfiles are large binary files that must exist on a filesystem for Git to access them. The simple approach of having an HTTP server in front of a repository on disk has a very low ceiling. Ideally you'd want the repository to exist on many disks and many machines (this lets you run many Git operations in parallel, and keeps your repository available when a server crashes). But how do you do that?

There are broadly three possible approaches to accomplish this, in increasing order of complexity: distribute the filesystem, distribute the packfiles, or distribute Git itself.

[...]

Spokes was originally developed at GitHub around 2013, and it has since become an industry standard. Most Git hosting services use a variant of the Spokes approach (application-level replication for Git repositories) in their architecture. The main reason Spokes has worked well for many years is that it made three fundamental choices that, over time, have been proven to be optimal:

[...]

Spokes is a consensus-based distributed system. It works by storing several copies of your Git repository on different servers. Whenever you push new data, an orchestrator fans out your push so that every instance of your repository receives a copy. The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.

[...]

This scalability constraint also applies the other way. When agents work with Git repositories at scale, they often operate outside of a monorepo by creating vast numbers of small repositories, many of them throwaway, and most of them barely touched. Spokes struggles here because it still requires three replicas for every one of these repositories. Three mostly idle replicas, which cannot be trimmed down because then the system wouldn't be fully consistent and data loss would be possible. With three-phase commit, the floor is always too high, and the ceiling too low.

Another flaw, impossible to see up front, but painfully obvious after having suffered through it, is that Spokes can be rough to operate at scale. Because the repositories on disk are always the source of truth for consensus, every copy of every repository is very important. You have to treat repositories as pets, not cattle.

[...]

Continuity is a simple system (a system cannot be easy to operate if it is not simple). The core primitive behind it is a write-ahead log, which we store in S3-compatible object storage. In production, we run directly on S3, but we designed it so it can be deployed on any cloud.

[...]

The local copy of the repository is, of course, a normal Git repository stored on a very fast NVMe drive. We do the same thing that Spokes does because I think Spokes got that exactly right. It allows us to reuse all the amazing OSS work of the Git community, including the upstream Git client and its many performance optimizations. It lets us focus on shipping new features, instead of doing weird stuff with Git.

[...]

We've seen that one thing that makes a Spokes cluster hard to operate is that it's very important to keep track of the location of every repository on each server. Continuity does this very differently. Where does every repository live? The answer is "anywhere". It doesn't matter! We treat repositories like a warm cache on disk, but the source of truth is always the write-ahead log in S3. The system is stateless, and there are no routing tables (and no relational database to operate — hashtag blessed). If a repository is missing from the local disk when accessed on a host, we just materialize it from the WAL. We can do this very efficiently, but of course we don't want to do this all the time, because it'd be wasteful. In production, we use rendezvous hashing to map a repository ID to the list of nodes where we expect it to be. All the state we require to route repositories is the repository ID and the current set of healthy nodes in a cluster. But if this state gets out of sync (e.g., a node becomes unhealthy), that's perfectly fine too. We'll just materialize the repository on whichever node comes next.