5 Amazing Reasons To Choose Monolithic Version Control
What is your favorite thing about working at BuzzFeed?
Since I started a few months ago, I’ve heard this question a lot. If I’m talking to someone who never spent a Friday night recompiling their Linux kernel, I might say something like all the animated gifs in official email correspondence or the coffee machine that automatically grinds beans on-demand.
But if you catch me in a sincere moment, I will say without hesitation: monolithic version control.
Choosing a repository structure is like choosing a god. You can debate the technical pros and cons ad nauseam, but it is partially a cultural decision and partially a leap of faith. Inertia is usually a factor, too. It’s like any other holy war:
Reasonable people can disagree on Vim vs. Emacs but also collaborate on the same repository. You can even disagree on tabs vs. spaces if you never edit the same file (or even if you do). But you can’t disagree on repository structure. A choice is necessary.
The default choice is to use separate repositories. A small repository is free to create and easy to maintain in 2017. If, like me, you are old enough to remember CVS, you know this wasn’t always the case.
On the other hand, maintaining a monorepo is not easy. It requires thought and coordination. You cannot make up your own standards, choose your own build system, or pick your own workflow. You need to agree on a lot upfront. It may seem oppressive, especially if you’re used to working on a small team with its own repository.
Everyone also needs to be aware of how their actions in the monorepo affect others. Even at the moderate scale of hundreds of services, technical issues begin to crop up more frequently. And when they do, they affect the whole company, not just one team.
For example: If you store some binary files in an individual Git repo, it usually works fine. But if everyone is doing that in a Git monorepo, the repo size might soon become unwieldy.
You will face scaling challenges no matter what repository structure and version control system you choose. But if you use individual repositories, usually you will face fewer problems initially. So why should you defy inertia and choose a monorepo? Isn’t it easier, safer, and faster to just keep everything separate?
Here’s some amazing things that will happen when you work on a monorepo.
1. You will have access to everything.
The first thing I realized about monolithic version control is that it’s the ultimate onboarding hack.
When you have no idea what you are doing, where anything is, or what the right way to do something is, reading other people’s code can answer many of your newbie questions. The more examples you have, the more likely you’ll find what you’re looking for. When every team’s code is already checked out and a simple grep away, looking for answers this way becomes a new reflex.
By contrast, in organizations with many separate repositories, another team’s code is frequently treated as a foreign entity. It’s a black box at the other end of an arrow in an architecture diagram. If you get curious, you can start asking questions:
- Can you tell me where your repo is?
- Can I have read access to your repo?
- Can I submit a pull request to your repo?
But the more curious you get, the more likely you are to hear the answer no.
If you work for a spy agency, maybe the default answer should be no. But usually source code doesn’t need to be so classified. A monorepo sets a cultural norm that the default answer to Can I have access? is yes, at least when it comes to code. If you want curiosity to be rewarded rather than stifled, yes is the better default.
2. You will always know what the right version is.
My favorite rule of the BuzzFeed monorepo: Master is golden.
Code can’t be merged to master until it’s been code reviewed and verified on staging. Soon after that, it’s typically released to production.
This means as an outside observer of another team’s service, I don’t need to know anything special about their workflow, versioning, or release schedule. It also means I don’t have to do my least favorite thing and the thing I’m most likely to forget: manually bumping version numbers.
The only version that matters is HEAD of master.
Software systems are complex. It is a full-time job to understand how a system interacts at just one commit hash. When you introduce the additional complexity of dependency resolution, it becomes many times more difficult to debug problems.
For shared libraries, you may still want to use semantic versioning. But at the service level, it’s much more sane to have a unified view of the live system.
3. You will become a better person.
My second favorite rule: Write good commit messages.
When I’m developing code, the last thing I’m thinking about is commit messages. Sometimes I write bad ones. And I know I’m not the only one who does this.
If you have your own repository, it can be tempting to leave those bad commit messages in as you merge to master. This is especially true when you’re pressed for time and working on a live system. I’ve done this on countless occasions.
But when your commits will be shared with the entire company, it becomes important to clean up your history. Each commit should be a self-contained feature or bug fix. And each commit message to master should explain what the change does in detail. At BuzzFeed, we use this guide as a starting point to writing better commit messages.
4. You will build things you never dreamed of.
“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.” — Leslie Lamport
Here’s a counterintuitive claim: The more micro your services are, the more compelling monolithic version control is.
If you have a few large applications, go ahead and put them in a few separate repositories. But if you have hundreds of microservices, you should put them in a monorepo.
Just because your services are micro doesn’t mean the total amount of running code needed to develop a new feature end-to-end is micro. The surface area of what you need to understand is the same or greater.
The more monolithic a system is, the more likely that surface area is limited to a single application that you already know how to build and run. But in a microservices architecture, you cross into different services frequently. You become affected by “a computer you didn’t even know existed.”
If you want to build a multi-service feature or debug a systemic problem, eventually you’ll face an existential crisis: Hell is other people’s build processes.
If you only have a few large applications, maybe it’s worth spending an afternoon creating the right build environment for all of them.
But if you have hundreds of microservices all with different build processes, it’s not feasible to run them all locally. You might give up and say I’m blocked and file a ticket with another team. Or maybe you’ll launch code to staging and see what happens when your code interacts with its upstream and downstream dependencies for the first time.
This makes for a long and error-prone development cycle. Here’s a different approach:
- Use a monorepo that everyone has access to.
- Leverage VMs and containers to standardize how to build and run services in a way that works both locally and in production.
- Enforce that standard via continuous integration in the monorepo.
Now the “computer you didn’t even know existed” is just another service you can see, run, and modify right on your laptop. And if you want to modify it permanently, you can submit a pull request.
It’s possible to be this strict about standards without a monorepo, but I’ve seen it promised a million times and delivered exactly never. Once you have one rogue repository, you will soon have many more. Your standard will eventually be lost and forgotten.
5. You will find strength in unity.
Synergy is an easily-mocked word, but it has a genuine meaning: an interaction that creates an effect greater than the sum of separate efforts.
If someone tells you a light bulb manufacturer and a fishing company can create synergy in a joint-venture, you might want to short their stock. But delivering production software systems is fundamentally about creating synergy through the interactions of code that was built by many different people.
Because good software development and system design is frequently about modularization, it’s easy to forget the final product is not about how great the individual pieces are. Sooner or later, your code will need to work together. Different levels of testing can help predict how services will interact, but the only true test is actually running them.
To me, that’s the fun part. I can’t build all of BuzzFeed by myself. But I can build a small piece of it. Seeing how that piece fits into the greater whole is my greatest motivator, and it’s most easily accomplished with a monorepo.
