A few years ago I was lucky enough to lead a small team that got to develop a large new distributed system. The project was/is quite successful, quickly replacing the previous, scaling up well, and being stable. During early code reviews I often had a couple of pieces of feedback:
- Never hold a mutex while doing anything that may block.
- Don't set deadlines on RPCs.
These were informed by working on and debugging previous systems like
Bigtable and other complex distributed systems. I repeated them so much, that eventually they were transcribed as Maloney's Laws. Over the next few years probably 30+ engineers contributed to the project, many went on to new projects and some even began to cite these laws to other engineering teams.
Now I know that these laws as stated here may be overly-simplistic. But given the way Google builds systems internally, they are good rules of thumb. Of course like all rules, there are exceptions.
I will explore the logic behind these two rules in follow up posts.
