I’ve recently begun work on a follow-up to my 2009 book Coders at Work. While Coders is a collection of interviews about the craft of programming with fifteen great programmers such as Donald Knuth, Ken Thompson, and Jamie Zawinski, the new book, tentatively titled We Learned a Lot: Hard-Won Lessons from Building the Modern Web, will collect stories from the world of software but with a focus on the mid-90s to today. The engineers and managers who came of age in this era had lots of chances to learn lessons the hard way. Among the challenges they faced: dealing with massive changes in scale as their infrastructure moved from someone’s computer under their desk to hundreds or thousands of someone else’s computers in the cloud and adapting to a world where complex software was built on top of an ever-shifting foundation of open source libraries, languages, and frameworks.
My goal is to find the stories that capture the true nature of our field: that software development is messy and does not always proceed by clean logical steps from problem to solution; that even the best of us are often feeling our way in the dark. Some of these stories may provide clear lessons; others may just provide the reader the solace of knowing that they’re not the only one who has ever found themself in such messy situations. The book will be aimed primarily at working programmers though should also be accessible to enthusiastic amateurs.
I’m currently interviewing folks for this book. Unlike Coders at Work, I think this book will be more reported and shaped than straight Q&A interviews. Thus I am looking for anyone who has a story, or stories, from a a career building software. These may be stories where you are the hero—the one who figured out the bug or got the system back on line—but the best stories will capture the missteps and confusion along the way. It is also of course fine if you’re not the hero or even if you were the goat. Humility is very appealing. At any rate, in collecting and telling these stories, I aim to always remember that most decisions probably made sense at the time to the people who made them and everyone was trying to do a good job.
If you have a go-to story or stories you tell about things going remarkably or amusingly pear shaped, I certainly want to hear those! Otherwise, here are a few things to think about:
What is something you know now that you didn’t at the beginning of your career that new engineers still need to learn?
What is something you know now that you didn’t earlier on because basically nobody knew, i.e. things that we had to learn as in industry?
What is something you believe now about building software or being on a team that earlier in your career you didn’t believe or even believed the opposite?
What is something that many people believe that you think is wrong?
What was something you were very right about and couldn’t convince others of, only to be proved correct by subsequent events?
What was something you were very wrong about, maybe even something people warning you that you were wrong about, and how did that play out?
If you can think of any of these, what were the episodes in your career when you learned important things or came to your current beliefs? Or perhaps when you suffered the consequences of your prior lack of knowledge or your old, mistaken beliefs. If you have contrarian beliefs now, what were the times you experienced things working well despite doing them in a way most people would consider wrong.
I’m interested in any story that captures something true about building software but a few broad themes I suspect will be important are:
What did we have to learn to take the web from a box under a desk to the cloud?
Software can go wrong so many ways. Everything from a single misplaced character that ruins everyone’s day to elaborate cascading failures in a distributed system. What were the worst reliability problems you’ve seen—or created? What was the most elusive bug you’ve ever had to track down? How do you test code that is part of a complex distributed system?
If things weren’t hard enough, the modern web has had to operate in an increasingly hostile environment. While no fun to live through, getting hacked often teaches some important lessons.
In the immortal words of Bullwinkle, this time for sure! Sometimes a big rewrite seems like the only way out of some terrible predicament. And occasionally it actually is. But it’s almost always a high-risk maneuver. Have you seen one go well? Or horribly wrong?
What made the best teams you were on great? Or the worst teams so bad? Why did we ever think it was a good idea to talk about ninjas, rock stars, or 10x engineers? Have we figured out anything about how to interview and hire engineers?
Computers would probably be fine if we could just keep humans away from them. But as soon as you have users, they’ll start doing the darnedest things. What surprising things have your users done?
There are no doubt others. At any rate, if you’re willing to chat with me, hit me up at peter@gigamonkeys.com or DM me on Twitter @peterseibel and we can set up a time, either to Zoom or perhaps to meet in person if you’re in the Bay Area and vaccinated. And if you know someone else who you think has some good stories, please refer them to this page!
In addition to Coders at Work I also wrote Practical Common Lisp, published by Apress in 2005. When not writing books I have worked as both a developer and a manager, most recently as the Director of Engineering at the Democratic National Committee from 2017 to 2020. Immediately prior to that, I worked at Twitter as a Director of Software Engineering and a Senior Staff Engineer. I got my start in software in 1995, learning Perl on the job while working on the Mother Jones website, fresh out of college with an English degree.