Software Projects: Sink or Swim — Why the Owner Matters Most

The technical aspects

I’ve been developing software professionally for a decade now and have been writing code since I was a kid. Through my experiences across personal projects and professional endeavors, I’ve learned that building a software project that lives for years comes down to four key areas, with one being more important than all the rest:

• Consistent Patterns

• Good Test Hygiene

• Well Maintained Documentation

• The Benevolent Dictator

Each of these areas is crucial for the long-term health and survivability of a project.

However, while you can have deficiencies in the first three areas and your project can still pull through with sheer willpower, number four is absolutely non-negotiable. Without a benevolent dictator, someone who is responsible for initially establishing and guiding the first three areas, your project is doomed. This is the crux of what I’ve learned, and I’ll spend this post discussing the first three points and setting up that fourth point which will be discussed in depth in part 2. I would have discussed it in this post too, but when I started writing this as one blog post I got to a point where it was over 3000 words long so splitting it up became necessary.

Pattern Consistency

First, what is a pattern exactly? A pattern is a repeated abstraction or implementation detail throughout a project that helps clearly articulate what the code is doing and provides easily identifiable structure and functionality during software development. Establishing common patterns in your project helps with a few things. They provide common ground ideas developers can use when contributing to the project, while for established contributors, they help provide clear signposts when reviewing contributions or coaching people in the project. Developing patterns helps create a shared understanding in the project that all contributors can rally around. In short, they’re a living representation of how the project works and how developers think about the project. Long-time contributors can easily point to specific patterns for new contributors to help them onboard into the project-specific norms.

Establishing common patterns specifically in the codebase is one of the most impactful ways to ensure your project is maintainable long-term. These patterns provide a roadmap for structuring and organizing code, making it easier for new contributors to understand and for existing members to navigate and modify the codebase without introducing errors or inconsistencies. So how can you establish patterns for your project? There are a number of ways.

Drawing on Experience: Drawing on experience, both your own and others, for establishing your project’s patterns can significantly enhance maintainability. Books are an excellent resource for learning and applying proven design patterns.

For example:

• Design Patterns: Elements of Reusable Object-Oriented Software by the Gang of Four introduces several design patterns that can be adapted to various projects. This book is foundational for understanding object-oriented design and applying reusable solutions to common software design problems.

• Designing Data-Intensive Applications by Martin Kleppmann provides insights into designing robust and scalable data systems, addressing issues such as data storage, processing, and consistency.

• SQL Anti-Patterns: Avoiding the Pitfalls of Database Programming by Bill Karwin discusses common pitfalls in database design and how to avoid them, which is crucial for maintaining efficient and reliable database systems.

Those are just three books (and prolific ones at that) I’ve personally referred to over the years as I’ve grown on my software engineering journey. If you choose to use these books I would recommend going to specific chapters as reference material versus reading them straight through.

Learning from Blogs

Blogs are another great way to learn about how others design their projects. They often provide real-world examples and practical advice that you can apply to your own work. Some notable blogs include:

• Rachel by the Bay: Offers insights into various technical challenges and solutions, often based on personal experiences and practical scenarios.

• Dropbox Tech Blog: Shares detailed technical articles about the engineering challenges and solutions at Dropbox, providing a glimpse into the practices of a large-scale tech company.

• Discord Engineering Blog: Provides in-depth articles about the engineering behind one of the most popular communication platforms, highlighting best practices in system design and optimization.

• Martin Fowler’s Blog: Features articles on software design, architecture, and agile methodologies, authored by one of the long-time voices in the software engineering community.

• Dan Luu’s Blog: Very excellent writing detailing a number of technical and non technical topics across many years of work at companies like Google and Twitter.

Other Projects

I’d be remiss if I also didn’t mention reading code from projects or people in those projects you admire. Plenty of open source projects have been around for longer than I’ve been alive at the ripe old age of 29 and they have stood the test of time. Be selective about this though, as they’re just as likely to have adopted context specific terrible patterns as good ones! Many newer projects even have discords where you can join and talk with everyone involved about their software development patterns. You’ll forgive me if I shamelessly plug our discord here as one of those places: https://discord.gg/YZmbcxG3ya

Utilizing ChatGPT

More recently, ChatGPT has become a close second to me versus trawling the web for me in bouncing around design ideas. While I don’t ask it to write code or take its replies to heart without question, it can enumerate many design patterns for incredibly specific situations making brainstorming and refining ideas more productive. For example, I developed a fairly complex project with consistent patterns while having a series of back and forth rubber duck sessions with ChatGPT.

I will mention one pitfall of using ChatGPT is if you don’t have the depth of experience to know when it’s simply spitting out nonsense you’re going to have a bad time if you design your project around its suggestions. There were multiple times I hand waved what it was replying with while building Dart, but it can legitimately provide things you’ve forgotten about or hadn’t considered from its breadth of knowledge which I found pretty invaluable. If you’re a bit newer to the field I’d recommend going with books, blogs and practical experience for a while before bouncing architectural ideas with ChatGPT, or at the very least cross check suggestions across the web which is something I still did even when something seemed plausible.

Linting for Consistency

Slightly aside from true code patterns, but important consistency nonetheless: Linting a codebase automatically with common established styles is an absolute must. It ensures that the code follows a consistent style guide, reducing the cognitive load on developers and maintaining a uniform codebase. Linting should run automatically on commit and as part of your merge process. Many languages have commonly established style rules.

For example:

• Python has PEP8, which provides guidelines for writing readable and maintainable Python code. Black and Darker are great tools that can automatically rewrite your code to conform to these standards. Black will apply the PEP8 rules project wide, while Darker supports partial updates to the codebase so you can incrementally update larger existing projects.

• JavaScript/TypeScript projects often use tools like Prettier and ESLint to enforce style rules and code quality standards.

For languages without an established community style, settling on a set of style rules early on is important. Tools like Prettier come with a reasonable set of rules out of the box, making it easier to adopt consistent styles.

Patterns in programming are not universal and often the product of one or a few people. They can vary significantly between different types of organizations (and even within organizations), such as startups versus FAANG companies. Therefore, while consistent patterns are important, it’s also crucial to intentionally review and evolve those patterns when it makes sense, ensuring that the project remains adaptable and responsive to new community insights, organizational needs, technological advancements and changes in the industry. Easy right?

Testing

Without test coverage, your project will grow into a ball of spaghetti where one thread pulled breaks another thread elsewhere. Ask me how I know. Testing ensures that changes to the codebase do not introduce new bugs and that the code remains maintainable over time.

Test Coverage

Testing does not mean every single little thing in your codebase needs a test. Integration testing core business logic that touches many areas is better than aiming for 100% test coverage with isolated unit tests that only tell you single components work in a vacuum.

Focus on:

• Core Business Logic: Integration tests for business logic ensure that different parts of your system work together as expected, catching issues that might arise from the interaction between components.

• High Risk Features: Tests for high risk features help catch bugs early and ensure that essential functionality remains intact, providing confidence that critical parts of the system function correctly.

Development and Testing

Development does not need to be test-driven for tests to be useful. The key is that tests are always added with a feature and not at a separate time. This approach ensures that tests are relevant and aligned with the current state of the codebase. By incorporating testing into the development process, you create a safety net that allows for easier refactoring and feature additions.

First-Party Testing

Tests should be written and maintained by the team or person building the project. Third-party QA has its place as another line of defense for production code, but for maintainability and cohesion as a project grows, first-party tests are a must. They help new (and old) contributors safely make changes without breaking core functionality and reinforce established patterns by codifying certain design expectations.

Benefits of First-Party Testing

• Understanding and Ownership: The team that develops the code has the best understanding of its intricacies and is therefore best positioned to write effective tests.

• Rapid Feedback: Immediate feedback from tests during development helps catch issues early, reducing the cost and effort of fixing bugs later.

• Documentation: Tests serve as active documentation of the expected behavior of the code, providing examples of how different parts of the system interact.

Documentation

Speaking of documentation, it is a highly valuable but often under appreciated part of software development. If you are looking to build a project that survives the test of time you may feel free not to write and maintain high quality documentation at your own peril. Most people already understand the value to people maintaining the project, but judge that it’s not valuable enough to prioritize for one reason or the other. I’m not going to discuss the team benefits of documentation because frankly I’ve never seen someone swayed by it if it’s not incentivized already. I will instead discuss the external value documentation provides which hopefully has a better chance of making an impact.

Documentation is the first way many prospective users of your software, both in industry and open source will ever be introduced to what you’re doing and how it can potentially provide value to them. If your documentation is lacking users will have a harder time getting started and the risk they bounce off the project entirely is much higher than it needs to be. I think that sometimes developers overlook the external factors of documentation and think that because they have all the context in their head they don’t need to write it down. Even people who do write documentation often don’t include enough detail because of how close they are to the project, or write said docs clearly enough to make it easy to work with their software. I know because I run into it all the time and it adds a lot of friction.

Recently, I was working with well known software to make distributed LLM (Large Language Model) training easier and their documentation is existent, decent even, but it misses a lot of key details where if you’re working at a lower level with their software it is insufficient. I needed to go into their code and reverse engineer the details more often than was enjoyable. In fairness this is a hard problem to balance against development priorities and available hands when your project grows to be so big, but that’s why a strong documentation culture needs to be built from the very beginning.

A Few Practical Tips For Documentation

• Use clear short sentences

• Define all jargon the first time you use it

• Bullets and numbered lists are good

• Graphs and diagrams are important
  • Interactive media is a huge plus and require a lot of time, but a solid static image is fine

• Flow and order are important, starting from simpler concepts and building up to detailed expert sections. Reference should be at the end in a ToC (table of contents).

• Take whatever level of detail you think is good and go even more detailed
  • Following building simpler to more expert concepts, perhaps split by different sections so users can choose what level of depth they need

• Provide a mixture of reference material and guides

• Example projects of what people can do with your software are a must

A resource I don’t think many people are aware of but I used myself a few years ago, Google provides a few short open online courses focused on technical writing. You can find those courses here. Additionally If you want examples of good technical documentation take a look at Stripe, their documentation has been best in class forever. Hopefully the reasoning and tips provided here are enough to kick start you on writing good technical documentation for your project, it is paramount for growth and success.

Conclusion

In this part one we discussed what patterns are, how to establish them and where to draw inspiration. We also discussed testing, who should own testing, whether being test driven is super important and what are some non obvious benefits of testing. Finally we discussed documentation, the importance to new users, practical tips and some small courses where you can learn to improve your own technical writing. These areas are key to establishing a project and ensuring it can grow and thrive over time. I couldn’t cover every possible angle or this would have turned into a book, but hopefully this provides some starting points and a little food for thought as you go on to manage your own technical projects.

There is a lot to think about here and not every tip and trick has to be done at once, but when creating a project you want to grow you should try as hard as possible to get these things right. This is as much to attract people to use the project as it is to set a foundation for contributors to eventually take on responsibility and ownership themselves. Early on in a project’s life cycle all of the areas discussed so far have one thing in common. They are usually initially established and set up and guided by one person. This person is the benevolent dictator. If you do it right as the benevolent dictator you should be gradually ceding responsibility to others on the project and create a whole bunch of people who can help grow and maintain the project for a long time to come. In part 2 I’ll discuss the social side of being the benevolent dictator, handling the role and how to eventually minimize your need from others on the project so the team around the project can thrive without you if need be.

A Quick Plug For BismuthOS

Hey, hi, how are ya? I’m Ian, the author and one of the co-founders of BismuthOS an AI enabled Cloud platform that helps you get your Python project built and launched quicker and with less hassle than any other solution. We have serverless functions, a NoSql DB and blob storage sort of like S3. Another really cool thing is we also have an AI assistant that understands how to use and build stuff with our platform as well as tools from the broader Python ecosystem. If you hate dealing with YAML or working with the AWS console we can help you out. Check out the platform here at www.bismuthos.com