Best practices quick guide: Developing maintainable software

In today's world, software teams are expected to deliver features rapidly. The ever-increasing demand for delivering solutions quickly pushes software developers to de-prioritize aspects of supportability and maintainability. These usually go into the backlog as tech debt. In my 20+ years in tech I have seen many teams get stuck in this tech debt quicksand, but I've also seen many teams handle maintainability right. Let us look at how development teams can help ensure that the software solutions they deliver not only meet current requirements and best practices, but are also maintainable, so they do not become irrelevant quickly.

ISO 25010, which defines the model for software product quality and software quality in use, defines maintainability ‘as the quality characteristic that represents the degree of effectiveness and efficiency with which a product or system can be modified to improve it, correct it or adapt it to changes in environment, and in requirements.’ A highly maintainable software solution/product must possess the following qualities:

• Modularity - The product is composed of discrete components such that a change to one component has minimal impact on other components.
• Reusability - The product makes use of assets that can be re-used in building other assets or in other systems.
• Analyzability - The impact of an intended change on the product, diagnosis of deficiencies, causes of failures or identification of the components that need to be changed can be analyzed effectively and efficiently.
• Modifiability - The product can be effectively and efficiently modified without introducing defects or degrading existing product quality.
• Testability - The test criteria can be established effectively and efficiently, and the product can be tested to determine whether those criteria have been met.

Ensuring that your software solution incorporates these qualities guarantees that you will be able to:

• Add new features or fix issues without introducing new bugs.
• Improve performance or other attributes.
• Adapt to a changed environment efficiently.
• On-board new developers quickly.

The need for producing maintainable software solutions does not need justification. In a perfect world, where all the feature requirements are laid out clearly and the development team gets all the time that they need to deliver, producing a maintainable solution would be a desirable challenge. But, in reality, developers are usually delivering under a time crunch. The short iterations of the SCRUM methodology and the need to produce ‘visible’ features for the business team and customer can demand a lot of tenacity on the part of developers. Otherwise it becomes difficult to hold on to maintainability when delivering a software solution.

 Take a simple example of displaying a list of values. How could we implement this list? The easiest option would be to hard code the list based on the current requirements.  But such a solution requires a change to the code if we need to add a new value, new unit tests at a minimum, and a new build and deployment.

 In contrast, consider creating a database table with a set of options. The impact of a change would be much lower and may not require a new build and deployment. However, the upfront cost of developing a new table in the database and managing the asset will take time and resources. And for this reason, it is easy to go with the hardcoding option, despite the drawbacks. This creates a tech debt.

As Jon Bodner, Distinguished Engineer at Capital One says, “There's nothing fundamentally wrong with tech debt. Like all debt, it's just borrowing from your future self to make life better today. The problem comes when you have too much (tech) debt and can't pay yourself back. Development teams are going to incur tech debt. It's simply a matter of understanding that they have to include the payment schedule in future development plans.”

If it falls on the shoulders of the development team to ensure that the tech debt is kept in check, it becomes necessary to develop guidelines which can help them balance the accumulation of tech debt while estimating time to deliver.

In my role as Engineering Manager at Capital One I work to impress the following standards on my teams to ensure that we are delivering maintainable software solutions.

Designing a maintainable solution, calls for a modularized solution with reusable components. Targeting highly reusable components and modularization of every single feature will require expert developers, thereby increasing cost. But, these aspects will be beneficial in the long run due to the decreased cost of maintenance and flexibility to make changes. A good design should strive to balance these aspects against the requirements of the product. While most of these aspects can be handled by the product leveraging a good framework, every developer must still take these aspects into consideration while writing code.

Every module must include meaningful unit, functional, and regression tests.

• Unit tests - Unit Testing may be frowned upon as it adds a lot more time to development, but it is the first line of defense when trying to make changes to code, irrespective of whether you wrote that code earlier or not. Good unit test provides an unassailable way to prove that the code works, and hence eliminates the worry of breaking something when you try to make changes. In his article Improve Java Code Coverage and Quality with Unit Tests and JaCoCo, Jon Bodner talks about how unit tests can be leveraged to ensure that code is properly tested.
• Functional tests - Automating functional tests will help with quick and accurate validation of the requirements. These will also help in validating that any new changes to the software did not break the existing functionality.
• Regression tests - Not every functionality needs to be tested with every deployment, especially when the software is huge and has a large set of features that require testing. In such a case, it is always good to have a set of slower, more comprehensive tests that you run to "smoke test" and validate that a new build works. This will save testing time while verifying that the changes didn’t break existing functionality in a different piece of the application.

All good software systems must have a good logging scheme, and this logging must be done with a purpose. Every log event must be comprehensive containing  meaningful information. In his article Logging Wisdom: How to Log, Emil Stenqvist states that software programs must write log as if it is a journal of its execution: major branching points, processes starting, etc., errors and other unusual events. Logs must contain messages that describe what’s going on, along with the relevant context as key-value pairs. They must include relevant identifiers such as request ID’s, PID’s, user ID’s, etc.

Logs must be written so that they capture the data that is meaningful for the purpose that it is written for. Logging needs must be identified at the time of feature grooming. Few examples of motivations  for logging are to:

• Capture Business Metrics
• Capture User navigation for triaging an issue
• Capture application events for monitoring application performance and health

In my career, I have seen logs like ‘I am here’. That is great, but where is the information about:

• Who you are
• Where exactly you are and
• Why you are there

Without these being available, this log is just a waste of drive space.


The motivation for logging must drive the details that go into logging. For example, if a log event is written when a user sees an error message, it is important to log the user ID, date, and time of the error, as well as the details of system state or data that resulted in the user seeing that error message. It is important to be careful not to store sensitive information in logs or encrypt them if they are needed.

Error messages displayed to the user must help the user understand why they received the error and what steps they can take to resolve the error. In addition, if these errors are caused by system problems, then all relevant data to understand what caused the error must be logged in the application logs. This will help the support teams to quickly identify why the error happened.

Effort must be made to make messages unique so when the user has questions about it, support teams can quickly provide an answer rather than trying to identify which one of the many reasons could have caused the issue.

Ensuring that infrastructure and application monitoring is designed and implemented at the time of application development is a key criterion to making good maintainable software. While infrastructure monitoring can be handled by monitoring aspects like memory, CPU utilization, number of instances, etc., application monitoring requires deeper understanding of the application domain and the instrumentation of the application. In his blogpost Logging V. Instrumentation, Peter Bourgon talks about when to use logging versus when to use instrumentation to ultimately increase the system observability.

Application monitoring must focus on proactively identifying the degradation in availability and performance of the application. A downtick or uptick of the monitored parameters must be watched. A highly scalable application may endure a DDOS attack and the attack might go unnoticed in the infrastructure monitoring. Whereas, if the application is being monitored for user traffic, then the DDOS attack will be rendered as a spike in users hitting the platform, which can be leveraged to alert support teams about a possible abnormality.

In addition to tracking the real user transactions, a good monitoring set up will also try to proactively track availability and performance using synthetic transactions. Synthetic transactions are performed by monitoring tools and are useful in understanding the degradation of the systems even when there are no users using the system.

To summarize, a sound application and infrastructure monitoring design will contain:

• Monitoring tools to read logs and perform real time user monitoring, as well as proactive synthetic monitoring of transactions.
• Dashboards that show:
  • Trend & thresholds for application performance & availability for the system, and individual components.
  • Frequency and count of different error messages. Example: errors displayed to the user or errors when APIs are being tried multiple times, indicating degradation in the API performance.
• Alerts when the thresholds are breached.

Good documentation is required for developing and maintaining a good maintainable software solution. Some of the documentation that will be good to have right from the start are:

• Coding standards - Include standards for naming conventions, styles, requirements for unit tests, functional tests, level of comments in code, etc. Programming languages are starting to adopt standards for this, others have standards defined by large companies. So, adopt these standards rather than re-inventing the wheel.
• Contribution guidelines - This becomes very important if the project code is updated by more than one team, or even within the team, as not all the contributors are likely to be committers to the project.
• Project governance - Include how many reviews will be needed to commit the code, version control, who will be committers vs contributors, decisions when there is conflict on design, etc.
• Feature descriptions - This is one of the key details that gets buried in design drawings and JIRA stories, making it difficult to know what exactly a module is intended for, especially when the developers and product managers change. It is  beneficial to maintain high level documentation on the features, making note of some of the key decisions and discussions that went into their design and development. This includes:
  • Infrastructure and Architecture Document
  • Deployment procedure
  • Support FAQs

Like anything in this world, even the best of intentions to tackle maintainability does not guarantee that you won’t have tech debt, which you will have to tackle retrospectively. It can be a tough sell to ask the product team to support a refactor work, that doesn’t render any new functionality but is aimed at solely making a developer’s life better and making code changes in future less buggy.

 A high incident ticket count, the need to change error messaging for the customer’s benefit, or a need to enhance functionality are all times when a development team can easily tackle their tech debt items to increase maintainability. Aside from these opportunities, I suggest that every development team must strive to ask for a 10-20% allocation to work on tech debt items.

Irrespective of whether a project is big or small, setting up good development practices with a focus on enhancing the maintainability of a software solution helps the customers, business, development, and support teams by reducing the time and effort required for supporting and enhancing features. I hope that this article helps you focus on maintainability best practices  which are often overlooked during software development.