An Approach to Reducing Technical Debt — Tackle or Pay as You Go
Highway snow plowing is one of the coolest orchestrations of the Tackle-as You-Go phenomenon I have observed. The synchronized and efficient nature of the plowing process at high speeds over long distances is a joy to watch. However, municipalities certainly do not wait for the end of the winter season or for the weather to get a little warmer to get rid of snow on the roads. To put it mildly, not attending to snow on the streets during the winter and clearing it only at the end of the season would introduce significant inefficiencies in the transportation system.
Snow clearing during the winter season is also a real-world example of the famous “Clean-As-You-Go” philosophy. Professionals across industries such as hospitality, construction, medicine (surgery), finance, and automotive are constantly practicing the Clean-As-You-Go philosophy. The general premise is a continuous elimination of waste resulting from activities in a process. In your kitchen, you are cleaning as you prepare a meal; when renovating your home or installing that new bathroom in your basement, you are cleaning after putting up the drywall in readiness for painting; a surgeon is cleaning out bodily debris during surgery; we pay (or should be paying), financial debt as move along the journey of life and not till we get to the end of the road; the banks will not function and doing so will also not be the best legacy to leave behind! The Kanban system pioneered by Toyota is all about improving productivity and reducing waste within an industrial process.
Summing it all up, the common goal is to avoid accumulating waste during a process and only ridding of this waste at the end. Why?
- It is a lot more expensive to get rid of all waste in the end.
- Working amidst waste increases the risk of injury, significantly lowering productivity and the organizations’ bottom line.
On the flip side, operating on systems that produce waste without a well-conceived strategy on waste elimination, such as the case with nuclear power plants, incurs a significant cost burden on future generations. For example, a December 2018 report from the Department of Energy estimated $493.96 billion as the total cost to eliminate nuclear waste and also estimates this cleanup to last for 50 years!
Operating on systems that produce waste without a well conceived formula on waste reduction or elimination incurs a significant cost burden on future generations.
How does all of this relate to Software Engineering, you may be asking?
In Software Engineering, there is the well-known concept of technical debt. Technical debt, in simple terms, is an accumulation of work that we need to do. We need to do this work to overcome architecture limitations in today's solution resulting from prior decisions made. Unattended technical debt adversely compromises several software quality factors, including lead time, reliability, etc.;- technical debt adversely compromises engineering and operational excellence. Technical debt results in lost revenue. Here is a homorus example of a system that has accumulated a sufficient amount of technical debt tracked in person-hours wasted to fix a bug in a particular area of code.
Accumulating technical debt is an ongoing industry challenge with many approaches on its reduction. Garner’s research indicates that companies with a strategy to reduce technical debt will ship 50% faster, resulting in increased revenue.
Companies with a strategy to reduce technical debt will ship 50% faster resulting in increased revenue.
All software systems include some form of technical debt. It is simply the nature of our craft coupled with constantly shifting business priorities and processes primarily driven by software systems. Architectural or design decisions today could quickly render a software solution obsolete without extra care or design considerations. As such, we should strongly advocate design-first and collaborative design approaches to ensure the architectural underpinnings of software systems are rock solid and forward-looking — we should not rush to write code even though this is the most joyous part of the craft for many software engineers. Design-first or collaborative design approaches is undoubtedly one easy and cost-effective way to prevent the accumulation of more technical debt.
Strongly advocate design-first or collaborative design approaches to ensure the architectural underpinnings of our software systems are rock solid and forward looking — do not rush to write code.
Now that we have established that all software systems contain and continue to accrue technical debt, what are some of the things we can do to ensure an upper bound?
We certainly do not want technical debt to continue increasing exponentially as this is one very plausible outcome of an overly complex and feature-packed system. With such unmaintainable systems, we compromise operational excellence, safety and our bottom line. Unfortunately, there is no easy cure for such systems and we usually end up paying a lump-sum payment with accrued interest via a re-write or through a bad accident. For example, the Boeing 737 Max accident that killed 346 passengers resulting in a grounding of the aircraft from 2018–2019, can be attributed to technical debt (known design flaw) that the Boeing engineers did not address.
The method I propose borrows from the philosophy and practice of ‘Cleaning as You Go.’ It emphasizes a continuous awareness of technical debt, a continued conversation about such debt with all stakeholders, including PM (not just engineering) and a continued discussion and negotiation of tackling the debt as we go. The proposed strategy in line items as follows:
- Track debt by immediately creating JIRA tickets for each new architectural compromise discovered. We should then convert these JIRA stories into engineering-driven epics with a well-defined scope. Be deliberate in the story definition and quantify a business justification for why we need to address this debt in favour of a specific product feature. For example, a technical debt item in one company involved upgrading our software stack from .NET 2.0 to .NET 4.0, mainly to take advantage of security releases made available in .NET 3.5. It started as a story that converted to an epic and reached completion several months later. What is essential here is to track the debt as part of your product backlog.
- Avoid premature optimizations and instead embrace a democratic design-first and collaborative approach fostered by an RFC process. Through a well-thought-out workflow, an engineer should take on a problem, understand the problem, define its scope, formulate a solution, evangelize the solution to key stakeholders before committing a line of code. While we recognize one can not have all the answers and corner-cases covered in the initial design discussions, there is a sure guarantee on forward-looking design when engineers go down this route instead of jumping straight into code.
- Whenever possible, pivot to addressing new architectural limitations discovered while working on a feature. Avoid duck tape solutions as much as possible, mainly when you have found a significant flaw in your design. Such design flaws could arise from sudden changes in business requirements or an engineering discovery after the initial iteration. Here is a real-world example: we recently created a Golang Authorization SDK to centralize API authorization logic spread across our microservices. This SDK is required to be modular and extensible (think the O in SOLID). It would support retrieving the authorization token using different strategies based on different portions of the HTTP request, including but not limited to a secured cookie or the HTTP Authorization header. During a collaborative or peer review process where the engineer presented the initial pull request, the team identified an architecture limitation that would compromise backward compatibility and prevent easy extension of the SDK. We surely could push this iteration through. But we also could not afford to break backwards compatibility and compromise the extensibility of the SDK. So the team worked hard and diligently to re-engineer the library, releasing a solution that addressed the architectural limitation. The engineer leading this effort was very content and confident with the resulting quality of the initial release of the SDK. Zero debt incurred!
- Actively follow the Boy Scouts Rule. Engineers should always be encouraged to leave the codebase cleaner than they met it. Leaving the codebase cleaner could include simple changes such as improving variable and method names, adding unit tests where easily applicable, documenting obscure system workflows discovered from reverse engineering activities while quashing difficult bugs, and other subtle initiatives to improve code maintenance.
- Embrace change, take risks and over-communicate. Discovering new debt and taking on this debt mid-sprint is risky. Please do not shy away from these risks but instead carefully quantify them. If we do (1) above well, PM would have sufficient visibility and understanding to communicate a potential scope creep to executives. For example, it was scope creep and risk when we decided to build an OAuth 2.0 based Authorization Microservice, which would serve as the basis for current and future granular access control. But taking on this risk was essential not only to properly achieve the simple use-case of hiding a UI settings panel based on a particular user privilege but also to position the solution to meet future authorization needs. The engineering-driven epic was not an immediate PM feature. However, the team did a good job communicating and educating PM and external stakeholders on why this scope creep was justified. This service, as designed, is meeting new access control solution requirements with little lead time. Again zero debt incurred!
All software systems have some amount of technical debt. All teams possess a certain risk tolerance. But organizations need to avoid kicking this debt down the road as it accumulates interest and becomes even more expensive to pay off while adding additional features. Future generations usually pay for this price with slower lead times and further compromises. Instead, formulate a strategy that keeps your technical debt front and center while slowly chipping at it to avoid exponential growth as it affects the bottom line.
Happy Technical Debt Tackling.
