This blog post presents the relationship between Lean Engineering and DevOps.
What is the essence of Lean Engineering?
You can learn about the Lean movement by reading about its pioneers like Deming and GE’s Jack Welch. This movement was centered on the manufacturing industry and was attempting to find solutions to its problems. This is a culture of Continuous Improvement, focused on quality, that can essentially be applied to other domains, not just hardware manufacturing. Concepts such as optimizing the value stream, reducing the number of handoffs between process steps, reducing the work batch size, amplifying feedback loops, pursuing perfection, etc. can easily be applied to other domains, including Software Engineering.
What is the link with Software Engineering?
When you think about Software Engineering, you can conclude that the Value Stream concept defined by Lean is again applicable to Software development. Let’s see the history of Software Development processes and the links that we can establish with Lean.
Waterfall development
In the early days of Software Development, the waterfall process was king (I guess that for a little while, there was no process at all, but I am going to skip that part for now!). This resulted in many handoffs between the various stages of the development; from Requirements definition to Architecture and Design to Development to Testing, to Deployment. The whole process could easily take months/years on large projects. With no real up-front risk management, and more importantly, no feedback loop until the very end, it is no surprise that most development projects were running late, over budget. But even worst, they were failures, because not meeting the customers needs and with low quality built in. Pretty similar to the manufacturing production lines before the Lean movement.
Iterative development
Then, the software development industry’s first reaction was to reduce the batch size by doing the same waterfall steps essentially, but with smaller iterations, a small as 3-4 weeks instead of years. A little bit of requirements, the most risky ones, a little bit of Design, some implementation, some tests, etc. to produce a first iteration of the product. Processes such as Rational Unified Process (RUP) were then major improvements over Waterfall. The issue was there was still no involvement of the actual customers until the very end of all iterations. It was still very possible to deliver the wrong thing.
Agile
Then Agile and variants came in, with Scrum, etc. The intent was to involve the actual customers, or their closest representatives, into every iterations (or sprints). To help ask the right thing at the beginning of the sprint and to validate the results at the end of each sprint. The concepts of being Business Value driven appeared, increased collaboration, better feedback loops were added. This was all good. It was ensuring that we would do the right thing. Not that we would do the thing right however… this was kind of lacking or forgotten (even if Extreme Programming (XP) was focused on practices, this was not widely used). The practices and craftsmanship to produce good quality software was not at the core of the Agile movement. Also kind of missing in Agile transformation in the enterprises: Agile focused mostly on Dev, excluding Ops people from the whole process. Agile teams developed software to be released and operated in production by non Agile teams.
DevOps
Then the DevOps movement came in to address one of the biggest handoff still present after the Agile revolution: the handoff between Dev teams and Ops teams. With the Ops people now in the loop, other concepts, complementing Agile ones, mostly centered on practices, pushed for no handoffs at all and automation of everything between code commits to deployments in production, amplification of feedback loops, complementing pretty well the Agile values and principles. This was creating a culture for learning Organizations and not just isolated Agile development teams.
Very interesting books on DevOps
If you have some free time, and if you are working in the field of Software Development – no matter what your role is – I invite you to read The Phoenix Project. You will discover via this novel, where you will most likely recognize yourself, the applicable concepts of Lean to Software Engineering. Here’s an overview:
The First Way: Deliver Fast(er)!
The first way is about optimizing the time it takes between the start of the work and its delivery: the Lean lead time. In the DevOps field, this essentially means automate everything. Use Infrastructure as Code to provision the environments automatically, use TDD, BDD to test things automatically, use Continuous Integration (CI) and Continuous Delivery (CD) to integrate and release products automatically in small increments. Essentially about Do the Thing Right. This is huge in reducing the handoffs.
The Second Way: Amplify feedback loop!
The second way is about creating and amplifying the feedback loop. It serves little purpose to follow the first way, to deliver things that your customers don’t want or don’t need! By having the right feedback loop and by ensuring that the return is transparent and visible to all, you put in place the required ingredients to develop the right thing for your customers. Essentially about Do the Right Thing. This is huge in reducing waste, i.e. work that you need to redo, because not adapted to your customers’ needs.
The Third Way: Creating Learning Organizations
The third way is about ensuring that all the teams work together toward a common goal, permitting fast experimentation. We do not want to optimize the first and second way at the small team level, we need to do it at the multiple teams or global organization level. Essentially about creating the condition for a learning organization, one where post-mortem of incidents are done, in a transparent fashion, where people look into optimizing the global results and not just the result of their own local team. By continuously experimenting and by doing it faster, your organization can actually beat the competition. This is true Agility!
Conclusion
For many years, I thought that all those analogies between Hardware Engineering and Software Engineering were not so useful. To some extent, I still think that not everything can be compared between those two domains. I however encourage you to inspire yourself from other domains when it makes sense, and the Lean movement certainly provides multiple lessons that the Software Engineering world could follow. The whole DevOps movement is pretty much based on the Lean principles.
Another interesting book, filled with great examples, is The DevOps Handbook, a sort of sequel to The Phoenix Project. If you are currently playing a role in your organization’s journey towards a DevOps culture, no doubt you will find in there a lot of good material to help you. This book is filled with examples and tricks on how to conduct a DevOps transformation in your business. No matter what you role or title is in your organization, the DevOps principles can be applied and bring benefits. Enjoy your DevOps journey!