Summary of Software Development in the 21st Century: an excellent presentation (again!) by Martin Fowler

Martin Fowler has established himself as a reference in the world of software development. He has written excellent books, defined useful concepts, published an inspirational blog, etc. Last week, I came across something that was truly an inspiration to me and will probably be for most of you as well: a three-part presentation given by Fowler himself, at the 2014 XConf. The presentation is entitled Software Development in the 21st century. I invite you to view it, but in case you do not have the time, here’s a summary of the content along with some of my notes.

Part 1: Agile Essence and Fluency

The origins of Agile or Why Agile?

In this presentation, Fowler takes the time to provide some context for the creation of the agile movement. He first takes us back to the pre-agile, waterfall world, where projects were delivered late, with unhappy users, buggy software, unhappy developers, etc. The answer then was to

take control of the plan using plan-driven processes such as CMMI, Ratio Unified Process (RUP), etc.

The measure of success then was did the development go according to the defined plan. If so, it was considered a success. After all, it was completed on time and on budget.

But was it the correct yardstick to measure success? Is it possible that following a plan end up with a software product not meeting the customers’ needs? It is one thing to be on time and on budget, but it is something entirely different to deliver the best possible value to customers.

Then along came Agile

To address this problem and focus more on delivering true value to end users, the agile movement was launched with the Agile Manifesto.

It shifted focus from a process-first to a people-first approach and from predictive planning, (which is doable when you are well aware of all your requirements – very rare in software projects!) to adaptive planning, when you continuously adjust your requirements based on feedback received from users.

The measure of success shifted from how the plan was executed to how happy its users were with software that delivered true business value.

Truly Agile?

Agile is not always properly executed. Often, people do not realize this shift between predictive planning and adaptive planning. The successful execution of a plan depends on the stability of its requirements. Therefore, some less agile groups started to try to stop the development of requirements instead of adapting to them. An example of such behavior is when your organization puts in place Configuration Control Boards (CCBs). Such boards actually reject new requirements in order to stabilize the old ones. Not exactly agile… especially if the new ones are requirements from your end users. As Mary Poppendieck, author of the book Lean Software Development: An Agile Toolkit, said:

A late change in requirements is a competitive advantage

Therefore, if you reject those late changes, you are in fact rejecting this competitive advantage; another, more agile business, might use this to its advantage against you!

… and people?

I have seen companies try to force agile on its people, trying to standardize how story points are calculated so they can compare teams and see which one is better. True agility accepts that the teams are responsible for finding the best possible process to deliver maximum value to customers. This is why retrospectives are so important in agile. A team can even decide that SCRUM is not the best process in a certain context. Accepting this fact is truly agile.

The Agile fluency model

Similar to a maturity model, the fluency model includes the following levels to assess how agile a team or an organization is:

• Level 0: you are not agile yet
  • Your cultural shifts from process-first to people-first and from predictive planning to adaptive planning are not started yet.
  • About 15% of teams are at this level
• Level 1: Adoption of Agile Management Practices
  • You have started to shift, and you use tools such as Kanban boards, product and sprint backlogs, iteration development, retrospective meetings, etc.
  • Benefits: increased visibility on what is truly going on in the software product development
  • Metric: Progress reports on business value
  • Time to reach this level: about 2-6 months
  • About 45% of teams are at this level
• Level 2: Technical Practices added to Management Practices
  • You have started to add technical practices such as Test-Driven-Development (TDD), Continuous Integration (CI), Continuous Delivery and/or Continuous Deployment (CD), refactoring practices, and most Extreme Programming practices
  • Benefits: Fewer defects in software, higher productivity
  • Metric: the pace at which the team ships to its customers/market
  • Time to reach this level: 3-24 months; a bigger investment
  • About 35% of the teams are at this level
• Level 3: Organization Optimizations
  • Work across teams is better, teams can go as far as to reorganize themselves.
  • Benefits: better product development decisions, even across multiple teams
  • Time to reach this level: 1-5 years
  • About 5% of the teams are at this level
• Level 4: Theoretical Level
  • Teams can report on how they impact the organizational goals.

Levels 0-2 can be implemented at the team level. Levels 2-4 can be implemented at the overall organization level.

The key takeaway from this model: Agile adoption takes time; it is a step by step approach.

Part 2: Continuous Delivery

A deployment pipeline is the series of steps from development environments steps to production environments steps. For example:

• Step 1: Can the newly developed software increment be built on a machine other that the developer’s?
• Step 2: are the low-level unit tests successful?
• Step 3: are the higher-level integrated/functional tests successful?
• etc. up to the actual deployment of a developed software increment.

This could be a deployment pipeline.

Continuous Delivery or Continuous Deployment?

Fowler took the time to distinguish between both CDs. Continuous Delivery means that you can deploy to production environments while Continuous Deployment does deploy automatically to production environments. I also strongly agree with him that deploying software should be a business decision and not a technical decision. Therefore, I would tend to prefer and recommend Continuous Delivery over Continuous Deployment.

What are the ingredients to do Continuous D?

Here is the list of elements considered by Fowler as required to implement any sort of deployment pipeline:

• Automation: of everything from build, to tests, to deployment in various environments (staging, demo, production, etc.), involving the automatic provisioning of required machines.
• Solid Configuration Management: source code, database schemas, scripts, etc. This is essential to rebuild and redeploy any past versions of the software.
• DevOps culture: Development teams and operation teams must work together to make this work.
• True Continuous Integration: i.e. all developers integrate their changes with all others on a daily basis in the main branch, all integrations are tested, and fixes to failed integrations occur within 20 minutes (i.e. nobody on the team has a more important job than fixing this).

Based on my own experience, I would think that True Continuous Integration is the most difficult one to achieve, followed by Solid Configuration Management. Without a doubt, until you have those ingredients in place, continuous delivery is very difficult to achieve.

Benefits of Continuous Delivery

Here’s Fowler list:

• Decreases risks by deploying smaller changes, more often
• Gives a sense of real progress (this works not only at a developer’s desk, but also potentially in a production environment)
• Provides an excellent opportunity for user feedback

I have never seen a team or a project manager not like the possibility of automatically deploying any single change that a team develops. It is certainly worth the effort.

Part 3: Microservices

The last part concerned the Microservices concept. Essentially, one can see microservices as an answer to the Service Oriented Architecture (SOA) and Enterprise Services Bus (ESB) concepts. In fact, microservices can be seen as a subset of SOA. The thing is that SOA/ESB brought complexity to middleware and that this complexity made Continuous Integration and Continuous Delivery complex.

What are microservices?

The following are characteristics of microservices. Some are shared with SOA, but others refine SOA even more. Microservices:

• Help you achieve componentization via services (replaceable and upgradable)
• Are organized around business capabilities
• Are at product level, not project level
• Use smart end-points and dumb pipes
• Offer decentralized governance
• Enforce decentralized data management (you never share the data directly, but via the services that wrap the data)
• Require infrastructure automation
• Must be designed for failure (because they happen as always in distributed systems)
• Must use evolutionary design principles

When to use them and their Prerequisites

Since distribution and asynchronicity increase complexity, you should use microservices only when you need them. When in doubt, do not use them. If you need them, ensure you have the following prerequisites:

• Rapid, automatic provisioning
• Basic infrastructure monitoring
• Rapid, automatic application deployment
• DevOps culture

Conclusion

First, material from Fowler is almost always good. He is not dogmatic and has a pragmatic approach to today’s software development issues. I personally never consider it a waste of time to read or listen to the material he produces. Second, once you decide that you want to be agile, you should not forget about the technical aspects of the process. Too often, only the management practices are explored or considered. The second and third parts of Fowler conference are perfect examples of technical aspects that can make a team truly agile.