Previous post on DevOps Basics

What is Continuous Integration?

At the basis, Continuous Integration, or CI, is the concept of continuously integrating team members’ code into the main branch of your code repository. The continuous aspect refers to the idea to do this with small increments of code, continuously, every time new code is committed to the code repository. Part of this integration, and crucial part I would add, is the idea to validate those increments with regards to multiple aspects: build validity, unit tests pass-fail, coverage, security vulnerabilities code scans, etc. When properly done, this integration process takes just a few minutes so that it is easy, and painless, for developers to submit code increments and get a fast feedback on its validity before continuing their development efforts.

Why CI?

Let’s take an example of a team of 10 developers. They all work on different features/user stories of the same product. They all start the product improvement project today with 10 new features to be added to the code base, one per developer. The first thing they typically do, to isolate their development from other team members, is to create a feature development branch. This results into 1 main branch (sometimes called “master” or “trunk”) and 10 feature development branches. Let’s say for the sake of example, that the features will all take a 2-weeks sprint to complete. Since developers are trying to avoid big bang integration efforts at the end of the sprint, they all rebase their development branches daily by pulling the main line additions into their feature branches. But since all developers are not completely finished with coding their new features, they keep them isolated in their development branches and they don’t commit or push this unfinished code into the shared main line. So essentially, when they rebase their branches daily during the sprint, they obtain nothing from the other developers (who are not done either). This will result in a lot of large commits near the end of the sprint, resulting in the big bang integration we wanted to avoid in the first place, with potentially conflicting development efforts and waste. The key to avoid this are 2 important practices:

1. The ability to commit to the main line unfinished code using Feature Toggles and
2. Adhesion to Test Driven Development (TDD) principles.

But how can I commit unfinished code into the main line?

To permit often and early integration of everyone’s code, you need to commit to the main line all your code increments, daily, even unfinished. When you do this, you give a preview to all other developers of what is coming their way (and you get theirs as well as a bonus!); if there are conflicts between some of the features being developed, you detect them earlier and you can adjust to them and reduce the produced waste to a maximum of one day of work. The best way to “hide” those unfinished features, is to use the concept of Feature Toggles. A feature toggle, sometimes called Feature Flags, is a way to hide a given feature, sometimes because it is not complete yet. When the management of the toggle needs to be performed at build time, we talk about a static toggle; when management of the toggle can be performed at run-time, we talk about a dynamic toggle. Both types of toggles have their value and use cases. This is a very important technique to eventually reach Continuous Delivery/Deployment stages. They offer you mechanisms to be able to do things such as A/B testing, and Canary roll-outs:

• A/B testing permits to try a variant of a feature on a portion of your customers while trying another variant on another portion and then decide using factual metrics which variant performs best and continue with it from then while
• Canary rolls-outs permits you to use the previous version of your application on most of your users and use the new version of your application on a selected small portion of your users and increase gradually traffic towards the new version if all goes well. Eventually, only the new version will exist. As an example, you could decide to direct company employees to the new version first and decide to extend to external customers only when working properly internally.

See my next post on this, but if you cannot wait, I suggest you start with this excellent blog post from Martin Fowler explaining in great details all the different types of toggles and how to implement them: https://martinfowler.com/articles/feature-toggles.html.

Test Driven Development

The next most important technique to prevent/minimize impacts of code conflicts in CI is Test Driven Development (TDD). This technique is about developing test code before the feature production code. Here are the TDD process steps:

1. Decide on the smallest possible feature increment you could develop to add business value
2. Develop a new test for this increment only
   • Note that at this point, the new test will fail since no associated production code was developed yet for it to pass
3. Develop just enough production code to have the new test pass
4. Refactor both test and production code
5. Rinse and repeat steps 1-4 until feature complete state is attained and then start again with another feature

Here’s how it looks like graphically (image created by Xavier Pigeon under CC license):

By adhering to the TDD process, you ensure that test coverage, i.e. the quantity of production code that is covered by tests, will always be excellent, providing you and your team with the required safety net to go faster and faster without suffering from it. That essentially means that as your team members will commit their new code into the shared branch, all the previously created tests, for all the currently existing production code base up to now, will be used to validate that the new increment of business value does not interfere with any of the previously built business value. And again, if something fails, because of your daily CI process, it will be the result of a 1 day development work effort, maximum.

In my experience, failure to adopt good testing practices was always front and center of development teams having velocity issues:

1. If you cannot go fast without quality
2. and if you cannot develop quality without good testing
3. then, you cannot go fast without good testing

Pretty simple logic 😉 However, even if simple, it takes a large amount of discipline and rigor to adopt good testing practices – but I promise you, they always pay off!

Wait, but why have you not talked about CI servers (like Jenkins) yet?

This is probably the biggest pitfall with CI; people put in place a CI server, such as Jenkins, and then think they are doing CI. CI is a lot more than a CI server, as described above. Obviously, a CI Server is a good tool to adhere to CI principles, but this will not be sufficient. Frequent commits to the main line from all developers, as well as excellent unit tests coverage, are pre-requisite to efficient CI. As soon as you have those aspects integrated in your team mindset, then, you can go with designing a CI pipeline for your product. Before selecting a CI server to implement that pipeline, let’s review what important parts of the CI pipeline should be present:

CI part 1: Continuous Build

When adhering to CI principles, the CI pipeline activities are triggered on code commits in your code repository. When well done, code increments proposed by team members to be integrated in the main line, are evaluated before their actual merge into it. The merge will be done only if all the following steps performed with success and all the established quality gates are passed.

The Continuous Build part is required for code in programming languages requiring compilation, as opposed to interpreted languages. Once the build is done, produced versioned artifacts should be pushed to an artifacts repository (Artifactory, Sonatype Nexus are good options) for consumption by next steps of your CI-CD pipeline. The main concept here is to adopt the BuildOnce-ReuseOften concept:

1. Your application code is built once on code commit
2. The CI environment tests the built artifacts (see CI part 2 below)
3. The very same tested artifacts are promoted to the next stage of the pipeline to test some other aspects, like performance, and
4. Eventually, if all gates are passed, the exact same built artifacts are promoted into your production environment.

The corresponding anti-pattern would be to build your application, to test it in your CI environment, and then to rebuild it in pre-production to run performance tests and to rebuild it once again to deploy it in Production. When doing this, you possibly release in production something different from what you actually tested in CI and Pre-prod environments. It also has the disadvantage to consume more ressources and to take more time to go through the complete pipeline.

CI part 2: Continuous Testing

The unit tests produced using TDD we referred to just above play their role in this part. All the unit tests are ran, as part of the CI build, to validate that the code increment proposed did not break any existing part of the code base. This is the safety net we discussed about. When the testing coverage is too low, you cannot count on this to intercept issues that enter the code base, but with a great coverage (about 70%+), there is a high chance that undesired side effects in the code logic will be caught at this step. As soon as a test fail, entering what is called the pipeline red state, it should act as a lean manufacturing “stop the line” concept and it should be everyone’s responsibility to bring back the build into a green state.

CI Part 3: Continuous Inspections

When the build is successful, and all the unit tests pass, the next step is the Continuous Inspections. This is where the code is scanned with static code analysers. You use various kinds of scanners, for various kind of information that static code analysers can reveal. For example:

• Did the team adhered to our defined team coding standard, to the team’s rules?
• Did we insert security vulnerabilities in our code increments?

It is a good practice to publish the results of the tests and the inspections into a tool such as SonarQube, which could be used to manage your code technical debt. It supports many integrations with lots of tools for various technical stacks.

CI Tools: CI servers

In order to orchestrate your CI pipeline activities, you will need a CI Server. There are multiple CI servers/services out there; here’s a list maintained on wikipedia

Here’s my shorter list that you can look into, along with the reasons I think they are worth looking into:

• Jenkins: probably the most popular one (Open Source Code Repo), huge community, However, 1st CI generation tool, not designed for Continuous Delivery or pipeline as code at first. Blue Ocean interface is a well deserved improvement and worth looking into. Jenkins-X is also an interesting development being done to send Jenkins into the CI-CD modern era.
• GO-CD: Designed by CI-CD creators, open-sourced a little too late (https://github.com/gocd/gocd); Jenkins took the space first. This is probably the best 1st generation CI tool I know. Good support for pipeline as code, pipelines were designed up front into the solution as first class citizens, not an afterthought.
• ConcourseCI: Second generation CI server, Docker based, Full pipeline as code, Open Source as well
• Cloud-based ones: TravisCI (code repo), CircleCI (code repo),: Cloud based could be a good thing if you have simple use cases and you do not want to host your servers
• There is also AWS and its CodeCommit, CodeBuild, CodePipeline solution, especially interesting if you are already completely integrated in the AWS ecosystem.

But no matter which one you select, the important thing is that they support you CI pipeline needs.

One advice here: select one that support the concept of pipeline as code so that you can easily reconstruct your CI-CD pipeline from your code repository. Too often I have seen teams lose everything when their Jenkins server died and all they got was an old backup (sometimes never tested… and not working!) or even nothing to restart their pipeline. They had their production code in their repo, but not their pipeline definition as code in their repo. This is possible to be done in Jenkins, GO-CD, ConcourseCI, and most serious CI servers out there.

Overview of seen concepts

Here’s a nice overview picture of CI process taken from http://www.pepgotesting.com/continuous-integration/ :

1. Developers commit their code increments into Source Control Server
2. CI server fetches those changes from Source Control Server and builds, tests and inspects the proposed changes
3. The proposed increment is either respecting the quality gates (build passes) or not (build fails)
4. Feedback is provided back to all stakeholders
5. Rinse and Repeat!

Typical CI challenges

Based on experience, here are typical challenges that people designing and putting in place CI pipelines will encounter:

• Not enough tests to stop the line; code progresses, even when not satisfying proper quality gates. You go fast, but with not enough quality checks.
• Build takes too long (more than 5-10 minutes)
• Tests take too long (more than 5-10 minutes)
  • Unit tests should run fast, they should run isolated from their dependencies using test doubles techniques (stubs, mocks, etc.)
• Team members are not properly notified of stop the lines events (missing open areas dashboards and/or good notification systems, etc.)
• As discussed just above, not using CI pipeline as code, not having backups of your CI configuration if not adhering to pipeline as code, and not testing your backups, is an anti-pattern to avoid

References on CI

If you want to progress on the CI journey, I have put many references within the above text, but here are some more items to dig into:

• Continuous Integration: Improving Software Quality and Reducing Risk  : The official CI book (a Martin Fowler Signature Book)
• Martin Fowler’s post on CI; Essentially recommends the following practices to adopt CI:
  • Automate Your Builds
  • Make your Builds self testing (TDD, etc.)
  • Everyone commits to mainline every day
  • Every commit triggers a build of the shared mainline
  • Fix broken build immediately
  • Keep the build fast
  • Test in a clone of Production Environment
  • Make it easy for everyone to get the latest build artifacts (using an artifact repository)
  • Everyone can see what is happening (using build radiators, etc.)
  • Automate Deployment
• CI certification test developed by Martin Fowler and Jez Humble: 
• Feature Branching from Martin Fowler
• GIT Flow is also an interested branching model to adopt, especially if you cannot use feature toggles easily in your current stack

I consider all those references as being essential to adopt CI in practice. Consider it an investment of your time if you have never read them yet; you won’t be disappointed 😉

Conclusion

Continuous Integration is probably the most important set of practices related to adopting DevOps. However, going fast and automating things does not make much sense if automated testing is insufficient or absent and if true integration between developers does not truly take place. Developing using TDD and CI certification test, are very important steps in the right direction to adopt CI and eventually embrace a full DevOps mindset. Once mastered, you can reach to my next post of the series: Continuous Delivery!