Failing builds when developers misbehave (code coverage)

Developers. They’re kinda like little children sometimes, aren’t they? You have to keep your eye open on ’em…

Joking aside though, code coverage is an important aspect in software development.

code coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high code coverage, measured as a percentage, has had more of its source code executed during testing which suggests it has a lower chance of containing undetected software bugs compared to a program with low code coverage.

Low code coverage typically may occur when new code was implemented but tests were not added for the new code just yet.

One way to potentially solve it is developing the TDD way, but not all teams like that. So another solution is using tooling in the pipeline to fail a build when code coverage drops. One such tool is the open source SonarQube.

The prerequisites are to:

  • Install the SonarQube server on a host machine
  • Install the appropriate Sonar scanner on the same host machine
  • Install the “Quality Gates” Jenkins plug-in
  • Install the “SonarQube Scanner for Jenkins” plug-in
  • Install the SonarJS plug-in in SonarQube’s Update Center

As for failing the build, I have set up a Quality Gate in the SonarQube dashboard to error when the coverage metric is below 80%.

My implementation is as follows:

stage ("SonarQube analysis") {
   steps {   
      script {
         STAGE_NAME = "SonarQube analysis"

         withSonarQubeEnv('SonarQube') {
            sh "../../../sonar-scanner-2.9.0.670/bin/sonar-scanner"   
         }

         // Check if coverage threshold is met, otherwise fail the job
         def qualitygate = waitForQualityGate()
         if (qualitygate.status != "OK") {
            error "Pipeline aborted due to quality gate coverage failure: ${qualitygate.status}"
         }
      }
   }
}

withSonarQubeEnv('SonarQube') refers to my SonarQube configuration in Jenkins > Manage Jenkins > Configure Jenkins > SonarQube servers where Name is set to SonarQube.

Now, don’t start with 80% coverage right away, let it simmer… start with 40% or 50% and increase it by 10% each passing week. This way developers will be ‘forced’ to add tests to their code as they progress with their work, but not be totally pressured about it and will have enough time to cover everything with time.