Tag: jenkins

Dynamic parallel stages in Jenkins

Jenkins Declarative Pipeline v1.2 added support for Parallel Stages, which is a great and easy way to, well, run multiple aspects of a job – in parallel.

In most cases this would suffice and you could author a simple parallel block as described in the Jenkins documentation (See blog post.

But what if you need to generate multiple parallel runs in a job – dynamically?
One way would be as follows.

Lets assume a have a package.json that defines different script executions, such as:

"scripts": {
"test1": "jenkins-mocha --recursive --reporter mocha-multi-reporters --reporter-options configFile=config/mocha-config.json test/test1.js",
"test2": "jenkins-mocha --recursive --reporter mocha-multi-reporters --reporter-options configFile=config/mocha-config.json test/test2.js"
},

In order to run these in parallel you could do this in a stage:

packageJson = readJSON file: ('package.json')
tests = packageJson.scripts
listOfTests = tests.keySet()

for (int i = 0 ; i < listOfTests.size() ; i++) {
test = listOfTests[i]
parallelTests[test] = {
sh "npm run $test"
}
}

parallel parallelTests

This would then generate a similar flow as depicted below (replace “Test on Linux/Windows” with “test1” and “test2”).

If one of the parallel executions will fail, this will then fail the job once all parallel runs have finished.

Testing for node modules vulnerabilities

To make sure the codebase doesn’t get creeped in with various vulnerabilities in its node modules, one way to achieve this is by creating a Jenkins job to check on a daily basis against a continuously updated database of known vulnerabilities.

One such database is provided by the Node Security community and can be easily integrated using the Node Security Platform (nsp) node package.

Create a new Pipeline-type job with the following implementation (just the required parts, you may need to add some more pieces to fit it in):

  • root-of-project-must-have-node-modules-folder – a pre-existing job folder containing a cloned repository with its node modules folder
  • Packages Vulnerability Checker – the name of this job (will be created once the job is run)
#!groovy

pipeline {
   stages {
      stage ("Check for vulnerability") {
         steps {
            script {
               def vulStatus

               // Check dashboard node modules
               sh '''
                  cd ../root-of-project-must-have-node-modules-folder
                  nsp check > "../Packages Vulnerability Checker/test-results.txt"
                  nsp check
               '''

               vulStatus = readFile('test-results.txt').trim()
               if (vulStatus != "(+) No known vulnerabilities found") {
                  slackSend (
                     color: '#F01717',
                     message: "@channel $JOB_NAME: <$BUILD_URL|Build #$BUILD_NUMBER> vulnerabilities found in Dashboard node modules. Check build logs."
                  )
               } 

               // additional folders to check...    
            }
         }
      }
   }
}

Creating a changelog in a pipeline job

In my pipeline, for the master branch, I’ve decided that I’d like to keep track of what each new build will contain – for debug, traceability, auditing purposes and what not…

After the flow of tests > Docker build & push, Kubernetes deployment & verification have all passed, this is the time to generate the changelog, as the last task for the pipeline after everything else has passed successfully.

Here’s how it’s done (partial snippet):

sh '''
   changelog=$(git log `git describe --tags --abbrev=0 HEAD^`..HEAD --oneline --no-merges)
   jq -n --arg tagname "v0.0.$BUILD_NUMBER"   \
      --arg name "Release v0.0.$BUILD_NUMBER" \
      --arg body "$changelog"                 \
      '{"tag_name": $tagname, "target_commitish": "master", "name": $name, "body": $body, "draft": false, "prerelease": false}'  |
   curl -d@- https://github.ibm.com/api/v3/repos/my-org-name/my-repo-name/releases?access_token=$JENKINSBOT_GHE_ACCESS_TOKEN_PSW
'''

As you can see, you will need jq installed for this.

The end result is quite nice:

Checking for vulnerabilities in Docker container images

It’s very likely that you are using a Docker container image for cloud native application. In which case, you’re probably also worry for possible security vulnerabilities in your base image.

As part of my daily work (I work for IBM), I use the IBM Bluemix Container Registry service. This service offers a Vulnerability Advisor which can let you know if there are any identified vulnerabilities in the image and also offer a detailed report. Nice.

I decided to create a Jenkins job that will check daily for such issues.
Here it is.

Note: since this is part of an IBM Bluemix service, use of the Bluemix CLI and Container Registry plug-in is required.

#!groovy

pipeline {
    stages {
        stage ("Check for vulnerability") {
            environment {
                JENKINSBOT = credentials('${JENKINSBOT_USERNAME_PASSWORD}')
            }
            steps {
                script {
                    // Login to Bluemix and the Bluemix Container Registry      
                    sh '''      
                        bx login -a ... -c ... -u $JENKINSBOT_USR -p $JENKINSBOT_PSW        
                        bx target -r ...    
                        bx cr login
                    '''

                    // Check for image vulnerability
                    isVulnerable = sh(script: "bx cr images --format '{{if and (eq .Repository \"registry.ng.bluemix.net/certmgmt_dev/node\") (eq .Tag \"6-alpine\")}}{{eq .Vulnerable \"Vulnerable\"}}{{end}}'", returnStatus: true)

                    if (isVulnerable == 1) {
                        slackSend (
                            channel: "...",
                            color: "#F01717",
                            message: "@iadar *Vulnberability Checker*: base image vulnerability detected! Run the following for a detailed report: ```bx cr va registry-name/my-namespace/node:6-alpine```"
                        )
                    }
                }
            }
        }
   }
}

Verifying a Kubernetes Deployment in a Jenkins pipeline

One of the tests I’ve been missing in my pipeline was a crucial one. Verifying that the deployment finished – successfully. e.g. that the pods were updates with a new Docker container image, but also restarted successfully(!).

It’s pretty simple I suppose, but here is my interpretation.
The following should sit inside a step:

// Verify deployment
sh '''
   export KUBECONFIG=/.../kube-config-dal12-mycluster.yml
   kubectl rollout status deployment/serviceapi --namespace=\"my-namespace\" | tail -1 > deploymentStatus.txt
'''

def deploymentStatus = readFile('deploymentStatus.txt').trim()
echo "Deployment status is: ${deploymentStatus}"
if (deploymentStatus == "deployment \"serviceapi\" successfully rolled out") {
   sh "rm -rf deploymentStatus.txt"

   // Additional logic
   )
}

else {
   error "Pipeline aborted due to deployment verification failure. Check the Kubernetes dashboard for more information."
}

A cleaning strategy for a Docker registry

With the DevOps pipeline maturing and deployment of multiple containers for multiple micro-services taking place, it became evident quite quickly that space is running out and a cleaning strategy is needed.

One way to do this is to clean the repository from images that are older than a set number of days, say, 5 days.

I am using the Bluemix Container registry so bx cr can simply be replaced with docker.

In the pipeline, use:

sh '''
    timestamp=$(date +%s -d "5 day ago")
    bx cr images --format "{{if ge $timestamp .Created}}{{.Repository}}:{{.Tag}}{{end}}" | xargs -r bx cr image-rm
'''

I use the above snippet after I have successfully built the Docker container image > pushed it to the registry and updated the image (in my case, in the Kubernetes cluster).

So, I first save in a shell variable the date value of 5 days ago. Then, using the Go format command (Docker uses Go templates) I iterate through the image repositories and compare the repository creation date with the value in $timestamp. Once it is “5 days old, or more” I delete it.

The enclosed {{.Repository}}:{{.Tag}} is important. It makes the image name and tag values available for the piped command that follows it.

xargs -r ensures the piped command will not execute if no result is passed to it (e.g., no images are >= 5 days old).

For production scenario you may want to ensure you images quota is big, so you could store images for cases where you might need to rollback, and adjust the script accordingly, or possibly also use your own storage solution for Docker container images such as jFrog Artifactory or Nexus Repository, etc.

Additionally, I also docker rmi 0.0.$BUILD_NUMBER the Docker container image that I build at the very beginning of the deployment stage of the pipeline as the image is pushed to the registry, and so there is no need to store it twice: in the build machine and in the registry.

Deploying to a Kubernetes cluster

As you may know, Kubernetes is all the rage these days. Kubernetes. Its feature list is impressive and it is no wonder why it is the go-to system of orchestrating your containers.

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

I wanted to share my pipeline for building and updating containers in a Kubernetes cluster. In fact it’s quite straightforward. The pipeline includes: building a Docker container image, pushing the image to a container registry and updating the container image used in a Pod.

My environment is based in IBM Bluemix, so some commands will not apply…

stage ("Publish to Kubernetes cluster") {
   environment {
      JENKINSBOT = credentials('credentials-ID')
   }

   when {
      branch "develop"
   }

   steps {
      script {
         STAGE_NAME = "Publish to Kubernetes cluster"

         // Login to Bluemix and the Bluemix Container Registry
         sh '''
            bx login ...
            bx cr login
         '''

         // Build the Docker container image and push to Bluemix Container Registry
         sh '''
            docker build -t registry.../myimage:0.0.$BUILD_NUMBER --build-arg NPM_TOKEN=${NPM_TOKEN} .
            docker push registry.../myimage:0.0.$BUILD_NUMBER
         '''

         // Check for image vulnerabilities - applies only if you have such a service...
         isVulnerable = sh(script: "bx cr images --format '{{if and (eq .Repository \"registry.../myimage\") (eq .Tag \"0.0.$BUILD_NUMBER\")}}{{eq .Vulnerable \"Vulnerable\"}}{{end}}'", returnStatus: true)

         if (isVulnerable=="true") {
            error "Image may be vulnerable! failing the job."
         }

         // Apply Kubernetes configuration and update the pods in the cluster
         sh '''
            export KUBECONFIG=/home/idanadar/.bluemix/plugins/container-service/clusters/certmgmt/kube-config.yml
            kubectl set image deployment myimage myimage=registry.../myimage:0.0.$BUILD_NUMBER --record
         '''

         // If reached here, it means success. Notify
         slackSend (
            color: '#199515',
            message: "$JOB_NAME: <$BUILD_URL|Build #$BUILD_NUMBER> Kubernetes pod update passed successfully."
         )
      }
   }
}

Notes:
* I use $BUILD_NUMBER as the means to tag the image.
* I use a pre-defined export... to configure the session with the required configuration for the kubectl CLI to know which cluster to work with.
* The Bluemix Container Registry provides image scanning for vulnerabilities!
* I use kubectl set image ... to update the image used in the Pod(s). Works great with the replica setting.

More on Kubernetes in a later blog post.

Explicitly triggering a Jenkins job using a keyword

Testing is important. To that end a developer should implement unit tests to assure any implemented code does what it is meant to do, and also integration tests with networking mocks to, for example, make sure endpoints do what they’re supposed to do. Then there are end-to-end tests to assure that the code works properly in tandem with other players.

It’s good to set end-to-end tests suites to run continuously according to a schedule to capture errors that managed to sneak it. It’s also good to trigger those end-to-end tests explicitly when you know that you’ve added risky code and would like the extra check.

Similarly to how I’ve implemented skipping builds in declarative pipelines, I have implemented the same concept here as well:

post {
    // Run end-to-end tests, if requested
    success {
        script {
            if (BRANCH_NAME == "develop") {
                result = sh (script: "git log -1 | grep '.*\\[e2e\\].*'", 
returnStatus: true)
                if (result == 0) {
                    build job: '****/master', wait: false
                }
            }
        }
    }
    ...
}

Once a job for a micro-service has finished its run successfully, I am checking if the git commit log contains the keyword “[e2e]”. If yes, this triggers a run of a job that does the end-to-end testing. Note that this is a multi-branch job and so need to specify both the job name and the branch name.

Failing builds when developers misbehave (code coverage)

Developers. They’re kinda like little children sometimes, aren’t they? You have to keep your eye open on ’em…

Joking aside though, code coverage is an important aspect in software development.

code coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. A program with high code coverage, measured as a percentage, has had more of its source code executed during testing which suggests it has a lower chance of containing undetected software bugs compared to a program with low code coverage.

Low code coverage typically may occur when new code was implemented but tests were not added for the new code just yet.

One way to potentially solve it is developing the TDD way, but not all teams like that. So another solution is using tooling in the pipeline to fail a build when code coverage drops. One such tool is the open source SonarQube.

The prerequisites are to:

  • Install the SonarQube server on a host machine
  • Install the appropriate Sonar scanner on the same host machine
  • Install the “Quality Gates” Jenkins plug-in
  • Install the “SonarQube Scanner for Jenkins” plug-in
  • Install the SonarJS plug-in in SonarQube’s Update Center

As for failing the build, I have set up a Quality Gate in the SonarQube dashboard to error when the coverage metric is below 80%.

My implementation is as follows:

stage ("SonarQube analysis") {
   steps {   
      script {
         STAGE_NAME = "SonarQube analysis"

         withSonarQubeEnv('SonarQube') {
            sh "../../../sonar-scanner-2.9.0.670/bin/sonar-scanner"   
         }

         // Check if coverage threshold is met, otherwise fail the job
         def qualitygate = waitForQualityGate()
         if (qualitygate.status != "OK") {
            error "Pipeline aborted due to quality gate coverage failure: ${qualitygate.status}"
         }
      }
   }
}

withSonarQubeEnv('SonarQube') refers to my SonarQube configuration in Jenkins > Manage Jenkins > Configure Jenkins > SonarQube servers where Name is set to SonarQube.

Now, don’t start with 80% coverage right away, let it simmer… start with 40% or 50% and increase it by 10% each passing week. This way developers will be ‘forced’ to add tests to their code as they progress with their work, but not be totally pressured about it and will have enough time to cover everything with time.

Skipping builds in a multi-branch job

Skipping builds in Jenkins is possible, there’s a “CI Skip” plug-in for Jenkins, but this plug-in doesn’t work with multi-branch jobs… How can you skip builds then in such a case? Here’s my take on it. Of course, feel free to modify it for your  pipeline’s needs.

I originally wrote about this in my Stack Overflow question: https://stackoverflow.com/questions/43016942/can-a-jenkins-job-be-aborted-with-success-result

Some context
Lets say a git push to GitHub was made, and this triggers either the pull_request or push webhook. The webhook basically tells Jenkins (if you have your Jenkins’ master URL set up there) that the job for this repository should start… but we don’t want a job to start for this specific code push. To skip it I could simply error the build (instead of the if statement below) but that’d produce a “red” line in the Stage View area and that’s not nice. I wanted a green line.

  1. Add a boolean parameter: 
    pipeline {
 parameters {
     booleanParam(defaultValue: true, description: 'Execute pipeline?', name: 'shouldBuild')
 }
 ...
  2. Add the following right after the repository is checked out. We’re checking if the git commit has “[ci skip]” in the very latest commit log. 
    stages {
 stage ("Skip build?") {
     result = sh (script: "git log -1 | grep '.*\\[ci skip\\].*'", returnStatus: true)
     if (result == 0) {
         echo ("This build should be skipped. Aborting.")
         env.shouldBuild = "false"
     }
 }
 ...
}
  3. In each of the stages of the pipeline, check… e.g.: 
    stage ("Unit tests") {
 when {
     expression {
         return env.shouldBuild != "false"
     }
 }

 steps {
     ...
 }
}

If shouldBuild is true, the stage will be run. Otherwise, it’ll get skipped.