In this article we will see how to implement a DevOps pipeline with Jenkins that can handle concurrent builds. Some stages of a  pipeline require resources that can only be used by one build at a time.

For example we might have a single test or staging environment. Instead of limiting the number of concurrent builds of our pipeline we will marshal the access to resources with the “lock” functionality.

The test lab:

So this is what our desired pipeline looks like. The build stage creates the build artifacts from the sources and might process code analysis and unit tests.

The QA stage verifies the build artifacts by running them in the integration test and performance test environments. There is only one integration test environment and one performance test environment, so we have to make sure those are only used by one build at a time. The tests should run in parallel to accelerate execution.

The build artifacts are rolled out into the staging stage if all tests pass successfully. The setup of this environment is very similar to our production.

The production stage performs a rollout into the production environment if everything looks good in the staging environment.

The challenge:

The duration to pass each stage might be very different. So maybe the build stage takes just one minute while running the QA stage takes 15 minutes.

Even though there can be only one build in some of our environments at any time, we don’t want to constraint our pipeline to run only one build at a time. Instead each stage should start processing consecutive builds once it’s free.

So in our example the build stage could process 15 builds (maybe we have a lot of developers checking in new code) while meanwhile the QA stage processes one build. This gives our developers fast feedback from the build stage each minute.

Since our performance test stage is slower, the builds queue up at the performance test stage. Now obviously we do not want our performance test stage to run a build that is already superseded by newer builds. Instead it should pick up the newest build and discard all older builds.

The solution:

Controlling the flow of concurrent builds with Jenkins is nowadays quite easy. Check out this article for an introduction on Controlling the Flow with Stage, Lock, and Milestone.

We can use “lock” to throttle the number of concurrent builds in a defined section of the Pipeline. And we can use “milestone” to automatically discard builds that are superseded by newer builds.

Here is a Jenkins Pipeline that puts it all together. Just replace the placeholders with the actual pipeline steps and you should be ready to go. Doesn’t really matter whether the constrained resources reside. As long as we can reach them with our Jenkins Agents we are good to go.

/**
 * A Jenkins DevOps Pipeline that allows concurrent builds
 * with constrained resources
 * requires the "Lockable Resources plugin"
 * Author: Lars Berning
 */

stage('Build') {
  node {
    echo 'Building artifacts...'
    // generate build artifacts, unit tests and code analysis
    sleep 2
    // store build artifacts for consecutive stages (i.e. stash
    // or versioned artifact repository)
    echo 'Building artifacts finished'
  }
}

// milestones are used to discard obsolete build, if a consecutive
// build passes a milestone, it will discard all older builds that
// have not yet passed the milestone
milestone 1
stage('QA') {
  // we only have one QA environment so to limit concurrency to
  // a single build we can use locks
  // if another build reaches a lock, it will "wait" until the
  // resource is available again
  // inversePrecedence instructs the lock to pick the most recent
  // build first
  lock(resource: 'qaEnvironment', inversePrecedence: true) {
    // this milestone in combination with the inversed lock above
    // discards all queued superseded builds
    milestone 2
    // run integration and performance testing in parallel
    parallel("Integration Tests": {
      node {
        echo 'Running integration tests...'
        // get build artifacts (i.e. from stash)
        // setup test environment
        // run integration tests
        sleep 5
        // store test results
        echo 'Running integration tests finished'
      }
    }, "Performance Tests": {
        echo 'Running performance tests...'
        // setup and run performance tests
        sleep 10
        // store test results
        echo 'Running performance tests finished'
    })
  }
}

milestone 3
stage('Staging') {
  lock(resource: 'stagingEnvironment', inversePrecedence: true) {
    milestone 4
    node {
      echo 'Deployment to staging environment...'
      // setup and deploy staging environment
      sleep 5
      echo 'Deployment to staging environment finished'
    }
    // wait for user verification before pushing to production
    input message: 'Deploy to production?'
  }
}

milestone 5
stage ('Production') {
  lock(resource: 'productionEnvironment', inversePrecedence: true) {
    node {
      echo 'Deployment to production environment...'
      // setup and deploy production environment
      sleep 5
      echo 'Deployment to production environment finished'
    }
  }
}
Implementing a DevOps Pipeline in Jenkins with concurrent builds
Tagged on: