Introduction

It's well-known that Jenkins' popularity and developer affection have been steadily declining over the years. It’s difficult to maintain, has the plugin system that causes a lot of headaches; and don’t get me started on Groovy. DevOps folks really like tools that “just work”, and Jenkins sometimes is the opposite of that.

However, you may find many reasons to still want to learn and use Jenkins - big enterprises still use it, it lives as a legacy software. In my case, I’ve become interested in hosting a CI/CD tool locally - and surprisingly the choices are not as abundant as one might expect. While cloud-based CI/CD solutions like GitHub Actions, GitLab CI, and CircleCI dominate the modern development landscape, local CI/CD tools are relatively limited.

The purpose of this article is to share the set up I’ve gone through. The emphasis will be to counter some ill advice found on the Internet - especially securing Docker API access, which is surprisingly common, to the point of being scary. However, some people rightly point out that it’s a bad practice and we’re going to do it properly here.

What I’ll cover:

• install Docker Engine in Linux (Fedora)

• use asymmetric encryption to secure access to Docker API

• deploy Jenkins control plane in Docker

• deploy Jenkins Docker cloud

• demonstrate usage of Docker-in-Pipeline in Jenkins

Docker installation and initial set-up

Docker terminology can be slightly confusing. By 'Docker installation,' I mean installing several components:

• Docker engine

• Docker build

• Docker compose

The commands are taken from the documentation:

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

After installation we need perform some steps.

First of all, let’s start docker daemon and enable it (so that it starts automatically after boot):

sudo systemctl start docker.service
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Next, we need to handle privileges:

The Docker daemon binds to a Unix socket, not a TCP port. By default it's the root user that owns the Unix socket, and other users can only access it using sudo. The Docker daemon always runs as the root user.

If you don't want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group.

In our case, the docker group should already be there, so let’ just add the user:

sudo usermod -aG docker $USER

Log-in/log-out (or restart), and you should be able to run docker container without becoming superuser:

docker run hello-world

Let’s configure logging. I’ve used this:

sudo vi /etc/docker/daemon.json

And put there:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  }
}

Final step is creation of a new bridge network:

docker network create jenkins

Let’s inspect it and note the gateway IP - we’ll need that later.

docker network inspect jenkins | jq '.[0].IPAM.Config[0].Gateway'
"172.18.0.1"

By deafult, “bridge” network is created. If you want some background on what it is, check out this documentation page.

Creation of a new Docker network also adds a new network interface on your host - you can confirm it by running ip addr command:

11: <if-name>: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    ...
    inet 172.18.0.1/16 brd 172.18.255.255 scope global <if-name>
       valid_lft forever preferred_lft forever

Any request send to the gateway will effectively arrive at the localhost’s interface, which is exactly what we’ll use.

Secure Access to Docker API

Why do we need to access Docker API securely?

The eventual plan is to run Docker containers (spun dynamically) as agents. To achieve this, the control needs to send a request to a Docker host that will provision a container on which our job will run. In our case:

• Control plane will run on a Docker container itself (container running on my localhost)

• The Docker host is also my localhost

The question is - how does the control plan communicate with the Docker host? The underlying system is the same (my localhost), so it should work seamlessly, right?

Well, no, not exactly. As we can read here:

By default, the Docker daemon listens for connections on a Unix socket to accept requests from local clients. You can configure Docker to accept requests from remote clients by configuring it to listen on an IP address and port as well as the Unix socket.

The Control Plane container and my localhost (that is running dockerd) may be considered “remote” in this scenario.

When configuring Docker Cloud in Jenkins, we’re going to need to put “Docker Host URI”. In our case, it will be the gateway of the Docker network - I’ll come to that part later, in Jenkins Docker Cloud set-up. Bottom line is, we’ll communicate with Docker on the host via API calls, as using Unix socket will not be possible (there is a trick with sharing socket as a volume, but I’ll leave that).

Docker API can be configured to be open to anyone - a bad practice, examples of which I linked in the introduction. This anti-patter is achieved by running the deamon like this:

/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375

Or supplying the following parameters in the /etc/docker/daemon.json file:

{
    "hosts": ["unix:///var/run/docker.sock", "fd://", "tcp://0.0.0.0:2375"],
}

Now, anyone with access to any interface of a Docker host can send a request there. They can run a container with filesystem of the host attached, and viola - the host is effectively open to root login (with no password) to anyone on the network. Docker docs state:

It's critically important that you understand the security implications of opening Docker to the network. If steps aren't taken to secure the connection, it's possible for remote non-root users to gain root access on the host.

If not that way, then how we can secure Docker API? Certificate, of course!

Overview of how to secure Docker API

I assume the reader is at least somehow familiar with how asymmetric encryption and TLS works. A basic understanding is all we need, such as what's covered in this article.

1. First we have to create a CA - Certificate Authority on our local machine.

2. Then we generate a public and private key pair for the Docker API (2).

   • Normally we’d need to send a certificate signing request (CSR) containing te public key and other identifying details to the CA. But we’re in a local context here, so it’s not needed.

   • Next, the CA will take steps to validate the applicant’s identity and the right to claim credentials such as domain names for server certificates or email addresses for email certificates in the CSR. Again, in our local context this doesn’t happen.

3. If validation is successful, the CA issues the certificate containing the details and public key from the CSR. In our case, we obtain pair of keys for the Docker API.

4. Then we generate a public and private key pair for the Jenkins Control Plane.

5. We sign the CSR using our CA.

6. We modify the Docker deamon parameters to only accept connections on the API from clients providing a certificate trusted by your CA.

In summary, we’ll create CA, 2 pairs of keys, sign them by CA and later use to configure Jenkins Docker Cloud.

My steps follow the official documentation with some changes/additions.

Steps for securing Docker API

1. Create CA.

First we generate CA key.

openssl genrsa -aes256 -out ca-key.pem 4096

And then put some passphrase:

Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Then we generate CA cert:

openssl req -new -x509 -days 3650 -key ca-key.pem -sha256 -out ca.pem

req: This subcommand is used to create and process certificate requests.

-x509: This option specifies that the output should be a self-signed certificate instead of a certificate request. Without this option, the command would generate a CSR that would need to be signed by a Certificate Authority (CA) to become a valid certificate.

-key ca-key.pem: This option specifies the private key file (ca-key.pem) to be used for signing the certificate.

2. Generate private/public key pair (for the Docker API).

Create a private key:

openssl genrsa -out server-key.pem 4096

Create a CSR:

openssl req -subj "/CN=piotrs-fedora" -sha256 -new -key server-key.pem -out server.csr

In summary, this command generates a new CSR with the Common Name "piotrs-fedora", using the private key from server-key.pem, and saves the CSR to server.csr using the SHA-256 hash algorithm.

Normally here we’d have 2 more steps:

• sending CSR to the CA (not happening, same machine)

• validating CSR (nothing needs validating here)

3. Sign CSR - Docker API key.

Since TLS connections can be made through IP address as well as DNS name, the IP addresses need to be specified when creating the certificate. Which IPs to choose? We’ll use the gateway of the Docker network we created before. If you don’t have it hand, just run the command I laid out in the previous section.

For example, to allow connections using 172.18.0.1 and 127.0.0.1:

echo subjectAltName = DNS:piotrs-fedora,IP:172.18.0.1,IP:127.0.0.1 >> extfile.cnf
echo extendedKeyUsage = serverAuth >> extfile.cnf

Sign CSR:

openssl x509 -req -days 3650 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem   -CAcreateserial -out server-cert.pem -extfile extfile.cnf

This command signs the CSR (server.csr) with the CA certificate (ca.pem) and CA private key (ca-key.pem), generating a signed certificate (server-cert.pem) valid for 10 years. The SHA-256 hash algorithm is used for signing, and additional certificate extensions are specified in the extfile.cnf configuration file. If a serial number file does not already exist, it will be created.

4. Generate private/public key pair (for the Jenkins Control Plane).

Key:

openssl genrsa -out key.pem 4096

Cert:

openssl req -subj '/CN=client' -new -key key.pem -out client.csr

To make the key suitable for client authentication, create a new extensions config file:

echo extendedKeyUsage = clientAuth > extfile-client.cnf

5. Sign CSR - Jenkins Control Plane key.

openssl x509 -req -days 3650 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial -out cert.pem -extfile extfile-client.cnf

6. Modify the docker deamon parameters.

Remove unnecessary files:

rm -v client.csr server.csr extfile.cnf extfile-client.cnf

Let's move CA and Docker server keys to another location.

sudo mv ca-key.pem ca.pem ca.srl server-cert.pem server-key.pem key.pem cert.pem  /etc/docker/

Modify /etc/docker/daemon.json so that it looks like:

❯ cat /etc/docker/daemon.json 
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  },
  "hosts": ["unix:///var/run/docker.sock", "fd://", "tcp://0.0.0.0:2376"],
  "tlscacert": "/etc/docker/ca.pem",
  "tlscert": "/etc/docker/server-cert.pem",
  "tlskey": "/etc/docker/server-key.pem",
  "tlsverify": true
}

The last is removing parameters from the deamon’s unit file so that only those from the daemon.json file are used:

sudo systemctl edit docker.service

So that ExecStart is like in override file that opened is like:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

To protect your keys from accidental damage, remove their write permissions. To make them only readable by you, change file modes as follows:

 chmod -v 0400 /etc/docker/ca-key.pem /etc/docker/key.pem /etc/docker/server-key.pem

Certificates can be world-readable, but you might want to remove write access to prevent accidental damage:

 chmod -v 0444 /etc/docker/ca.pem /etc/docker/server-cert.pem /etc/docker/cert.pem

Restart the docker deameon and we’re done with this part!

sudo systemctl restart docker.service

Jenkins and Docker Cloud Setup

Jenkins Control Plane on a container

I use the official Jenkins image and the command is based on the official docs. Description of the parameters is also there (I removed some due to the usage of host’s Docker API as opposed to a separate container).

docker run \
  --name jenkins \
  --restart=on-failure \
  --detach \
  --network jenkins \
  --publish 8080:8080 \
  --publish 50000:50000 \
  --volume jenkins-data:/var/jenkins_home \
  jenkins/jenkins:latest

Next step is running post-installation wizard as described in the docs.

To get the initial password, use:

docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

Docker Cloud

First we have to save the key pair generated for Jenkins in Jenkins Credentials:

• in Client key put content of the file /etc/docker/key.pem

• in Client Certificate put content of the file /etc/docker/cert.pem

• In CA cert put content of the file /etc/docker/ca.pem

Right now if we were to open “Cloud” section in “Manage Jenkins”, we’d see:

We need to go to Plugins → Available Plugins and install Docker plugin:

Now we can proceed with Docker cloud set-up:

Now we have to verify connectivity to the Docker API:

• The URI is the IP of the gateway of our Docker network. Alternative way could be supplying DOCKER_HOST variable when running the container and putting it here.

• If you get permission denied error, make sure you supplied correct credentials (keys and certs).

Even though we have connectivity to the Docker host, we still don’t have any container images that could be used in our jobs. As we read in the Docker plugin docs:

You need a docker image that can be used to run Jenkins agent runtime. Depending on the launch method you select, there's some prerequisites for the Docker image to be used

We’ll connect via SSH, so we’re going to pick an image with SSHd running, for the purposes of this article, not to overcomplicate, let’s use jenkins/ssh-agent

Using jenkins user because the docs state:

When using the jenkins/ssh-agent Docker image, ensure that the user is set to jenkins

Note: if you’re using an image built locally, set “Pull strategy” to “Never pull”, otherwise Jenkins will try to pull your local image from Dockerhub.

The 2nd template will use a different connection method for demonstration. I put the following values:

Testing Docker cloud

Let’s test if our Docker Cloud works:

Open “Configure” in our new project:

There are only 2 fields we need to modify:

Restrict where this project can be run → Label Expression: basic-agent

And we add a single build step that echoes a string:

It should finish successfully:

Let’s re-create the same Freestyle project, but this time let’s put label basic-agent-attached to confirm the different connection method to agent works as well.

The job should finish successfully as well:

Docker with Pipeline

What is it?

In the previous section, we created a Docker Cloud and 2 templates, effectively allowing us to schedule jobs on different “types” of agents. Let’s pause here for a second - try to think of limitations of such approach. What if we needed to utilize a tool that was not present on any of the Docker images used in templates? Unfortunately, we’d have to add a new template, new label and then use the new label in our job. There’s an easier and more convenient way, though.

Docker with Pipeline allows us to supply a Dockerfile within Jenkinsfile. If you need refresher on Dockerfile, check out this page:

Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.

Jenkinsfile in turn:

Jenkinsfile is a text file that contains the definition of a Jenkins Pipeline and is checked into source control.

Jenkins provides a way also to write pipeline scripts (Jenkinsfile syntax) directly in the project configuration (without the need to check the file into source control). You wouldn’t use it for any half-serious scenarios, but for our demo it will work just fine and that’s what I’ll use later.

Putting it together, Docker with Pipeline allows us to dynamically spin up containers that don’t have to be pre-built. No more worrying about adding new templates to Docker Cloud!

Docker with Pipeline normally assumes that agents are capable of running Docker, let’s look at the docs:

By default, Pipeline assumes that any configured agent is capable of running Docker-based Pipelines. For Jenkins environments that have macOS, Windows, or other agents that are unable to run the Docker daemon, this default setting may be problematic.

Fortunately, there’s also option to use a remote Docker server (in our case, API accessible on localhost):

By default, the Docker Pipeline plugin will communicate with a local Docker daemon, typically accessed through /var/run/docker.sock.

To select a non-default Docker server, such as with Docker Swarm, use the withServer() method.

Docker with Pipeline set-up and example

First, let’s ensure we have Docker Pipeline plugin installed.

Start with creating a new project, this time make sure to use “Pipeline”:

Once created, open “Configure” tab. Note that we’ll specify virtually all configuration (such as agent label and build stages) in the Pipeline → Definition section.

If we were to run a variation of the script that I linked, it would fail:

node {
    docker.withServer('tcp://172.18.0.1:2376', 'jenkins-keypair-for-docker-api') {
        docker.image('jenkins/agent:latest-bookworm-jdk17').withRun('-p 3306:3306') {
            sh 'echo Hello'
        }
    }
}

The result is:

Started by user user
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/docker_with_pipeline_demo
[Pipeline] {
[Pipeline] withDockerServer
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker run -d -p 3306:3306 jenkins/agent:latest-bookworm-jdk17
/var/jenkins_home/workspace/docker_with_pipeline_demo@tmp/durable-dd328bc8/script.sh.copy: 1: docker: not found
[Pipeline] }
[Pipeline] // withDockerServer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE

Why is that? Well, it boils down to how Docker Pipeline plugin works. Our Jenkins server needs a Docker client installed so that it sends request to our remote Docker Server (with Docker deaemon). Only client is needed - Jenkins server will not run any containers.

This issue could have been prevented if I actually followed documentation on how to deploy Jenkins in Docker, instead of using jenkins/jenkins:latest image directly! Let’s build an image for our Jenkins. Please note the line in which docker-ce-cli is installed.

FROM jenkins/jenkins:latest
USER root
RUN apt-get update && apt-get install -y lsb-release
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \
  https://download.docker.com/linux/debian/gpg
RUN echo "deb [arch=$(dpkg --print-architecture) \
  signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \
  https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
RUN apt-get update && apt-get install -y docker-ce-cli
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean docker-workflow"

Build it (on our localhost of course):

docker build -t jenkins-with-docker-cli .

Now let’s stop our current container, run the new one, of course mount the same volume so that we don’t lose any of our config:

docker stop $(docker ps --filter "ancestor=jenkins/jenkins:latest" --format "{{.ID}}")
docker container rm $(docker ps -a --filter "ancestor=jenkins/jenkins:latest" --format "{{.ID}}")

Run the new container:

docker run \
  --name jenkins \
  --restart=on-failure \
  --detach \
  --network jenkins \
  --publish 8080:8080 \
  --publish 50000:50000 \
  --volume jenkins-data:/var/jenkins_home \
  jenkins-with-docker-cli:latest

Now it works as expected:

node {
    docker.withServer('tcp://172.18.0.1:2376', 'jenkins-keypair-for-docker-api') {
        docker.image('jenkins/agent:latest-bookworm-jdk17').withRun('-p 3306:3306') {
            sh 'echo Hello'
        }
    }
}

Started by user user

[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins
 in /var/jenkins_home/workspace/docker_with_pipeline_demo
[Pipeline] {
[Pipeline] withDockerServer
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker run -d -p 3306:3306 jenkins/agent:latest-bookworm-jdk17
[Pipeline] sh
+ echo Hello
Hello
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker stop ca125a3e5a45ac8b002c71f0b2d86122569823f4e6513158e9e98f2c80978ba9
ca125a3e5a45ac8b002c71f0b2d86122569823f4e6513158e9e98f2c80978ba9
+ docker rm -f --volumes ca125a3e5a45ac8b002c71f0b2d86122569823f4e6513158e9e98f2c80978ba9
ca125a3e5a45ac8b002c71f0b2d86122569823f4e6513158e9e98f2c80978ba9
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withDockerServer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

There was last point I want to mention. The snippet form the documentation I picked uses so-called scripted pipeline. Declarative syntax is easier, does the job and I personally prefer it. Could we use remote Docker host method and declarative pipeline?

Yes, but it involves a couple steps more - we need to set env variables for the Jenkins to know how to access remote host. The variable list is here.

• DOCKER_HOST

• DOCKER_CERT_PATH

The first one is easy, we’ve already used it: tcp://172.18.0.1:2376.

The second one is more tricky, since we store key/cert pair as Jenkins Credentials, how to access them in a way that’s understandable by Docker?

Let’s use Snippet Generator to get to know how to access Jenkins Credential:

Expanding the question mark near the “variable”:

Name of an environment variable to be set during the build.
Its value will be the absolute path of the directory where the {ca,cert,key}.pem files will be created.
You probably want to call this variable DOCKER_CERT_PATH, which will be understood by the docker client binary.

The third part is ensuring that we use the agent that has Docker Client in stage when we need to call remote Docker API. For that let’s follow this.

Assign label control-plane to our Jenkins server:

Use label control-plane in Docker pipeline:

And use piepeline code:

pipeline {
    agent none

    environment {
        DOCKER_HOST = 'tcp://172.18.0.1:2376'
        DOCKER_TLS_VERIFY = '1'
    }

    stages {
        stage('Initialize') {
            agent {
                label 'control-plane'
            }
            steps {
                withCredentials([dockerCert(credentialsId: 'jenkins-keypair-for-docker-api', variable: 'DOCKER_CREDS')]) {
                    script {
                        env.DOCKER_CERT_PATH = "${env.WORKSPACE}/docker-certs"
                        sh 'mkdir -p $DOCKER_CERT_PATH'
                        sh 'cp -r $DOCKER_CREDS/* $DOCKER_CERT_PATH/'
                    }
                    stash includes: 'docker-certs/**', name: 'docker-certs'
                }
            }
        }
        stage('Use Docker') {
            agent {
                docker {
                    image 'busybox:latest'
                }
            }
            steps {
                unstash 'docker-certs'
                sh 'echo Hello'
            }
        }
    }
}

It succeeds, and indeed I’ve seen a container (busybox) spun up on my localhost.

Well, we’ve arrived at quite complex solution to a simple problem - typical for Jenkins.

• We specify a few env variables for Docker so that we use remote endpoint (instead of trying to run Docker container directly on another container that’s hosting Jenkins server)

• Then we save the certificate and key to a certain directory that we stash. Stashing is not really necessary if we use the same agent (and we use Jenkins server for stages), but better to put it in case we used different agent to call remote Docker API in the 2nd step.

• Finally we use Docker Client on the Jenkins Server to call remote Docker API that’s on my localhost - accessible via Docker bridge network.

Summary

Effective setup allows us to use 2 methods of spinning up containers acting as agents dynamically - Docker Cloud and Docker with Pipeline. Both have their advantages actually, some discussion can be found here. Localhost’s Docker Engine is used to pull/create images and run containers.

I’d like to mention that for me this set up was more of a training exercise than anything else. When you run Jenkins locally, you probably don’t need to throw additional hurdles under your feet, probably for most cases you just treat your localhost as a static node and that’s it.

Indeed it seems quite exotic when we look at it - Jenkins in a container, calling localhost’s Docker API. Something more natural would be to run Docker-in-Docker - have an additional container and run containers there (official docs have it in the installation guide).