CI/CD Goat

Here’s my write-up for CI/CD Goat, a purposely vulnerable CI/CD environment with a set of CTF challenges, created by Cider.

Write-up

I deployed the environment on my machine using the instructions from the project’s readme:

curl -o cicd-goat/docker-compose.yaml --create-dirs https://raw.githubusercontent.com/cider-security-research/cicd-goat/main/docker-compose.yaml
cd cicd-goat && docker compose up -d

There are 10 challenges in total, each inspired by Alice in Wonderland.

White Rabbit

I’m late, I’m late! No time to say “hello, goodbye”! Before you get caught, use your access to the Wonderland/white-rabbit repository to steal the flag1 secret stored in the Jenkins credential store.

As directed by the challenge, I visited Gitea and logged in with the provided credentials: thealice:thealice. I cloned the white-rabbit repository and took a look at the Jenkinsfile.

I added an arbitrary line to the Jenkins file to test if I could write to the repository and to test if I could see my changes reflected in the output of the Jenkins job.

Unfortunately, I didn’t have write access to the main branch…

… but I could push my change to a separate branch. I created a PR and waited for the job to start.

The job finished within a couple of seconds and I could see my change reflected in the Jenkins job output:

Now that I had a way to execute commands, I needed to get the value of the flag1 credential.

Unfortunately (for me) it’s not as simple as echo-ing the flag1 credential as Jenkins would mask the value of the credential with several * characters.

This article describes many ways an attacker can dump credentials and bypass the credential masking logic.

I made a second change to my branch, which added one of the suggested methods in the article to print out the value of a given credential, comma delimited.

After the job finished, I had the flag.

Mad Hatter

Jenkinsfile is protected? Sounds like an unbirthday party. Use your access to the Wonderland/mad-hatter repository to steal the flag3 secret.

For this challange, there are two repositories: mad-hatter and mad-hatter-pipeline.

mad-hatter-pipeline contains one file: a Jenkinsfile, which the mad-hatter repository uses as part of it’s build pipeline.

The interesting part of the Jenkinsfile appears when we get down to the make stage:

stage('make'){
    steps {
        withCredentials([usernamePassword(credentialsId: 'flag3', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
        sh 'make || true'
        }
    }           
}

This stage runs the make command with the value of the flag3 credential available to make under the variable FLAG.

I didn’t have write access to this Jenkinsfile, but I did have write access to the mad-hatter repository.

I took a look at the Makefile in mad-hatter:

The Makefile makes a request to http://wonderland:1234/api/user, using the flag3 credential (available via the $FLAG variable from the Jenkinsfile). I wondered if I could mimic this request, but point it to my machine instead.

I set up netcat to listen for connections to port 9000 on my machine and I added a line to the Makefile which would make a request to my machine.

I pushed the change to a new branch named test and clicked “Scan Multibranch Pipeline Now” so that an associated job for my branch would be created.

I ran the job and watched the log file. Within a few seconds the code I added in the Makefile was executed:

I checked my terminal and I could see flag3:

Duchess

If everybody minded their own business, the world would go round a deal faster than it does. Does it apply to your secrets as well? You’ve got access to the Wonderland/duchess repository, which heavily uses Python. The duchess cares a lot about the security of her credentials, but there must be some PyPi token left somewhere… Can you find it?

The first thing I noticed about the duchess repository was how many commits there were: 696. This fact, combined with the prompt from the challenge text: “but there must be some PyPi token left somewhere…” got me thinking about leaked secrets.

I cloned the duchess repository and used gitleaks to scan the repository to see if anything came up.

docker pull zricethezav/gitleaks:latest
docker run -v /home/ben/cicd-goat-repos/duchess:/path zricethezav/gitleaks:latest detect --source="/path" -v

I scanned the output and spotted the token (the flag for this challenge):

gitleaks also provides the commit the secret was committed under:

Caterpillar

Who. Are. You? You just have read permissions… is that enough? Use your access to the Wonderland/caterpillar repository to steal the flag2 secret, which is stored in the Jenkins credential store.

For this challenge, there is one repository: “caterpillar” and two jobs on Jenkins: “caterpillar-prod” and “caterpillar-test”.

I first took a look at the “caterpillar” repository. I spotted the Jenkinsfile and identified the interesting part:

I didn’t have write access to this repository, so I moved my focus over to the Jenkins jobs.

Running the “caterpillar-prod” job can be triggered manually from Jenkins. The “caterpillar-prod” job runs against the main branch, meaning that it will also execute the deploy stage in the project’s Jenkinsfile:

stage('deploy') {
    when {
        expression {
            env.BRANCH_NAME == 'main'
        }
    }
    steps {
        withCredentials([usernamePassword(credentialsId: 'flag2', usernameVariable: 'flag2', passwordVariable: 'TOKEN')]) {
            sh 'curl -isSL "http://wonderland:1234/api/user" -H "Authorization: Token ${TOKEN}" -H "Content-Type: application/json" || true'
        }
    }
}

I could see the “deploy” step being executed in the logs, although the wonderland hostname did not resolve:

The “caterpillar-test” job was interesting because it was a “Pipeline Branch project”. As stated on the job’s page:

There exists a job/project on Jenkins, that is waiting for branches to appear in the repository, but I couldn’t create branches on the “caterpillar” repository… but I could potentially create a branch if I forked the repository and then opened up a PR on the “caterpillar” repository.

My change would be to remove the “when” expression from the Jenkinsfile, and also point the curl request at my machine instead of wonderland.

I created a PR with my change and saw a branch job appear in Jenkins… but the job failed.

The logs showed that the flag2 credential wasn’t available. Maybe it was only available to the “caterpillar-prod” job? I went back to the PR and noticed something interesting:

I couldn’t merge the PR, but something could. I made a change to my PR to put back the “when” clause, to get a passing build. Now I just had to figure out how to merge my change.

I did a little research, and there are several ways you can configure Gitea to merge changes after successful builds (which you might want to do in certain pipelines/use cases). One of those ways is with the Gitea API.

I remembered that I had seen an error referencing a “Gitea access token” in the Jenkins logs:

...
The recommended git tool is: NONE
Warning: CredentialId "gitea-access-token" could not be found.
Cloning the remote Git repository
...

The Jenkins job was trying to pull an access token but it either wasn’t being stored as a credential, or the credential ID was wrong. I edited my change to perform a little more recon, part of which involved running the env command from the Jenkinsfile. That’s when I spotted the token:

Now that I had the token, I could merge the PR via the Gitea API, providing the head_commit_id, which was just the latest commit on the PR.

Checking back in Gitea I could see my PR had been merged.

I set up netcat again to listen on port 9000 and ran the “caterpillar-prod” Jenkins job.

Switching back to my terminal, I saw flag2 had been sent to my machine:

Cheshire Cat

Some go this way. Some go that way. But as for me, myself, personally, I prefer the short cut. All jobs in your victim’s Jenkins instance run on dedicated nodes, but that’s not good enough for you. You are special. You want to execute code on the Jenkins Controller. That’s where the real juice is! Use your access to the Wonderland/cheshire-cat repository to run code on the Controller and steal ~/flag5.txt from its file system. Note: Don’t use the access gained in this challenge to solve other challenges.

For this challenge, I had access to the “cheshire-cat” repo, which I could push branches to and create PRs on, and the “cheshire-cat” Jenkins job, which was a “Pipeline Branch project”.

Given the note on this challenge (and the fact I’d gone this long without popping a shell on Jenkins), I thought I’d try my hand at gaining shell access.

Given this pipeline’s setup, it was a case of raising a PR with a reverse shell in it, and waiting for the connection back to my machine (via netcat) when the job runs.

I had a shell, but I couldn’t find any trace of flag5, and I couldn’t find a way to escalate to the Jenkins Controller from inside agent1.

I took a step back and did a little research.

After a little Googling, I learnt that I could set the agent manually via the Jenkinsfile. The “cheshire-cat” Jenkinsfile was setting the “agent” value to “any”, meaning that the job would execute on any available agent.

There was technically another “agent” though: the Jenkins Controller. Navigating to the home screen in Jenkins listed the two agents: “Built-In Node” (the one I wanted to access) and “agent1” (the one I already had access to). The “Built-In Node” was set to EXCLUSIVE, meaning that jobs would only run on it when specified. Fortunately for me, the agent’s label can be seen in the URL when viewing the agent in Jenkins.

I updated my PR to specify the “(built-in)” agent as part of the Jenkinsfile, set up netcat again, and waited for the job to run.

Within a few moments the job started, and I had access to the Jenkins Controller:

flag5 was inside a file conveniently named “flag5.txt”, found in the “jenkins_home” directory.

Twiddledum

Contrariwise, if it was so, it might be; and if it were so, it would be; but as it isn’t, it ain’t. That’s logic. Flag6 is waiting for you in the twiddledum pipeline. Get it.

For this challenge, I had access to a “twiddledum” job on Jenkins, and two repositories: “twiddledum” and “twiddledee”. I didn’t have write permissions to the “twiddledum” repository, but I did have write permissions to the “twiddledee” repository.

The “twiddledum” job clones the “twiddledum” repo and runs node index.js. “index.js” is a file within the “twiddledee” repo: “twiddledum” requires “twiddledee”.

The output of node index.js was “twiddledee - 1.1.0”

I could see this code in the “twiddledee” repository:

I took a look at the “package.json” file in the “twiddledum” repo and could see a reference to the “twiddledee” repo.

NPM has a few “interesting” features when it comes to dependency management, and one of them is around automatically pulling in dependency updates. This article explains the details, but in short:

Instead of specifying the exact version to be installed in package.json, npm allows you to widen the range of accepted versions. You can allow a newer patch level version with tilde (~) and newer minor or patch level version with caret (^).

The “twiddledee” version specified in “package.json” was ^1.1.0. If I released a new minor or patch version of “twiddledee” then “twiddledum” would automatically pull in the update.

I made a simple change to “index.js” in the “twiddledee” repository to confirm my suspicion and released a new version of “twiddledee” on Gitea: 1.1.1

I ran the “twiddledum” jenkins job and saw my code reflected in the logs:

I made a second change and released a second version of “twiddledee”, but this time I included code to initiate a reverse shell, taken from PayloadsAllTheThings.

I found the flag under the environment variable FLAG6.

Dodo

Everybody has won, and all must have prizes! The Dodo pipeline is scanning you. Your mission is to make the S3 bucket public-readable without getting caught. Collect your prize in the job’s console output once you’re done.

This challenge has one repository (with push directly to “main” enabled) and one job. Running the job sets up an S3 bucket via Terraform.

The code to set up the S3 bucket is found in “main.tf”:

I set the “acl” value to “public-read” and committed the change. The job ran, but it failed.

Something called “checkov” was blocking my change. Checkov is a static code analysis tool that detects common misconfigurations in Terraform, CloudFormation, Kubernetes, etc.

Because I had full write access to the “dodo” repo, I could turn off the Checkov checks by supplying an empty config file. From the docs:

I added a “.checkov.yaml” file, with the config to suppress all failures.

I committed the change, ran the build again, and had the flag.

Hearts

Who stole those tarts? Your goal is to put your hands on the flag8 credential. But not so fast… These are System credentials stored on Jenkins! How would you access THAT?! A permission to admin agents is something you might find useful…

I guessed from this challenge that I needed to assume the role of another user on Jenkins. I could see a list of users in Jenkins: who is “knave”? Clicking through to the knave’s profile shows that they can control admin agents.

I initialled tried to use hydra to brute force knave’s password but I couldn’t seem to get the validation check working, so I instead switched to ZAP. I could tell that I had a valid username and password combination because the response size was different: an invalid response returns 402 bytes, and a valid response doesn’t.

I signed in as “knave” and created a new node. I could see the credential I wanted was being used to authenticate via SSH. I set the host to my machine’s IP and set up a listener on my machine to listen on port 22.

I saved the configuration and waited for the agent to start.

I saw the connection back to my machine but it wasn’t going to be as easy as setting up netcat. This output didn’t contain the credentials.

OpenSSH (a commonly used SSH server) doesn’t output confidential connection information, even if logging is set to “verbose”. To get more information about the request, I set up a honeypot using Cowrie. This was very easy to set up via Docker:

docker run -p 22:2222 cowrie/cowrie:latest

I reset the node and waited for the connection back to my machine. This time I could see more details about the connection, including the credentials (the flag), used to connect.

Dormouse

Is “I breathe when I sleep” the same thing as “I sleep when I breathe”? If you steal secrets when you hack pipelines, does it mean you hack pipelines when you steal secrets? Leave that nonsense aside. Hack the Dormouse pipeline. Steal flag 9. Good luck.

This challenge has one repository and one pipeline job. The “thealice” user doesn’t have write access to the “dormouse” repository.

Taking a look at the Jenkinsfile shows an interesting stage:

stage ('Unit Tests') {
    steps {
        sh "pytest || true"
        // lighttpd is also accessible at http://0177.0.0.01:8008/reportcov.sh
        withCredentials([usernamePassword(credentialsId: 'flag9', usernameVariable: 'USERNAME', passwordVariable: 'FLAG')]) {
            sh """curl -Os http://lighttpd/reportcov.sh
            chmod +x reportcov.sh
            ./reportcov.sh
            """
        }
    }
}

A file is downloaded from lighttpd and executed. If I could gain overwrite reportcov.sh in some way then I would have a code execution path.

lighttpd is running on port “1111” on my machine (as configured by CI/CD Goat’s docker-compose file)

Where does this reportcov.sh file come from? Looking around on Gitea highlighted a “reportcov” repo:

I couldn’t commit directly to the “main” branch but I could create a PR. I took a look at the Jenkinsfile inside the “reportcov” repo and spotted a section that emails the admin when a PR was opened on the repository:

stages {
    stage ('Send notification') {
        steps{
            script {
                try {
                    sh "echo Pull Request ${title} created in the reportcov repository"
                    mail bcc: '', body: '', cc: '', from: '', subject: "Pull Request ${title} created in the reportcov repository", to: 'red_queen@localhost'
                }
                catch (Exception err) {
                    currentBuild.result = 'SUCCESS'
                }
            }
        }
    }
}

It was this part that got me thinking about potentially injecting code via the PRs title:

subject: "Pull Request ${title} created in the reportcov repository"

Further down in the Jenkinsfile was the upload step for the “reportcov.sh” script:

stage ('Deploy') {
    steps {
        sh "set +x && echo \"${KEY}\" > key && chmod 400 key && set -x"
        sh 'scp -o StrictHostKeyChecking=no -i key reportcov.sh root@lighttpd:/var/www/localhost/htdocs'
    }
}

It took a few attempts and a little local testing, but I eventually managed to inject a curl command into the Jenkinsfile via the PR’s header:

Now that I had a PoC working, I raised another PR which would send the key to my machine via a POST request body:

To capture the request I set up a listener using a script from Ahmed-Galal.

I created a malicious “reportcov.sh” script that would make a request back to my machine including the flag in the request (when executed by the dormouse job). I copied the script onto the lighttpd machine using scp and the SSH key.

I set up the listener again, ran the dormouse main job on Jenkins, and saw the flag in my terminal.

Mock Turtle

Have you seen the Mock Turtle yet? It’s the thing Mock Turtle Soup is made from. Can you push to the main branch of the mock-turtle repo? Do what’s needed to steal the flag10 secret stored in the Jenkins credential store.

This challenge has one repo “mock-turtle”, a pipeline job to run checks against each PR raised on Gitea, and a “main” job, which runs against the “main” branch.

I checked out the repo and took a look at the Jenkinsfile:

The Jenkinsfile will automatically merge the PR the build is associated with if the following conditions are met:

check1: The number of “words” added is the same as the number of “words” removed (as determined by wc -w)
check2: The version number is valid (as determined by the regex "^\\d{1,2}\\.\\d{1,2}\\.\\d{1,2}$")
check3: A file matching “version” (as determined by grep version) was changed.

I realised (after some initial tinkering) that raising a PR with a simple malicious change to the Jenkinsfile wasn’t going to work. This is because each PR appeared to run against the Jenkinsfile in “main”. In other words: I couldn’t just add a line that spits out the flag and get the job to run, so I had to abide by the three checks and merge my PR, as the challenge stated.

I raised a PR with two changes. The first change was to add a “version1” file which would satisfy check3. The second change was to add a withCredentials block above the existing code that would send the “flag10” credential to my machine via curl.

To satisfy check1 I removed a few lines of code from the Jenksinsfile making sure to keep the code in the Jenkinsfile valid. In retrospect, I could have removed a few words from the README or something.

After I raised the PR I switched over to Jenkins to check everything was working, and within a few seconds, my PR was merged.

I set up netcat to listen on port 9000 and ran the “main” job. I had the final flag.

Final thoughts

CI/CD Goat is a must for anyone working regularly with deployment pipelines. The challenges are well structured, with the challenge text and hints giving you just the right amount of information to get to the solution. I can also recommend the Application Security Podcast episode (featuring Omer and Daniel from Cider) if you want to hear more about the top 10 CI/CD security risks.

>> Home

2022/05/15