CICD-Goat Setup and Easy Challenge walkthrough (WhiteRabbit, MadHatter, Duchess)

Intro

Continuous Integration / Continous Delivery(/Deployment) (CI/CD) is becoming a major part of organisations world wide as technology stacks become more efficient and complex, often requiring CICD pipelines to leverage privileged credentials to build environments. As we see more environments move to Infrastructure as Code (IaC), we are seeing more privileged access being required by critical pipelines.

With this change, more and more on my Red Teams I am finding that CICD platforms are an exceptional route to take, due to their level of privilege and the lack of detection technology that exists for CICD abuse. This has often taken the form of getting access to the platform by compromising some form of development account (or just self registering as any user from within the organisation), reading repositories and understanding the projects, creating new branches, changing pipeline YAML files to abuse runners or secrets, extracting authentication material for cloud providers and then compromising production cloud environments.

Whilst researching CICD attack chains I stumbled across CICD-Goat, a lab which has currently 11 challenges to complete with ranging difficulty that cover the CICD Top Ten Vulnerabiltiies. Given that this appears to be quite a niche area to exploit on Red Teams, I wanted to cover a blog series on the challenges to demonstrate how these weaknesses in CICD can come about and what they can mean for your organisation.

Setup

The setup for these challenges simply involves installing Docker Desktop on Windows, downloading the docker compose file from the repository and then spawning the machines.

mkdir cicd-goat; cd cicd-goat
curl -o docker-compose.yaml https://raw.githubusercontent.com/cider-security-research/cicd-goat/main/docker-compose.yaml
get-content docker-compose.yaml | %{$_ -replace "bridge","nat"}
docker compose up -d

Once this is started (which may take some minutes), you should be able to hit the following web portals:

CTFd http://localhost:8000 to view the challenges:
Username: alice
Password: alice

Jenkins http://localhost:8080
Username: alice
Password: alice

Gitea http://localhost:3000
Username: thealice
Password: thealice

GitLab http://localhost:4000
Username: alice
Password: alice1234

Once connecting to CTFd you should see the login page.

Once logging in with the above credentials we see the challenges contained.

Level 1 - White Rabbit

Clicking on White Rabbit we get the following message:

Logging in to the Jenkins instance we see a bunch of Jenkins jobs.

Once of these jobs is called Wonderland-White-Rabbit and clicking into the job and looking around, we can see it is pulling from a Gitea repo but the pipeline currently doesn’t output anything. Hovering over the reference to main in the log output provides the URL http://localhost:3000/Wonderland/white-rabbit/src/branch/main.

Logging in to Gitea we can see 7 repositories as the user thealice.

Looking at the white rabbit repository we can see that a Jenkinsfile is present which would control the stages on the jenkins pipeline.

This Jenkinsfile will end up being where we need to load in the credential we want to steal from the jenkins store and then print it out, bypassing sanitisation.

To do this, we first need to locate what the name of the secret is within Jenkins. From the scenario challenge we know it is likely to be called flag1, but it would be good to confirm that through other usage.

Looking at the credential store within Jenkins we don’t see any as our user does not have permission to view them. However, when facing these scenarios in Red Teams, this information is found by looking through other repositories and finding references to secrets that may exist. Looking through all the repositories on Gitea, there was a repository for mad-hatter-pipeline, which contained a Jenkinsfile that pulled out a secret called flag3.

Adding a similar line to the white-rabbit repo, we could update the Jenkinsfile. This couldn’t be done on the main branch so instead a new branch was made and a pull request was created from our branch to main.

Looking at Jenkins, we could see that the PR generated a pipeline trigger even without approval.

By clicking into the PR we can then click on the build history.

We can then look at the console output.

Scrolling down to the bottom we can see the error message as to why our build failed.

From this we now know that the secret name is correctly flag1 but the type of the secret is Secret Text where as we used the jenkins reference for a UsernamePassword secret. Going back to our white-rabbit repo on our newly created thealice-patch-1 branch, we can tweak the Jenkinsfile to now call the correct type of secret. We will also need to pipe it to something like base64, as the Jenkins output log will sanitise secrets. A common and easy way to bypass this is to use base64 to encode the string and then decode it from the log.

Then after a couple minutes another build will be triggered within jenkins since the webhook will see that an update was made to the PR without having to create a new one.

Going in to the latest execution we can look at the console output and find the reference to our added code. We can now see a base64 string on the output (sanitised so to not spoil the flag).

This can then be copied and decoded, either on the terminal using base64 -d <FLAG> or using an online decoder. You can then enter the decoded flag on the CTFd scoreboard to make sure it is correct.

Mad Hatter

Looking at the brief for the challenge we can see that the previous attack path will not be possible since we can no longer modify a Jenkinsfile. We can see what is within Gitea for the challenge to see how we may approach this.

We can see two repositories. One of them is the pipeline which will be the most interesting, since if we cannot modify the pipeline definition itself, then to access jenkins secrets we will need to modify the pipeline indirectly through a pulled in resource. Looking at the mad-hatter-pipeline repo we see the following Jenkinsfile.

We can see that one of the steps involves taking the flag3 credential and leveraging it, storing a username / password combination into the variables USERNAME and FLAG respectively. This shows us that we want to be executing code after that step, to ensure the variables exist when our code runs. The code within that stage runs the command make. This is useful for us since make will execute commands based on a makefile, which we may have control over.

Looking inside the repo mad-hatter, we can indeed see that a Makefile is present.

Inside the Makefile we can see that a curl command is being executed leveraging the $FLAG variable.

By attempting to edit the file we can also see that there is no protection of the file and that we can edit directly to main.

We then edit the Makefile to echo out the variable in the same way it is being used already and commit to main.

Once commited, this ended up not causing a pipeline to run within Jenkins. Through trying to figure out why several steps were taken, including creating new branches, new PRs, new merges etc. Eventually it seemed that just having a new branch with the changes and no PR is what was needed for jenkins to recognise the build and trigger a pipeline execution.

Looking through the build output we can see the base64 string again.

This can then be decoded and submitted.

Duchess

The challenge for this one suggests minding their own business and how that may apply to secrets and that a token was left somewhere. Looking at the repository for the challenge showed that it appeared to be a forked repo for pyJWT.

There were no Jenkins pipelines for the repository and there were 696 commits in the history. Looking through the history revealed an interesting commit on the last entry of the second page.

Looking inside the commit we can see a pypi token which was removed for this commit from the repo.

This token could then be submitted to the scoreboard.

Summary of Attacks

Through the Easy challenges of the CICD-Goat project, we have exploited two of the top 10 vulnerabilities for CICD.

White-Rabbit:

• Direct Poisoned Pipeline Execution (D-PPE) utilised to inject new steps into the Jenkins pipeline via a Jenkinsfile to load in secrets and reveal them. This is CICD-SEC-4 on the top 10 list.

Mad-Hatter:

• Indirect Poisoned Pipeline Execution (I-PPE) utilised to inject malicious code inside a Makefile which is then utilised by a Jenkins pipeline after loading in sensitive credentials. This is also classed under CICD-SEC-4 on the top 10 list.

Duchess:

• Abuse of Insufficient Credential Hygeine to reveal credentials which were removed from the repo, but still exist within the git history. This is CICD-SEC-6 on the top 10 list.

These issues are ones that I have seen through many organisations, often with privileged access being eventually gained as a result. Some key actions that can be done to help mitigate this risk are:

• Ensure pipeline definition files such as Jenkinsfiles are well secured and that only limited trusted resources can be used to alter these files and that they must have approvers to do so.
• Ensure credentials and secrets are managed securely. This is much easier said than done and will depend heavily on your environment, however, ensuring that your platform is only storing credentials it requires and ensuring that those credentials have the least possible privilege to perform their function will help.
• Ensure credentials and secrets can only be accessed by the repositories requiring them and make sure access to those repositories is limited to only those that require it.
• Ensure proceeses are in place such as pre-commit checks to prevent sensitive material ending up on git repositories accidently, if they are then be sure to rotate credentials and know they will be in the history.