Please leave your details and we'll let you know when it's up and ready to use.
Do you know what happens under the hood of your CI? Without deep understanding, you might be vulnerable to innovative supply chain attacks. This article describes such an attack.
Caching is used to speed up processes. Instead of building or downloading, again and again, a package, it simply and automatically keeps artifacts for reuse. However, caches might be poisoned. For instance, a malicious tool used in a test workflow can poison its cache. Later, another workflow using the same cache might be affected. If this workflow has higher privileges, this is in fact a method of pivoting an attack. In this post, we report on an experiment attack on a GitHub CI pipeline, but a similar logical vulnerability exists in other CI products.
The attack plays out as follows:
In response to our disclosure, GitHub said they don’t have a plan to harden the cache feature against this type of attack.
We propose mitigation by signing the cache’s content hash value cryptographically and verifying the signature before use. To learn more about the attack and mitigations, keep reading.
often reuse the same outputs or downloaded dependencies from one run to another (for example, the packages downloaded by package and dependency management tools such as Maven, Gradle, npm, and Yarn. These packages are usually kept in a local cache of downloaded dependencies).
To optimize CI runs, GitHub grants access to a cache resource that can be used across the pipeline via an API. The entries in the cache are a key-value combination, where keys are string-based, and values are files or directories one would like to cache.
Using the action/cache Git action anywhere in the CI will run two steps: one step will take place during the run process when it’s called and the other will take place post workflow (if the run action returned a cache-miss).
Access restrictions provide cache isolation and security by creating a logical boundary between different branches (for example: a cache created for the branch Feature-A [with the base main] would not be accessible to a pull request for the branch Feature-B [with the base main]).
The cache action first searches cache hits for a key and restores keys in the branch containing the workflow run. If there are no hits in the current branch, the cache action searches for the key and restores keys in the parent branch and upstream branches.
Access to a cache is scoped by branch (current and parent), meaning access is provided to all workflows across runs of said branch.
Another important note is that GitHub does not allow modifications once entries are pushed – cache entries are read-only records.
GitHub scopes allow specifying exactly what type of access is needed for various tasks. GitHub’s CI is scoped in a variety of ways, each with its own set of values and features:
GitHub cache scope has been defined in a way that can break some of the other scope restrictions (ex: GitHub cache can affect multiple workflows).
We used an example CI that included two workflows. This example shows how an attack can pivot from a low permission workflow to a high permission one.
The unit-test workflow uses a malicious action that adds a cache entry with malicious content by changing a Golang logging library (go.uber.org/zap@v1) to add the string, ‘BAD library’ to the application artifact description.
Next, the release workflow uses this poisoned cache entry. As a result, the malicious code is injected into the built Golang binary and image. The cache remains poisoned until the entry key is discarded (usually triggered by dependency updates). The same poisoned cache will affect any other workflow, run, and child branch using the same cache key.
In the test we performed, we managed to inject the string ‘BAD library’ into the image description:
This was in version 0.4.1. Next, we updated the tag and rebuilt the image several times, and observed that ‘Bad library’ remained in the description.
Github’s response to our disclosure was that the Git action, action/cache behaves as expected and they have no plans to tighten the scoping of the cache.
Although the GitHub team doesn’t consider this behavior problematic, we advise DevSecOps practitioners to be wary of this attack vector.
In this post, we outlined a cache poisoning attack in CI workflows, which is hidden from the eyes of the DevSecOps team, and discussed mitigations.
Mikey Strauss / March 30, 2022