Continuous integration and continuous delivery (CI/CD) misconfigurations discovered in the open source TensorFlow machine learning framework could be used to orchestrate supply chain attacks.
In a report released this week, Praetorian researchers Adnan Khan and John Stawinski said that attackers could abuse these misconfigurations to “compromise TensorFlow’s build agent through malicious pull requests, thereby targeting TensorFlow on GitHub and PyPi.” version for supply chain disruption.”
Successful exploitation of these issues could allow an external attacker to upload a malicious version to a GitHub repository, obtain remote code execution on a self-hosted GitHub runner, or even retrieve the tensorflow-jenkins user’s GitHub Personal Access Token (PAT).
TensorFlow uses GitHub Actions to automate software build, test, and deployment pipelines. A runner refers to a machine that executes jobs in the GitHub Actions workflow. It can be self-hosted or hosted by GitHub.
“We recommend that you only use self-hosted runners with private repositories,” GitHub states in its documentation. “This is because forks of public repositories may be used to create pull requests that execute code within a workflow. Dangerous code is running on the managed runner computer.”
In other words, this allows any contributor to execute arbitrary code on a self-hosted runner by submitting a malicious pull request.
However, this does not pose any security issues for GitHub-hosted runners, as each runner is ephemeral and is a clean, isolated virtual machine that is destroyed at the end of job execution.
Praetorian said it is able to identify TensorFlow workflows executed on self-hosted runners and subsequently find forked pull requests from previous contributors that automatically trigger the appropriate CI/CD workflows without the need for approval.
Therefore, an adversary who wants to trojanize a target repository can fix a typo or make a small, legitimate code change, create a pull request for it, and then wait until the pull request is merged before becoming a contributor. This will allow them to execute code on the runner without raising any red flags by creating malicious pull requests.
Further inspection of the workflow logs revealed that not only was the self-scrambling executor non-ephemeral (thereby opening the door to persistence), but the GITHUB_TOKEN permissions associated with the workflow had broad write permissions.
“Because GITHUB_TOKEN has Contents:write permission, it can upload the version to https://github[.]com/tensorflow/tensorflow/releases/,” the researchers said. “An attacker who compromises one of these “GITHUB_TOKEN” could add their own files to the release assets. “
On top of that: Write permissions can be weaponized by secretly injecting malicious code into a feature branch and merging it into the master branch, pushing the code directly to the TensorFlow repository.
That’s not all. A threat actor could steal the AWS_PYPI_ACCOUNT_TOKEN used in the publishing workflow to authenticate to the Python Suite Index (PyPI) registry and upload a malicious Python .whl file, effectively poisoning the suite.
“An attacker could also use permissions on GITHUB_TOKEN to compromise the JENKINS_TOKEN repository secret, even if the secret is not used in workflows running on the self-hosted runner,” the researchers said.
Following responsible disclosure on August 1, 2023, project maintainers addressed the flaws on December 20, 2023, requiring workflow approval for all forked pull request submissions and changing the GITHUB_TOKEN permissions to only Read to run on a self-hosted runner.
“As more organizations automate their CI/CD processes, similar CI/CD attacks are increasing,” the researchers said.
“AI/ML companies are particularly vulnerable because many of their workflows require significant computing power that GitHub-hosted runners cannot provide, hence the popularity of self-hosted runners.”
The revelation comes as two researchers revealed that multiple public GitHub repositories, including those associated with Chia Networks, Microsoft DeepSpeed, and PyTorch, were vulnerable to malicious code injection through their own GitHub Actions runner.
