Agentic AI systems can be used to do a variety of things autonomously on behalf of a human user: open or manage bugs, generate code, submit pull-requests, and (apparently) even complain about rejection. In May, a Fedora developer discovered that an allegedly rogue agent had been pestering the project in a number of ways: reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer. It also submitted a number of pull requests (PRs), some accepted, to several upstream projects. The Fedora account associated with the agent has had its group privileges revoked and the messes have been mopped up, but the motive behind the agent's actions is still a mystery.
On May 27, Adam Williamson copied
Fedora's developer and testing mailing lists on a message to Nathan
Giovannini about what appeared to be an unsupervised agentic AI system
under Giovannini's control. "It's great that you're trying to fix
things, but the results seem to be kind of erratic.
"
Williamson said that he was still looking through the history of
Giovannini's actions in Bugzilla, but had already spotted a number of
problems. For example, Williamson had found dozens of instances of
Giovannini's agent assigning Bugzilla entries to his account after submitting allegedly related
pull
requests to upstream projects, or closing
a bug after a PR was merged
into an upstream project. In some cases, the agent simply closed bugs
with comments
that either restated the original bug or were, as Williamson said of
this comment,
"superficially plausible, but problematic in other ways
".
In addition, Williamson said that Giovannini (or his agent) had
submitted patches that were incorrect and then "replied to
objections with LLM-generated justifications that eventually
overwhelmed the maintainer into merging the fix
". The agent, as
GitHub user "nathan9513-aps", had
submitted a pull
request for the Anaconda
installer used by Fedora and other Linux distributions. The PR's
description claimed it was a fix for an Anaconda
bug that would cause installation to fail, but the patch actually
preserved a kernel option passed on the command line that seemed to
have nothing
to do with the actual bug.
The agent's GitHub account has since been disabled. It now shows up in conversations on GitHub as "ghost", which is the platform's default placeholder for user accounts that have been deleted. Thus, it is difficult, if not impossible, to piece together a full trail of all the agent's actions on GitHub.
Williamson said, rather diplomatically, that the agent's actions were not
"having a positive impact on Fedora or the upstream projects
",
and suggested that Giovannini adjust the agent to be "substantially
less autonomous
". He specifically asked that the agent not assign
bugs to Giovannini, change their state, or "post confident
assertions or specific action recommendations
" without human
review.
Later on May 27, Williamson said
that Giovannini had replied to him privately to say that his
credentials had been compromised and that he was not the one behind
the AI system. "Obviously we should therefore treat any actions it
has taken with suspicion
", Williamson said. He planned to review
the bugs touched by Giovannini's account "even more
aggressively
", and asked for help from others to review them as
well.
A reply
later that day, ostensibly from Giovannini, said that he was able to
regain access to his GitHub and Fedora accounts "and I am currently
securing and reviewing all involved systems and credentials
". The reply
said his GitHub account was "nathangiovannini99". Williamson
replied
that the GitHub account was only an hour old, and that the recent
emails to the list and sent to Williamson privately did not seem like
messages Giovannini had sent in earlier interactions with the
project.
Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
Williamson said that he had reviewed account activity in Bugzilla by "nathan95" from this year, and found suspicious activity, such as severity and priority changes to a bug with no justification, beginning on April 7, in bug 2416721. Activity before that appeared legitimate, he said, and none of the activity that he had seen so far looked outright malicious.
He also identified another GitHub account, "leurus27-boop", as likely being associated with the same agentic AI. That account is still active, and has submitted a PR to the openSUSE Commander (osc) command-line interface for the Open Build Service as well as a PR to the lxqt-policykit repository. That project is used to extend the privileges of the LXQt desktop's lxqt-admin GUI tools for administering operating-system settings such as user and group configurations.
Williamson said that it would be good to look
through any other actions by the related accounts and warn other
projects that they should review anything that had been submitted by
them. Williamson seems to have followed up on each PR to warn
other maintainers "the whole situation is extremely
fishy
". Kevin Fenzi said
that he had removed the nathan95 user from any groups it had been in,
so it should no longer have the permission to reassign or close
bugs.
Martin Kolman, a member of the Anaconda team, said
the events were "really problematic
" even if not malicious. The
team had spent a lot of time reviewing PRs from what seemed to be an
eager contributor: "while it started to look off after a while, all
the replies were still like this - a bit weird, but still
*plausible*
". He also theorized that it could be an attacker
working their way up to malicious activity, much like the XZ backdoor:
Unfortunately, for an actual attack the preparatory phase could (and for the Xz attack did) look very similar - a new contributor slowly gaining trust in the community, getting in harmless changes and building up to the point when the attack payload can be injected (or the changes not actually being harmless if combined the right way).
So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here.
Chris Adams said that the commit to Anaconda should be inspected and probably reverted immediately. Kolman replied that it had been reverted. He also confirmed that the LLM-generated PRs had made it into the Anaconda 45.5 release on May 26. They were reverted in the Anaconda 45.6 release on June 2.
The targets certainly suggest that it may have been a prelude to an attack of some sort; an operating-system installer, a utility for escalating user privileges, and a tool for interacting with a build system all seem like promising avenues for inserting malware or hijacking systems.
It's disconcerting that what appears to be an AI agent has had so much success after gaining access to a human contributor's accounts. It seems that an AI agent with access to an account with a legitimate history of interacting with projects stands a good chance of persuading busy maintainers to accept questionable contributions. Happily, Williamson caught this before it became a bigger problem. Let's hope that other human maintainers are as observant.