This Week in AI Security - 15th October 2026

All right. Welcome back to another episode of This Week in AI Security, coming to you for the week of the fifteenth of October twenty twenty six. And we are back on our kind of normal service here, meaning fifteen-ish minute episodes. We've caught up from a little bit of the backlog, but we've got some really great and interesting stories to go through this week across about three different themes that we're going to get into, a little bit of the kind of political, legislative, regulatory side, a little bit of attacks, and a little bit of more of what we've seen over the last few months. So without any further ado, let's get started.

First, we've got a story. Attackers are using custom GPTs to implement ClickFix attacks in a new RAT campaign. And RAT campaign here is remote access Trojans. And this is threat actors abusing custom GPTs, GPT, general purpose transformer, to deliver remote access Trojan malware through ClickFix attacks. So it highlights how trusted AI platforms are becoming a vehicle for social engineering techniques. Kudos to the team over at Huntress for kind of identifying this. So they talk about threat actors building malicious custom GPT experiences that funnel users into ClickFix workflows to deploy these remote access Trojans. ClickFix is two years old and unsophisticated. And so what it does is it disguises a malicious command as a CAPTCHA or a routine verification step, and persuades the victim to paste it into their own terminal. So that'll be just kind of a string of characters and text that to a lot of users will kind of look like, you know, mumbo jumbo. And so they think, oh, this is just one of those, like, are you a human verification routines? So I need to just, you know, click command, run in terminal or whatever the case is. I can't remember the exact hotkey on Windows right now. Paste this in and click enter. And so what ends up happening is that the user is the one actually installing the payload. It shifts all the heavy lifting to the victim, bypasses controls designed to stop unauthorized software installation. This is not a novel attack chain. The GPT is actually just a wrapper here, so it leans on the credibility of the ChatGPT ecosystem. And so that's why it specifically calls out custom GPTs. So these are like kind of a lookalike domain type of thing. Um, and trust is really like the kind of boundary that gets abused in these types of attacks, where users will look at it and be like, well, it looks like ChatGPT, feels like ChatGPT, I trust ChatGPT, therefore that's fine. All right. Moving on to our next story.

And we've got an Amazon Bedrock AgentCore vulnerability, I would call it a vulnerability. Kudos to the team over at Zenity on this. So they found that in AWS Bedrock AgentCore, and that is the platform on Amazon Bedrock, and Amazon Bedrock is kind of the AI as a service offering on the AWS platform. Uh, but an agent in the AWS AgentCore runs in a Firecracker VM that didn't sufficiently isolate the network. And that makes it so that any tool that is able to make an HTTP request to the agent turns the agent into a server side request forgery primitive. So if you ask the agent to fetch 169.254.169.254, which is a localhost IP address, you can actually return the temporary IAM credentials of the execution role from the instance metadata service running from that microVM. And that default execution role is generally scoped to have access to all AgentCore resources in the entire account. And that is very often due to the ease of use kind of trade off that is very often made. And it's something that we've pointed out before, that it's not always the AI itself or the LLM that gets abused. Our previous story this week also kind of highlighted that it's a lot of the plumbing or infrastructure around it, or, in the previous story, the trust around it. And so when you think about this trade off between ease of use and security, what you're observing here is that, you know, kind of the all AgentCore resources permission is granted so that as you're building this AgentCore agent, you don't run into continuous denies on various credentials being blocked from accessing, let's say, access to read a file out of an S3 bucket or access to start another subprocess or whatever. And so by getting that set of stolen credentials off of the instance metadata service through that server side request forgery request, those credentials allow for enumeration of every single agent, pulling the ECR image, reading the source code, invoking other agents, reading private conversations across every agent, stealing Secrets Manager values, and creating events on Bedrock AgentCore memory to plant memories that could persistently hijack agents' future behavior. So the disclosure timeline was really, really interesting. Reported on the twenty fifth of December twenty twenty five, so on Christmas Day. IMDSv2 was enforced on fourteenth of February, and AWS reported that they were actively working on the role on the twenty fifth of February. The report was closed as, quote, informative, end quote, on the twelfth of April. And on the twenty second of June, the default role permissions were still unchanged. So that's six months from disclosure. So also kudos to the guys over there for, like, working with AWS to not put this out into the open, because typically that is a generally accepted best practice timeline where you can kind of go live with this. Substantially hardened as of the twenty ninth of September. AWS then told The Register that Zenity, quote, misinterprets documented behavior as a vulnerability, end quote, and that a developer error would be needed. So meaning that the AgentCore agent would need to be misconfigured in order to enable this behavior. I would say that is somewhat disputed from the disclosure that comes out here. But again, just pay attention to the plumbing around this for some of the vulnerabilities that can be present in those environments. All right, moving on to our last kind of set of stories for this week. And this is around agent behavior, breakouts, etcetera, etcetera.

So we've got a story out here actually documented by the company to whom it happened, XMTP Labs. Their CEO, Shane Mack, who built a personal CFO agent on Grok in late August for a monthly financial review. He gave it read only access to his checking and savings, and told it to message him in a private group chat he'd named my personal exec team. On Thursday, the first of October, the monthly report ran, posted the report into the XMTP company Slack channel exec team, instead of posting to the my personal exec team. His head of product DM'd him, quote, hey, heads up, I think you meant to post the XMTP bank balance, end quote. The colleague first read it as company financials. Uh, it was actually personal financials, because there was a line item in there about building a barn on his property and blowing his monthly spending target. Uh, also exposed were his checking balance, his savings balance, his biggest expenses, and his over target spend. And the root cause, as per Grok's own investigation, quote, it was doing exactly what I told it to do, end quote. Sorry, quote, it confused the destination, end quote. Two chats with effectively the same name. And although he'd built several separate agents, quote, underneath the hood they are actually all the same connections, end quote. And that last bit I will say is true, and it's something to very much keep in mind around connecting tools to agents right now, which is to say, once you build a connector between an agent that you have and a set of tools that it might be using. So in this case, Grok to Slack. Or, for instance, here at FireTail, we do a certain number of workflows and we have connected, for instance, our own internal agents to Slack. The permissions that you grant there are permissions that govern that entire connection. A lot of these connectors don't have the granularity to go down to the level of being able to say, you can post into channel one but not into channel two. The interesting thing was that the read only permission wasn't honored. The destination wasn't honored either. So we spend a lot of our time on what agents can read, and not a lot on where they can write to or what they can do. But I would say that that's going to become an increasing area of focus moving forward. So a little bit of a learning lesson here, a story that we should all kind of take a look at and then ask ourselves, as we're deploying these agents, especially for projects like personal productivity, what does it have access to? And governing all of that becomes a real strong question in a lot of organizations moving forward. All right. Moving on.

We have a pretty interesting story out of the UK around AI systems covering up their misbehavior. And this one gets a little bit technical, so bear with me for a second. So an organization called METR tested whether an agent inside an evaluation environment could edit the transcript that a human reviewer sees. So basically what's going into the logs, which is what a human could potentially review afterwards with the help of AI. A researcher found a vulnerability in Inspect, which is the UK AI Safety Institute's framework, which is widely used across the safety evaluation ecosystem, in about ten minutes. This vulnerability is a MathJax ref expression that breaks out of its attribute to inject an SVG animate element with an onbegin handler running the agent's JavaScript in the viewer's origin. All of that is to say, basically, it can rewrite what appears to you versus what is actually stored in the system. So it can rewrite displayed actions and intercept the viewer's own download button. And just to be clear here, this is a proof of concept and not an observed case. But what is interesting is when you put this in the context of some of the other cases that we've discussed recently, things like disclosures around the open face event, things like the continued, um, disclosures that OpenAI is having to make, again, to other companies whose environments might have been, um, let's say, intruded upon by various agents in safety evaluation environments. Um, looking at the logs is actually the key thing. And if you combine that with another topic that we've touched on any number of times on the show, which is that anywhere that you can have text, that text becomes an attack vector. And so the observability logs can also become an attack surface. So METR explicitly says that it has not seen agents exploit this in any evaluation. It only changes what the viewer rendered. It doesn't change the underlying database record, that stays intact. METR has now patched this, and now shipped an untrusted mode that disables rendering of agent output, so you can know for certain that this could not be intercepted. And the architecture here is really the key thing. Like I said, we've talked about any number of text based environments where, you know, whether it be logs, whether it be commit comments, whether it be titles of commits, etc., that can become an attack surface in that. So treating text as inputs, as part of an attack surface or as part of a threat model, becomes a really compelling environment. And again, when we see instances where AI agents have moved forward to try to satisfy a goal without being given any constraints, that is where we see them blowing past security guardrails and blowing past ethical guardrails and things like that. So that's where this risky behavior comes up. The logs are the only place that you can detect it. And if there were something aligned to the agent's goal that convinced it that it should actually manipulate the logs, this is proof that that is mathematically possible. Now, this will probably get fixed by others, and so it's great disclosure to have out there into the community for other companies who are building around that. All right, moving on to our second to the last story for the day.

This is a story from Anthropic around Anthropic now pulling live internet from every internal evaluation until it can confirm its monitoring will catch this class of behavior. This was found by a transcript review begun in July. Claude Mythos Preview hit an error on a university hosted analysis tool, explored the site, found a script that would return any file, pulled that script's own source code, spotted an injection flaw in the source code, and used it to run commands on the server in order to finish the assigned calculation. So it very much aligns with what I just described. If there is a goal that an agent is trying to reach, and it doesn't have constraints around what is and is not allowable behavior, this is what an agent can do. So just to be clear, this is not a jailbreak of the LLM. This was an agent actually following its goal oriented behavior very much as it was told to do. Mythos 5 also read the settings file of a local government map site sent to browsers, lifting working access tokens, querying the back end directly. Other events from Anthropic, Opus 5 and Mythos 5 both used free URL shorteners to beat fetch tool length limits. The da.gd operator emailed Anthropic about it before Anthropic cut it on their own, so that is the URL shortener that they used. Some affected sites were US government, federal and local. The White House was briefed. Anthropic said, mission is, quote, alignment training is not yet sufficient or fully robust on its own, end quote. They had focused alignment work on coding and are only now extending it to search and computer use. So that's that tool invocation that we talk about in agent environments. The human cost of this, on the eighteenth of July at eleven twenty seven p.m., Haiku 4.5, which was told to interact with randomly selected websites, landed on PhillyUnsolvedMurders.com and submitted an invented eyewitness tip to the Philadelphia PD on an unsolved homicide, citing a description the page never contained. So this is one of these, you know, so-called hallucinations, which is effectively, as one of our previous guests on Modern Cyber labeled it, don't call them hallucinations, that lets them off too easy. This is a bug. This is a flaw in the LLM that did this. Anthropic didn't discover this until the twenty eighth of September, a seventy two day blind spot or lack of disclosure. The tip was spam filtered so investigators never saw it, kind of containment by luck more than anything else. The Philadelphia Police Department said, quote, the two month delay in detecting and reporting the incident to the city is unacceptable, end quote. Philadelphia Police Department has never been an Anthropic customer. This is the first story, though, where the blast radius lands on someone who isn't a user. So really kind of just interesting set of behaviors here. And so Anthropic is effectively, you know, doing a lot more investigation into unintended moral actions in their own training and implementing some controls around that.

And that really ties us into our last story of the week, which is four hundred and twenty signatories from thirty four countries, led by Professor Toby Walsh of the University of New South Wales AI Institute. It also includes some real luminaries in the AI space, Geoffrey Hinton, Yoshua Bengio, Dan Bowness, Renee Devdas. They have three core asks. Organizations testing cyber capable models should default to air gapping. Second, an international body should be established to standardize containment and alignment practice for less restrictive setups. And third, containment failure incidents must be reported to that body and eventually made public for learning and accountability. So Professor Walsh's framing is deliberately unglamorous. Quote, we wouldn't have had the Hugging Face attack if they just air gapped the test environment from the internet. We wouldn't have had the Medicare hack if they'd been careful about giving the agents minimal permissions for the task it was needed to do. I mean, these are all incredibly basic things, end quote. And that is true. These are all very, very basic cyber controls. When we talk about cyber first principles and fundamentals, we talk about network micro-segmentation, we talk about least privilege. These are all core cyber concepts for any environment. And the letter really refuses the usual either or. It argues for both better alignment and better sandboxing. And that either or, in case you're curious what that means, it's really kind of that trade off question. It's like, either we can get rapid advancement on AI agents, or we have to do all the cyber testing step by step along the way. And they're arguing that basically both are possible.

So that is the story to leave it with. If you're curious about any of those, as always, we'll have links to the story in the show notes for today's episode. We will also pick a story from this week to do a little bit of a deeper dive on. So stay tuned for that. And if you are worried about your agents going off the rails and you not having the logs to detect them, please do definitely talk to us here at FireTail. We can build you that AI agent audit trail that you're looking for and help you analyze and catch any of that behavior so you don't become one of these companies as well. With that, we will say thank you so much for listening, and talk to you next week on This Week in AI Security. Bye bye.

‍