This Week in AI Security - 23rd July

All right. Welcome back to another episode of This Week in AI Security, coming to you for the week of the twenty third of July 2026. And a little bit like last week, we've got a little bit of a lighter load, but we've got a couple of stories that I want to spend a little bit more time on. So without further ado, let's dive in and let's get through some of the quick stories first.

So first off is a report that OpenAI's latest AI model likely has similar cyber vulnerabilities to the one that led to US export controls on Anthropic's Fable, according to the British Standards Agency. So we're talking about a new family class, GPT-5.6 Sol, and it has cyber vulnerabilities that are apparently a risk and have prompted some governments to look at export controls on it. The flip side of this is that there are conflicting reports that actually say it's less about the vulnerabilities, and it's more about the cyber capabilities of the model. Just something to watch out for. And I do expect in the continued kind of geopolitical debate about keeping models ahead of, let's call them geopolitical adversaries. This is going to be a continuing theme. We're going to see this pop up again and again. And it particularly pops up in the realm of both vulnerabilities. So risks to infrastructure, communications, data sets, etc.. But again, especially on the capabilities side, with both sides seeming to want to have control of the most and, you know, the most cyber capable models for offensive potential offensive purposes, let's call it like that.

Moving on to the next, which is a related story. And a Russian hacker who goes by the handle of trim, has jailbroken Claude and turned it into a pen testing platform. And so what we're looking at here is we know that there's the Claude Mythos family, which is still under somewhat restricted access. The public version of it has limited cyber capabilities, etc.. But one of the things that we've talked about a couple of times in recent episodes is that these are guardrails, not so much the removal of actual capabilities from models. So these are things that, as we've also discussed, guardrails can be worked around with enough time and persistence. And so the jailbreak technique here is something called context warming. And some combined with something called ghost reset, which deletes sessions and reframes refusals as a network drop and reprompt. Now trim is claiming a ninety percent success rate in the combination of these two techniques. So think about it like this. I kind of warm the conversation by explaining my goals, my role, maybe providing some other kind of training capability around the prompt that I'm about to inject. And then I reset that conversation by saying, oh, I timed out. I had a network issue and kind of picked that back up. And it looks like there may be a vulnerability window in there where the guardrails don't kick back in.

Now, continuing on the topic of AI powered kind of vulnerability hunting, again, along the lines of Mythos and that theme of AI being able to find vulnerabilities in code and in applications very, very quickly, the team over at Capital One has produced an open source tool that they are calling VulnHunter, and as its name suggests, it hunts for vulnerabilities and pieces of software. So it is looking for, uh, code paths, identifiable exploits. It will also map out attack paths. It will propose targeted remediations. It comes from a developer first mindset. So it is looking across a lot of, you know, structural flaws and the business logic and the design of the software. It does currently require your own Claude Code environment and Claude Opus 4.8 or higher. It is open source on GitHub. And so I think this is actually a great thing. So teams that are constructing software, you know, definitely have a look at this. If this is something that you are interested in improving the quality of your software, then taking a little bit of a pause for a second away from the kind of hardcore technology aspect and focusing on some things around content.

There is a Nature commentary warning that hundreds of AI generated bird photos have been discovered on a website called iNaturalist that hosts six hundred and ten million images or more. And another site called Macaulay Library. The true scale. The number of photos is a little bit unknown. So these are fakes. There are AI tools that are designed to make photos look better, but they can also do things like superimpose birds in places that they don't belong, such as in this example of a toucan in the Antarctic. Obviously, a tropical bird from Brazil is not going to pop up there. This is much more about data integrity, but the the theme that I want to pick on here is that these AI editors are popping up for almost every type of content, everything from documents to images to videos, audio, etc.. The topic of deepfakes, we've talked about enough on the show here, but where I think this actually gets interesting is when you start thinking about kind of the circular loop here. Here's what I mean by that. Imagine that you wanted to train an AI model on bird photos today. Well, you would probably go to sources like these to look for training data. But if the training data is now polluted with AI, let's not call it hallucinated, but let's call it mocked or or faked data. What quality are you going to get on your training? And so that is something to keep an eye out for, especially if you or your organization are going down that path of building data sets for training purposes.

All right. Now, getting back to a couple of other things. One of the things we've talked about any number of times is it's not the AI itself, it's the infrastructure or the applications around it. So the ServiceNow AI platform has a flaw that has now been observed for exploit in the wild using unauthenticated code execution. Hat tip over to the folks at diffused who disclaimer are personal friends of mine, and they're confirming seeing this from some honeypots that they run. So this is a multiple sandbox escape gadget that chains together a number of vulnerabilities. Self-hosted instances of ServiceNow are the primary target. ServiceNow says on their hosted platform, they don't see any, um, any evidence of this. And it's actually a really interesting argument because there's always this tension around security for, uh, self-hosted versus SaaS based platforms and any number of risks. And I won't go into the pros and cons of the two, but, you know, there is potentially an argument to be made to looking at a vendor hosted platform as being the most up to date and, you know, quicker to patch than you might be on your own self-hosted version of a piece of software.

All right, now I want to get into the two big stories of this week where I want to spend a little bit of time. So we have talked about prompt injection on the show. I don't know how many times by now, and we've talked about many, many different aspects of it. We've talked about everything from kind of, let's say, clever but simple techniques to use hidden ASCII or Unicode characters. We've talked about emoji smuggling, where malicious prompts are embedded in emojis, where the LLM will actually interpret the text behind the emoji, rather than what's actually visible from the emoji's image file. And on those same lines, we've also talked about sources of malicious prompts in the general web. Anything from malicious prompts being embedded on web pages that AI agents are crawling to, anywhere in IDE environments where we see things like, you know, review comments, pull requests, uh, the, uh, payload query parameters in a URL where a malicious prompt has been embedded there and all of these different places.

Now, one of the most interesting kind of new representations of this is from a group of researchers from Seoul National University in South Korea. A paper published on July sixth identifying something that they are calling ADI. And what does ADI stand for? It stands for ETA agent Data injection. Excuse me, agent data injection. Let me say that one more time. And the main idea is basically like this. Just like with the emoji smuggling example we see on the screen an emoji file. Let's say a happy face. But behind the scenes, a lot of emojis are represented in markdown as something like, let's say colon, smiling face colon. And it's interpreted as that image file in the context where we're consuming it, let's say maybe in Slack or something like that. Similarly, when we look at a web page and we see a button like buy now, what we're actually seeing is a HTML rendering of the button. And then behind the scenes there is a button description. And also behind the button description, there's a hyperlink to what is supposed to happen. Now that could be a JavaScript action. Like add this item to the cart. It could be moved to another web page or a URL anchor or something like that. It has a lot of possibilities.

So now what they're finding is that, well, our eyes read what's on the page. The agent actually reads the totality of it. The agent will read not only the label on the button by now, but it'll read whether there is a button name in the HTML code or there is something in the button hyperlink that says ignore all previous instructions and whatever. And so they tested a few different behaviors, including things like click on next review, which would actually buy the book that they were looking at, and a couple of other things like that to really prove the point. So what's interesting about this, this is a form of prompt injection, sort of, but it's not really a prompt. It's again kind of hiding or embedding malicious instructions in various places that are out of human purview but are within the scope of agent purview.

So think about it like this. LLMs read the delimiter punctuation, sort of by guesswork. Our HTML browser reads them in strict format. You know, we look for an open bracket, a close bracket. We look for an open parenthesis, a closed parenthesis, an open quotation mark, a close quotation mark. All of those things have meaningful constructs to the web browser that is doing the HTML rendering and will affect, again the display of that buy now button that we see, but LLMs read through guesswork. So if something looks more or less accurate, it's going to pass down the page. But the LLM is going to read all of that. So a lack of escape quotes, a lack of close brackets, etc. is not going to be a problem for the LLM. They will see a structure and they will interpret that and they will then execute on that. So it's not really an architectural or a code level weakness. This is just kind of how it goes with text based instructions for LLMs going.

All right, moving on to our last story and really the biggest story of the week, if not the biggest story of the year to date so far. And that is the breach of Hugging Face, supposedly by an agent. And that does look to be confirmed as of the time of publication on July twenty third, 2026. So let's dive into this story with a little bit of detail here. So the story starts with the Hugging Face team disclosing a cyber event. And they published this on July sixteenth. So about a week ago now that they had a security incident and the security incident identified an unauthorized access to a limited set of internal data sets and to some credentials used by services.

So it started with an in the environment where the AI platforms are uniquely exposed, which is the data processing pipeline. And this is an environment where LLMs are ingesting data for training purposes, a malicious data set abuse two code execution paths in the data set processing. So this is a remote code dataset loader and a template injection in a data set configuration that instructs the LLM on how to ingest the data set. And inside there is a set of malicious prompts that runs code on a processing worker. In that set of instructions was an escalation to node level access, harvesting of cloud and cluster credentials and lateral movement, and descend into internal clusters.

Now, the interesting thing is that Hugging Face on their own reports that this campaign was run by an autonomous agent framework appearing to be built on a agentic security harness, where the LLM at the time of disclosure was not known. And we're going to come back to that point in just a sec. Now, Hugging Face took a set of remedial actions, including fixing the root vulnerability, which was something inside the data set code execution path that's been fixed. Not no disclosure on exactly what happened there. That's probably a piece of internal proprietary software. That's fine. They also cleared out any threat actor access rotated credentials. Et cetera. Et cetera.

Interesting follow up items. Now they talked about doing AI driven analysis of the 17,000 recorded events over the roughly two day period. And that allowed us allowed them to reconstruct the pipeline, extract the indicators of compromise, map the credentials touched and figure out what was real and what was actually decoy activity. And they did this in a matter of hours, and they did it through using their own agents. A couple of really interesting things. So one of the things that they tried to do was they tried to do this on frontier models that reportedly have the, the best cyber capabilities, but it actually failed because there was a lot of exploit payloads in the log files that they could view from that data pipeline environment with the compromised data that was injected in there. And what they found was a lot of like command and control artifacts and malicious payloads and things like that that kept getting blocked by the safety guardrails.

So what they had to do instead was actually run the forensic analysis on the GLM 5.2 model that comes from XAI, makers of the Chat Z or Chat Z platform. And this is an open weight model out of a firm from China. And I can't remember off the top of my head which firm is behind that model. So that was really interesting. They did mention that there is actually a secondary benefit because they used an open weight model that they hosted themselves, so they didn't have to send anything outside to another company. Everything constrained within their environment. So it's really interesting to notice that some of those guardrails can actually be limitations in a cyber forensic analysis.

And now moving on to things that we have learned since then. Now about five days later, the team over at OpenAI put out a blog post of their own announcing a partnership with Hugging Face to address the security incident during the model evaluation. So they basically disclosed, yes, it was us. And what they found that it was their new 5.6 Sol model, which we talked about at the beginning of today's episode or this week's episode. And the model was put into a training environment where it was specifically given tasks to evaluate its cybersecurity capabilities. So there's a common framework used for evaluating cyber capabilities of different model. It's something called Exploit Gym. We'll try to link to it from today's show notes, but it's an open source package with contributions from a number of researchers at the University of California, Berkeley, as well as from Anthropic. And it looks like participants also from OpenAI and probably other some leading AI research firms.

So Exploit Gym is given a set of tasks, and what it is meant to do with this is to go out and try to accomplish different things. So it's trying to accomplish advance exploitation, using complex attack paths to quantify the cyber capabilities. So these are one of the things we've talked about on the show in the past is that when you give agents more broadly scoped sets of instructions or goals without constraints on how they should accomplish them, you might experience interesting behaviors. And so in this environment, the new model spent a lot of time trying to find a way to open internet access to solve the evaluation problem. So the interesting thing was that the model found a zero day vulnerability in a proxy software in order to gain external internet access, because the model was supposed to be closed for evaluation purposes.

Next, the model performed a series of privilege escalation and lateral movement in the research environment in order to get to that internet access that it had now opened up via that zero day exploit. Then it found that Hugging Face was probably the target that it should go after, and it probably guessed that based off of previous research that's in the training data set that says like, hey, Hugging Face has a repository of lots of models that you can go to use for various purposes. So the model searched for and found ways to gain access to an environment inside Hugging Face. And it's unclear at this point whether it gained the Hugging Face access via a zero day vulnerability. Um, in fact, I correct myself. It is known that it used stolen credentials in zero days to find a remote code execution path on Hugging Face that has been disclosed to the Hugging Face team as well. And the interesting thing is that this was actually discovered by OpenAI through an anomaly investigation. All of this has been communicated. And the interesting thing is now obviously these firms are cooperating on it.

Um, so the really interesting things to go into here, many, many interesting things to go into. So let's try to take them one at a time. First is kind of the goal setting versus constraint aspect of this. And we've talked about this a little bit. So I won't dwell on this point too much. But if you give an agent a goal without a lot of constraints or instructions, depending on the capability of the agents, it will either be super creative or it might be limited creative. It might burn through its entire token budget. All of that really depends. And so that depends very much on the environment in which you run the agent, the LLM configuration behind it, etc., etc.. So that's the first obviously in this case, the agent was given a very broad goal, almost no constraint, and basically an unlimited token budget as well as unlimited tool utilization. Because if you think about what goes into discovering and exploiting a zero day, there's a lot that has to happen there. So there needs to be the discovery, but there also needs to be the crafting of the exploit, the creation of an environment to then run that exploit in order to trigger that vulnerability in the proxy server and gain that external internet access, as well as all the network movement that needs to happen through there. So think about what is your agent environment configuration? What are the tools access? Do you have limits on access that could limit the blast radius, etc.?

Second is in this evaluation from both sides. What's been really interesting to see is that both sides actually realize the value of the telemetry data. So in the case of the Hugging Face response to the intrusion, what they found is, you know, 17,000 log files in a pretty short period of time against a very limited environment. That's a lot to get through. But obviously the agentic processing allowed them to get through that very, very quickly. But the key thing here is they had the data, they had the log files, they had all the telemetry on the agent execution that was necessary in order to do that evaluation. That's really the key point. And in fact, it's mirrored on the other side by OpenAI and their report that they looked at this from an anomaly perspective, and they saw an anomalous utilization, whether that was the token spike, whether that was the tool invokes unclear. And they're not disclosing that. And that's perfectly fine. But they did disclose that it was through the anomaly behavior or the anomalous behavior analysis and identification that they looked at.

And then the last thing to really think about is how are you going to evaluate your agent? And so, you know, we talked just a minute ago about telemetry from a cyber forensics and investigation perspective, but there's a secondary part of this, which is, you know, at the end of the day, this Exploit Gym is used to measure the cyber capabilities of a model. How do you measure the cyber capabilities after the event? Do you do it through something like capture the flag and you see like, okay, you had a set of tasks, how many of them completed correctly? But that's not going to be the end goal of a lot of agents that are being built by companies today. A lot of companies today are going to be looking at, hey, I have an agent for a very particular business process, and what I want to do at the end is evaluate whether my agent successfully completed that business process. And so you think about the constraints on the environment of the agent, but you have to think about how do you measure the agent. We've talked a lot about prompt open endedness. And if you're interested in that topic, please do hit me up. I'm happy to discuss it. It's an area we've been doing a lot of research into over here at FireTail. But it is something that I think a lot of organizations are not necessarily thinking about right now. You've got a goal. How do you measure the goal? How do you gather the data that allows you to measure that goal? And it's just in my mind, reinforces the need for AI observability and collecting all that log data at a layer where you can actually do meaningful analysis on it. If you're collecting a lot of, let's say, like network or application telemetry. That may be well and good, but if it doesn't have a lot of the LLM or application payload level kind of information, then it's really less useful for evaluating that. And similarly, from the Hugging Face side, they clearly had that data because they saw malicious prompts in log files, which would again exist at the application layer, not so much at the network layer. And that's why the frontier models, commercial frontier models blocked a lot of that analysis.

So a ton to get through here. I think all in all, I would have to agree with a lot of the analysis that has been out there that this is kind of a watershed moment. And it really does prove that we are entering an era of of potential identity attacks on infrastructure. And for organizations that are, you know, about to unleash agents of their own on their environment, whether those agents may get, you know, let's say, out of your environment by exploiting a zero day with the cyber capabilities that the OpenAI 5.6 Sol model had, maybe that's unlikely for your environment, but what is it to. What is there to constrain it from potentially accessing data on the network where it exists that it might think is useful or not really think, but it might kind of calculate probabilistically, would be useful for accomplishing the task that it's been given. So this is where I think, again, you really have to lean into that observability aspect. And then doing some kind of post run analysis of what your agent performance looks like.

So a long episode. In the end, this may end up getting broken up into two episodes. So you may be listening to part two of this already, but I thank you for your time and patience. We're going to follow up this story as new news breaks around it, and we'll try to publish some of our own thoughts about how observability really supports your ability to move forward on any kind of LLM based application, whether it's agentic workflows, etc., for your own environment. And if nothing else, thank you so much for listening to this week's episode or episodes, plural, and we will talk to you next week. Bye bye.