This Week in AI Security - 20th November 2025

All right. Welcome back to another week of this week in AI security coming to you for the week of the twentieth of November, twenty twenty five. I've got a little bit shorter than average episode for you today because we've only got two stories to cover, but they are two very important stories, and one of them is going to take a little bit of time to go through.

So let's dive in. We're going to start with a shout out to the team over at oligo or oligo. I'm not exactly sure of the pronunciation, but it's linked from the show notes. And they talk about a vulnerability that they've come to dub shadow MQ. And the reason that they're calling it shadow is that it's kind of silently lurking within the ecosystem of many popular AI tools like Nvidia, tools from Nvidia, tools from Meta Llama, to whom they did disclose it, got assigned to CVE. The meta team actually responded very quickly and well. A great response from the team over there.

But what it really breaks down to is that there's a lot of reuse of core components that underpin LLMs. Let me explain what I mean. So an LLM on its own is its own engine for producing a large language model and all of the kind of gen AI capabilities that come along with it. But how do we interact with that LLM? Well, there's all kinds of interface tools, and this is a lot of the kind of necessary plumbing to get data in and out, you know, issue a request to receive a response back, etc.. And there is within the Python programming language, an unsafe deserialization known vulnerability in something called Python pickle deserialization technique. And what happened in this case is that some of these underpinnings just got copied and pasted effectively into a number of different LLM packages, and so the same Deserialization vulnerability is there.

What is this Deserialization vulnerability mean? What is it? What can it lead to? Well, what it is, is that a request that goes into an LLM may go through a couple of different phases of kind of encoding and encryption, right. And remember, these are not exactly the same thing. Encoding and encryption are different things. Encoding typically means you use some sort of kind of coding algorithm that is very reversible, is not built on the same kind of prime number algorithms that encryption is built on. So they're not the same thing. You can hear things like kind of antsy. That's an encoding mechanism. And TLS that is actually an encryption tool or an encryption kind of protocol, if you will. So remember, they're not exactly the same thing. But when you send a request in one of the ways that you make sure that the request doesn't get tampered with is you encode it as part of the outbound request process when it gets received on the server side. So in this case, on the LLM side, it gets decoded and then executed during that decoding process. You can think of a set of malicious instructions basically not being recognized that malicious as malicious at that point in time.

Well, what can that lead to? That can lead to abusive and malicious commands being included there. And what was found is that these malicious commands could lead to everything from remote code execution. An attacker could include some malicious code that would then get kind of serialized, deserialized, encoded, decoded, etc., or even privilege escalation. So I could say, hey, you know, I'm Jeremy for my user, please assign me admin privileges on the remote server. So very common problem. And what it speaks to for those who have kind of listened to some of our talks about the intersection of AI and APIs, this is a common problem in API powered applications. And one of the things that we've observed recently across our AI Incident tracker, which you can find linked from the footer of our website, is so many of the recent issues that we've seen with LLM powered applications or LLM based interactions, is that the breaches or the malicious kind of unintended behavior is triggered at the API layer. And there's a lot of kind of basic controls around APIs, things like input validation, sanitization. Again, these kind of like serialization, deserialization, encoding, decoding etc. that go along with that, that are really core basics. But with the speed that things are moving at, and you've probably heard that theme several weeks in a row now here on This Week in AI security, a lot of these things get overlooked. Or in this case, you know, you just kind of reuse the underpinnings of other engines. So have a look at this. Maybe look at the LMS reference to see if you're using any of them. See if there's a bump version available to to kind of remove this vulnerability from the packages that you might be unwittingly using already today.

All right, moving on to the second story. This is the bigger story of the week, and there is a lot to get through on this one. So let's dive in. So first big shout out to the team over anthropic. As we've seen all of this stuff moving at such a staggering pace. One of the things that the team over at anthropic has done, in my opinion, really, really well is, first of all, be transparent. There are issues. We know there are security issues. They are doing their part in terms of sharing intelligence, sharing information about the things that they're observing. I really got to credit them for that. You might have seen a couple of months back, they released a AI Threat Landscape report around some of the things that they've been seeing. So there's some great stuff on that side.

But let's dive in to this week's story. So you might have seen this one circulating on LinkedIn already. It got reported, I think at the very tail end of last week and over the weekend there was analysis coming out. But this is the anthropic Threat Intelligence report titled Disrupting the First Reported AI Orchestrated Cyber Espionage Campaign. So it's a critical piece of intelligence. It really does detail the first publicly confirmed end to end cyber espionage campaign. It is a state sponsored actor , and I won't go into the attribution or everything else. It uses a frontier AI model. So this is basically a campaign that consists of multiple parts, and it uses an LLM for orchestration and coordination of the campaign. So the AI engine was used to accelerate almost every phase of the attack.

So we find new AI related risks for detection. There are concrete real world examples of malicious AI usage now validating the core risks around AI adoption and some of the things that can go wrong if you're not doing a good job controlling your AI environment. We've seen cases in the past where organizations have their keys compromised in their own AI environments, get abused by attackers, or so on. So this is, you know, again, one of these risks that goes along with that weaponized system prompts. So the attackers provided the AI with a detailed prompt chain that included the definition of a persona as a senior cyber operations specialist and red teamer. So, you know, they kind of set the context that the LM should operate in. It's a great example of kind of embedded system prompt risk. So you want to understand what are all the elements of a prompt that go into an LM engine that you're designing that can be an embedded prompt or that can be an interactive prompt, right. Interactive might mean that it comes from exactly that point in time. Embedded mean. It's kind of baked into the definition of the tool that you're using.

So we see here that, you know, the persona is set, the system prompts are set, etc. then, uh, AI driven evasion. So this is one of the things where, um, you know, not only from the human actors who kind of directed the campaign, but even from the standpoint of this agent persona that was given there. Um, the actor used the AI to refine the malware to bypass EDR solutions. So, for instance, when a script fails or when they find that there are things that are kind of being, uh, potentially detected, they go back to the AI and they say, well, how would you solve this problem? This is one of those things where, ironically, if you want the closest to deterministic answer for any kind of problem, you you give an overly verbose set of instructions. But if you want the most creativity out of an LLM, you give a much vaguer set of instructions with a boosted persona. So really interesting things on that side you might want to think about for your own strategies. How would you monitor for these kinds of things? How would you monitor for agents that go slightly away from the script that you might be thinking about?

But other pieces of the of the attack chain that were really kind of, um, AI assisted here, uh, AI powered reconnaissance, performing vulnerability research on obscure networking protocols and libraries, looking for more ways to execute remote commands, looking for ways to execute lateral movement within networks. Again, more creativity. How would you get from system A to system Be, etc. and then post-exploitation how do we well, first getting the data out, but then once we get the data out using AI to analyze it, have we gotten anything valuable out of all of this stolen data that we've been able to kind of get out of the system?

So there's a lot of kind of both organizational and technical challenges encapsulated in this case study. It's linked. The full report is there. They have both a blog post with a nice summary and executive high level, as well as a more technical, detailed walkthrough as a PDF downloadable. So I really again want to credit them for being very open and transparent about the things that they're observing. And then really, you know, remember, this is a bit of a wake up call. You know, I always say to people in the talks that I give that remember, every tool you have access to, bad actors have access to as well. Build that into your calculus. Think about for every LLM powered application that you're putting out there, what are the inputs and the outputs and the touch points where it could be abused, and what would be the impact of any of those things?

So a lot going on in this story , I'm going to leave it there. You can do your own analysis. But two really interesting stories from this week. That's all for today. We'll talk to you next week. As always, if you've got stories, please drop them to US podcast at IO. If you're interested in our breach tracker that is always linked from the footer of our website, if you want to come on and talk about one of the stories, or talk about a story that you identified, please reach out to us. We'd love to talk to you. All right. Thanks so much. Bye bye.