This Week in AI Security - 4th December 2025

All right. Welcome back to another episode of This Week in AI security, coming to you for the week of the fourth of December twenty twenty five. We've got three stories to get into today. Well, two stories in one event, so let's dive in.

First is a recent Mixpanel security incident that affected OpenAI. And at first glance, when you hear the name OpenAI, you think, oh, this is probably really bad. This is a breach of probably or arguably the most popular AI platform that is out in use today. But when you peel back the layers, there's only a couple of things that, to me really kind of stand out from this event. Number one is this is a third party incident. So this is really kind of what you consider third party risk affecting a particular organization. Number two, kudos to them for disclosure and transparency around the incident and good disclosure around what data was and was not involved here. So the disclosure from OpenAI makes it clear that, yes, customer account information might have been made available, such as users email addresses, other PII around that, but prompts and data that were accessed or that were fed into ChatGPT and other open AI systems was not impacted by this. So vendor breach. Third party breach? Yes, you can put it in that category. The other thing we'll call out from a retail perspective is just as we have mentioned in many, many previous episodes and many, many previous discussions of AI related data and breach incidents, the API was the attack surface yet again in this incident. So check this out. Nothing too big to see here, but just a reminder you're integrating these third parties. Almost no service goes online today that is self contained. There are always API based integrations into third party services that makes them a popular attack surface for AI.

All right, moving on into the second story for today. This was a really fascinating academic research paper we've highlighted here in the visuals. If you're looking at this on YouTube or if not, I would encourage you to jump over and have a look at this over there. Or watch it on the website where we highlight an ASR, and you might remember that term from past episodes that the attack success rate of the various malicious prompts, using poetry as the vehicle for introducing them. And the title of the paper, I think is adversarial poetry with AI systems, or with Llms or with ML systems. And the point of it is this, you know, we've talked in the past about things like the hamburger, the so-called hamburger attack, which really kind of embeds a malicious prompt in a inside a series of five requests. And typically in the hamburger attack, you would see that being like five in five different languages. And the theory is that by the time the LLM gets to prompt number three or four, which might be the malicious one, the guardrails in the context are kind of forgotten in a way by the LLM. And effectively in this paper they're laying out. Successful demonstration of the fact that this works with poetry as well. You don't even need to use five different languages. You just embed malicious prompts somewhere in a poem, and you kind of change the context of the prompting and of the token consumption of the LLM. And so the guardrails again tend to get kind of forgotten or bypassed, or they're somehow no longer in effect.

What really stands out to me from this is just how big the difference is between the baseline ASR, where malicious prompts are just given as straightforward malicious prompts, and where they are poetry, ASR, where the malicious prompts are embedded somewhere inside a poet, inside poetry as a delivery vehicle. Thankfully, I would say that the two most popular llms being adopted by enterprise, which are those from OpenAI and anthropic, are actually lowest on the list for the difference in the ASR and the overall ASR rate with kind of, you know, three to seven percent increase. Um, that does mean, however, that three to seven percent of the time, more frequently, those malicious prompts would be successful against llms from those providers. But some of the providers in here, the difference is as high as sixty percent plus, meaning malicious prompt on its own might only succeed ten percent of the time, but a malicious prompt embedded in a poem might succeed up to seventy percent of the time. So really fascinating stuff. And we, you know, we've covered some of these research papers over the last several weeks around prompt injection problems, around the math behind kind of the translation from natural language into the token consumption, and how there's a lot that gets lost in there and how, based on the mathematical principles, prompt injection can always be successful because of that very thing. And this is just kind of proof of concept that this is a very concrete example of a way to embed malicious prompts. This is a delivery vehicle that will succeed more often than without it.

All right. And then moving on to the last story for today, which is the one I'll probably spend the most time on. I was fortunate enough to be able to attend the twenty twenty five Ascent Community Summit, advancing AI security. This is a summit run for the first year, co-hosted by Paladin Global Institute and Paladin Capital. Disclosure Firetail is funded by Them and Georgia Tech and the Georgia Tech Research Institute. There were a number of great speakers from different organizations. Certainly Georgia Tech and the academic world were well represented, but I also wanted to call out the fact that a number of public service and public sector institutions were involved. Everything from the Department of Energy, Department of Defense, etc. all of those had some representation, whether current or former representatives, Pacific Northwest labs, any number of other institutions. And the event itself really focused on kind of a what's the current state of affairs for AI security? B what are some of the kind of underlying assumptions that we as a community have around access to AI systems and how we should be using them? And then c focusing on kind of three critical industries. And in this case it was defense. What I will broadly term infrastructure. There was talks around electricity and water treatment and all of these kind of critical infrastructure components and then healthcare. And certainly I would think I think we all agree that these are three very, very important sectors to focus on all areas where AI can have a massive impact, but also areas where there's critical data at play and where the risks are actually really high.

So I'm just going to share some of my thoughts from some of the observations. And some of the talks were live streamed and others were really under Chatham House rules where, you know, unfortunately, I won't be able to share anything from the speakers who requested kind of in the room availability of the content that they were, that they were sharing. So a couple of thoughts at a high level. Of course, a lot of the discussion was security of the AI systems themselves, securing the usage of those AI systems and then applying AI systems for security. I'm not going to go too deep into any of those themes. I think anybody who's listening to this show or watching this show will probably have already thought about or really explored those areas, but it was great to just kind of frame the whole day with that backdrop. Always think every step, every sector, every potential data source. You know, all three of those things need to kind of come into account.

One of the other really interesting things was thinking about what are the capabilities required in order to benefit from AI. And, you know, Michael Steed with his keynote address, he really talked about some of them, and I captured four of them. I think I might have missed one, but they're they're really kind of four key capabilities. Number one is math. These LLM systems are built on math. And we've talked about that on the show before. So I won't go into that too deeply. Number two is compute. You need a broad compute infrastructure in order to run LLMs and AI systems. And so you've got to have those as kind of baseline technical requirements. And then three you need data. It's broadly assumed, although not widely confirmed, that most LLM systems have trained on every volume or every piece of publicly available data in the world or on the internet, and most have also trained on private data sets. If you think about something like a meta llama, it has access to Facebook data, for instance, that probably outsiders wouldn't. Right. So there's there's training data of all different types that have gone into the various engines. And number four is people you need to have the people to run these systems and to actually know how to leverage them for best possible effect. So without those core requirements, you can't get very far.

So just a good reminder and then some positives around some of the recent developments coming out of the public sector, like the appointment of a new national cyber director , the discussion around the formation of a formal cyber force. And this is something that, you know, Mikko Hypponen and I have talked about before as to whether this is going to come into being, broadly speaking, or perhaps on a state by state basis. I think it's fair to say, though, that once one or more of the call them the global kind of superpowers, if you will, or whatever passes for a global superpower these days does implement a cyber force. It won't be long before all the others do as well. And then three like things are just moving so much faster. And we've talked about that many, many times on on this week in AI security. So we won't get into that, focusing on some of the critical industries.

Some of the notes that I took are there's a lot of talk about the criticality of the supply chain for AI. So think about those things that we just said and what goes into them. So I need compute infrastructure. Well what does that mean. It means we need data center infrastructure. Well, what does that mean? It means I need the real estate. I need the facilities. I need the buildings. I need the electricity. I need the cooling, I need the bandwidth. I need all of those things in order to power and run those data centers. And that is having some societal impact in in areas where real estate prices are going up, where the price of electricity is going up, where the grid might be locally tapped out to fuel the demand for these data centers, and where things like noise and continuous hums are starting to cause societal effects on the communities around them. So kind of an unintended consequence of the AI boom that's going on right now.

Another note on that topic was that one of the speakers pointed out that AI is very, very heavily venture capital subsidized right now. We will never get access to AI systems as cheaply as we are getting it right now. So that does kind of motivate a lot of organizations to think about how can I maximize what we might be able to get down the road by investing right now. To that note, there was a lot of discussion around a topic that we've discussed on modern cyber in the past. Who who has the upper hand with AI being so widely available? Is it attackers or defenders? And a lot of thought around, you know, the attacker mindset being more creative and more kind of focused on overcoming obstacles in order to get the data that they're looking for, as opposed to a defender mindset that might be a little bit more. Check the box, or only thinking about the attack vectors that are present in the threat model that's being examined. So, you know, a lot of interesting discussion around them, around those topics.

Some other things that came up kind of regularly. Um, you hear a lot of talk around, you know, what is the perimeter in the modern enterprise or the modern organization? With the advent of cloud and certainly the rise of third party AI systems, you know, we might be putting data into third party systems. We might be doing all these different things with our with our data that kind of erode the the kind of perimeter of what an organization's cybersecurity or cyber control is. And you hear things like identity being, being identity as the new perimeter being thrown around pretty loosely. But a lot of the talk today was actually around the data being the key asset. And the key thing to really focus on in utilization of an adoption of AI systems.

A couple of other things that were discussed, certainly the DARPA Cyber Challenge and the DARPA, DARPA, AI Cyber Challenge was really interesting. Hearing from the winning team, from someone from the winning team, Team Atlanta, who took the four million dollars prize around why their research was actually probably awarded. And I think one of the things that I took away from it in particular was the highest true positive rate was a huge differentiating factor. It's easy to apply AI to find small problems, Plums, but filtering through problems in cybersecurity, getting that signal to noise ratio right is a long standing problem. There are so many cybersecurity tools and systems that are known for generating too many alerts and too much volume, and cutting through that noise to eliminate false positives is a real game changer for practitioners. So hearing some perspective on that was really, really interesting.

Other things that are that were really kind of discussed regularly, um, consumer adoption in a lot of organizations is actually dragging them down the AI adoption path , you know, when the workforce is already using these tools on a regular basis that really prompts or can prompt organizational change. And this is something that we see very often at Firetail. We know that it is kind of user use of AI systems that forces the cybersecurity teams and leadership at organizations to start to think, okay, we need to get our arms around it. And that's why, you know, shadow AI discovery is such a big focus of what we do over here. So it was great to hear some kind of consensus around that being a real challenge point for that.

Uh, there were some really interesting talks around, um, energy water treatment and critical infrastructure. Those were a little bit more under the Chatham House rule. So we can't go into those. But I will say, I think the last section of the day around healthcare was also really eye opening for me. Um, one of the use cases I never thought about was kind of doctor assisted or sorry, AI assisted human in the loop, uh, applications in the healthcare domain and some of the statistics around the increased efficacy of doctors who are using AI systems, in particular in areas like radiology, can really be beneficial in game changing. And yet at the same time that some of the most sensitive PII Phi that's out there. And so there's really some tension and some trade offs that have to be thought about in the healthcare domain space.

All right. We'll leave it there for today. I really, uh, got a lot out of today's session. I would encourage anybody to kind of look for content around that. We'll have the event itself linked. Some of the sessions were live stream. I think some will be recorded. Oh, also, Dundee West from GlaxoSmithKline shared two frameworks for framing the AI problem that I thought were really interesting. One is the center of gravity framework and the other is the hard problem framework. Um, he did announce that his slides will be shared on LinkedIn, so I would encourage you to look him up. You can find his name on the slide here. Dundee West, assistant general counsel at GlaxoSmithKline, GSK. And I think, you know, one of the analogies that he said was, you know, his job is to allow the organization to move forward, but he has to be the brakes on a very fast moving car. And that's maybe a way to think about it, is you want to enable maximum speed adoption for what the organization can tolerate, but have the ability to respond and react very, very quickly if needed.

All right. Apologies for going long on that last section, but I again, I did find the day really interesting to frame some of this stuff. Hope you enjoyed this week's episode. We'll talk to you next week on another episode of This Week in AI security. Bye bye.