This Week in AI Security - 15th January 2026

Happy New Year. Welcome to twenty twenty six. It's Jeremy coming back to you with another episode of This Week in AI security. I hope all of you had a great holiday season and maybe managed to get a little bit of a break I did myself. So for that reason, we've got a long episode today because while I took a break, I can tell you that AI certainly did not. We've got a lot of stories to get through. This is going to be a slightly longer weekly episode than usual, and I actually wanted to start it off in a little bit of a different vein.

Instead of only reporting on AI security incidents, I wanted to kind of check in on the state of AI around the world right now, and kind of what some of the predictions are coming into twenty twenty six. And I want to just start off with this report from Netskope around open AI usage, based on some sampling of data that they did across their customer base. And what they're finding is that actually AI consumption is up about three x from a year ago. The number of prompts, the volume of prompts issued by users is up actually around six x. So this is some statistic. If you thought that AI might be slowing down heading into twenty twenty six, it appears that that is not the case.

And in fact, what we're seeing is more and more use cases of AI are creeping into new devices all the time. So Boston Dynamics, who is famous for their spot dog, they're flipping robots. All kinds of robotic devices have announced that their next gen humanoid robot will have DeepMind, an AI service from Google embedded in the application. And in fact, that's not all. Nvidia announced a new Alpamayo model that allows autonomous vehicles to think like a human. This is kind of interesting to me, because this is one of the first times that we're seeing a model specifically launched less as a general purpose, but now an LLM for a specific purpose.

And let me just take a second to kind of explain why I think this is important and interesting. Before the advent of the large language model. A lot of AI, so-called AI initiatives were really machine learning initiatives. And so they were specialized data sets, and they were specialized applications of artificial intelligence used in one domain. What made LLM so powerful is that they actually finally broke away from that and created a general purpose AI and or artificial intelligence platform. And now we're kind of coming full circle. We've gone from, you know, specially trained, uh, systems based on machine learning algorithms into a general purpose algorithm. And now we're going to a general purpose algorithm, but specifically trained for a specific use case.

And I think that's pretty interesting because in theory, it should combine the best of both worlds, remove some of the restrictions around, let's say, like lateral thinking that older ML models might have faced and now get to AI, but apply specific training data sets and other capabilities and maybe features. There's there's not that much detail to go off of, but for a specific use case. So that'll be interesting to keep a tab on and see if there's going to be more of these types of initiatives coming out, whether from Nvidia from other providers.

Another interesting thing is, you know, we hear a lot about agentic use cases and AI agents. So these are, you know, AI systems, really pieces of software that are designed for a specific use case and then given tasks and given the autonomy to execute those tasks within certain guidelines or maybe guardrails. And what Google has announced is a new protocol to facilitate commerce using AI agents. So how is this interesting or how is this relevant? Well, we all use protocols every day in things that we do, like browsing the web. If you think about HTTP or Https, the P in there stands for protocol and the HTTP is Hypertext Transfer Protocol, and the S on the Https stands for secure.

And so what that really is, is it's the way that your web browser communicates with a website to transfer the text or the images and so on that get rendered on the screen in front of you that has been suitable for the web up till now. What Google is saying is that with commerce going forward, there are going to be enough use cases where I, for instance, am going to enable some agent to go shopping on my behalf, whether that's booking tickets or putting together a travel itinerary or just, you know, do my grocery shopping for me or whatever the case may be, that in fact, they need a new way for that interaction between my agent and the stores to happen.

So it could be that my agent for grocery shopping communicates with one of my local grocery stores, Wegmans or Giant. In the case of the two local ones close to me, and there may be decisions to make. So, for instance, I say I want chicken breasts. It may not know what size, how many, and so on. It may need to make some decisions around that. And a protocol is emerging around my shopping agent and maybe the store's agent communicating with each other. So again, a specialized use case for specific scenarios. And we'll see how this evolves, because I do think that there's going to be more and more of these protocols potentially coming not just around commerce, but other areas as well.

And finally, on the kind of general purpose thing, New York and California tend to lead the US in terms of regulation and compliance standards. New York, obviously, being the headquarters of a lot of banks in New York City, tends to have a lot more sway in terms of financial service regulations. And in fact, a friend of mine who works in the compliance field really says, you just got to watch financial service regulation coming out of New York to see what's going to happen across the rest of the world. That hasn't always proven to be true, but it is a good leading indicator of what could be coming at a broader scale.

And so the governor of New York, Kathy Hochul has signed something called the Raise act, and that is responsible AI security. And I can't remember what the E stands for. I should have had it in my notes here as I'm recording today. But effectively it is a responsible and secure AI initiative to regulate AI safety for systems that get put out there for general purpose use. This probably won't affect internal LM systems that you might be using inside an organization, but suffice it to say that if you're in New York or subject to New York regulation and you're using an LLM powered application that you might be using for things like customer service or for chatbots or things like that. And again, you do have some New York exposure. This may well apply to you. It's going to be interesting to watch. If, for instance, California follows suit, other states, Virginia and Colorado have have typically been at the state level in, uh, in the absence of a national regulation here in the US, which currently, again, there is no US regulation at this point.

So it'll be interesting to watch. Now Next story from a regulatory perspective, is Italy talking to Mehta and telling Mehta to suspend its policy that bans rival AI chatbots from WhatsApp? And the context here is that, of course, Mehta is the owner of the WhatsApp service that is used by billions of people around the world, myself included, for various communications, and it is a rich, rich source of data. It is also a rich source of customer interactions, and I've been in many places as I've traveled where restaurants take reservations over WhatsApp, customer service is given over WhatsApp messages, etc. and Mehta, as the owner of WhatsApp, wants to be the sole channel of AI chat bots that can go on there.

If you think about customer support scenarios or commerce scenarios or things like that, it could lead to a situation where only agents or chatbots built on one of Meta's AI services, like llama, is allowed, and Italy is telling meta that this is not suitable under EU regulations for competition and openness of ecosystems. So again, another interesting story that'll be kind of curious to watch. A lot of the other big providers of Llms would similarly be subject to regulations like this. So Google has its Gemini models, Amazon has its Nitro models and sorry, its Nova models and many others. Microsoft, of course, is one of the leading investors in OpenAI and ChatGPT.

So there's a lot that could be happening here because there is, again, a lot of intersection between, you know, the the platforms that have broad access. If you think about Google, its advertising, its search, etc. if you think about Amazon, its e-commerce, if you think about Microsoft, its a lot of B2B use cases. It's also things like email and accounts on the Microsoft three hundred sixty platform. So they have a lot of potential for pseudo monopoly power into various channels. And it'll be really interesting to watch how this plays out as well.

Now moving on to some of the more specific stories. We've got a number of stories to cover this week that are. I'm going to go through somewhat quickly because they're kind of new things. Um, but they're based on things that we've covered in the past. This first one, though, is really interesting. There is a claim of a leak of the system prompts from most of the major lmz. We've got this linked in the show notes. It's a GitHub repo right now where the user claims to have been able to extract the system prompts from a lot of these llms. What's interesting about these system prompts is, you know, a lot of the times, understanding a system prompt helps you understand how an LLM will behave or interact with inputs and prompts from users or systems that are talking to them.

However, what's been interesting so far and what's out there, is that these system prompts tend to be very, very general purpose, relatively short, and not all that informative. They don't really so far seem to give a lot of information in terms of how these LMS could be compromised or could be vulnerable. So something to keep an eye on. Again, this is claimed not necessarily verified. And we've got the link for you to check out for yourself if you are so interested. All right.

Moving on into other stories. We've talked about AI browsers a number of times here on This Week in AI security and some of the fundamental flaws in them. And really interesting story. In late December, OpenAI researchers came out and said, look, it may always be the case that these are vulnerable to prompt injection. If you want to have a browser or a agent or a genetic powered browser that can go and do things on your behalf, you are kind of inherently always giving it a level of permission and telling it to go read things. And it may always read things that are instructions that it then executes, which is where prompt injection can come in, or as it's sometimes called, indirect prompt injection because it's not coming directly from a user input. It's coming as a kind of an after effect or a side effect of something that the AI browser is already doing.

So interesting story. And it does kind of make you pause for a second and ask yourself, you know, what level of comfort do you have in kind of delegating authority and permission to something like a browser to go do things on your behalf? And are you okay if you know that there is kind of an inherent vulnerability in a system like this? Uh, next is the abuse of LMS. Since this headline went out there of French and Malaysian authorities cracking down on Grock for generalizing sexualized deepfakes. In fact, this story has evolved. We're now at a point where I believe Malaysia, Indonesia and other countries are temporarily banning, uh, XAI and Grock LMS because they have been known to generate sexualized deepfakes. So the abuse of the LMS systems, in the absence of guardrails that prevent things like this or the ability to circumvent those guardrails, uh, does lead to real world consequences, at least on a country by country basis. We'll have to see again how this plays out.

Next is something we've talked about here on the show before, which is that, you know, it's not always the AI system itself that's vulnerable. It's very often things like the plumbing and the architecture and the connective tissue around them. There is a platform called N810. It is kind of a workflow automation platform. I've sometimes sort of pronounce Nathan. There is a ten out of ten maximum severity CVE identified in this system that allows for remote, unauthenticated control over locally deployed instances.

The reason that I brought this up in this week in AI security news, is that this is one of the most common workflow automation tools that I've seen people using for building agent or Agentic powered workflows. So it's a great kind of workflow editor where you say, hey, go execute. Step one bring back. Take the output of step one, hand it off to step two. Go on. Well, that kind of step one, handoff to step two, etc. is in many cases orchestrated by this n810 software in a mac severity like this that allows for remote control or unauthenticated access and control, is really a crucial severity vulnerability for any of these AI powered systems. So again, it's not always the LM. It's very often the kind of the connective tissue, the plumbing around it.

Moving on. We've talked about IEDs. We've talked about ID disaster or disaster. However that was pronounced a vulnerability. Again, very, very high severity for kind of co-pilot. You know, VB code platforms however you want to think about them. But code generation platforms Amazon just released their own called Cairo. It's a dev. Dev. Uh, there was again a confirmed vulnerability on this. Um, super high severity requires attention, CVE assigned, etc. and, you know, in previous, uh, ID related vulnerabilities, what we saw was things like command injection, exposure of credentials, keys and things like that.

This one came down to folder names, so you could embed folder names with injected commands in those folder names. And so there's, you know, a lot of creativity from the attackers and finding ways to kind of exploit these systems and not necessarily, um, a ton of thought from the people developing them on all the potential ways that this could be abused. I myself have a particular soft spot for knowing about folder names, uh, or a particular sensitivity to folder names. Because if you've listened to modern cyber for a little while, you may know that on our breach series, I shared the worst breach I ever had or experienced as a security practitioner, and it came down to folder name related items.

So moving on. One of the other things that we've talked about on the show before, and in fact, some research from Firetail around Ascii smuggling, um, there have also been follow ups to that around emoji smuggling, the ability to embed malicious commands inside emojis or emoji encoding. And in fact, the Google AI Google Gemini AI chatbot has other vulnerabilities, um, where they trust AI outputs and then use them to continue in. Some of those AI outputs can have invisible instructions. Again, very similar to SK smuggling. So nothing new here. Again, just more of the same, I suspect. And I continue to believe that there are more of these kind of semi malicious ways to embed bad instructions that we just haven't gotten to yet because nobody has bothered to test for them.

Docker has issued a new version of its Docker desktop agent. Um, this was in response to a vulnerability around um. Pause here. Note to Alan this is at the fifteen twenty nine mark. You're going to need to do some editing out here. Let me pause for a sec. Okay. and a little bit more of the same around Eid environments. Docker, as you may know, is a very popular, uh, kind of server virtualization technology used for running kind of virtual machines or containers and different environments.

This is similar to I disaster. This is oh, this is another, um, common tool used for desktop of desktop testing of software as it's being developed. And there's a prompt injection or indirect prompt injection vulnerability in here. This is again very similar to I disaster with the indirect prompt injection being a set of kind of malicious instructions uh through metadata and describe this repo types of queries that could do things like extract credentials, API keys, network information, etc.. Uh, so very similar to things that we've seen in the past, again, just affecting other parts of the broader ecosystem around developing with, uh, you know, with, uh, AI powered IDE environments or test environments or build environments in this case.

All right, moving on. We've got a vulnerability in Lang Chain that exposed millions of apps. So Lang chain is again another very popular platform for building a powered applications. And this is, uh, this is related to API serialization and deserialization, which we've covered on the show before. So again, just more on that theme of it's not necessarily the LM, it's the things around the LM kind of speaks to the importance of, you know, some standard security controls like input validation and normal, uh, sorry, sanitization. Um, and in this case, you know, a well crafted, uh, prompt could go and expose API keys and secrets. And the reason that this one is so big or so, uh, critical, is that Lang Chain is, again, a very popular platform used by millions of applications already today. It's one of the easiest platforms to get up and running and build on, uh, with something called Lang Chain Corp. Uh, awesome.

So That's a lot of the variations on a theme of things that we've talked about before from previous episodes, and hopefully you're getting some of what we're talking about here at Firetail when we talk about, you know, it's not always the LM, you got to think about the holistic view of what you're building. And that's why we think that some of the work that we're doing about tying APIs and AI together can be really meaningful and helpful to our customers. But we've got a couple of new stories as well.

So a new unrestricted AI tool has just been launched, and there have been versions of this before. You might have heard of something called fraud GPT or worm, GPT, etc. these were kind of, um, customized forks of open source llms that were released in twenty twenty three or twenty twenty four, where the specific intent was to enable cybercriminals to get around things like guardrails and to do faster generation of malware, uh, things like that. And so a new one called dig AI, was released in late December twenty twenty five. Um, sorry, it was actually released in September of twenty twenty five. And it was finally some some data around it surfaced in December around, uh, some of the stats and the usage of it.

Ten thousand prompts were a performed within the first twenty four hours of operation or availability of this dig AI. So it definitely shows that there is an appetite in the kind of, you know, criminal underworld to find ways to benefit from LLM and AI and AI automation. And I'll just kind of remind you something that we say very often is like, remember that criminals have access to all the same tools as you. They just have different intents. They have different goals for how they are going to use these systems. But they're all looking for productivity. They're all looking for the fastest way to the lowest hanging fruit in order to exploit people, customers, environments, exfiltrate data, whatever the case may be.

All right, last but not least, hackers actively exploiting AI deployments as ninety one thousand attack sessions are detected. So this is a complex campaign of big numbers. And there were really two types of activities that were observed. The first is a campaign using llama to launch something called CRF and CRF. If you're not familiar, stands for Server Side Request Forgery.

The effectively what they're doing is finding ways to relay requests via an LLM that will then kind of disguise the malicious intent behind them, and use the LLM as the launching point, and see whether the service at the other end is able to pick up the fact that it's a malicious payload that is being relayed via the LLM. So that is a real problem. The second campaign is potentially even worse. So it's looking for commercial AI APIs that are open. And when you look for these AI APIs that are open, what you see is any number of unauthenticated API requests.

Combine that with the risks associated to LMS, so that can be anything from prompt injection to misinformation to, you know, planting false information inside the LM, submitting bad training data, anything that can go into that. And again, just to kind of reiterate, and I think I've probably said it ten times during this week's episode, it's not always the LM, it's very often the plumbing around it. So it kind of, uh, really reinforces the importance of things like controlled APIs around LMand so on. And, and also just reinforces the fact that, again, criminals are using LMS actively, just as you are or your organization may be.

All right. Like I said, longer episode than usual. We had a lot to get through. Hopefully this has been good for you. If you've got stories, remember please do submit them. We'll get back on a regular weekly schedule from now on and back to kind of our normal eight to ten minute episodes on a weekly basis. Look forward to talking to you next week. Thanks so much. Bye bye

‍