Breach Series 1 - Jeremy Snyder of FireTail

Welcome to the first episode of, modern cyber breach series. I'm your host, Mike McCabe, the CEO of Cloud Security Partners, and I'm guest hosting today because our main host, Jeremy, is in the guest chair. Welcome, Jeremy. Thank you so much, Mike. I'm really excited for this series.

Yep. Jeremy is the founder and CEO of, FireTel. Prior to FireTel, Jeremy worked in m and a at Rapid7 and before that at DV Cloud, AWS, and other companies. Jeremy started his career with thirteen years in cyber and IT operations. Indeed.

It's been a ride. Today, we're gonna talk about some of, your experience with some breaches that you went through. This whole series is about breaches and kind of getting real world stories about that because it's a very useful topic for people to hear more about. I guess to start off, do you wanna set the context for kinda, you know, where it happened, the kind of company, things like that? Yeah.

To be honest, I don't remember the exact year, but I can tell you it was between 02/2002 and 02/2004. I don't remember if it was 02/03, or '4, which of those years that it was. I was working for a software company. We were in the language technology space. We were probably about a hundred person company at the point in time when this happened, and our clientele was primarily larger enterprise customers.

So we worked with some international kind of organizations. We worked with some very large, very brand name blue chip customers, publicly traded companies, what have you. And we were a smallish team on the IT and cyber side, we were probably only about eight to 10 people around the world. Given the time frame, of course, we ran our own data centers, nothing in the cloud, there was no cloud at that point in time. So, you know, we had a data center location, ironically enough, in Northern Virginia data Center Alley, which I always tell people, like, you do not realize just how much of the Internet is in Northern Virginia.

But, you know, we were in one of these data centers all the way back then, and, yeah, that's that's really the time frame and what the company was like. Gotcha. And, when this breach happened, kind of, what was your first indicator, and and what was the story after that? Our first indicator sucked candidly because our first indicator was the bill from our hosting provider. And at this point in time, we were averaging something like $7.08 k a month, between our cage at the, Colo facility and our bandwidth allocation on top of that.

We had a little bit of variation around that. What it really depended on, honestly, was we were kind of an early, I won't call it a SaaS company because really, like, it we didn't have a lot of the SaaS principles on our software platform. We were really more of a client server solution. So every customer you could think of it as every customer had a set of dedicated servers that we ran in the data center for them. And Yeah.

Frankly, we only did that because our software was kind of frail. It didn't stay up and running on its own very easily. And so, you know, what we we had actually started by deploying the software into customer data centers, and we got a lot of complaints and pushback, like, hey, this thing is crashing all the time, and we would we tried to write some, you know, auto restart scripts, literally, you know, MS DOS command line, you know, kind of auto run boot script type stuff. Dating myself a little bit, but, you know, this is a while ago, more than twenty years ago. And frankly, like, it just wasn't good enough.

So we took over hosting responsibilities, we put all this stuff into a data center. The variation around our bandwidth was typically because of files. So I worked in the language technology space. We worked on very large files. At that time frame it was a lot of like Internet help files.

It was desktop publishing files. Probably this will be a little bit of a throwback for some of our audience, but if you ever heard of software like PageMaker, FrameMaker, QuarkXPress, you know, we would get these very large, for the time, very large, you know, twenty, fifty, hundred megabyte files, that customers were working on. And I know it sounds kind of funny in like the 2025 time frame that you couldn't just like upload these files, but we actually had to set up an FTP server. And, an FTP server for customers to interact with our customer support team, and so they would load up files that would have problems, with our software for one reason or the other. And anyway, we get this end of the month comes and we get this bandwidth bill, and instead of it being, like, in the seven to eight k range, it's, like, 28 k.

And we're, like, what the heck is going on? And our provider is, like, hey, bill is due. And so we start digging into it, and, you know, we start then looking at the, graphs for traffic by IP address, and we do we look at our NAT tables on our our, firewalls, and we trace it back to our FTP server. And I have to say that, like, we did something that I think a lot of companies do, which is we looked at all of the hardware or all of the equipment that we had, and we're like, we had this old school seven height unit, compact server. It did have hot swappable disks, but they were the old generation of hot swappable disks.

So it was, like, incompatible with any of the rest of our server fleet where we had standardized. So all of our customer servers were, like, kind of standard. I can't remember if they were Compaq or HP or what, but, you know, at that time, like, this server was kind of a standalone dinosaur server that we really didn't have much use for other than we needed a place for an FTP service. And it was kind of like deemed to be low criticality, it was also low in terms of CPU utilization or CPU requirements. All you needed was, you know, let's call it 20 gigabytes of space to to kind of swap files back and forth between customer support and customers.

And so, I should also mention by the way, this server was so old that we couldn't install Windows NT four on it. It was NT 3.5 and it could not be upgraded because of the CPU architecture. So it was like like I said, dinosaur, oldest server in our fleet. We thought we were super clever, that we found a great way to reuse this server, make it productive, keep it in service, etcetera. No more, like, service packs coming out for NT 3.5.

So it was what it was. You know, end of life operating system that we were continuing to run, which, honestly, was a terrible risk to take, but, you know, we didn't know better at the time. So we created an anonymous FTP upload. Like and I know that sounds might sound kinda funny to some people listening right now. It's like, okay.

Anonymous download. Sure. No. No. We created an anonymous FTP upload.

No login required so that any customer anywhere around the world could just drop their files onto there. We did it in a way where we, IAS at the time, if you ever worked with it, you might recall it. Actually, the permissions are file, system level permissions to run IAS. Mhmm. So you have a service account that runs the IAS process itself, but in order to actually serve up, like, the web server or the FTP server in this case, you set those permissions on a file, folder structure within the file system.

So we set a set of permissions that we thought were very appropriate for the task at hand. Anybody can write. Nobody can read. Okay? So when you connected to this FTP service, you would actually get an error message, and we we told all of our clients about this.

We sent them screenshots like, don't worry, when you connect you will get this error message, and that error message is cannot list files. You will still be able to drop your file into the FTP server. That was our design. We were very proud of it too by the way, I can tell you that. Like, it took us some time to figure out what this folder structure was like.

Long story short, what we came to learn was that there was effectively a vulnerability within the Windows operating system structure. And it comes down to kind of the 16 bit architecture of the system at the time. You might remember that conventions around files and file names at that time, eight dot three, if you've ever heard that phrase, You know, file names were typically no more than eight characters with a three, character extension on the file. And then kind of a limit of 256 characters in a path. And that is like a directory folder, you know, c colon slash windows slash IAS slash blah blah blah blah blah and so on down the line.

Turns out, that is not hard enforced. So you can create subfolder after subfolder after subfolder after subfolder, and you could do that with our anonymous write capability, and it broke the permission system. So the permissions are no longer enforced after that 256 character limit. So what ended up happening to our server is some people found it. They just created a series of foot subfolders all named hyphen.

So you would just see a folder name dash, subfolder dash, subfolder dash, repeat 250 however many times. And then somebody uploaded a bunch of pornographic video content onto our server. And that is how we actually found out about it. It was the bandwidth of all the downloads. Upload was free, upload was kind of included in our colo bill, but download is what we paid for.

And so, you know, I don't remember how many hundreds of gigabytes that translated to at the time, but we became the unwitting host of a porn server. That is yeah. That's a good story. That's a good story. It didn't feel great at the time, I can tell you.

Like, in hindsight, it's kinda funny to look back on, but, you know, it wasn't wasn't great. Yeah. I'm sure. Yeah. That's, that does feel like a little bit of a throwback to the more fun and innocent times of, hacking, but I can see from a business perspective, it's not it's not great.

So back then, those things were a little bit different, not cloud APIs, not not the same kind of logging we have now. Kind of what was your what was your next steps after you kind of discovered what was going on? Yeah. Obviously, once we got the bill and we mapped it back to the server, our first steps were to log into the server, pull system logs, pull access logs off of IIS, Internet Information Service, I think was what IIS stands for. I'm not a % sure about that anymore.

But, that was that was really what we saw. And and, you know, kind of as I said, it was just hundreds of gigabytes of download. So we saw requests coming from all over the place, you know, IP addresses from all across the country. So it was hard to attribute it to any one actor. In fact, even on the upload side, it maps back to some, you know, dynamic IP address of somebody's cable modem or DSL home connection at that point who who created all of this nonsense and then uploaded the original content.

So it's really hard to kind of, like, go after somebody. I suppose we probably could have pursued law enforcement. Mhmm. But those were our first steps is, like, you know, get on the server, pull all the logs. In our case, thankfully, one server, one set of logs to go look at, not like, you know, other breaches that I've heard about from people where you're talking about like multi system chained access, that gets really hard to trace.

Right? So thankfully, pretty easy to track down, very easy to find the offending folder structure. Turned out, by the way, hard to get rid of the offending folder structure, because it broke the permission system again, trying to actually delete that path and all of this, sub directories and all of the content would just like fail. Again and again. We tried it inside Windows, we tried it, from the, DOS command line, both steps failed.

So what we ended up having to do was actually take a backup of the server and restore it without this set of folders. So we had to like wipe all the disks, restore this, and set this. Then what we found, you know, a new set of permissions to apply to the folders that restricted the creation of subdirectories, and that ended up kind of like giving us some peace of mind. But also, like, honestly, we weren't monitoring IIS log files, you know, and in fact, we have been rotating them on a regular basis. I could think we had an automatic seven day overwrite because otherwise, they get they get big and they fill up a disk and, you know, if you've been around for a while, you know that that used to be a very very real problem.

And that used to be the kind of, like, two AM drive to the data center, because your pager went off kind of thing that, like, nobody wanted to do. So it was a it was a super common thing to do was to overwrite your log files. Yeah. Yeah. For sure.

Interesting. So what were kind of the lessons learned for you and your career based on this experience? I mean, definitely the importance of logging and the importance of actually doing something with your logs. Prior to that, that very same company, I'd been at that company for, like, five, six years at that point, and we had basic logging turned on everywhere. Nobody ever did anything with it.

Nobody ever looked at it. Nobody ever reviewed the logs, you know, so building in some kind of a regular log review process was pretty important. File system backups, you know, if we hadn't been able to take a backup from before the breach and then restore it without the kind of infected I I use infected in kind of air quotes there because it wasn't really a virus or a malware infection. It was literally, you know, just kind of a set of files that broke a Windows permission system. But if we hadn't had the backups, I think, you know, we would have been in in we probably would have been in reformatting the whole server and trying to set the whole thing up from scratch mode.

I would also say, looking back on it, like, do not use end of life systems, you know, especially not for anything mission critical. You wanna use it for an internal test dev server, like, I get it. That's fine. I think, you know, as long as you've got good, let's call it good network segmentation and protect it and make sure it's not on the Internet. But anything end of life has so many vulnerabilities in it.

This being the least of, you know, what could have happened to us. Right? Like, it it could have been so much worse in the sense of somebody could have established persistence. Somebody could have used that to get into many other systems inside the network. We were a pretty flat network topology at that time, and I'm sure there was at least one service account on that server that would have had permissions to other things in our domain.

So, you know, a lot of those basics around, again, backup segmentation, you know, just reviewing the logs. Those were are probably my big three. And and and then again, like, do not use end of life stuff or anything that is production. Just don't. Bad idea.

And especially in this day of cloud, like, there's no reason for it. Right? Like Yeah. You know? So yeah.

Yeah. Yeah. Good lessons to learn for sure. Definitely, the the basics will always will always, maybe not save you, but definitely help protect you. So always good to do those for sure.

Yeah. Yeah. Cool. Cool. Before we wrap up, anything else you wanna kinda tell people about this this, event or anything you learned from it?

Well, look, I think, you know, from my own experiences, this is one of kind of two major breaches that I've experienced in my career. Another one came a couple years later. But I think some of it is you have to go through some of these experiences to form strong opinions that then really, like, do form a part of your guiding principles going forward in your career. And you know, everywhere I've gone since then, do we have logging? Are we doing anything with the logs?

Do we have basic network segmentation? Is there something that we're doing in our network topology that you could scratch your head and look at and be like, there is no logical reason for these two things to be able to talk to each other. You know, let's just, for safety's sake, let's just separate them, whatever. And and, yeah, this whole thing about, like, old systems, it is so much better to retire the systems. Yeah.

And by the way, one other thing I will say, kudos to our provider at the time. We didn't always have the best of relationships with them as a colo provider. They kind of continuously jacked up rates over those years, but when this happened, they were very understanding and worked with us, and they didn't waive the whole bill by any means, but we kind of, like, agreed on on kind of a negotiated payment point, which for us as a company was actually, you know, that would have been massively damaging to our budget. Having to pick up, like, four x our monthly bill would have would have been a real problem. So maintaining good vendor relationships is probably another thing that I would actually say is kind of overlooked, but can be very important, you know.

Yeah, yeah, good points. Yeah. Cool. Well, thank you so much for sharing your story with us. It's really interesting, and, I love the kind of throwback to yesteryear and different hacks and things like that.

Looking forward to hearing more stories and, more guests to talk about breaches. Awesome. Thank you so much for hosting today's episode, Mike.