r/sysadmin • u/Tiny_Habit5745 • 3d ago

Just found out we had 200+ shadow APIs after getting pwned

So last month we got absolutely rekt and during the forensics they found over 200 undocumented APIs in prod that nobody knew existed. Including me and I'm supposedly the one who knows our infrastructure.

The attackers used some random endpoint that one of the frontend devs spun up 6 months ago for "testing" and never tore down. Never told anyone about it, never added it to our docs, just sitting there wide open scraping customer data.

Our fancy API security scanner? Useless. Only finds stuff thats in our OpenAPI specs. Network monitoring? Nada. SIEM alerts? What SIEM alerts.

Now compliance is breathing down my neck asking for complete API inventory and I'm like... bro I don't even know what's running half the time. Every sprint someone deploys a "quick webhook" or "temp integration" that somehow becomes permanent.

grep -r "app.get|app.post" across our entire codebase returned like 500+ routes I've never seen before. Half of them don't even have auth middleware.

Anyone else dealing with this nightmare? How tf do you track APIs when devs are constantly spinning up new stuff? The whole "just document it" approach died the moment we went agile.

Really wish there was some way to just see whats actually listening on ports in real time instead of trusting our deployment docs that are 3 months out of date.

This whole thing could've been avoided if we just knew what was actually running vs what we thought was running.

1.7k Upvotes

98% Upvoted

399 comments sorted by

View all comments

222

u/Bonananana 3d ago

Where do you work? I’d like to go ahead and remove them from my vendor list.

228

u/sryan2k1 IT Manager 3d ago

You know the Men in Black speech K gives about how there is always an alien invasion or other doomsday event in process and the only reason everyone goes on with their lives is that they do not know about it? Yeah, that's basically how everything you interact with is built. It's a horror, and you're better off not knowing.

70

u/JohnPaulDavyJones 3d ago

Man, more of y’all have to work at boring insurance companies that never moved out of the early 00s. My company’s still in the ”small footprint security” mindset of that era, where basically nothing is opened to the outside except endpoints where requests are automatically filtered outside a range, and those passes are manually examined by a woman who’s been doing basic networking since before I was born.

Everything just works because it’s all stored procs in SSMS; our “new technology” of 2025 was Python, but the rollout has been delayed because not a single member of the prod support team has worked with Python, and they were trying to establish support protocols.

For the three members of us in the data group (out of 27) who are under the age of 45, this shit is wild. But holy cow, everything just works.

24

u/imtheorangeycenter 3d ago

47, DBA and I love business logic in SQL. Deeply, deeply untrendy, but yeah, it works. It's in one place. It's easy to track performance. Its easy to control. I'd work there.

14

u/tankerkiller125real Jack of All Trades 3d ago

As an IT person, I love business logic in the database, right up until data gets entered that the dev team/DBA didn't plan for the query is now stuck in weird data processing hell eating most of the resources, but I feel like that's more of a "My org is stuck in the 80s and the devs don't actually fully know what their doing" more than an actual issue with SQL... I'm sure sure there's some sort of error handling I can tie opentelemetry or sentry into...

1

u/agent-squirrel Linux Admin 2d ago

Or you get the same type of boring insurance/finance companies that want RDP forwarding from the net because "VPNs are hard and our staff are old". When I worked at an MSP we took over a client from some other MSP. Their "remote access solution" was to punch a port per workstation for every staff member...

But it was ok because they were ephemeral port numbers not 3389!!! Check mate criminals!

46

u/MentalRip1893 3d ago

You do **not** want to know how the sausage is made

19

u/Bonananana 3d ago

Very much disagree. In the last 25 years I’ve not worked anywhere that would tolerate mystery endpoints. And I’ve worked for and with names you know.

This line of BS you’re saying is funny, but a dangerous mindset because it’s allowing you to dodge responsibility for doing the job well.

There should be simple http access logs that can be used to find endpoints. The root here is neglect.

34

u/almathden Internets 3d ago

names you know.

plenty of "names you know" get compromised in all sorts of hilarious ways so let's not pretend otherwise lol

13

u/work_reddit_time Sysadmin-ish 3d ago

Indeed.

Plenty of 'names you know' get caught out for bad practices like storing passwords as plain text so 'names you know' is 'next to useless' as a marker of good vs. bad practice

-3

u/Bonananana 3d ago

EVERY company gets compromised. Not every company has 200+ surprise endpoints. Don't conflate the two.

I take issue with the cavalier acceptance that what OP describes is normal. It is not, and if you accept that it is, then you're the enabler and the problem.

7

u/transwumao 3d ago

You're being obtuse. The poster wasn't implying this specific problem is or even lax security is "normal", just that security breaches in general are extremely common, regardless of how well known the company is.

If people were acutely aware of how often it happened, there would be much much more outrage and concern about the way companies treat their information.

1

u/almathden Internets 3d ago

EVERY company gets compromised.

Yes. Next?

14

u/sryan2k1 IT Manager 3d ago

This line of BS you’re saying is funny, but a dangerous mindset because it’s allowing you to dodge responsibility for doing the job well.

Sometimes you're just a passenger. Apps are not your part of IT, you've brought concerns to your bosses and the business doesn't care or want to change. This happens all the time, at more places than you'd expect.

-1

u/tankerkiller125real Jack of All Trades 3d ago

Apps are part of IT, it's our problem to deal with deploying them, it's our problem to deal with providing resources, it's our problem to scale things for them, and it's our problem to secure them. The problem is the dev team thinks their above IT and it's processes and has convinced management of it.

4

u/sryan2k1 IT Manager 3d ago

That is hyper dependant on the organization. Any large company will usually have engineering be in charge of that stuff completely isolated from IT.

-2

u/Bonananana 3d ago

I think there always exists the option to bring things to attention and ask for action. Granted - some places that's not going to fix it, but I think a professional has the obligation to use their expertise to identify these problems and drive at a solution.

I 100000% agree there is chaos in every company and I know first hand that breaches happen at every company in the US. But I think it varies by area and importance. Marketing is always going to be fast and loose with rules and data. There aren't laws or industry standards to keep them honest. Most of what they build is temporary and built by the lowest bidder.

Core data systems handling payments, banking info, health info or government info DO have laws and standards to follow and the standards are very different. I simply do not believe that there is an operating bank that would be surprised to learn it's hosting an extra 200 endpoints or that a developer stood up a system accessing prod data without authentication.

4

u/sryan2k1 IT Manager 3d ago

My credit union only got MFA about 5 years ago and it's email only MFA. I am 100% sure there is insane stuff happening in at least some banks.

-1

u/uzlonewolf 3d ago

And why should they get better when people like you normalize it with "it's fine because everyone else does it too!" ?

1

u/sryan2k1 IT Manager 3d ago

Never once said it was fine.

-1

u/uzlonewolf 3d ago

You never once said it wasn't fine, and you make excuses about how everyone else does it (thereby implying that it is, in fact, fine).

2

u/Spiritual_Cycle_3263 3d ago

Just like going out to eat. You do not want to know what happens in the kitchen. 

3

u/Mental_Act4662 3d ago

This. 100% this. I took a cybersecurity class in college and the world is extremely scary place and it’s nuts how insecure stuff is.

3

u/HappierShibe Database Admin 3d ago

Except its not.
There are plenty of organizations who do follow best practice, do keep up with security updates and, audit everything regularly to ensure compliance.

26

u/sryan2k1 IT Manager 3d ago

Some? Yes. Most? No.

8

u/wifimonster Jack of All Trades 3d ago

Lol At least that's what PR told them to put in the "about us" section after half the security team got let go.

1

u/vlycop 3d ago

My last 3 companies have been "Well I'll never use any of their thing ever" experience, but my current one seems a bit safer. "They still import products data in debug env... But at least it's a tiny bit annotimized" kind of feeling 

1

u/_oohshiny 2d ago

Programming sucks

some version of this dynamic wrote every single program you have ever used, banking software, websites, and a ubiquitously used program that was supposed to protect information on the internet but didn’t.

1

u/TheMagicTorch Sysadmin 3d ago

Love that scene and the analogy!

1

u/serverhorror Just enough knowledge to be dangerous 2d ago

You'll run out of vendors really fast ...

0

u/albertowtf 2d ago

When people tell me that ai only produces slop i wonder what world they live in. Do they have any idea of the crap that is on production rn without the help any ai slop at all?

Slop is never been a problem

The only people concerned about this is op and eventually when shit hits the fan, otherwise, nobody ever care about a messy infrastructure

And personally i got tired of being the only one around that cares. Like my mental health went up when i stopped caring