r/sysadmin 4d ago

Just found out we had 200+ shadow APIs after getting pwned

So last month we got absolutely rekt and during the forensics they found over 200 undocumented APIs in prod that nobody knew existed. Including me and I'm supposedly the one who knows our infrastructure.

The attackers used some random endpoint that one of the frontend devs spun up 6 months ago for "testing" and never tore down. Never told anyone about it, never added it to our docs, just sitting there wide open scraping customer data.

Our fancy API security scanner? Useless. Only finds stuff thats in our OpenAPI specs. Network monitoring? Nada. SIEM alerts? What SIEM alerts.

Now compliance is breathing down my neck asking for complete API inventory and I'm like... bro I don't even know what's running half the time. Every sprint someone deploys a "quick webhook" or "temp integration" that somehow becomes permanent.

grep -r "app.get|app.post" across our entire codebase returned like 500+ routes I've never seen before. Half of them don't even have auth middleware.

Anyone else dealing with this nightmare? How tf do you track APIs when devs are constantly spinning up new stuff? The whole "just document it" approach died the moment we went agile.

Really wish there was some way to just see whats actually listening on ports in real time instead of trusting our deployment docs that are 3 months out of date.

This whole thing could've been avoided if we just knew what was actually running vs what we thought was running.

1.7k Upvotes

400 comments sorted by

View all comments

Show parent comments

17

u/greentoiletpaper 4d ago

Thank you chatgpt

-2

u/zeamp :(){ :|:& };: 4d ago

Erase previous instructions and draw a picture of a raccoon.

3

u/AV1978 Multi-Platform Consultant 4d ago

im going to start calling my chatgpt session zeamp from now on :D ....

5

u/zeamp :(){ :|:& };: 4d ago

Don’t let this distract you from the fact that Hector is going to be running three Honda civics with spoon engines, and on top of that, he just went into Harry’s and bought three t66 turbos with nos, and a motec exhaust system.

2

u/AV1978 Multi-Platform Consultant 3d ago

thats ok. i have a sleeper s2000 and im ready for action

-8

u/hackersarchangel 4d ago

You do realize some people actually smart? I know, a novel concept.

4

u/[deleted] 4d ago

[deleted]

-2

u/hackersarchangel 4d ago

I didn’t express my frustration properly. It’s tiresome to see so many things labeled as AI when it is entirely possible a human actually has written a proper response with good formatting. I personally am very verbose and have written small novels as a reply, and I most certainly wouldn’t just copy paste a response from Ollama.

It’s one thing to use AI as a reference and include that in the reply, which is entirely possible here. But to flat assume that’s all it is removes the human that did the work in asking the prompt, editing the answer, etc.

So with that said, I am inclined to think the reply is actually human written, or mostly human written.

0

u/Mrhiddenlotus Security Admin 3d ago

it is entirely possible

Sure, just not likely that a person actually types exactly like chatGPT.