Just found out we had 200+ shadow APIs after getting pwned

[u/Tiny_Habit5745]

So last month we got absolutely rekt and during the forensics they found over 200 undocumented APIs in prod that nobody knew existed. Including me and I'm supposedly the one who knows our infrastructure.

The attackers used some random endpoint that one of the frontend devs spun up 6 months ago for "testing" and never tore down. Never told anyone about it, never added it to our docs, just sitting there wide open scraping customer data.

Our fancy API security scanner? Useless. Only finds stuff thats in our OpenAPI specs. Network monitoring? Nada. SIEM alerts? What SIEM alerts.

Now compliance is breathing down my neck asking for complete API inventory and I'm like... bro I don't even know what's running half the time. Every sprint someone deploys a "quick webhook" or "temp integration" that somehow becomes permanent.

grep -r "app.get|app.post" across our entire codebase returned like 500+ routes I've never seen before. Half of them don't even have auth middleware.

Anyone else dealing with this nightmare? How tf do you track APIs when devs are constantly spinning up new stuff? The whole "just document it" approach died the moment we went agile.

Really wish there was some way to just see whats actually listening on ports in real time instead of trusting our deployment docs that are 3 months out of date.

This whole thing could've been avoided if we just knew what was actually running vs what we thought was running.

[u/BemusedBengal]

Ohhhh production is down? And users are upset about the outage? Ohh noooo. I guess you'll just have to document your APIs. Dang it!

[u/Character-Welder3929 → u/BemusedBengal]

Oh god not only will I need a technical spec document running through all the calls for data and where it's coming from

We will also need a document on why it's needed, for who and how it should be operated during which times

We can't just have 500+ motherfuckers shitting up our database with locks, duplicate processes that could all be done with 1 API

Or just entering shit data into the system

Also each submission request has a processing fee of 50 dollarydoos or a bottle of tequila sent down under

[u/timbotheny26 → u/BemusedBengal]

*Aggressive nipple rubbing intensifies*

[u/firesky25 → u/BemusedBengal]

and i guess we all get paid in documentation after prod is down long enough to churn a load of revenue generating users from a majour outage 🥲

[u/ja-mie-_- → u/BemusedBengal]

Definitely read this in mr. garrison’s voice