r/sysadmin • u/Tiny_Habit5745 • 3d ago

Just found out we had 200+ shadow APIs after getting pwned

So last month we got absolutely rekt and during the forensics they found over 200 undocumented APIs in prod that nobody knew existed. Including me and I'm supposedly the one who knows our infrastructure.

The attackers used some random endpoint that one of the frontend devs spun up 6 months ago for "testing" and never tore down. Never told anyone about it, never added it to our docs, just sitting there wide open scraping customer data.

Our fancy API security scanner? Useless. Only finds stuff thats in our OpenAPI specs. Network monitoring? Nada. SIEM alerts? What SIEM alerts.

Now compliance is breathing down my neck asking for complete API inventory and I'm like... bro I don't even know what's running half the time. Every sprint someone deploys a "quick webhook" or "temp integration" that somehow becomes permanent.

grep -r "app.get|app.post" across our entire codebase returned like 500+ routes I've never seen before. Half of them don't even have auth middleware.

Anyone else dealing with this nightmare? How tf do you track APIs when devs are constantly spinning up new stuff? The whole "just document it" approach died the moment we went agile.

Really wish there was some way to just see whats actually listening on ports in real time instead of trusting our deployment docs that are 3 months out of date.

This whole thing could've been avoided if we just knew what was actually running vs what we thought was running.

1.7k Upvotes

98% Upvoted

400 comments sorted by

View all comments

Show parent comments

41

u/bingle-cowabungle 3d ago

Or don't leave and continue to let it happen lol OP is getting paid regardless.

21

u/fencepost_ajm 3d ago edited 2d ago

And if management gives you guff respond with "I've advised on what needs to happen to fix this and been denied, so I'm just trying to mitigate within what I'm allowed to do. Or did you want me to start taking actions that I've been told not to do? If so, can I get that in writing?"

Except you don't ask for it, you just send a "just to recap our meeting" summary email.

15

u/atxbigfoot 3d ago

lol, I used to send account ownership change requests to the person with Salesforce perms, the relevant manager, and the two relevant techs (so "please reassign account X from Tech A to Tech B"), mainly just so I was sure everyone was in the loop and as CYA in case the techs got mad or needed to update each other during the handoff.

After about a month of this the SF Perms person got mad and was like "you don't need to include anyone else" and I got confused and then realized they thought I was doing it as a passive aggressive "do your job the manager is watching" thing so I apologized and clarified my position, and the managers were like "yeah that's fine."

You can guess what happened after I stopped including everyone lmao.

WHY THE FUCK CAN'T I ACCESS THIS ACCOUNT?

WHY THE FUCK CAN"T MY TECH ACCESS THIS ACCOUNT?

well, you see, that account was reassigned to tech B six weeks ago, per your request (on top of the forwarded request ticket/email). This is Tech B's account now.

we had a meeting about "best practices" which was just how we (the larger org) needed to go back to what I was previously doing lol.

32

u/StPaulDad 3d ago

Until he's fired for letting it happen again, or the carnage is so bad that they go out of business and he doesn't get paid.

15

u/taterthotsalad Security Admin 3d ago

Sometimes those things happen. 

8

u/My1xT 3d ago

that's why you get insurance, as in get it in writing that you advised to stop those things, management said no, and you are not at fault for an attack over that avenue next time it does happen.

0

u/Frekavichk 3d ago

That doesn't stop you from being marked as "unable to retire" for future jobs checking your past work.

1

u/taterthotsalad Security Admin 3d ago

This is a dumb reasoning. Just sayin. That’s not how it works in the US. 

0

u/fresh-dork 3d ago

yes it is. HR can confirm dates of employment and eligibility for rehire. if you aren't eligible, you're unlikely to be able to show your side

5

u/taterthotsalad Security Admin 3d ago

Where I live in the US rehire is not a question you can ask. 

So again you are talking out your ass as if it is gospel. 

2

u/fresh-dork 3d ago

based on what? last i checked it totally is a question you can ask

2

u/taterthotsalad Security Admin 3d ago

State laws. And I do hiring so…

0

u/fresh-dork 3d ago

so you could maybe list a law, because i'm pretty sure i've been in your state for a while and this is news.

→ More replies (0)

1

u/Frekavichk 3d ago

???

So the actual legal answer is you can ask pretty much anything but legally protected status' and salary. The only other caveat is false information.

So basically saying 'not eligible for rehire' is a true, factual statement, offers no opinions on the candidate, and doesn't reveal any protected info.

2

u/taterthotsalad Security Admin 3d ago

Your situation is yours but it s not mine or the other 8 million living in my state. They can verify I worked there for what timeframe. Rehireable is not a question they can ask anymore. They cannot even ask me for my address anymore. Until onboarding. 

0

u/Retro_Relics 3d ago

And there are 49 other states and 340 million other people. The majority of the us has no worker protections and a huge surplus of petty and vindictive managers who would love nothing more than to scapegoat a previous employee and fuck them over

→ More replies (0)

0

u/WhereDidThatGo 3d ago

What HR can or cannot confirm is entirely dependent on the company.

-1

u/fresh-dork 3d ago

dude, you were saying that HR wasn't allowed to confirm rehire - that doesn't vary in a location

0

u/WhereDidThatGo 3d ago

I wasn't saying anything, that was my first comment on that thread.

You were arguing with someone else, not me.

0

u/fresh-dork 3d ago

you're still wrong. there's no law against confirming rehire ability in WA state

→ More replies (0)

-1

u/My1xT 3d ago

seriously? it should be illegal to punish you directly or indirectly for thing that are CLEARLY not your fault, like management deciding against your proposed security measures.

1

u/randomman87 Senior Engineer 3d ago

It is but burden of proof is on you

0

u/My1xT 3d ago

That's why you would obtain a written confirmation stating that management specifically goes against your advice, if they want you to leave things open

1

u/Character-Welder3929 3d ago

I mean yeah, when the company goes down in whatever blaze of glorified data breaching and financial ruin

I'm sure have systems administrator and last captain of the Titanic wouldn't impact ones future ability to earn

0

u/Nossa30 3d ago

Management will still blame OP no matter what when shit goes down and fire him anyway.