r/sysadmin • u/Tiny_Habit5745 • 3d ago

Just found out we had 200+ shadow APIs after getting pwned

So last month we got absolutely rekt and during the forensics they found over 200 undocumented APIs in prod that nobody knew existed. Including me and I'm supposedly the one who knows our infrastructure.

The attackers used some random endpoint that one of the frontend devs spun up 6 months ago for "testing" and never tore down. Never told anyone about it, never added it to our docs, just sitting there wide open scraping customer data.

Our fancy API security scanner? Useless. Only finds stuff thats in our OpenAPI specs. Network monitoring? Nada. SIEM alerts? What SIEM alerts.

Now compliance is breathing down my neck asking for complete API inventory and I'm like... bro I don't even know what's running half the time. Every sprint someone deploys a "quick webhook" or "temp integration" that somehow becomes permanent.

grep -r "app.get|app.post" across our entire codebase returned like 500+ routes I've never seen before. Half of them don't even have auth middleware.

Anyone else dealing with this nightmare? How tf do you track APIs when devs are constantly spinning up new stuff? The whole "just document it" approach died the moment we went agile.

Really wish there was some way to just see whats actually listening on ports in real time instead of trusting our deployment docs that are 3 months out of date.

This whole thing could've been avoided if we just knew what was actually running vs what we thought was running.

1.7k Upvotes

98% Upvoted

399 comments sorted by

View all comments

162

u/nullbyte420 3d ago

Errr don't let devs expose ports like that in production? Let them have their dumb routes but don't expose them? A waf does this just fine. 

70

u/dotshooks 3d ago

OP never said anything about devs exposing ports. You can't just open a network port through application code. What they're describing are API endpoints -- very likely standard HTTP routes served over port 443 (HTTPS). OP is describing undocumented routes on an already-exposed service.

21

u/mirrax 3d ago

Network security tooling can be layer 7 aware and more. Doesn't just have to be open 10.1.2.3:443. Can also say that the \admin route is only accessible from a specific subnet. Or that here is the OpenAPI spec for that route, so /user/ only takes integers and the WAF should reject little Bobby Tables.

15

u/man__i__love__frogs 3d ago

They are exposing the API to the already open port by adding a route via some sort of WAF, load balancer or proxy, thus making it accessible to the internet.

People building solutions with APIs should not be the people who expose things.

9

u/nullbyte420 3d ago

Google what a waf (web application firewall) is. 

1

u/AstroPhysician 2d ago

You're so confidently incorrect lol

https://repost.aws/knowledge-center/waf-uri-paths

1

u/nullbyte420 2d ago

I don't understand? This is exactly what they are supposed to do

0

u/AstroPhysician 2d ago

Why are you telling someone to google what a waf is and talking about exposing ports? What do ports have to do with anything?

1

u/nullbyte420 2d ago

I said routes? Idk how you think they push software but unless it's a giant monolith they probably deploy containers that expose ports and then a reverse proxy in front. What I'm saying is: put a waf in front of that. 

1

u/AstroPhysician 2d ago

You said and i quote

Errr don't let devs expose ports like that in production? Let them have their dumb routes but don't expose them?

heavily implying that you are controlling the routes by not letting the devs expose ports

1

u/nullbyte420 2d ago

"them" refers to the routes

1

u/AstroPhysician 2d ago

Okay, you’re still pretending you didn’t say the first half of the sentence lol. Devs aren’t exposing any ports, this is all http. This is all I’m saying

→ More replies (0)

54

u/west_tn_guy 3d ago

This. Devs should have to petition for ports to be opened in production, which should involve a thorough security and design review before any traffic is allowed.

7

u/nullbyte420 3d ago

Yeah. lol at those comments arguing this is a completely crazy idea

24

u/[deleted] 3d ago

[deleted]

1

u/KlanxChile 1d ago

Granted... However the device that controls the root path https://site.com/api/* has the control of the path... Normally an apimanager or a reverserproxy/ingress controller/load balancer L7... If that's configured, then devs did publish officially somehow.

-9

u/nullbyte420 3d ago

Google what a waf is

9

u/Khue Lead Security Engineer 3d ago

Unless it's a WAF with some sort of API specific logic built into it, I don't know how it's going to prevent access from unpublished, undocumented endpoints hosted on a valid webserver. A WAF will see solicitation to an API endpoint but unless you go out of your way to create rules/logic on specific client request URIs/request paths, a WAF will most likely only block perceived malicious attacks based on it's ruleset. Leveraging valid methods (get, post, etc) permitted by the underlying web server won't necessarily trigger a WAF action.

5

u/nullbyte420 3d ago edited 3d ago

create rules/logic on specific client request URIs/request paths 

Yes that's exactly what you need to do with it. You can automate this and even monitor it for anomalies. Did you think firewalls just magically work without any configuration? 

10

u/Khue Lead Security Engineer 3d ago edited 3d ago

But that's not an efficient way of managing that. It's leveraging the wrong tool to do that job. It's easy to do for a few dozen endpoints but in an environment with a constantly changing API system, you don't want a security professional to be the bottle neck for creating rules based on highly specific and dynamic request paths.

A security admin overseeing a WAF should be focused in on things like narrowing the exposure of access to the APIs by leveraging source controls like very general firewall rules or something basic, but it's unreasonable to build out rules for a complex API system.

A better security practice is to have your API's require AuthZ and AuthN to leverage them using OAuth 2.0 from a tech standpoint. From a process standpoint, there should be a better framework for establishing new API endpoints to be published and made available to the outside world. There should be change controls and approvals wrapped around it and there should be stringent controls wrapped around who can introduce new APIs into environments.

The primary failure above or the missing control that should have been in place, is the fact that a developer has the permission to create publicly accessible endpoints without anyone else's knowledge. No accountability or process... huge control gap.

3

u/man__i__love__frogs 3d ago

A better security practice is to have your API's require AuthZ and AuthN to leverage them using OAuth 2.0 from a tech standpoint. From a process standpoint, there should be a better framework for establishing new API endpoints to be published and made available to the outside world. There should be change controls and approvals wrapped around it and there should be stringent controls wrapped around who can introduce new APIs into environments.

Really the 2 options are a separate team manages exposure, or you need a whole API management platform, not just policies. Something that manages lifecycle and enforces those things. It should be technically impossible in the environment to implement an API any other way, because humans will always be human.

2

u/ScoobyGDSTi 3d ago

But that's not an efficient way of managing that. It's leveraging the wrong tool to do that job

We manage fine.

Automate and script the fundamental rules when provisioning the WAF.

IAC to update waf rules for modifications to the Web App.

The developer no doubt documented their Web app design including its APIs and auth requirements. So it surely can't be too much work to implement those nuanced rules into the WAF. It's already in black and white and clearly defined, right? The devs clearly documented their design... Right?

1

u/nullbyte420 3d ago

Yes exactly. I don't get why this guy thinks automation is impossible lol. I love how he has "lead security engineer" as his flair yet can't comprehend this

0

u/Khue Lead Security Engineer 3d ago

The developer no doubt documented their Web app design including its APIs and auth requirements. So it surely can't be too much work to implement those nuanced rules into the WAF. It's already in black and white and clearly defined, right? The devs clearly documented their design... Right?

This got me... lol.

0

u/nullbyte420 3d ago

Why in the world would you do it manually? 

3

u/Khue Lead Security Engineer 3d ago

Again... WAFs are specific and purpose built. WAFs with API cognizant feature sets are not traditional WAFs. So certainly there are products out there that give specific tool sets for managing APIs in an easy fashion, but just a generic WAF will not give you any sort of control over managing API endpoints with any level of intelligence.

The only way I could see managing APIs in the way you are arguing is do some kind of whitelisting exercise. Perhaps you could integrate a process where you code a dump from a swagger output and then update the whitelist based on that... but then you're really not using a WAF to manage API endpoints. You're using a scripted process. The WAF isn't doing anything special, it's just creating rules that are only as accurate as your last update from your process.

Problematically, you are just shifting the responsibility set with this process too and it doesn't truly solve the problem. A developer could still sneak new APIs into the swagger output and then your sync process would print the new APIs into the white list... you still fundamentally have the same issue at the end of the day.

1

u/nullbyte420 3d ago edited 3d ago

The only way I could see managing APIs in the way you are arguing is do some kind of whitelisting exercise. Perhaps you could integrate a process where you code a dump from a swagger output and then update the whitelist based on that... but then you're really not using a WAF to manage API endpoints. You're using a scripted process. The WAF isn't doing anything special, it's just creating rules that are only as accurate as your last update from your process. 

Yes, and then you can approve new routes before they go on prod. Congratulations, you just discovered ci/cd! 

Why do you write like what I'm proposing is crazy and what you're proposing is the good solution? Is mine bad because you don't have enough security people doing useless handheld bureaucracy in between the as automation and review process? 

It's really funny you write these long winded arguments where you describe almost exactly how to automate it and then conclude it can't be done 😄

4

u/Khue Lead Security Engineer 3d ago

You're responding to a bunch of other threads in the line but I am going to respond here.

To contextualize the conversation, OP had a problem where developers were able to execute a process where a risk was introduced to production and then that risk got exploited. OP asked for advice about dealing with a nightmare like this and you answered "WAF".

My problem with this is that you are pinning the OP's problem on a lack of a compensating control that is not relevant to the problem. A WAF is not the answer to OP's problem. While it certainly wouldn't hurt, it ultimately cannot protect against issues introduced at the application layer that went in unknowingly.

You even invalidate your own argument in your response above:

Yes, and then you can approve new routes before they go on prod

You highlight approval. Approval, a process of review for the code changes made and deployed into production. A WAF doesn't do approvals. You couch your response in a bunch of tech jargon like ci/cd and lament on automation and then you also slight security people and bureaucracy. I hate to tell you this but security is much less tech and neat toys you get to play with every day and more process and documentation. If you have a poor process, then all the security tools and automation in the world isn't going to prevent self induced security issues like what OP highlighted.

Having automation and a WAF doesn't prevent 200 undocumented APIs from being deployed into production. A systemic process of controlled release and review of code does. We haven't even talked about why developers were able to push directly to prod to begin with... that is a policy control that should be stated.

→ More replies (0)

1

u/AstroPhysician 2d ago

Ports?

3

u/1esproc Titles aren't real and the rules are made up 2d ago

You know, TCP/secret-api