API4:2019 Lack of Resources & Rate Limiting

In the previous post we covered API3:2019 Excessive Data Exposure, which was the third post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

You can read the official document from the OWASP Project team here.

Tanya on stage
Who here has perfectly secure APIs? What? No hands? – OWASP Global AppSec, Ireland, Feb 2023

Before diving into this one, I want to briefly discuss bots online.

An Internet bot, web robot, robot or simply bot, is a software application that runs automated tasks over the Internet, usually with the intent to imitate human activity on the Internet, such as messaging, on a large scale. – Wikipedia

While bots can be great (I used to have an automated message for all new twitter followers, to greet them and make them feel welcome), they can also be quite bad (slowly eating away at our defenses, with automated requests).

Bots are one of the quiet enemies of APIs. They hike up our cloud bills, they make our APIs seem unresponsible or slow, and they can be used to brute force an API that is not properly protected.  Because APIs can do so many things, it is possible for them to eat up all sorts of resources on your network, such as CPU, storage and memory on the host of the API, or whatever the API is calling.

There are all sorts of ways that APIs can have their limits tested, including: uploading very large files or amounts of data, making several requests at once, requesting huge amounts of data (above what the system or supporting infrastructure can handle), etc.

Setting Boundaries

The OWASP API Security top ten team recommends setting limits on the following settings:

  • Execution timeouts
  • Max allocable memory
  • Number of file descriptors
  • Number of processes
  • Request payload size (e.g., uploads)
  • Number of requests per client/resource (this is also called resource quotas)
  • Number of records per page to return in a single request response

So how do we avoid this happening to our APIs?

  • We can set throttling limits, to slow down requests that all come from the same source .
  • We can add resource quotas, limits to how many requests someone can make, and then they have to wait a time period to start making requests again.
  • Docker containers has several options built in for adding the limits described earlier in this article, as suggested by the team that maintains this great OWASP project.
  • Send messages back to whoever is calling the API ‘too much’, informing them they’ve reached the limit, and that now they must wait.
  • Design your API to ensure it takes into account if requests are “too big”. This is something threat modelling could help with, but ideally you would start with looking at each function and thinking about this as a problem your API will face at some point. Design with this in mind.
  • Also design into your API maximum amounts of data that it can accept and that it can return to the caller.  This might mean breaking a large request into multiple responses or blocking it altogether. This is something you should talk to your team about, ideally during the requirements or design phase(s) of your project.

They provide several very helpful resources, which you can find here:

OWASP Resources!

Even more resources!

In the next blog post we will be talking about API5:2019 Broken Function Level Authorization!

API3:2019 Excessive Data Exposure

In the previous post we covered API2:2019 Broken User Authentication, which was the second post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

#RSAC 2022

Excessive data exposure is something that web applications can face, not just APIs. That said, because web-based APIs are basically services on the web, they can be abused even more easily to exfiltrate sensitive data than a regular web app. It’s easy for an attacker to find APIs (just connect to any web app or mobile app using a web proxy and see the API calls for yourself!), to call them, and then look at the responses to see if anything being sent looks potentially sensitive. For instance, if the data in a field passed back is named “password”, “sin” or “secret”, you’re most likely onto something.

Using a web proxy to watch the API calls go back and forth is sometimes called “sniffing”, but no matter what you call it, it’s easy to do! Anyone with the tiniest amount of web-app hacking training can do this on day one. This means this threat is prevalent (happens all the time) and very dangerous (because unsophisticated attackers can easily execute it).

Some APIs are *supposed* to return sensitive data. This vulnerability is when sensitive data is exposed to someone it should not be (for instance, someone who is not a valid user, seeing another user’s sensitive data, or for whom that specific data should not be shown due to their role within the system). Since whether data is sensitive in nature is not obvious to automated testing tools, it can be a bit more difficult to identify than other types of vulnerabilities.

* Note: occasionally the vulnerability rears its head via poorly-generated and/or overly-populated responses. For instance, the API delivers the entire table worth of data, which includes sensitive information, but then the client-side front-end sifts through it and only reveals the non-sensitive/appropriate data to the end user. Unfortunately, if the API call is not encrypted in transit, this means a malicious actor could see all of the data if they were sniffing the API at the time.

How do we avoid this?

Let’s look at some great advice from the project team (I may have added a bit onto their list):

  • Never rely on the client side to filter sensitive data. By this we mean, only return the data you need to return! Don’t send a ton of stuff you do not need to, then let the GUI/front end decide what to show the user. Make these important decisions on the server.
  • Classifying then label all your data. If you know immediately when you look at something that it is sensitive, it’s automatic to treat it in a certain way. Educate your developers and other areas of IT on how to classify data, and to ask the security team if they aren’t sure.
  • In the design of your API, add user stories and/or threat models around this potential vulnerability. Making protecting sensitive data part of your design.
  • Review the responses from the API to make sure they contain only legitimate data, data that the specific user (or users with that role inside your system) are allowed to access.
  • Back-end engineers should always ask themselves “who is the consumer of the data?” before exposing a new API endpoint. Or better yet, perform threat modelling on your data flows, THEN design.
  • Avoid using generic methods such as to_json() and to_string(). Instead, cherry-pick specific properties you really want to return. You do not need to return everything. In fact, it’s better for your cloud bills to return only what you need, even if it requires a bit more programming.
  • Classify sensitive and personally identifiable information (PII) that your application stores and works with, reviewing all API calls returning such information to see if these responses pose a security issue.
  • Implement a schema-based response validation mechanism as an extra layer of security. As part of this mechanism define and enforce data returned by all API methods, including errors.
  • Perform strict linting on your API definition file, to ensure you have input validation built-in, by default, for every variable.

In the next blog post we will be talking about API4:2019 Lack of Resources & Rate Limiting!

API2:2019 Broken User Authentication

In the previous post we covered API1:2019 Broken Object Level Authorization, which was the first post in this series. Feel free to start there and work your way back to here if you want! Or just keep on reading.

You can read the official document from the OWASP Project team here.

Tanya Janca presenting "Shifting Security Everywhere" at OWASP Global AppSec, Dublin, Ireland, Feb 2023
Tanya Janca presenting “Shifting Security Everywhere” at OWASP Global AppSec, Dublin, Ireland, Feb 2023

Broken User Authentication is a vulnerability that can affect any web app or API that has user accounts, which is a very large percentage of applications currently on the internet. The app or API asks the user “Who are you?” (authentication) and if the user is able to trick the app into ‘recognizing them’ when they are not a valid user, that is broken authentication.

When someone exploits this vulnerability, they have access to a system they should not, and quite likely, sensitive data of that user, plus the ability to change account settings of whatever account they have broken into. This is a total nightmare for the user whose account has been compromised, not only the security team.

The most common ways this vulnerability is exploited would be via credential stuffing (trying thousands of stolen credentials to see if any of them work), and brute force attacks (allowing end users to try over and over again to guess a password and not locking them out when they deserve it).

Other types of attacks of this nature include:

  • Allowing users to make very weak passwords, such as “Winter2023”, which is likely the English-speaking-world’s most popular password this very moment. </sigh>
  • Sending auth-tokens in insecure ways, such as in an unencrypted/insecurely configured cookie, or the URL parameters
  • Not checking if your auth tokens are valid, every-single-call/page/action/time
  • Using auth tokens incorrectly/with poor config/very old versions they shouldn’t use anymore
  • Following all the right programming steps, but then sending it using very insecure encryption, such that it the data can be decrypted
  • Insecure storage of passwords (passwords should be salted and hashed, possibly peppered, but never plain text or encrypted)

How can I ensure this never happens?

  • Follow auth best practices and *ideally* use an API gateway. API Gateways can handle a lot of this for you, and that makes life WAY BETTER. Writing your own is very complex, time consuming, and potentially risky.
  • Test. Test this manually, and with tools. Test this thoroughly and if you make changes test it again.
  • Threat model this or make a security user story about this. It’s a user story/potential threat to almost every app and API on the net. Take it very seriously.
  • This OWASP Resource from the Cheat Sheets Project is extremely helpful: OWASP Authentication Cheatsheet

Advice from the OWASP Project team (it is also excellent!)

  • Make sure you know all the possible flows to authenticate to the API (mobile/ web/deep links that implement one-click authentication/etc.)
  • Ask your engineers what flows you missed.
  • Read about your authentication mechanisms. Make sure you understand what and how they are used. OAuth is not authentication, and neither is API keys.
  • Don’t reinvent the wheel in authentication, token generation, password storage. Use the standards.
  • Credential recovery/forget password endpoints should be treated as login endpoints in terms of brute force, rate limiting, and lockout protections.
  • Use the OWASP Authentication Cheatsheet. It’s very good!
  • Where possible, implement multi-factor authentication.
  • Implement anti brute force mechanisms to mitigate credential stuffing, dictionary attack, and brute force attacks on your authentication endpoints. This mechanism should be stricter than the regular rate limiting mechanism on your API.
  • Implement account lockout / captcha mechanism to prevent brute force against specific users. Implement weak-password checks.
  • API keys should not be used for user authentication, but for client app/ project authentication.

In the next blog post we will be talking about API3:2019 Excessive Data Exposure!

The OWASP API Security Top Ten: API1:2019 Broken Object Level Authorization

The following blog series will be on the OWASP API Security Top Ten, 2019 version. The new version is coming out shortly, and we will add even more to this series when the time comes.

What is “The OWASP Top Ten”?

OWASP is an international non-profit, with a huge community, over 100 active open source projects, and over 300 in-person chapters around the world. The thing they are most well known for is “The OWASP Top Ten”, an awareness document created to teach the world about the most dangerous risk to web applications. Over the years, various project teams have created more “Top Ten” documents for Serverless, IoT, and APIs!

You can visit the project page here, and see the PDF from the project team here. Thank you very much to the project team who worked so hard on this, especially the leaders: Erez Yalon, Inon Shkedy and Paulo Silva. We Hack Purple applaud your efforts to help make the internet a safer place!

Now down to business! Let’s dive into #1 on this infamous list.

"Shifting Left is not enough." Tanya Janca presenting at OWASP Global AppSec, Dublin, Ireland, Feb 2023
“Shifting Left is not enough.” Tanya Janca, presenting at OWASP Global AppSec, Dublin, Ireland, Feb 2023

API1:2019 Broken Object Level Authorization

Back in the day, PenTesters used to be able to “minus 1” from any USER ID located in the URL parameters and they could quite often see the previous user’s account data. The URL parameters would have “userid=622”, the tester switches it to “userid=621”, and suddenly they were reporting a vulnerability. Fast forward to today, and although most Web Apps don’t fall for this trick anymore, unfortunately APIs are often in a state where changing the parameters a bit can fetch all sorts of data that they should not be able to access. It’s very easy using a intercept web proxy to sniff the API calls, see the parameters, and change the value of one of them. This vulnerability is the most-often exploited and most-damaging of everything listed in the API Security Top Ten document, which is why it’s #1.

But how bad is it really?

It’s pretty bad! It can result in sensitive data exposure (confidentiality broken), changed data (integrity ruined), deleted data (no availability) and even complete account takeover. No one wants this for their users.

How does this happen?

When the IT world moved from monolithic applications into the world of micro service architecture, a few things were lost in the translation. Monolithic applications have been performing session management for a very long time, meaning it knows what state it is in, and thus they can see when an attacker tries to access things it should not. Session management (keeping track of which user is logged in, and that access they are allowed) isn’t native to RESTful APIs. REST APIs are *supposed* to be stateless. And that creates a problem for us since we need to keep track of this in order to avoid this vulnerability.

Every time any user asks for access to anything, we should validate that they are 1) still the user we think they are (we validate the session) and 2) that they are allowed access or have permission to see what they want to see or do what they want to do. Every single time we must check. That one time we forget, is when PenTesters and malicious actors alike do a happy dance, because we have left open a hole in our armour.

How do we avoid this?

As you might have guessed, the OWASP API Security Top Ten Project Team has some thoughts on the matter! We’ve added some thoughts to each item below, please see the original document for a more-succinct description.

  • Implement a proper authorization mechanism that relies on the user policies and hierarchy. There are tools/products that you can buy that perform these functions for you. This is one of those programming things that is difficult to get right, and most security folks recommend you buy a well-trusted solution over attempting to build your own, due to complexity and cost.
  • Use an authorization mechanism to check if the logged-in user has access to perform the requested action on the record in every function that uses an input from the client to access a record in the database. Check every single time.
  • Use random and unpredictable values as GUIDs for records’ IDs. Do not use incremental, guessable, numbers. If a GUID or other record ID is sent to your system that your never issued, trigger an alert and block that IP address immediately, because your API is under attack.
  • Write tests to evaluate the authorization mechanism. Do not deploy vulnerable changes that fail the tests. Ensure the entire project team understands that failing any of these tests blocks all releases to production until they are fixed. You must pass these tests to get to prod.

Helpful links from the OWASP API Security Top Ten Project Team!

In the next article we are going to discuss API2:2019 Broken User Authentication!

From the OWASP API Security Top Ten!