API10:2019 Insufficient Logging & Monitoring

In the previous post we covered API9:2019 Improper Assets Management, which was the 9th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization. You can see the formal document from the OWASP API Security Top Ten Team, here.

Photo by Ales Krivec on Unsplash

Years ago, I worked at Microsoft, as a developer advocate. A few of us were working together, creating demos, for ‘Microsoft Ignite The Tour’. I recall two of my colleagues asking me why Azure couldn’t detect an SQL injection attack they had done as part of their demo. I checked their settings and explained that logging and monitoring was turned off. If monitoring is off, that means there was no observation of the application. How would an attack be noticed if they had removed Azure’s ability to watch what as happening? I also explained that since they had also turned off logging, that would mean that an incident responder would have nothing to investigate. They would not only miss the attack happening, but they would also never be able to find out later what had happened if they investigated. It had been turned off to save money (we didn’t want to spend a small fortune just on demonstrations, we had a budget). They turned both logging and monitoring back on, tried the attack, and immediately Azure went into red alert. All was well for the developer advocates and our demos.

Imagine finding your data on the dark web for sale, and not even knowing how it got there. Obviously, this has never happened to me before at a client site… If it had, I would tell you how incredibly frustrating it is not to be able to explain what happened, and therefore ensure it could never happen again. You can’t do that if you have no logs AND no monitoring. Again, this is totally hypothetical and definitely did not happen to me or any of the clients that I have worked with.

– Not me

Back when I was a full-time developer, I remember asking to turn on logging. I had asked the client during the requirements phase, explaining why I wanted it (so we could provide better reliability, and investigate any outages, there was no security slant for me, at the time). The client had agreed immediately. Then we got into the costing phase, trying to calculate how much the final project would be. When the client saw how much logging was going to cost, it was cut immediately. I had this happen several times as a dev, always being told it was a cost-saving measure. Although I didn’t love this decision at the time, it wasn’t a hill I was going to die on.

Fast forward 8 years to when I got my first AppSec job. I recall us having a security incident, and me being able to search through the logs and find the attack in about 30 minutes (the logs were HUGE, and I didn’t have log viewing software, that’s why it took so long). I learned powershell that day, or, the basics of powershell, and wrote a script to de-obfuscate, so then I could see the exact attack commands. It took me quite bit longer to figure out a perfect timeline, and where our AppSec program had broken down to allow this vulnerability into prod…  But that said, I realized that logs were so incredibly valuable for investigating a security incident, there’s just no other way you can find our exactly what happened without them!

Over the years I have learned that 1) working in incident response is absolutely fascinating and 2) I become far, far too stimulated to do incident response work on a regular (full-time) basis. I have a lot of respect for people who do that type of work full time.

– Tanya’s Thoughts on Incident Response
Photo by Charlotte Harrison on Unsplash

Around this time, I also learned that sometimes attackers will modify the logs, erasing their tracks as part of the attack. One of the ways that logs can be manipulated is via attacks against user input fields, where the attacker bypasses the input validation, that input is logged, and that type of attack is referred to as “log injection”. Attacks against the integrity of our logs is the reason I always go on and on about why we need to protect our logs, and back them up to a secure location. Ideally logs should be protected because they are *sensitive information*, they are literally evidence that could be used one day in court. We should treat them as the precious resource they are, a living record of all that has happened to our applications.

When an API, or any other IT system, has logging and monitoring turned off, or have improperly configured or insufficiently protected their logs, this vulnerability applies. It can apply to any IT system, but APIs are the focus of this blog series, and thus we shall concentrate on how to find, avoid, and solve this problem in APIs specifically.

Let’s talk specifics!

  1. Turn on monitoring. Give your monitoring system contact info for the correct people (it should not go to an unmonitored inbox, or phone number that no one answers). Someone needs to receive the alerts, otherwise why bother to pay for monitoring…
  2. You should log every activity that has to do with a security control, even failed ones. Logins, log outs, changes to privilege, account creation or deletion, password changes, authentication and authorization, input validation, system errors (especially if the global exception handler gets called), changing the contact info for the account, etc.
  3. Do not log sensitive info. Examples of sensitive info: complete credit card numbers, name + home address, name + date of birth, SIN/SSN, the text entries for failed password attempts (those are often typos that would allow you to guess the password), anything that could identify the person (PII) from the log data alone, personal health data + name or other identifying info, anything else that qualifies for your specific organization, system or customers.
  4. Do log: user ID, time stamp, what the user was trying to do, if they succeeded or not, their IP address and any other identifying information you can get about the user’s computer.
  5. Ideally your logs would be formatted so that your SIEM is able to consume them. It’s not very common for organizations to feed their custom app logs into the SIEM, but I hope this changes over time. It’s incredibly helpful.
  6. Your logs should not be stored on the web server or whatever your app lives on. It should be in a different place, inside/behind the firewall/perimeter. That location should not have execution privileges (read only), and only the incident response team should have access to this system.
  7. Monitor where your logs are stored. If an inappropriate account attempts to access this file server, initiate the incident response process immediately. Part of this process should be stopping whoever is accessing it, but then also investigating if these logs of been previously disturbed or altered in any other way. This might not be the first attempt to mess with your logs.
  8. Backup your logs! In a geographically differing place than your app server.

Download a PDF with more specifics for logging, error handling, and logging, here.

Advice straight from the OWASP API Security Top Ten Project Team:

  • Log all failed authentication attempts, denied access, and input validation errors.
  • Logs should be written using a format suited to be consumed by a log management solution and should include enough detail to identify the malicious actor.
  • Logs should be handled as sensitive data, and their integrity should be guaranteed at rest and transit.
  • Configure a monitoring system to continuously monitor the infrastructure, network, and the API functioning.
  • Use a Security Information and Event Management (SIEM) system to aggregate and manage logs from all components of the API stack and hosts.
  • Configure custom dashboards and alerts, enabling suspicious activities to be detected and responded to earlier.

This concludes the We Hack Purple blog series on the OWASP API Security Top Ten! Thank you to the volunteers of that project for all of their hard work to create this list and share this information with the world. Hopefully soon they will release the next version, and we can write more posts about their amazing research!

API9:2019 Improper Assets Management

In the previous post we covered API8:2019 Injection, which was the 8th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization. You can see the formal document from the OWASP API Security Top Ten Team, here.

Inventory Photo by Petrebels on Unsplash

Photo by Petrebels on Unsplash

Taking inventory is the first thing I do whenever I start or join an AppSec program. Figuring out all the applications and APIs that an organization has built, bought (COTS), or are using (SaaS), then doing a fast evaluation of the state they are in, is the best way to figure out where an organization is at regarding their security posture. Doing this helps me know just how much work we have to do and sets the stage for future conversations with management and the developer teams on how we can get them on track for securing all of their apps.

This vulnerability is the reason I start with inventory. I would argue that the majority of organizations around the planet do not have a current and accurate inventory of all of their web assets, including APIs. If they aren’t in the inventory, that means these assets aren’t being managed, which usually also means the security team does not have them on their radar. If the security team doesn’t know about an asset, how can they secure it? Not being in the inventory generally also means no testing, monitoring, logging, or documentation, at a minimum.

Taking inventory (regularly or continuously), and ensuring we properly decommission old versions of APIs when we release new versions, is the way we avoid this vulnerability. Then document all of it, or update documentation as you update your inventory. I realize this is easier said than done!

I recall a penetration tester telling me years ago that one of his tricks for finding vulnerabilities during an engagement was to try to call earlier versions of any APIs that were in scope. If there was a version 2.x, he would try to call version 1.x. He told me that at least once every year he would get a response of a phantom API. And that API was always a complete security disaster. He would earn his entire paycheck with that one Postman call.

PenTester Name Redacted

APIs and web applications that are not part of your inventory are generally also unmonitored, meaning no one is watching them to see if something goes wrong. They are often not behind a WAF, API gateway, or any other shielding that might protect them from common threats. If they are not a part of your inventory, there’s also a good chance that there’s no team in charge of maintenance, meaning no bug fixing is happening and technical debt is accruing. Lastly, it’s very unlikely that they are receiving regular security testing, or any type of security scanning, which can lead to all sorts of problems building up, invisibly.

Cheese melting in the hot sun. Image compliments of https://drawception.com.

Software doesn’t age like wine, getting better over the years. Software ages like cheese in the hot sun; extremely badly! The longer we do not update, test, or patch our software, the more likely it is to have vulnerabilities found within it. Without proper care, software accrues technical debt, which can make it even more difficult to fix security vulnerabilities, because you have to update so many different components (framework, plugins, operating system patches, etc.) in order to fix the real problem at hand (the vulnerability).

The risks of having APIs (or web apps) that are not a part of your inventory and maintenance plans has no bounds. Any type of vulnerability could happen, as no one is watching or paying attention, except perhaps malicious actors. Attacks upon such resources could result in damage to the availability of the system, sensitive data exposure, changes to the data leading to poor integrity, and worse. This makes the risk of this vulnerability very high. On top of no one knowing that the API exists and is live in production, there’s very likely to be little or no documentation about this API. This situation brings me to back to when I was a dev, and the DBA told me I wasn’t allowed to kill an old database server (I wanted to repurpose it), because there were a whole bunch of scripts on there. She said she has no idea which scripts did what, but she turned the server off once and “everything broke” (including payroll being missed for the entire company, yikes!). She said, “Do not touch, I don’t care why, buy a new server!” The DBA lady meant business, so I got a new server. That said… What if there had been documentation? This situation could easily happen to a company with unknown APIs running wild over their network…

Advice From the OWASP Project Team: ‘How to Prevent’

  • Inventory all API hosts and document important aspects of each one of them, focusing on the API environment (e.g., production, staging, test, development), who should have network access to the host (e.g., public, internal, partners) and the API version.
  • Inventory integrated services and document important aspects such as their role in the system, what data is exchanged (data flow), and its sensitivity.
  • Document all aspects of your API such as authentication, errors, redirects, rate limiting, cross-origin resource sharing (CORS) policy and endpoints, including their parameters, requests, and responses.
  • Generate documentation automatically by adopting open standards. Include the documentation build in your CI/CD pipeline.
  • Make API documentation available to those authorized to use the API.
  • Use external protection measures such as API security firewalls for all exposed versions of your APIs, not just for the current production version.
  • Avoid using production data with non-production API deployments. If this is unavoidable, these endpoints should get the same security treatment as the production ones.
  • When newer versions of APIs include security improvements, perform risk analysis to make the decision of the mitigation actions required for the older version: for example, whether it is possible to backport the improvements without breaking API compatibility or you need to take the older version out quickly and force all clients to move to the latest version.

In the next blog post we will be talking about API10:2019 Insufficient Logging & Monitoring.

API8:2019 Injection

In the previous post we covered API7:2019 Security Misconfiguration, which was the 7th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

Injection has been part of the original ‘OWASP Top Ten Risks to Web Apps’ list since the very beginning. Injection happens when an attacker is able to trick an application (or API) into executing malicious code. It does this by adding code to a place in the application where data belongs, and the app becomes confused, and then executes it.

Think of a search field at the top of any website. Imagine if instead of entering in your search term, you added a bunch of code. Then imagine the application executes the code you added. That code would execute with the full authority of that application, and all the same access, behind your firewall. It could result in damage to a database, a web service, your LDAP system, and potentially even worse, assuming there are other vulnerabilities the attacker can combine with this one.

Unfortunately, APIs are subject to this vulnerability, just like a regular web app. No front end does not protect us from injection.

Injection
Photo by Diana Polekhina on Unsplash

What types of injection exist?

If there’s code involved, someone will try to inject their own code. SQL, LDAP, or NoSQL queries, OS commands, XML parsers, and ORM are all potentially problematic (list provided by the OWASP API Security project team). Even Mongo DB databases, that don’t use the SQL language, are potentially vulnerable to NOSQL injection.  

Special note on XSS: Cross Site Scripting (XSS) is also a form of code injection, but it has it’s own classification because of the following reasons:

  • It occurs in the browser, as opposed to back on the server side like every other form of injection.
  • Only works with javascript (because that’s all browsers execute).
  • Is incredibly prevalent, so much so that OWASP felt it was necessary to give it it’s own category.
  • Has several defenses made just for this one vulnerability (cookie settings, and security headers).
  • Does not work on APIs, because they have no GUI front end, meaning no browser.

What do we do?

Hopefully by this point you agree that injection is dangerous and should be remediated as soon as possible if you find it in one of your apps. But how do we find it? How do we fix it? How can we ensure this never happens? Dear reader, secure coding is my favorite topic! Let’s go!

Finding Injection

First off you want to go through your APIs and figure out if you have injection. The most expensive way to do this would be to hire a PenTester to find and then test all the APIs. A cheaper and more sustainable way to do this would be to

  1. Buy a tool that can find all your APIs for you (No, I am not going recommend one at this time, there are several on the market of various qualities, and prices). This function is called “inventory” or “enumeration”.
  2. Run a SAST (static application security testing tool) on all of the APIs, ideally a next gen one, that has low false positives. Fix anything that says injection.
  3. Use a linter on your API, ensure you have completed your API definition file, as per the linter’s instructions. If you can find an API-specific linter, all the better.
  4. Run a DAST tool on the APIs that is made for APIs OR, use an old school DAST but first ensure you’ve linted your API perfectly, so it can hopefully do a good job. It will be easier and faster if you have an API-specific testing tool. Fix anything that says injection.

Fixing Injection

“That’s nice you told me to fix it. Exactly HOW do I do that?”

The first defense for injection is thorough input validation on any input to your app. This means data in the parameters, in a data field, in a hidden field, from an API you called, from the database, any input to your app needs to be validated that it is what you are expecting. What type is it? What size? What’s the content? Is it what we are expecting? If not, reject.

This is functionality is best performed using an approved list, on the server side. By ‘approved list’, we mean using a list of stuff you know is good, rather than a list of what you know is bad. It’s easy for malicious actions to get around a block list, using encoding, obfuscation, and other tactics. But if you give a regular expression and say “if it’s not in here, I’m just not having it”, bad things cannot get in.

As an example, imagine you have a username. It likely accepts numbers and letters. You could use a regular expression (REGEX) like this to say what is okay: [a-z,A-Z,0-9]. That’s an approved list or ‘accept list’. If instead you try to block bad characters such as <, >, ‘, “ and more, you (and your app) are in for a world of hurt.

The next thing you want to do is ensure you perform this check on the server side. Do not do it on the client side, and by this, I mean in the browser/JavaScript. Anyone with a web proxy can get behind your JavaScript in about 5 seconds, unfortunately. If you want to check in your JavaScript for speed, you can do that, in addition to checking on the server.

Input validation is defense number #1. Other defenses include:

  • Always using parameterized queries when making requests to any database (even non-SQL databases long mongo DB). It takes away the ability for it to be interpreted as code.
  • Use output encoding when you put stuff onto the screen. Some frameworks do this for you by default. It takes away and superpowers of the characters, before it puts them on screen, making XSS impossible. Okay, maybe this is only for XSS and doesn’t apply to injection in general, but I would still do it if I were you.

Preventing Injection

If we want to prevent injection (and a myriad of other vulnerabilities), follow this advice:

  • Have your development team take a secure coding course. It can be free or paid, formal or informal, live or recorded, interactive or lecture, the only important part is that they learn. Do the type of training that works best for you and your team.
  • Follow a secure system development life cycle (S-SDLC). Add security steps to each part of your SDLC, such as security requirements, code review, or threat modelling.
  • Ensure your application has thorough testing, which can mean any or several of the following: static analysis, dynamic analysis, manual code review, penetration testing, stress testing, performance testing, unit testing or any other testing you can think of!
  • Whenever possible, use modern and up-to-date frameworks that have security features built in. JavaScript frameworks like Angular and React have so many cool features that help protect your users! They aren’t just nifty dev tools, they can help you build stronger, tougher apps.
  • Never stop learning. Keep reading, studying, learning and hacking.

How To Prevent: OWASP API Security Top Ten Team Advice!

Preventing injection requires keeping data separate from commands and queries.

  • Perform data validation using a single, trustworthy, and actively maintained library.
  • Validate, filter, and sanitize all client-provided data, or other data coming from integrated systems.
  • Special characters should be escaped using the specific syntax for the target interpreter.
  • Prefer a safe API that provides a parameterized interface.
  • Always limit the number of returned records to prevent mass disclosure in case of injection.
  • Validate incoming data using sufficient filters to only allow valid values for each input parameter.
  • Define data types and strict patterns for all string parameters.

In the next blog post we will be talking about API9:2019 Improper Assets Management.

API7:2019 Security Misconfiguration

In the previous post we covered API6:2019 Mass Assignment, which was the 6th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

Security misconfiguration has been on the original OWASP Top Ten list (critical web app risks) for many, many years. It basically means lack of hardening, poor implementation, poor maintenance, mistakes, missing patches, and human error. There’s no difference between web apps and APIs for this; if the server and/or network has not been properly secured, your API may be in danger.

What can happen?

Someone has the giggles.

Because this category of vulnerability is so vague, the risk is anywhere from low to critical, depending upon what you misconfigured and how you misconfigured it. It could result in a complete system compromise, damage to the confidentiality, availably, and/or integrity of your system, and a plethora of other issues. It could result in as little as embarrassing error messages for the attacker, but no actual impact. That said, this vulnerability should not be taken lightly, it’s on this list for a reason.

How do we avoid such a fate?

Prepare for me to sound like a broken record:

  • Follow a secure system development life cycle that includes extensive testing of both the application later, but also the network and infrastructure layer.
  • Following the hardening guide for all infrastructure, middleware, COTS, and SaaS products
  • Scan (apps, network, infrastructure) continuously
  • Create and follow a fast and effective patching process
  • Monitor and log all apps, APIs and any other endpoints you have, for potential danger and/or attacks
  • Ensure you have access for configuring all of these systems locked down, using the principal of least privilege
  • Have an up-to-date and effective incident response (IR) process, and a well-trained IR team

I realize that this blog post is probably not only a bit underwhelming, but you may feel that I have greatly simplified how to avoid this problem. If you feel this way… You’re right. Creating and implementing an effective patch management process in an enterprise is HARD. Continuous scanning is HARD. Getting people to fix misconfigurations (or any vulnerability) that you’ve found is REALLY HARD. None of the things on the list above are easy. Let’s see what the Project Team suggests.

How To Prevent


The API life cycle should include:

  • A repeatable hardening process leading to fast and easy deployment of a properly locked down environment.
  • A task to review and update configurations across the entire API stack. The review should include: orchestration files, API components, and cloud services (e.g., S3 bucket permissions).
  • A secure communication channel for all API interactions access to static assets (e.g., images).
  • An automated process to continuously assess the effectiveness of the configuration and settings in all environments.


Furthermore: (From the project team)

  • To prevent exception traces and other valuable information from being sent back to attackers, if applicable, define and enforce all API response payload schemas including error responses.
  • Ensure API can only be accessed by the specified HTTP verbs. All other HTTP verbs should be disabled (e.g. HEAD).
  • APIs expecting to be accessed from browser-based clients (e.g., WebApp front-end) should implement a proper Cross-Origin Resource Sharing (CORS) policy.

OWASP References (The best kind of references!)

In the next blog post we will be talking about API8:2019 Injection.

API4:2019 Lack of Resources & Rate Limiting

In the previous post we covered API3:2019 Excessive Data Exposure, which was the third post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

You can read the official document from the OWASP Project team here.

Tanya on stage
Who here has perfectly secure APIs? What? No hands? – OWASP Global AppSec, Ireland, Feb 2023

Before diving into this one, I want to briefly discuss bots online.

An Internet bot, web robot, robot or simply bot, is a software application that runs automated tasks over the Internet, usually with the intent to imitate human activity on the Internet, such as messaging, on a large scale. – Wikipedia

While bots can be great (I used to have an automated message for all new twitter followers, to greet them and make them feel welcome), they can also be quite bad (slowly eating away at our defenses, with automated requests).

Bots are one of the quiet enemies of APIs. They hike up our cloud bills, they make our APIs seem unresponsible or slow, and they can be used to brute force an API that is not properly protected.  Because APIs can do so many things, it is possible for them to eat up all sorts of resources on your network, such as CPU, storage and memory on the host of the API, or whatever the API is calling.

There are all sorts of ways that APIs can have their limits tested, including: uploading very large files or amounts of data, making several requests at once, requesting huge amounts of data (above what the system or supporting infrastructure can handle), etc.

Setting Boundaries

The OWASP API Security top ten team recommends setting limits on the following settings:

  • Execution timeouts
  • Max allocable memory
  • Number of file descriptors
  • Number of processes
  • Request payload size (e.g., uploads)
  • Number of requests per client/resource (this is also called resource quotas)
  • Number of records per page to return in a single request response

So how do we avoid this happening to our APIs?

  • We can set throttling limits, to slow down requests that all come from the same source .
  • We can add resource quotas, limits to how many requests someone can make, and then they have to wait a time period to start making requests again.
  • Docker containers has several options built in for adding the limits described earlier in this article, as suggested by the team that maintains this great OWASP project.
  • Send messages back to whoever is calling the API ‘too much’, informing them they’ve reached the limit, and that now they must wait.
  • Design your API to ensure it takes into account if requests are “too big”. This is something threat modelling could help with, but ideally you would start with looking at each function and thinking about this as a problem your API will face at some point. Design with this in mind.
  • Also design into your API maximum amounts of data that it can accept and that it can return to the caller.  This might mean breaking a large request into multiple responses or blocking it altogether. This is something you should talk to your team about, ideally during the requirements or design phase(s) of your project.

They provide several very helpful resources, which you can find here:

OWASP Resources!

Even more resources!

In the next blog post we will be talking about API5:2019 Broken Function Level Authorization!

API3:2019 Excessive Data Exposure

In the previous post we covered API2:2019 Broken User Authentication, which was the second post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

#RSAC 2022

Excessive data exposure is something that web applications can face, not just APIs. That said, because web-based APIs are basically services on the web, they can be abused even more easily to exfiltrate sensitive data than a regular web app. It’s easy for an attacker to find APIs (just connect to any web app or mobile app using a web proxy and see the API calls for yourself!), to call them, and then look at the responses to see if anything being sent looks potentially sensitive. For instance, if the data in a field passed back is named “password”, “sin” or “secret”, you’re most likely onto something.

Using a web proxy to watch the API calls go back and forth is sometimes called “sniffing”, but no matter what you call it, it’s easy to do! Anyone with the tiniest amount of web-app hacking training can do this on day one. This means this threat is prevalent (happens all the time) and very dangerous (because unsophisticated attackers can easily execute it).

Some APIs are *supposed* to return sensitive data. This vulnerability is when sensitive data is exposed to someone it should not be (for instance, someone who is not a valid user, seeing another user’s sensitive data, or for whom that specific data should not be shown due to their role within the system). Since whether data is sensitive in nature is not obvious to automated testing tools, it can be a bit more difficult to identify than other types of vulnerabilities.

* Note: occasionally the vulnerability rears its head via poorly-generated and/or overly-populated responses. For instance, the API delivers the entire table worth of data, which includes sensitive information, but then the client-side front-end sifts through it and only reveals the non-sensitive/appropriate data to the end user. Unfortunately, if the API call is not encrypted in transit, this means a malicious actor could see all of the data if they were sniffing the API at the time.

How do we avoid this?

Let’s look at some great advice from the project team (I may have added a bit onto their list):

  • Never rely on the client side to filter sensitive data. By this we mean, only return the data you need to return! Don’t send a ton of stuff you do not need to, then let the GUI/front end decide what to show the user. Make these important decisions on the server.
  • Classifying then label all your data. If you know immediately when you look at something that it is sensitive, it’s automatic to treat it in a certain way. Educate your developers and other areas of IT on how to classify data, and to ask the security team if they aren’t sure.
  • In the design of your API, add user stories and/or threat models around this potential vulnerability. Making protecting sensitive data part of your design.
  • Review the responses from the API to make sure they contain only legitimate data, data that the specific user (or users with that role inside your system) are allowed to access.
  • Back-end engineers should always ask themselves “who is the consumer of the data?” before exposing a new API endpoint. Or better yet, perform threat modelling on your data flows, THEN design.
  • Avoid using generic methods such as to_json() and to_string(). Instead, cherry-pick specific properties you really want to return. You do not need to return everything. In fact, it’s better for your cloud bills to return only what you need, even if it requires a bit more programming.
  • Classify sensitive and personally identifiable information (PII) that your application stores and works with, reviewing all API calls returning such information to see if these responses pose a security issue.
  • Implement a schema-based response validation mechanism as an extra layer of security. As part of this mechanism define and enforce data returned by all API methods, including errors.
  • Perform strict linting on your API definition file, to ensure you have input validation built-in, by default, for every variable.

In the next blog post we will be talking about API4:2019 Lack of Resources & Rate Limiting!

API2:2019 Broken User Authentication

In the previous post we covered API1:2019 Broken Object Level Authorization, which was the first post in this series. Feel free to start there and work your way back to here if you want! Or just keep on reading.

You can read the official document from the OWASP Project team here.

Tanya Janca presenting "Shifting Security Everywhere" at OWASP Global AppSec, Dublin, Ireland, Feb 2023
Tanya Janca presenting “Shifting Security Everywhere” at OWASP Global AppSec, Dublin, Ireland, Feb 2023

Broken User Authentication is a vulnerability that can affect any web app or API that has user accounts, which is a very large percentage of applications currently on the internet. The app or API asks the user “Who are you?” (authentication) and if the user is able to trick the app into ‘recognizing them’ when they are not a valid user, that is broken authentication.

When someone exploits this vulnerability, they have access to a system they should not, and quite likely, sensitive data of that user, plus the ability to change account settings of whatever account they have broken into. This is a total nightmare for the user whose account has been compromised, not only the security team.

The most common ways this vulnerability is exploited would be via credential stuffing (trying thousands of stolen credentials to see if any of them work), and brute force attacks (allowing end users to try over and over again to guess a password and not locking them out when they deserve it).

Other types of attacks of this nature include:

  • Allowing users to make very weak passwords, such as “Winter2023”, which is likely the English-speaking-world’s most popular password this very moment. </sigh>
  • Sending auth-tokens in insecure ways, such as in an unencrypted/insecurely configured cookie, or the URL parameters
  • Not checking if your auth tokens are valid, every-single-call/page/action/time
  • Using auth tokens incorrectly/with poor config/very old versions they shouldn’t use anymore
  • Following all the right programming steps, but then sending it using very insecure encryption, such that it the data can be decrypted
  • Insecure storage of passwords (passwords should be salted and hashed, possibly peppered, but never plain text or encrypted)

How can I ensure this never happens?

  • Follow auth best practices and *ideally* use an API gateway. API Gateways can handle a lot of this for you, and that makes life WAY BETTER. Writing your own is very complex, time consuming, and potentially risky.
  • Test. Test this manually, and with tools. Test this thoroughly and if you make changes test it again.
  • Threat model this or make a security user story about this. It’s a user story/potential threat to almost every app and API on the net. Take it very seriously.
  • This OWASP Resource from the Cheat Sheets Project is extremely helpful: OWASP Authentication Cheatsheet

Advice from the OWASP Project team (it is also excellent!)

  • Make sure you know all the possible flows to authenticate to the API (mobile/ web/deep links that implement one-click authentication/etc.)
  • Ask your engineers what flows you missed.
  • Read about your authentication mechanisms. Make sure you understand what and how they are used. OAuth is not authentication, and neither is API keys.
  • Don’t reinvent the wheel in authentication, token generation, password storage. Use the standards.
  • Credential recovery/forget password endpoints should be treated as login endpoints in terms of brute force, rate limiting, and lockout protections.
  • Use the OWASP Authentication Cheatsheet. It’s very good!
  • Where possible, implement multi-factor authentication.
  • Implement anti brute force mechanisms to mitigate credential stuffing, dictionary attack, and brute force attacks on your authentication endpoints. This mechanism should be stricter than the regular rate limiting mechanism on your API.
  • Implement account lockout / captcha mechanism to prevent brute force against specific users. Implement weak-password checks.
  • API keys should not be used for user authentication, but for client app/ project authentication.

In the next blog post we will be talking about API3:2019 Excessive Data Exposure!

The OWASP API Security Top Ten: API1:2019 Broken Object Level Authorization

The following blog series will be on the OWASP API Security Top Ten, 2019 version. The new version is coming out shortly, and we will add even more to this series when the time comes.

What is “The OWASP Top Ten”?

OWASP is an international non-profit, with a huge community, over 100 active open source projects, and over 300 in-person chapters around the world. The thing they are most well known for is “The OWASP Top Ten”, an awareness document created to teach the world about the most dangerous risk to web applications. Over the years, various project teams have created more “Top Ten” documents for Serverless, IoT, and APIs!

You can visit the project page here, and see the PDF from the project team here. Thank you very much to the project team who worked so hard on this, especially the leaders: Erez Yalon, Inon Shkedy and Paulo Silva. We Hack Purple applaud your efforts to help make the internet a safer place!

Now down to business! Let’s dive into #1 on this infamous list.

"Shifting Left is not enough." Tanya Janca presenting at OWASP Global AppSec, Dublin, Ireland, Feb 2023
“Shifting Left is not enough.” Tanya Janca, presenting at OWASP Global AppSec, Dublin, Ireland, Feb 2023

API1:2019 Broken Object Level Authorization

Back in the day, PenTesters used to be able to “minus 1” from any USER ID located in the URL parameters and they could quite often see the previous user’s account data. The URL parameters would have “userid=622”, the tester switches it to “userid=621”, and suddenly they were reporting a vulnerability. Fast forward to today, and although most Web Apps don’t fall for this trick anymore, unfortunately APIs are often in a state where changing the parameters a bit can fetch all sorts of data that they should not be able to access. It’s very easy using a intercept web proxy to sniff the API calls, see the parameters, and change the value of one of them. This vulnerability is the most-often exploited and most-damaging of everything listed in the API Security Top Ten document, which is why it’s #1.

But how bad is it really?

It’s pretty bad! It can result in sensitive data exposure (confidentiality broken), changed data (integrity ruined), deleted data (no availability) and even complete account takeover. No one wants this for their users.

How does this happen?

When the IT world moved from monolithic applications into the world of micro service architecture, a few things were lost in the translation. Monolithic applications have been performing session management for a very long time, meaning it knows what state it is in, and thus they can see when an attacker tries to access things it should not. Session management (keeping track of which user is logged in, and that access they are allowed) isn’t native to RESTful APIs. REST APIs are *supposed* to be stateless. And that creates a problem for us since we need to keep track of this in order to avoid this vulnerability.

Every time any user asks for access to anything, we should validate that they are 1) still the user we think they are (we validate the session) and 2) that they are allowed access or have permission to see what they want to see or do what they want to do. Every single time we must check. That one time we forget, is when PenTesters and malicious actors alike do a happy dance, because we have left open a hole in our armour.

How do we avoid this?

As you might have guessed, the OWASP API Security Top Ten Project Team has some thoughts on the matter! We’ve added some thoughts to each item below, please see the original document for a more-succinct description.

  • Implement a proper authorization mechanism that relies on the user policies and hierarchy. There are tools/products that you can buy that perform these functions for you. This is one of those programming things that is difficult to get right, and most security folks recommend you buy a well-trusted solution over attempting to build your own, due to complexity and cost.
  • Use an authorization mechanism to check if the logged-in user has access to perform the requested action on the record in every function that uses an input from the client to access a record in the database. Check every single time.
  • Use random and unpredictable values as GUIDs for records’ IDs. Do not use incremental, guessable, numbers. If a GUID or other record ID is sent to your system that your never issued, trigger an alert and block that IP address immediately, because your API is under attack.
  • Write tests to evaluate the authorization mechanism. Do not deploy vulnerable changes that fail the tests. Ensure the entire project team understands that failing any of these tests blocks all releases to production until they are fixed. You must pass these tests to get to prod.

Helpful links from the OWASP API Security Top Ten Project Team!

In the next article we are going to discuss API2:2019 Broken User Authentication!

From the OWASP API Security Top Ten!

Conclusion: Security Champions

In the previous article we talked about Metrics, and in this article, I will conclude this series on Building Security Champions.

A few more tips:

  • Start by defining the focus of your program and what is expected from champions. Be realistic; you can only expect 1-4 hours maximum effort from them per week.
  • If someone is taking a security course, but they are not on the security team, they may make a good champion. Reach out and introduce yourself.
  • If the mantra of the security team is “it’s my job to help you do your job, securely”, “you’re my customer” or “I’m here to serve you”, that is very attractive. If your team is known as ‘the ministry of NO!’, you will have difficulty attracting volunteers until you turn over a new leaf.
  • Record every group session and save them. Create an on-boarding set of champion videos from these recordings, so you can auto-onboard new champions. Some of the videos can also be used to on-board new software developers or other IT staff.
  • Save all the videos so anyone who missed them can see them later. Offer up the list of videos to everyone at your organization, if appropriate.
  • Include a TTT (train the trainer) package so that your security champions can train their own teams as needed. For instance, if you want your champions to give training or talks to their own teams, have them follow your package. The package should contain 1) your slides, 2) demo information and instructions to set it up, 3) a video of you giving the talk/training, and, 4) a video of you explaining what you are trying to get across for each slide and the entire demo, spoken as though you are teaching someone to give the talk on your behalf. For an example of this, see mine!
  • PS… Feel free to give these talks yourself, at your own workplace.

Lastly, don’t stop. Don’t give up. Perseverance is the thing that will make this program work. As your program continues it will grow and the value you that you receive from it will also grow, scaling upwards over time. You and your organization can do this, all it takes is dedication and time.

Please feel free to email me with questions, or even better, tell me about your success with your own security champions program!

Security Champions: Metrics & Data

The previous article in this series is Recognizing and Rewarding Your Security Champions.

If you’ve followed my conference talks, you likely saw my Security Metrics That Matter presentation, and understand that I absolutely love data. Here’s a general list of security metrics that matter, if you don’t want to read the whole article or watch the entire talk.

You may wonder, why are metrics important? The answer is twofold.

  1.  We can use data and metrics to report up to our bosses and show them we are succeeding. It’s evidence that what we are doing is working, and how well it is working. You can then use that data again to ask for more resources (staff, tools, budget), a raise, or other changes.
  2. The second reason is so that we, ourselves, can improve. We want to improve our program, ourselves, and our results. When we measure our activities and their impacts, we can see which activities or methods produce better results. We can then use that information to change our approach, for the better.

It is important, however, that we do not become fooled by vanity metrics. Vanity metrics are numbers that make us look good, but don’t necessarily mean anything. My talk on this subject has several stories, but for now let’s just tell one.

I used to work somewhere, and we all wrote blog posts. We were measured on how many “clicks” we got. A colleague of mine got 10X the number of clicks that I did, and I asked him how he did it. He explained he got the most clicks on Reddit. I was unfamiliar with the platform but thought I would give it a try. First though, I asked for extra data: I wanted to know how long people were staying on our articles. It turned out that people were staying on my articles approximately 1.5 minutes (which means they were reading the whole thing), and on his they were staying an average of 1.5 seconds (which means almost no one was reading the article, they were just clicking the link. This is commonly known as a “bounce”.) The purpose of our jobs was to write articles to help customers know how to use our products, and this means a bounce wasn’t valuable. Armed with this new information, we started comparing different platforms, and it turned out almost all traffic from Reddit were ‘bounces’. I also noticed that my Twitter followers were significantly more likely to read the article when compared to LinkedIn, and LinkedIn got better results than Reddit. My colleague started focussing on sharing links on Twitter (he had more followers than I did), and I started trying to get more followers on the same platform. It turns out that measuring clicks was a vanity metric. The rest, as they say, is history.

Now for your security champion program metrics! Measure the following things so you can see what’s working and what is not. Don’t forget to report upwards about the ROI (return on investment) your champions program has produced!

  • How many new security champions you have attracted
  • Measuring program engagement: how many people attended an event, how many people reported issues to you, how many people asked questions,
  • Use the bug tracker for metrics on how many security bugs are being reported and fixed, especially if you have targeted a specific bug class. Also, count how many new instances of that type of bug appear, hopefully this number will be very low.
  • Instances where champions have told you about a security issue you would not have known about otherwise
  • If the champions report better work satisfaction and/or fewer missed days of work
  • Gather stories of your champs saving the day, providing help to their teammates, or anything else that makes for a good story-telling session for upper management.

Up next, I will share a few more tips that don’t fit into any of the previous categories and conclude this series. Please feel free to email me with any questions!

(Over)Communication With Your Security Champions

As mentioned in the previous article (Recognizing & Rewarding Your Security Champions), the most common reason for failure of a security champions program is the security team losing steam, and/or the champions losing interest. In this article, we will discuss a few ways to avoid this. The best way? Communication.

https://youtu.be/nm1MpTuSNyI

To start off with, pace yourself. Often when I speak to security teams who have a failed program, they tell me how they started off very strong. “We gave them 2 different trainings, 2 workshops, and 3 lunch and learns, all in the first three months. Then we were exhausted. We haven’t done anything with them in over a year.” This scenario is far too common.

To pace yourself, I suggest meeting with each champion once a month, for 30 minutes. Then hold one lunch & learn and send one email to the champions. This might not sound like much, but you must remember, they are already doing a full-time job for your organization.

In my 1:1 meetings I like to ask the following questions (adopted from Ray Leblanc’s Security Champions article on Hella Secure blog):

  • What are you working on?
  • What are you going to work on next?
  • Do you need any help?

Each of these questions is open-ended, with the hope that it will prompt a meaningful conversation. I usually take notes during the meeting, and then send them after to both of us, with any action items for either of us highlighted in bold. (Note: I’ve used this technique to get many of my previous bosses to do things for me. Set a reminder for a week from then, and then reply-all to that email chain and ask: “Any updates on these action items?” It works like a charm!)

In your lunch and learn (which does not need to be at lunch time, or involve food), teach them something you want them to know. Do not teach them things they do not need to know, unless they asked for that topic specifically. During this session you or a teammate can teach, or you can show them a training video you like, or even a recording of a conference talk that really hit home for you. If you show them something pre-recorded, ensure you watched it first, you don’t want to waste anyone’s time with death-by-powerpoint. The more fun you can make these sessions, the better. If you’re up for it, invite all of the developers and let everyone learn something new!

Woman running
Photo by Greg Rosenke on Unsplash

Ideas for lunch and learn topics:

  • The specifics on how to apply policies, standards and guidelines. This could be a secure coding workshop, or a threat modelling session.
  • Talks about the top vulnerabilities that you are seeing in your own products, including the risks they pose to your specific business model.
  • Workshops on how to use the tools that your team wants them to be responsible for. Especially how to configure them, how to validate results, and where to find information on how to fix what they find.
  • If they are responsible for design or architecture, give them secure design training.
  • Tell them about a security incident your team had, and how it could have been prevented (assuming you are allowed to share this information).
  • Hold a consultation on the new policy, standard, or guideline your team is considering publishing. Ask for their feedback, then adjust your documents accordingly.
  • Remember to take attendance (for metrics) and take notes of any questions for you to follow up.

The monthly email:

Sometimes you just don’t have time to do a lunch and learn event or hold 1:1s, but you still need to send a monthly email. The monthly email lets the security champions know what’s going on, and that they still matter to you. The program is still running, because you sent an email. If you don’t send this email, and you haven’t touched base in any other way, this leaves a space where your program may start to disappear.

The monthly email does not need to be fancy and doesn’t need to say a lot. Generally, the monthly email says:

  • What events are happening this month at your org (lunch and learn, all staff, any other meeting they should know about)
  • Any updates your team has (new policy, new tool, project updates, etc)
  • Anything interesting from the news that they may find valuable
  • Any local security events they may be interested in
  • Any podcasts, videos, blog posts or any other media that is relevant and you feel relates to them, about security (of course)

I live in Canada, and in Canada we are a country of immigrants. This means we have many, many different religions represented in most workplaces. In December, there’s Hannukah, Ramadan, Christmas, and more, and often people take time off for these special holidays. This means having a large meeting in December is darn-near impossible. This is the type of situation where you just send the monthly email! It could say something like the following:

Hello Security Champions!

As it is December and many of you will be off celebrating various holidays, we are not going to have any events this month. We also want to wish you happy holidays, and we hope you enjoy all the snow we got this past weekend!

In January we are going to boot the Champions program back up with a lunch and learn on XSS. As some of you are aware, we’ve found it in about 1/3 of our custom apps, and we want to stomp​ it out in the new year (with your help of course!) An invitation will arrive later this week.

In the meantime, please check out this XSS Deep Dive by Tanya Janca. We’re going to cover this topic a bit differently than she does, but it gives you a good idea of what we are up against.

Have a great December folks!

Sincerely,

The Security Team

My hope from this blog post is that you remember to continue to communicate with your champions. Don’t let your program slip, it will disappear faster than you think. When in doubt, send them an email and check in. Up next, we will discuss Metrics.

Recognizing & Rewarding Security Champions

If you’ve ever read the book The 5 Love Languages, or articles summarizing the 5 love languages, then you are aware that there are predictable patterns of how people respond to various acts of kindness. Someone’s “love language” is the specific type of kindness that they are most affected by. For example, someone for whom their love language is “words of affirmation” would respond very well to receiving a glowing performance review, a compliment on a new article of clothing, or accolades from their colleagues about a project they worked on.

The previous article in this series is Teaching Security Champions.

You may be wondering at this point if you accidentally clicked on an article from a women’s fashion magazine, not a technical article from We Hack Purple. But please have a bit more faith, and read on.

The 5 love languages are:

  1. Gifts
  2. Words of Affirmation
  3. Physical Affection
  4. Spending Quality Time
  5. Acts of Service

Two people sitting using laptops
Security Champions at work!

When we are creating a security champions program, it’s very important that we ensure they feel appreciated. We don’t want them to feel squished into doing two jobs, for only one paycheck. One of the biggest challenges that security team’s face when creating a champions program is having it fall apart after the first few months, either due to the security team losing steam, or champions losing interest. We need them to feel very aware of our gratitude, and interested in the program itself, for them to continue to want to serve the security team’s agenda.

As you likely already figured out, not all the love languages listed above are work appropriate. We can’t run around giving hugs or holding hands with other employees. That said, we can adopt most of them for work situations, so that we can show the champions they matter to us, in appropriate ways, that support our security program.

Below is a non-exhaustive list of several ideas to make your champions feel as valuable as you know they are for your program.

  1. (Security Related) Gifts
  • Physical or digital security-related gifts – books, videos, training, CTFs, perhaps a copy of Alice and Bob Learn AppSec?
  • Create a Certificate to put on their wall.
  • Stickers, posters or any other decoration that is security focused.
  • Tickets to a conference or training.
  1. Words of Affirmation
  • Make sure to put a note in their performance review about them being a champion.
  • Tell their boss every time they do something that makes a big difference.
  • Send them an email and tell them when they did something big, let them know that YOU saw.
  • Recognize them in front of their peers (special virtual background, star on their name is slack, etc.)
  • Digital badges for signature blocks.
  1. Physical Affection
  • High Fives are the only recommended form of physical affection that you should show another employee. High fives signal success, and your approval of whatever they just did.
    • *** And only do this if you are confident that the employee is comfortable. Please be mindful that some religions and cultures do not allow those of the opposite sex to touch each other and be respectful if this applies. Never push physical touching at work.
  1. Spending Quality Time
  • Giving them your time is a reward. When you do, give them your undivided attention (put your phone away), and turn your body towards them.
  • Let them see a new tool first, give them a “sneak preview” ahead of everyone else.
  • Let them help you make decisions. Ask for advice from them and feedback, then take it seriously.
  • Invite them to attend security events with you.
  • Whenever you meet with them, this is quality time. Ask them: What are you working on? What are you going to work on next? Do you need any help?
  1. Acts of Service
  • Help them with more than just security. Are you good at design? Help them with it! Are you great at presentations? Offer to let them practice in front of you. You don’t need to do this very often, just once can make a huge impression.
  • Make introductions, where appropriate. “Oh yeah, Chris from QA uses that tool, I’ll introduce you so you can learn.”
  • Find answers they need to security questions and problems. Never leave them hanging.

When people feel appreciated and valued at work, they work harder (many studies show this to be true). Your champions already have full time jobs on other teams, they are going above and beyond for you. Let them know that you are very aware of them, by always making them aware of it with your actions, not just your words.

In the next article we will discuss communication with your champions!