Category: AppSec

In the previous post we covered API9:2019 Improper Assets Management, which was the 9th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization. You can see the formal document from the OWASP API Security Top Ten Team, here.

Years ago, I worked at Microsoft, as a developer advocate. A few of us were working together, creating demos, for ‘Microsoft Ignite The Tour’. I recall two of my colleagues asking me why Azure couldn’t detect an SQL injection attack they had done as part of their demo. I checked their settings and explained that logging and monitoring was turned off. If monitoring is off, that means there was no observation of the application. How would an attack be noticed if they had removed Azure’s ability to watch what as happening? I also explained that since they had also turned off logging, that would mean that an incident responder would have nothing to investigate. They would not only miss the attack happening, but they would also never be able to find out later what had happened if they investigated. It had been turned off to save money (we didn’t want to spend a small fortune just on demonstrations, we had a budget). They turned both logging and monitoring back on, tried the attack, and immediately Azure went into red alert. All was well for the developer advocates and our demos.

Imagine finding your data on the dark web for sale, and not even knowing how it got there. Obviously, this has never happened to me before at a client site… If it had, I would tell you how incredibly frustrating it is not to be able to explain what happened, and therefore ensure it could never happen again. You can’t do that if you have no logs AND no monitoring. Again, this is totally hypothetical and definitely did not happen to me or any of the clients that I have worked with.

– Not me

Back when I was a full-time developer, I remember asking to turn on logging. I had asked the client during the requirements phase, explaining why I wanted it (so we could provide better reliability, and investigate any outages, there was no security slant for me, at the time). The client had agreed immediately. Then we got into the costing phase, trying to calculate how much the final project would be. When the client saw how much logging was going to cost, it was cut immediately. I had this happen several times as a dev, always being told it was a cost-saving measure. Although I didn’t love this decision at the time, it wasn’t a hill I was going to die on.

Fast forward 8 years to when I got my first AppSec job. I recall us having a security incident, and me being able to search through the logs and find the attack in about 30 minutes (the logs were HUGE, and I didn’t have log viewing software, that’s why it took so long). I learned powershell that day, or, the basics of powershell, and wrote a script to de-obfuscate, so then I could see the exact attack commands. It took me quite bit longer to figure out a perfect timeline, and where our AppSec program had broken down to allow this vulnerability into prod…  But that said, I realized that logs were so incredibly valuable for investigating a security incident, there’s just no other way you can find our exactly what happened without them!

Over the years I have learned that 1) working in incident response is absolutely fascinating and 2) I become far, far too stimulated to do incident response work on a regular (full-time) basis. I have a lot of respect for people who do that type of work full time.

– Tanya’s Thoughts on Incident Response

Around this time, I also learned that sometimes attackers will modify the logs, erasing their tracks as part of the attack. One of the ways that logs can be manipulated is via attacks against user input fields, where the attacker bypasses the input validation, that input is logged, and that type of attack is referred to as “log injection”. Attacks against the integrity of our logs is the reason I always go on and on about why we need to protect our logs, and back them up to a secure location. Ideally logs should be protected because they are *sensitive information*, they are literally evidence that could be used one day in court. We should treat them as the precious resource they are, a living record of all that has happened to our applications.

When an API, or any other IT system, has logging and monitoring turned off, or have improperly configured or insufficiently protected their logs, this vulnerability applies. It can apply to any IT system, but APIs are the focus of this blog series, and thus we shall concentrate on how to find, avoid, and solve this problem in APIs specifically.

Let’s talk specifics!

1. Turn on monitoring. Give your monitoring system contact info for the correct people (it should not go to an unmonitored inbox, or phone number that no one answers). Someone needs to receive the alerts, otherwise why bother to pay for monitoring…
2. You should log every activity that has to do with a security control, even failed ones. Logins, log outs, changes to privilege, account creation or deletion, password changes, authentication and authorization, input validation, system errors (especially if the global exception handler gets called), changing the contact info for the account, etc.
3. Do not log sensitive info. Examples of sensitive info: complete credit card numbers, name + home address, name + date of birth, SIN/SSN, the text entries for failed password attempts (those are often typos that would allow you to guess the password), anything that could identify the person (PII) from the log data alone, personal health data + name or other identifying info, anything else that qualifies for your specific organization, system or customers.
4. Do log: user ID, time stamp, what the user was trying to do, if they succeeded or not, their IP address and any other identifying information you can get about the user’s computer.
5. Ideally your logs would be formatted so that your SIEM is able to consume them. It’s not very common for organizations to feed their custom app logs into the SIEM, but I hope this changes over time. It’s incredibly helpful.
6. Your logs should not be stored on the web server or whatever your app lives on. It should be in a different place, inside/behind the firewall/perimeter. That location should not have execution privileges (read only), and only the incident response team should have access to this system.
7. Monitor where your logs are stored. If an inappropriate account attempts to access this file server, initiate the incident response process immediately. Part of this process should be stopping whoever is accessing it, but then also investigating if these logs of been previously disturbed or altered in any other way. This might not be the first attempt to mess with your logs.
8. Backup your logs! In a geographically differing place than your app server.

Download a PDF with more specifics for logging, error handling, and logging, here.

Advice straight from the OWASP API Security Top Ten Project Team:

• Log all failed authentication attempts, denied access, and input validation errors.
• Logs should be written using a format suited to be consumed by a log management solution and should include enough detail to identify the malicious actor.
• Logs should be handled as sensitive data, and their integrity should be guaranteed at rest and transit.
• Configure a monitoring system to continuously monitor the infrastructure, network, and the API functioning.
• Use a Security Information and Event Management (SIEM) system to aggregate and manage logs from all components of the API stack and hosts.
• Configure custom dashboards and alerts, enabling suspicious activities to be detected and responded to earlier.

---

This concludes the We Hack Purple blog series on the OWASP API Security Top Ten! Thank you to the volunteers of that project for all of their hard work to create this list and share this information with the world. Hopefully soon they will release the next version, and we can write more posts about their amazing research!

In the previous post we covered API8:2019 Injection, which was the 8th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization. You can see the formal document from the OWASP API Security Top Ten Team, here.

Photo by Petrebels on Unsplash

Taking inventory is the first thing I do whenever I start or join an AppSec program. Figuring out all the applications and APIs that an organization has built, bought (COTS), or are using (SaaS), then doing a fast evaluation of the state they are in, is the best way to figure out where an organization is at regarding their security posture. Doing this helps me know just how much work we have to do and sets the stage for future conversations with management and the developer teams on how we can get them on track for securing all of their apps.

This vulnerability is the reason I start with inventory. I would argue that the majority of organizations around the planet do not have a current and accurate inventory of all of their web assets, including APIs. If they aren’t in the inventory, that means these assets aren’t being managed, which usually also means the security team does not have them on their radar. If the security team doesn’t know about an asset, how can they secure it? Not being in the inventory generally also means no testing, monitoring, logging, or documentation, at a minimum.

Taking inventory (regularly or continuously), and ensuring we properly decommission old versions of APIs when we release new versions, is the way we avoid this vulnerability. Then document all of it, or update documentation as you update your inventory. I realize this is easier said than done!

I recall a penetration tester telling me years ago that one of his tricks for finding vulnerabilities during an engagement was to try to call earlier versions of any APIs that were in scope. If there was a version 2.x, he would try to call version 1.x. He told me that at least once every year he would get a response of a phantom API. And that API was always a complete security disaster. He would earn his entire paycheck with that one Postman call.

PenTester Name Redacted

APIs and web applications that are not part of your inventory are generally also unmonitored, meaning no one is watching them to see if something goes wrong. They are often not behind a WAF, API gateway, or any other shielding that might protect them from common threats. If they are not a part of your inventory, there’s also a good chance that there’s no team in charge of maintenance, meaning no bug fixing is happening and technical debt is accruing. Lastly, it’s very unlikely that they are receiving regular security testing, or any type of security scanning, which can lead to all sorts of problems building up, invisibly.

Software doesn’t age like wine, getting better over the years. Software ages like cheese in the hot sun; extremely badly! The longer we do not update, test, or patch our software, the more likely it is to have vulnerabilities found within it. Without proper care, software accrues technical debt, which can make it even more difficult to fix security vulnerabilities, because you have to update so many different components (framework, plugins, operating system patches, etc.) in order to fix the real problem at hand (the vulnerability).

The risks of having APIs (or web apps) that are not a part of your inventory and maintenance plans has no bounds. Any type of vulnerability could happen, as no one is watching or paying attention, except perhaps malicious actors. Attacks upon such resources could result in damage to the availability of the system, sensitive data exposure, changes to the data leading to poor integrity, and worse. This makes the risk of this vulnerability very high. On top of no one knowing that the API exists and is live in production, there’s very likely to be little or no documentation about this API. This situation brings me to back to when I was a dev, and the DBA told me I wasn’t allowed to kill an old database server (I wanted to repurpose it), because there were a whole bunch of scripts on there. She said she has no idea which scripts did what, but she turned the server off once and “everything broke” (including payroll being missed for the entire company, yikes!). She said, “Do not touch, I don’t care why, buy a new server!” The DBA lady meant business, so I got a new server. That said… What if there had been documentation? This situation could easily happen to a company with unknown APIs running wild over their network…

Advice From the OWASP Project Team: ‘How to Prevent’

• Inventory all API hosts and document important aspects of each one of them, focusing on the API environment (e.g., production, staging, test, development), who should have network access to the host (e.g., public, internal, partners) and the API version.
• Inventory integrated services and document important aspects such as their role in the system, what data is exchanged (data flow), and its sensitivity.
• Document all aspects of your API such as authentication, errors, redirects, rate limiting, cross-origin resource sharing (CORS) policy and endpoints, including their parameters, requests, and responses.
• Generate documentation automatically by adopting open standards. Include the documentation build in your CI/CD pipeline.
• Make API documentation available to those authorized to use the API.
• Use external protection measures such as API security firewalls for all exposed versions of your APIs, not just for the current production version.
• Avoid using production data with non-production API deployments. If this is unavoidable, these endpoints should get the same security treatment as the production ones.
• When newer versions of APIs include security improvements, perform risk analysis to make the decision of the mitigation actions required for the older version: for example, whether it is possible to backport the improvements without breaking API compatibility or you need to take the older version out quickly and force all clients to move to the latest version.

In the next blog post we will be talking about API10:2019 Insufficient Logging & Monitoring.

In the previous post we covered API7:2019 Security Misconfiguration, which was the 7th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

Injection has been part of the original ‘OWASP Top Ten Risks to Web Apps’ list since the very beginning. Injection happens when an attacker is able to trick an application (or API) into executing malicious code. It does this by adding code to a place in the application where data belongs, and the app becomes confused, and then executes it.

Think of a search field at the top of any website. Imagine if instead of entering in your search term, you added a bunch of code. Then imagine the application executes the code you added. That code would execute with the full authority of that application, and all the same access, behind your firewall. It could result in damage to a database, a web service, your LDAP system, and potentially even worse, assuming there are other vulnerabilities the attacker can combine with this one.

Unfortunately, APIs are subject to this vulnerability, just like a regular web app. No front end does not protect us from injection.

What types of injection exist?

If there’s code involved, someone will try to inject their own code. SQL, LDAP, or NoSQL queries, OS commands, XML parsers, and ORM are all potentially problematic (list provided by the OWASP API Security project team). Even Mongo DB databases, that don’t use the SQL language, are potentially vulnerable to NOSQL injection.  

Special note on XSS: Cross Site Scripting (XSS) is also a form of code injection, but it has it’s own classification because of the following reasons:

• It occurs in the browser, as opposed to back on the server side like every other form of injection.
• Only works with javascript (because that’s all browsers execute).
• Is incredibly prevalent, so much so that OWASP felt it was necessary to give it it’s own category.
• Has several defenses made just for this one vulnerability (cookie settings, and security headers).
• Does not work on APIs, because they have no GUI front end, meaning no browser.

What do we do?

Hopefully by this point you agree that injection is dangerous and should be remediated as soon as possible if you find it in one of your apps. But how do we find it? How do we fix it? How can we ensure this never happens? Dear reader, secure coding is my favorite topic! Let’s go!

Finding Injection

First off you want to go through your APIs and figure out if you have injection. The most expensive way to do this would be to hire a PenTester to find and then test all the APIs. A cheaper and more sustainable way to do this would be to

1. Buy a tool that can find all your APIs for you (No, I am not going recommend one at this time, there are several on the market of various qualities, and prices). This function is called “inventory” or “enumeration”.
2. Run a SAST (static application security testing tool) on all of the APIs, ideally a next gen one, that has low false positives. Fix anything that says injection.
3. Use a linter on your API, ensure you have completed your API definition file, as per the linter’s instructions. If you can find an API-specific linter, all the better.
4. Run a DAST tool on the APIs that is made for APIs OR, use an old school DAST but first ensure you’ve linted your API perfectly, so it can hopefully do a good job. It will be easier and faster if you have an API-specific testing tool. Fix anything that says injection.

Fixing Injection

“That’s nice you told me to fix it. Exactly HOW do I do that?”

The first defense for injection is thorough input validation on any input to your app. This means data in the parameters, in a data field, in a hidden field, from an API you called, from the database, any input to your app needs to be validated that it is what you are expecting. What type is it? What size? What’s the content? Is it what we are expecting? If not, reject.

This is functionality is best performed using an approved list, on the server side. By ‘approved list’, we mean using a list of stuff you know is good, rather than a list of what you know is bad. It’s easy for malicious actions to get around a block list, using encoding, obfuscation, and other tactics. But if you give a regular expression and say “if it’s not in here, I’m just not having it”, bad things cannot get in.

As an example, imagine you have a username. It likely accepts numbers and letters. You could use a regular expression (REGEX) like this to say what is okay: [a-z,A-Z,0-9]. That’s an approved list or ‘accept list’. If instead you try to block bad characters such as <, >, ‘, “ and more, you (and your app) are in for a world of hurt.

The next thing you want to do is ensure you perform this check on the server side. Do not do it on the client side, and by this, I mean in the browser/JavaScript. Anyone with a web proxy can get behind your JavaScript in about 5 seconds, unfortunately. If you want to check in your JavaScript for speed, you can do that, in addition to checking on the server.

Input validation is defense number #1. Other defenses include:

• Always using parameterized queries when making requests to any database (even non-SQL databases long mongo DB). It takes away the ability for it to be interpreted as code.
• Use output encoding when you put stuff onto the screen. Some frameworks do this for you by default. It takes away and superpowers of the characters, before it puts them on screen, making XSS impossible. Okay, maybe this is only for XSS and doesn’t apply to injection in general, but I would still do it if I were you.

Preventing Injection

If we want to prevent injection (and a myriad of other vulnerabilities), follow this advice:

• Have your development team take a secure coding course. It can be free or paid, formal or informal, live or recorded, interactive or lecture, the only important part is that they learn. Do the type of training that works best for you and your team.
• Follow a secure system development life cycle (S-SDLC). Add security steps to each part of your SDLC, such as security requirements, code review, or threat modelling.
• Ensure your application has thorough testing, which can mean any or several of the following: static analysis, dynamic analysis, manual code review, penetration testing, stress testing, performance testing, unit testing or any other testing you can think of!
• Whenever possible, use modern and up-to-date frameworks that have security features built in. JavaScript frameworks like Angular and React have so many cool features that help protect your users! They aren’t just nifty dev tools, they can help you build stronger, tougher apps.
• Never stop learning. Keep reading, studying, learning and hacking.

How To Prevent: OWASP API Security Top Ten Team Advice!

Preventing injection requires keeping data separate from commands and queries.

• Perform data validation using a single, trustworthy, and actively maintained library.
• Validate, filter, and sanitize all client-provided data, or other data coming from integrated systems.
• Special characters should be escaped using the specific syntax for the target interpreter.
• Prefer a safe API that provides a parameterized interface.
• Always limit the number of returned records to prevent mass disclosure in case of injection.
• Validate incoming data using sufficient filters to only allow valid values for each input parameter.
• Define data types and strict patterns for all string parameters.

In the next blog post we will be talking about API9:2019 Improper Assets Management.