Security is Everybody’s Job — Part 6 — The Second Way

The Second Way of DevOps is fast feedback. In security, when we see this we should all be thinking the same thing: Pushing Left. We want to start security at the beginning of the system development life cycle (SDLC) and ensure we are there (providing feedback, support and solutions) the whole way through!

Pushing Left, Tanya' Favorite Thing
Pushing Left, Tanya’s Favorite Thing

Fast feedback loops means getting important information to the right people, quickly and regularly. One of the main reasons that Waterfall projects failed in the past was the lack of timely feedback; no one wants to find out twelve months after they made a mistake, when it's too late to fix it.

The goal of security activities in a DevOps environment must be to shorten and amplify feedback loops so security flaws (design issues) and bugs (code issues) are fixed as early as possible, when it’s faster, cheaper and easier to do a better job. These DevOps people are really onto something!

Let’s go over several ideas of how to achieve this.

Activities to create fast feedback loops.

  • Automate as much as humanly possible. Inside or outside the pipeline, automation is key.
  • Whenever possible integrate your tools with the Dev and Ops team’s tools. For instance, have the issues found by your IAST tool  turned into tickets in the developer’s bug tracking system, automagically.
  • When you have a Pentest done, check all your other apps for the things found in the report, then add create unit tests to look for these things and prevent them from coming back.
  • Rename insecure functions or libraries as “insecure” with a wrapper, so programmers see immediately that there is an issue.
  • Add security sprints to your project schedule (to fix all security bugs in backlog).
  • Asking the Dev and Ops what they are concerned about (in relation to security), so you can fix any problems the security team might be causing them.
  • Add important security tests that are quick and accurate to the pipeline. For instance, scan for secrets in the code that is being checked in. That is an important test!
  • If an important security tests fail in the pipeline, the continuous integration server must break the build. Just like quality tests. This is loud feedback.
  • Create a second pipeline that doesn’t release any code, but runs all the long and slow security tests, then have the security team review the results after and turn the important things into tickets for the Devs.
  • Tune all security tools as much as possible and validate all results so that the feedback you are giving is *accurate*. There is no point of sending lots of feedback if half of it is wrong.
  • Work with developers to create negative unit tests (sometimes known as abuse tests). Create copies of regular unit tests, rename them with “Abuse” at the end, then add malicious payloads and ensure that your app fails gracefully and handles bad input well.
  • Have reports from your security tools automatically send their results to a vulnerability management tool such as Defect Dojo or Thread Fix to keep metrics and use them to improve all of your work. You need feedback too.

Be creative. Any way that you can get feedback faster to other teams is a huge win for your team too!

Security is Everybody’s Job — Part 5 — The First Way

The First Way of DevOps

The first “Way” of DevOps is emphasizing the efficiency of the entire system. Many of us tend to focus only on our part of a giant system, and get bogged down improving only our own contributions to the larger process. It’s rare that we stand back, look at the entire thing, and realize that if we helped another team or changed something small within our part, that it could improve other areas for the better. The first way of DevOps is about looking at the entire system, and making sure the entire thing is as efficient as possible. #speed

When we worked in Waterfall development environments security often acted as a gate. You had to jump through their hoops, then you were let through, and you could push your code to prod. Awesome, right? Not really. It was slow. Security activities took FOREVER. And things got missed. It was rigid and unpleasant and didn’t result in reliably secure software.

It may seem obvious to new developers that security should not slow down the SDLC, but I assure you, this concept is very, very new. When I was a software developer I referred to the security team as “Those who say no”, and I found almost all of my interactions with them left me frustrated and without helpful answers.

When we (security practitioners) think about The First Way, we must figure out how to get our work done, without slowing down all the other teams. They won’t wait for us, and we can’t set up gates. We have to learn to work the way they do; FAST.

Tools:

First of all, we need to use modern tooling that is made for DevOps pipelines if we are going to put anything into the CI/CD pipeline. Never take an old tool and toss it in there; no DevOps team is going to wait 5 hours for your SAST tool to run. Tune your tools and ensure you select tools that are made for pipelines if that is how you are going to use them. Whenever possible, only run your tools on the ‘delta’ (the code changed in that release, not the entire code base).

Tool Selection:

When selecting tools, remember that not every tool needs to be put in the pipeline. In fact, having tools that are out-of-band, but located on the ‘left’ of the SDLC, can offer even more value and save time. Examples:

  • Package management tools that only serve packages that are not known to be insecure (pre-approved by a security research team)
  • Adding security tests to your unit tests, which are often run before the code arrives in the pipeline (for instance, write input validation tests that ensures your code properly handles input taken from the XSS Filter Evasion Cheat Sheet)
  • Ddding security tooling to the check-in process, such as secret scans (don’t even let them check it in if it looks like there’s a secret in the code)
  • Scanning your code repository for known-insecure components. It’s just sitting there, why not use it?

Bugs:

This also means that security bugs should be placed in the same bug tracker or ticketing system that the developers and ops teams are using. They shouldn’t check two systems, that is not efficient.

Finding Vulnerabilities:

If at all possible, we should be providing and/or approving tools that assist in finding vulnerabilities in written code (both the code your team wrote, and the code from dependencies) and running code. This could be SAST + SCA + DAST, or it could be SCA + IAST (run during unit testing, QA and in prod). It could also mean manual secure code review plus a PenTest the week before going live (this is the least-efficient of the three options presented here).

Templates and Code Reuse:

If it makes sense, create templates and provide secure code samples, there’s no need to reinvent the wheel. Also, enable the developers and ops teams to scan their own code by providing tools for them (and training on how to use them safely and effectively).

Think Outside The Box

We (security) can no longer be a bottleneck, we must work to enable them to get their jobs done securely, in anyway we can. Examine your processes to ensure they are efficient; create a second asynchronous (which does not release to prod) pipeline to automate your longer tests; write your own tools if you absolutely have to. The sky is the limit.

Security is Everybody’s Job — Part 4 — What is DevSecOps?

In this post we will explore The 3 Ways of DevOps. But first, a definition from a friend.

DevSecOps is Application Security, adjusted for a DevOps environment.

Imran A Mohammed

DevSecOps is the security activities that application security professionals perform, in order to ensure the systems created by DevOps practices are secure. It’s the same thing we (AppSec professionals) have always done, with a new twist. Thanks Imran!

Photo by Marvin Meyer on Unsplash

Refresher on The Three Ways:

  1. Emphasize the efficiency of the entire system, not just your part.
  2. Fast feedback loops.
  3. Continuous learning, risk taking and experimentation (failing fast). Taking time to improve your daily work.

 

Let’s dig in, shall we?

 

1. Emphasize the efficiency of the entire system, not just one part.

This means that Security CANNOT slow down or stop the entire pipeline (break the build/block a release), unless it’s a true emergency. This means Security learning to sprint, just like Ops and Dev are doing. It means focusing on improving ALL value streams, and sharing how securing the final product offers value to all the other steams. It means fitting security activities into the Dev and Ops processes, and making sure we are fast.

2. Fast feedback loops.

Fast feedback loops = “Pushing Left” (in application security)

Pushing or shifting “left” means starting security earlier in the System Development Life Cycle (SDLC). We want security activities to happen sooner in order to provide feedback earlier, which means this goal is 100% inline with that we want. The goal of security activities must be to shorten and amplify feedback loops so security flaws (design/architecture issues) and bugs (code/implementation issues) are fixed as early as possible, when it’s faster, cheaper and easier to do a better job.

3. Continuous learning, risk taking and experimentation

For most security teams this means serious culture change; my favorite thing! InfoSec really needs some culture change if we are going to do DevOps well. In fact, all of IT does (including Dev and Ops) if we want to make security everybody’s job.

Part of The Third Way:

  • Allocating time for the improvement of daily work
  • Creating rituals that reward the team for taking risks: celebrate successes
  • Introducing faults into the system to increase resilience: red team exercises

We are going to delve deep into each of the three ways over the next several articles, exploring several ways that we can weave security through the DevOps processes to ensure we are creating more secure software, without breaking the flow.

If you are itching for more, but can’t wait until the next post, watch this video by Tanya Janca. She will explain this and much more in her talk ‘Security Learns To Sprint'.

Security is Everybody’s Job — Part 3 — What IS DevOps?

What IS DevOps?

There are many definitions of DevOps, too many, some might say. Some people say it’s “People, Processes, and Products”, and that sounds great, but I don’t know what I’m supposed to do with that. When I did waterfall I also had people, processes, and products, and that was not great. I thought DevOps was supposed to be a huge improvement?

I’ve heard other people say that it’s paying one person to do two jobs (Dev and Ops), which can’t be right… Can it? I’ve also been told once by a CEO that their product was “made out of DevOps”, as though it was a substance. I decided not to work there, but that’s another story. Let’s look at some better sources.

Wikipedia says:

DevOps is a set of practices that combines software development and information-technology operations which aims to shorten the systems development life cycle and provide continuous delivery with high software quality.

But what are the practices? Why are we aiming to shorten the SDLC? Are we making smaller software? What is ‘continuous delivery’?

To get to the bottom of it I decided to read The DevOps Handbook. 
Then I knew what DevOps was, and I knew how to do it. And I 
discovered that I LOVED DevOps.

According to the DevOps Handbook, DevOps has three goals.

Improved deployment frequency; Shortened lead time between fixes;

Awesome! This means if a security bug is found it can be fixed extremely quickly. I like this.

Lower failure rate of new releases and faster recovery time;

Meaning better availability, which is a key security concern with any application (CIA). Lower failures mean things are available more often (the ‘A’ in CIA), and that’s definitely in the security wheelhouse. So far, so good.

Faster time to market; meaning the business gets what they want.

Sometimes we forget that the entire purpose of every security team is to enable the business to get the job done securely. And if we are doing DevSecOps, getting them products that are more secure, faster, is a win for everyone. Again, a big checkmark for security.

Great! Now I think the DevOps people want the same things that I, as a security person, want. Excellent! Now: How do I *DO* DevOps?

That is where The Three Ways of DevOps comes in.

  1. Emphasize the efficiency of the entire system, not just one part.

In the next post we will talk more in detail about The 3 Ways (and how security fits in perfectly).

Security is Everybody’s Job — Part 2 — What is Application Security?

Application Security is every action you take towards ensuring the software that you (or someone else) create is secure.

This can mean a formal secure code review, hiring someone to come in and perform a penetration test, or updating your framework because you heard it has a serious security flaw. It doesn’t need to be extremely formal, it just needs to have the goal of ensuring your systems are more secure.

Now that we know AppSec is, why is it important?

For starters, insecure software is (unfortunately), the #1 cause of data breaches (according to the Verizon Breach Reports, 2016, 2017, 2018 and 2019). This is not a list that anyone wants to be #1 on. According to the reports, insecure software causes 30–40% of breaches, year after year, yet 30–40% of the security budget at most organizations is certainly not being spent on AppSec. This is one part of the problem.

The graph above is from the Verizon Breach Report 2017. Hats off to Verizon for creating and freely sharing such a helpful report, year after year.

If the problem is that insecure software causes breaches, and one of the causes is that security budgets don’t appear to prioritize software, what are some of the other root causes of this issue?

For starters, universities, colleges, and programming bootcamps are not teaching the students how to ensure that they are creating secure software. Imagine electricians whet to trade school, but they didn’t teach them safety? They twist two cables together and then just push them into the wall, unaware that they need two more layers of safety (electrical tape, and then a marrett). This is what we are doing with our software developers, we teach them from their very first lesson how to make insecure code.

Hello (insecure) World

Lesson #1 for every bootcamp or programming course: Hello World.
Step 1) Output “Hello World” to screen
Step 2) Output “What is your name?” to screen

Step 3) Read the user’s input into a variable (note: we skip teaching input validation)
Step 4) Output the user’s input to the screen with a hello message (note: we skip output encoding)

The above lesson teaches new programmers the best possible recipe for including reflected Cross Site Scripting (XSS) in their application. As far as I know there is not a follow up lesson provided on how ensure the code is secure.

“Hello World” is the most-taught lesson for starting a new programming language, I’m sure you’ve seen it everywhere. More like “Hello Insecure World”.

Although there has been some headway in universities and colleges recently, most of them barely scratch the surface in regards to application security.

So that's reason #2. Let's move onto reason #3.

Another issue that contributes to this problem is that security training for developers is costly. While this is true for all types of professional training, security training is significantly more expensive then other forms of technical training. A single course at the SANS institute (a well-respected training establishment that specializes in all things cyber), could cost an attendee as much as $5000-$7000 USD, for one week of training. There are other less-pricy options, such as taking a course when attending a conference, which usually range from $2000-$5000, however, those are of varying quality, and there is no standardized curriculum, making them a bit of a gamble. I’ve taken several trainings when attending various conferences over the years, and I’d say about 1/2 were good.

There are much cheaper alternatives to the options above (such as the courses from We Hack Purple and other training providers), and they are of very varying quality levels. I’ve seen both good free courses and some where I wish I could have my time back they were so bad. Most of them do not provide a curriculum to follow either, meaning it is often unclear to the student which other courses they should take in order to get the specific job they want. It is very easy to waste quite a bit of time and money; I know, that is how I started my AppSec career… Although I was quite lucky to have two professional mentors guiding me, which made it a lot easier. Not everyone has a mentor.

See how lonely she looks? She’s the ENTIRE security team! #WOCTechChat

Another cause (#4) of insecure software is that the security team is usually grossly outnumbered. According to several sources there is usually 100 software developers for every 10 operations employees for every single (1) security professional.

Let me repeat that. There are 100/10/1, Dev/Ops/Sec. With numbers like that you can’t work harder, you have to work smarter. Which is where we are going with this series.

Now we know the problem and several of the causes, what can we do about it? The short answer is DevSecOps, and the long answer is ‘read the rest of the blog series’.

For now though, let’s define DevSecOps, before we dive into what DevOps is, The Three Ways, and so much more, in the next article.

DevSecOps: performing AppSec, adjusted for a DevOps Environment. The same goals, but with different tactics and strategies to get us there. Changing the way we do things, so that we weave ourselves into the DevOps culture and processes.

Security is Everybody’s Job — Part 1 — DevSecOps

This is the first in a many-part blog series on the topic of DevSecOps. Throughout the series we will discuss weaving security through DevOps in effective and efficient ways. We will also discuss the ideas that security is everybody’s job, it is everyone’s duty to perform their jobs in the most secure way they know how, and that it is the security team’s responsibility to enable everyone else in their organization to get their jobs done, securely. We will define DevOps, ‘The Three Ways’, AppSec and DevSecOps. We will get in deep on the many strategies we can adjust security activities for DevOps environments, while still reaching our goals of ensuring that we reliably create and release secure software.

In summary; We will discuss how to make security a part of our daily work. 
It cannot be added later or after, it needs to be a part of everything.

But let’s not get ahead of ourselves, I have many more posts planned where I will attempt to sway your opinion my way.

Tanya Janca, also known as SheHacksPurple, presenting her ideas in Sydney Australia, 2019. Artwork by the talented Ashley Willis.

Before we get too deep into anything I’d like to dispel some myths. Look at the image below. This is how *some* security professionals see DevOps.

Slide credit: Pete Cheslock

This slide’s author, Pete Cheslock, is highly intelligent and experienced, this mention is not meant to insult him in any way. The slide is social commentary, it is not literal. That said, many people I’ve met truly feel this way; that DevOps engineers are running around making security messes where ever go, and that we (security professionals) are left to clean up the mess. I disagree with that opinion.

Luckily for me, my introduction to DevOps was at DevSecCon, where they introduced me to this image. Below you can see the security team teaching, providing tooling and enabling the magical DevOps unicorns in doing their jobs, securely. This is how I view DevOps; the security team enabling everyone, working within the confines of the processes and systems that all the other teams use.

Slide Credit: Francois Raynaud & DevSecCon

This series will be loosely based off a conference talk which I have delivered at countless events, all over the planet, ‘Security is Everybody’s Job’. You can watch the video here.

7 Places to do Automated Security Tests

When working in a DevOps environment security professionals are sometimes overwhelmed with just how fast the dev and ops teams are moving. We're used to having more control, more time, and more… Time!

Personally, I LOVE DevSecOps (the security team weaving security throughout the processes that Dev and Ops are doing). Due to my enthusiasm I am often asked by clients when, how and where to inject various types of tests and other security activities. Below is my list of options that I offer to clients for automated testing (there's lots more security to do in DevOps, this is only automated tests). They analyze the list together and decide which places make the most sense based on their current status, and choose tools based on their current concerns.

Man using computer
Photo by rupixen.com on Unsplash

Seven Places For Automated Testing

  1. In the IDE:
    • Tools that check your code almost like a spell checker (not sure what this is called, sometimes called SAST)
    • Proxy management & dependency tools that only allow you to download secure packages
    • API and other linting tools that explain where you are not following the definition file
  2. Pre-commit hook:
    • secret scanning, let's stop security incidents before they happen
  3. At the code repository level:
    • weekly schedule: SCA & SAST
  4. In the Pipeline:
 Must be fast and accurate (no false positives)
    • secret scanning – again!
    • Infrastructure as Code scanning (IaC)
    • DAST with HAR file from Selenium or just passive
    • SCA (again if you like, preferably with a different tool than the first time)
    • Container and infra scanning, + their dependencies
  5. Outside the Pipeline:
    • DAST & fuzzing, automate to run weekly!
    • VA scanning/infra – weekly
    • IAST – install during QA testing and PenTests, or in prod if you feel confident
    • SAST – test for everything for each major release or after each large change – manual review of results
  6. Unit-Tests:
    • take the dev's tests and turn them into negative tests/abuse cases
  7. Continuously:
    • Vulnerability Management. You should be uploading all your scan data into some sort of system to look for patterns, trends, and (most of all) improvements

You do not need to do all of these, or even half of these. But please do some of them. WHP will try to put out a course on this later on in the year!

Do YOU do testing in more places than this? Where, when and how? Do you 
have other tools you use that you find helpful? Please comment below 
and let us know!