DevSecOps Worst Practices – The Boy Who Cried Wolf

Photo by Lenny Kuhne on Unsplash

The first terrible practice that we will examine in this series is breaking builds on false positives. Whenever I explain this to people who are new to DevOps, I remind them of the story of ‘the boy who cried wolf’. In the age-old story, a young boy lies and says a wolf has come into the village he lives in, and scares all the villagers. The villagers snap into action, to protect everyone from the wolf. After a brief search they realize there is no wolf, and that the child was lying. The boy plays this trick on the villagers more than once, and the villagers become very angry with the boy. At the end of the story, a wolf does enter the village, and the boy attempts to warn everyone, but no one is willing to listen to him anymore. He has been labeled a liar; someone to ignore. But this time the wolf was real, and people were hurt. 

The takeaway of the story is that no one wins when trust is broken. People tell the story to children, to discourage them from lying. I tell the story to security professionals, so that we prioritize building trust with development teams, and thus avoid having our warnings ignored.

Originating from the word for a paper lantern, Andon is a term that refers to an illuminated signal notifying others of a problem within the quality-control or production streams. Activation of the alert – usually by a pull-cord or button – automatically halts production so that a solution can be found.

Toyota

DevOps pipelines are built to model real-life, physical assembly lines. Each assembly line has something called an “Andon cord”, which is pulled when there is an emergency to stop the line. The button or pull cord can save lives, and millions of dollars (imagine cars accidentally piling on top of each other and the potential cost). The cord is only pulled if something extremely dangerous is happening. When we “break the build” in a DevOps pipeline, we are pulling a digital Andon cord, which stops the entire process from continuing. And when we do this, we had better have a good reason.

When a test fails in the CI/CD pipeline, it doesn’t always break the build (stop the pipeline from continuing). It depends on how important the finding is, how badly it failed the rest, the risk profile of the app, etc. It breaks the build if the person who put the test into the pipeline feels it’s important enough to break the build. That it’s (literally) a show-stopper, and that they are willing to stop every other person’s work as a result of this test. It’s a big decision.

Now imagine you have put a lot of thought into all the different tests in your pipeline, and as to if they have the importance to break the build or just let it continue and send notifications or alerts instead. You and your team use this pipeline 10+ times a day to test your work, and you depend on it to help you ensure your work is of extremely high quality.

Now imagine someone from the security team comes along and puts a new security tool into your carefully-tuned pipeline, and it starts throwing false positives. All the time. How would that make you feel? Probably not very good.

Photo by carlos aranda on Unsplash

I have seen this situation more times than I care to count, and (embarrassingly) I have been the cause of it at least once in my life. While working on the OWASP DevSlop project I added a DAST to our Patty-the-pipeline module (an Azure DevOps pipeline with every AppSec tool I could get my hands on). One evening Abel had done an update to the code, and he messaged me to say my scanner had picked something up. I didn’t notice his email, then went to Microsoft to give a presentation for a meetup the next day and… Found out on stage.

When my build broke I thought “OH NO, HOW EMBARRASSING”. But then I had another thought, and proudly announced “wait, it did what it was supposed to do. It stopped a security bug from being released into the wild”. Then we started troubleshooting (40+ nerds in a room, of course we did!), and we figured out it was a false positive. Now that really was embarrassing… I had been trying to convince them that putting a DAST into a CI/CD was a good thing. I did not win my argument that day. Le sigh.

Fast forward a couple years, and I have seen this mistake over and over at various companies (not open source projects, made up of volunteer novices, but real, live, paid professionals). Vendors tell their customers that they can click a few buttons and viola! They are all set! When in fact, generally we should test tools and tune them before we put them into another team’s pipeline.

Tuning your tools means making lots of adjustments until they work ‘just right’. Sometimes this means suppressing false positives, sometimes this means configuration changes, and sometimes it means throwing it in the garbage and buying something else that works better for the way your teams do their everyday work. 

Photo by Birmingham Museums Trust

In 2020, I was doing consulting, helping with an AppSec program, and their only full time AppSec person proudly told me that they had a well-known first-generation SAST tool run on CI/CD every build, and that if it found anything that was high or above it broke the build. I said “COOL! Show me!” Obviously I wanted to see this awesomeness.

We logged into the system and noticed something weird: the SAST tool was installed into the pipeline, but it was disabled. “That’s weird” we both said, and went on to the next one. It was uninstalled. HMMMMM. We opened a third, it was disabled. We sat there looking and looking. We found one that was installed and running, but it was just in alerting mode. 

The next time I saw him his face was long. He told me that in almost 100% of the pipelines his tool had been uninstalled or disabled, except 2 or 3 where it was in alerting mode (running, but it couldn’t break the build). We investigated further to find out that the teams that had it in alerting mode were not checking the notifications, none of them had ever logged into the tool to see the bugs it had found.

To say the guy was heartbroken would be an understatement. He had been so proud to show me all the amazing work he had done. It had taken him over a year to get this tool installed all over his organization. Only to find out, with a peer watching, that behind his back the developers had undone his hard-earned security work. This was sad, uncomfortable, and I felt so much empathy for him. He did not deserve this.

We met with the management of the developer teams to discuss. They all said the right things, but meeting after meeting, nothing actually changed. After about 3 months the AppSec guy quit. I was sad, but not surprised at all. HE was great. But the situation was not.

I kept on consulting there for a while, and discovered a few things:

  1. The SAST tool constantly threw false positives. No matter what the AppSec guy had done, working very closely with the vendor, for over a year. It was not him, it was the tool.
  2. The SAST tool had been selected by the previous CISO, without consultation from the AppSec team (huge mistake), and was licensed for 3 years. So the AppSec guy HAD to use it.
  3. The AppSec guy had spent several hours a week just trying to keep the SAST server up and running, and it was a Windows 2012 server (despite being 2020, the SAST provider did not support newer operating systems). He also wasn’t allowed to add most patches, which meant he had to add a lot of extra security to keep ot safe. It was not a great situation.
  4. The developers had been extremely displeased with the tool, having it report false positives over and over, and they turned it off in frustration. It was not malice, or anger, they had felt they couldn’t get their jobs done. They really liked the AppSec guy. When I talked to them about it, they all felt bad that he had quit. It was clear they had respected him quite a lot, and had given the tool more of a chance because of him.  

It took over a year, but I eventually convinced them to switch from that original SAST to a next generation SAST (read more on the difference between first and second gen here). The new tool provided almost entirely true positives, which made the developers a lot happier. It also was able to run upon code check in, which worked better for the way they liked to do their work in that shop. When I had left, it was scanning every new check in, then sending an email to whoever checked the code in with a report if any bugs were introduced. Althought I didn’t have it breaking builds by the time I left, we went from zero SAST, to SAST-on-every-new-commit. And devs were actually fixing the bugs! Not all the bugs, but quite a few, which was a giant improvement from when I arrived. To me this was a success. 

Photo by Mech-Mind Robotics on Unsplash

Avoiding this fate…

To avoid this fate, carefully pick your toolset (make a list of requirements with the developers, and stick to it), then test it out first on your own, then with developers, before purchase. Next, test the tool manually with a friendly developer team and work out as many kinks as you can before putting it into a CI. Then put it in alerting mode in the Ci with that team, again, watching for issues. If it runs well, start adding it for more teams, a few at a time. Pause if you run into problems, work them out, then continue. 

Tip: You can also set up most static tools (ones that look at written code, not running code) to automatically scan your code repository. This is further ‘left’ in the CI/CD, because it is even earlier in the system development life cycle (SDLC). You can scan the code as it is checked in, or on a daily, weekly or monthly basis, whatever works best for you and your developers!

The next post in this series is Untested Tools.

API6:2019 Mass Assignment

In the previous post we covered API5:2019 Broken Function Level Authorization, which was the 5th post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

Tanya proudly displaying her Sec4Dev T-Shirt.
Tanya proudly displaying her Sec4Dev T-Shirt

This vulnerability is quite poorly named, it is not at all intuitive. When you first hear it, you might think “oh-oh, the API is called in a way where it mass assigns/creates/deletes records, but it should have only done it to one record”. That is not at all what this vulnerability is about, so please set that idea aside.

Mass assignment refers to linking the incorrect fields within a record in a data model to the parameters in an API, such that when the API is called a user is able to update fields they should not be allowed to. This only applies to one single record. Not sure where the word ‘mass’ comes in to play here, but ‘incorrect assignment’ might make a bit more sense if you want to think of it that way.

This situation can happen when we bind data that was provided by the user (and is therefore not trustworthy) to data models, without checking if those fields should be accessible to the user. Should the user be allowed to specific ‘role=admin’? Probably not. But if you bind the entire record (first name, last name, username, password, role), rather than just the properties the user is allowed to update, this can happen.

How does this happen?

Malicious actors can send a GET request to an API, and see if it sends more parameters than just the ones it sends on the PUT, UPDATE, DELETE or POST request. If the developer sends everything (the entire record) back to the front end, someone proxying the web app can easily see “Hey, look at all the fields are here! This is way more than I asked for! What a gold mine!”. Obviously, we do not want this.

Let’s Avoid This, Shall We?

You should never be updating any data or making decisions in your system with values provided by the user until after you have validated that the data is trustworthy/what you are expecting. Validating your inputs is OWASP 101, we all already know this!

You do not need to call default ‘get’ or ‘set’ constructor functions directly from the API, you can write your own functions to be called directly, or have code in-between the original call and the data, that only passes the parameters you need, after you validated the input you got from the user.
 

Let’s hear more advice from the OWASP Project Team!

  • If possible, avoid using functions that automatically bind a client’s input into code variables or internal objects.
  • Create an approved list of only the properties that should be updated by the client, then add checks against that list when performing data updates.
  • Use built-in features to block properties that should not be accessed by clients.
  • If applicable, explicitly define and enforce schemas for the input data payloads.

External Reference from the OWASP API Top Ten Project Team

CWE-915: Improperly Controlled Modification of Dynamically-Determined Object Attributes

In the next blog post we will be talking about API7:2019 Security Misconfiguration!

API5:2019 Broken Function Level Authorization

In the previous post we covered API4:2019 Lack of Resources & Rate Limiting, which was the fourth post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization. You can read the official OWASP listing for this article here.

Image of Tanya commemorating the infamous "Ottawa Sinkhole Van"
Image of Tanya commemorating the infamous “Ottawa Sinkhole Van

_________________________________

Different users within an application are often given different roles within that system. Roles can include administrator, auditor, approver, editor, etc. Often those roles can be organized in a hierarchy, referred to as levels, with the idea of one role being a level above or below another. For instance, the administrative user is usually at the top of the hierarchy, then a regular user and then at the bottom perhaps a guest user of the system.

Each different system role has different areas of the application’s functionality available to them, such as the ability to create or delete users, look through someone’s financial records, approve a new blog post, or edit a document that belongs to a team member. Each one of those features within an application is often called a function. When the application grants or denies access to features within itself, this is called ‘function level authorization’. It is (or is not) authorizing that user to access a specific function.

When we hear the term escalation or elevation of privilege, we mean that a user has moved up one or more levels in the hierarchy of the system. If this happens in a system, we generally consider this to be a serious (critical) vulnerability, that we would work hard to mitigate.

When function level authorization is broken within an API, as is the case with item #5 on this top ten list, this means users are able to successfully call functions that their user role should not have access to. Imagine someone who is an unauthenticated user on your system using an API to call a function that only administrators should have access to, they could cause all sorts of damage, such as deleting users, changing other user’s privileges (perhaps locking out the real administrators), or worse. Depending upon what the API is used for, they could negatively affect the confidentiality, availability and integrity of the system itself and its data. It could even negatively affect other systems that it interacts with. This is dangerous stuff.

What can we do about it?

How do we prevent this you might ask? Let’s go over a mix of our recommendations and those from the OWASP API Top Ten Project Team.

Tanya’s Advice

  • You must test every single HTTP method, for every function, for every user. I know, this doesn’t sound that exciting. But guess what? This is one of the best ways you can ensure you avoid this type of problem. Make a grid with user roles on one axis, and functions they are/are not allowed to access on the other. For each entry in your grid, go through all of the HTTP methods that are enabled on the server. Maybe get a nice beverage or snack, then put on some music, so this task can be at least somewhat enjoyable.
  • Every single function should check in with the authorization mechanism for your app before it does anything, to verify you are still you (authentication) and that you are allowed to use that function (authorization). This is the same as every page within a web app/GUI front end. Make it a habit.
  • One use one authentication and authorization system for all your APIs, if possible. If you use multiple authorization controls, especially within the same API, it’s very easy to make a mistake. Simplify your architecture whenever possible, you will be a much happier developer.
  • Deny access to everything, for every role, by default. Specifically grant permission only to roles that require such access. Apply least privilege (all the time, whenever possible, not just in this situation!).
  • If possible, buy a system that will perform authorization for you, and implement it carefully, following the advice from its makers. Whenever possible, we should follow the order of buy -> borrow -> build. Buy a system that is tried, tested and trust. If not buy then borrow, use open source or a 3rd party component that implements this functionality. As a last resort, write this complex functionality yourself, with a plan for extensive testing and long-term maintenance, as this is a system that many other systems will depend upon. If you don’t think you can maintain something like this in the long term, then you should be buying or borrowing, not building.

The OWASP Project Team’s advice:

  • The enforcement mechanism(s) should deny all access by default, requiring explicit grants to specific roles for access to every function.
  • Review your API endpoints against function level authorization flaws, while keeping in mind the business logic of the application and groups hierarchy.
  • Make sure that all of your administrative controllers inherit from an administrative abstract controller that implements authorization checks based on the user’s group/role.
  • Make sure that administrative functions inside a regular controller implements authorization checks based on the user’s group and role.

Helpful Links from OWASP!

In the next blog post we will be talking about API6:2019 Mass Assignment!

API4:2019 Lack of Resources & Rate Limiting

In the previous post we covered API3:2019 Excessive Data Exposure, which was the third post in this series. If you want to start from the beginning, go to the first post, API1:2019 Broken Object Level Authorization.

You can read the official document from the OWASP Project team here.

Tanya on stage
Who here has perfectly secure APIs? What? No hands? – OWASP Global AppSec, Ireland, Feb 2023

Before diving into this one, I want to briefly discuss bots online.

An Internet bot, web robot, robot or simply bot, is a software application that runs automated tasks over the Internet, usually with the intent to imitate human activity on the Internet, such as messaging, on a large scale. – Wikipedia

While bots can be great (I used to have an automated message for all new twitter followers, to greet them and make them feel welcome), they can also be quite bad (slowly eating away at our defenses, with automated requests).

Bots are one of the quiet enemies of APIs. They hike up our cloud bills, they make our APIs seem unresponsible or slow, and they can be used to brute force an API that is not properly protected.  Because APIs can do so many things, it is possible for them to eat up all sorts of resources on your network, such as CPU, storage and memory on the host of the API, or whatever the API is calling.

There are all sorts of ways that APIs can have their limits tested, including: uploading very large files or amounts of data, making several requests at once, requesting huge amounts of data (above what the system or supporting infrastructure can handle), etc.

Setting Boundaries

The OWASP API Security top ten team recommends setting limits on the following settings:

  • Execution timeouts
  • Max allocable memory
  • Number of file descriptors
  • Number of processes
  • Request payload size (e.g., uploads)
  • Number of requests per client/resource (this is also called resource quotas)
  • Number of records per page to return in a single request response

So how do we avoid this happening to our APIs?

  • We can set throttling limits, to slow down requests that all come from the same source .
  • We can add resource quotas, limits to how many requests someone can make, and then they have to wait a time period to start making requests again.
  • Docker containers has several options built in for adding the limits described earlier in this article, as suggested by the team that maintains this great OWASP project.
  • Send messages back to whoever is calling the API ‘too much’, informing them they’ve reached the limit, and that now they must wait.
  • Design your API to ensure it takes into account if requests are “too big”. This is something threat modelling could help with, but ideally you would start with looking at each function and thinking about this as a problem your API will face at some point. Design with this in mind.
  • Also design into your API maximum amounts of data that it can accept and that it can return to the caller.  This might mean breaking a large request into multiple responses or blocking it altogether. This is something you should talk to your team about, ideally during the requirements or design phase(s) of your project.

They provide several very helpful resources, which you can find here:

OWASP Resources!

Even more resources!

In the next blog post we will be talking about API5:2019 Broken Function Level Authorization!

What’s the difference between Product Security and Application Security?

Recently I have started seeing new job titles in the information security industry and the one that stuck out the most to me is product security engineer. I started seeing people who were previously called an application security engineer having their titles changed to product security, and I was curious. Some of you may remember that I had Ariel Shin on the We Hack Purple podcast, and although she does product security, and I did ask her a few questions about it, but I wasn’t satisfied. I wanted to learn more!

Image of a watch, to illustrate the idea of a product.
Photo by Daniel Korpai on Unsplash

I also started a Twitter thread, which you can read here.

From what I understand, after speaking to many people about this, product security means a person who is dedicated solely to the security of one or more products. This means that if the product has hardware and software, they must understand how to secure both hardware and software. They also need to be extremely well versed in the threats that it faces, the personalities of the users, and anything else that might affect the reliability, confidentiality, an integrity of that system.

An application security professional is concerned with securing the software of the entire organization. If they happen to only have one product, and the product is software, they could be called an application security professional or a product security professional. However, most of the time an application security engineer is expected to do projects with a broader scope, trying to secure several/all applications, trying to ensure that every project follows the secure system development life cycle, and all the other things you’ve heard me drone on about in this blog, in my talks, in my book, etc.

Whereas a product security professional dives extremely deep into one or more products. For example, imagine a company that does e-commerce. It has one gigantic site, where merchants and purchasers both use the site in different ways, but it’s one big system. It may contain APIs, a beautiful GUI front end, one or more databases, a serverless app, and maybe even an integration with Stripe to run the credit cards for them. This could be called one big product, and if a product security person was assigned to it, they would be expected to understand how the entire system works, and how to keep the system, its data, and all of its users safe.

From Adrian Sanabria we have this definition, which I also agree with:

Looking at it from a business/organizational perspective: AppSec is a sub-branch of infosec. Product security is a sub-branch of product.

Adrian Sanabria

Although you may not have heard of a product security professional who reports directly to the product group only (they often report to the information security team, but are embedded in the product team), this also makes a lot of sense. Embedding the product security person in with the product team helps ensure from the very first meeting that the product is secure. This is a huge #SecurityWin!

Continuing down this line of thought, this would mean that the product security person would also be responsible not only for the software, but the infrastructure it’s hosted on, the entire supply chain that leads up to the building of that software, hardware, deployment, etc. Way more than just the software component.

Product security includes the security features of products.   

Ray LeBlanc, of the Hella Secure Blog

Product security being responsible for the product itself having security features for the end users is also an interesting idea, which I had not thought of before Ray pointed it out. I like this as well.

Facts about Product Security

  • ProdSec professionals are embedded in the product team
  • Prodsec pros need to know:
    ⁃ Architecture and design
    ⁃ Threat modelling
    ⁃ Secure coding principals
    ⁃ Be able to use the basic Appsec toolset: DAST, SAST and SCA
    ⁃ How and when to hire a pentester
    ⁃ All the steps of the Secure SDLC, and arranges to do them or ensure they get these steps done (even if they hire out)
    ⁃ Any policies you have that apply to your product
    ⁃ Understanding the product inside and out

To echo/add: Product Security (aka Platform Security) could involve more complex external IAM functions, secrets and cryptographic infrastructure, very closely interlinked and overlapping depending on the org.

@vect0rx

Another resource that may interest you, a podcast with Anshuman Bhartiya on this topic: https://tromzo.com/podcasts/anshuman-bhartiya-product-security. He was also previously on the We Hack Purple Podcast, where we spoke about SAST.

I hope that clarifying the difference between #ProdSec and #AppSec has been helpful. Do you agree? Do you disagree? We’d love to hear from you in the comments below!

Conclusion: Security Champions

In the previous article we talked about Metrics, and in this article, I will conclude this series on Building Security Champions.

A few more tips:

  • Start by defining the focus of your program and what is expected from champions. Be realistic; you can only expect 1-4 hours maximum effort from them per week.
  • If someone is taking a security course, but they are not on the security team, they may make a good champion. Reach out and introduce yourself.
  • If the mantra of the security team is “it’s my job to help you do your job, securely”, “you’re my customer” or “I’m here to serve you”, that is very attractive. If your team is known as ‘the ministry of NO!’, you will have difficulty attracting volunteers until you turn over a new leaf.
  • Record every group session and save them. Create an on-boarding set of champion videos from these recordings, so you can auto-onboard new champions. Some of the videos can also be used to on-board new software developers or other IT staff.
  • Save all the videos so anyone who missed them can see them later. Offer up the list of videos to everyone at your organization, if appropriate.
  • Include a TTT (train the trainer) package so that your security champions can train their own teams as needed. For instance, if you want your champions to give training or talks to their own teams, have them follow your package. The package should contain 1) your slides, 2) demo information and instructions to set it up, 3) a video of you giving the talk/training, and, 4) a video of you explaining what you are trying to get across for each slide and the entire demo, spoken as though you are teaching someone to give the talk on your behalf. For an example of this, see mine!
  • PS… Feel free to give these talks yourself, at your own workplace.

Lastly, don’t stop. Don’t give up. Perseverance is the thing that will make this program work. As your program continues it will grow and the value you that you receive from it will also grow, scaling upwards over time. You and your organization can do this, all it takes is dedication and time.

Please feel free to email me with questions, or even better, tell me about your success with your own security champions program!

Security Champions: Metrics & Data

The previous article in this series is Recognizing and Rewarding Your Security Champions.

If you’ve followed my conference talks, you likely saw my Security Metrics That Matter presentation, and understand that I absolutely love data. Here’s a general list of security metrics that matter, if you don’t want to read the whole article or watch the entire talk.

You may wonder, why are metrics important? The answer is twofold.

  1.  We can use data and metrics to report up to our bosses and show them we are succeeding. It’s evidence that what we are doing is working, and how well it is working. You can then use that data again to ask for more resources (staff, tools, budget), a raise, or other changes.
  2. The second reason is so that we, ourselves, can improve. We want to improve our program, ourselves, and our results. When we measure our activities and their impacts, we can see which activities or methods produce better results. We can then use that information to change our approach, for the better.

It is important, however, that we do not become fooled by vanity metrics. Vanity metrics are numbers that make us look good, but don’t necessarily mean anything. My talk on this subject has several stories, but for now let’s just tell one.

I used to work somewhere, and we all wrote blog posts. We were measured on how many “clicks” we got. A colleague of mine got 10X the number of clicks that I did, and I asked him how he did it. He explained he got the most clicks on Reddit. I was unfamiliar with the platform but thought I would give it a try. First though, I asked for extra data: I wanted to know how long people were staying on our articles. It turned out that people were staying on my articles approximately 1.5 minutes (which means they were reading the whole thing), and on his they were staying an average of 1.5 seconds (which means almost no one was reading the article, they were just clicking the link. This is commonly known as a “bounce”.) The purpose of our jobs was to write articles to help customers know how to use our products, and this means a bounce wasn’t valuable. Armed with this new information, we started comparing different platforms, and it turned out almost all traffic from Reddit were ‘bounces’. I also noticed that my Twitter followers were significantly more likely to read the article when compared to LinkedIn, and LinkedIn got better results than Reddit. My colleague started focussing on sharing links on Twitter (he had more followers than I did), and I started trying to get more followers on the same platform. It turns out that measuring clicks was a vanity metric. The rest, as they say, is history.

Now for your security champion program metrics! Measure the following things so you can see what’s working and what is not. Don’t forget to report upwards about the ROI (return on investment) your champions program has produced!

  • How many new security champions you have attracted
  • Measuring program engagement: how many people attended an event, how many people reported issues to you, how many people asked questions,
  • Use the bug tracker for metrics on how many security bugs are being reported and fixed, especially if you have targeted a specific bug class. Also, count how many new instances of that type of bug appear, hopefully this number will be very low.
  • Instances where champions have told you about a security issue you would not have known about otherwise
  • If the champions report better work satisfaction and/or fewer missed days of work
  • Gather stories of your champs saving the day, providing help to their teammates, or anything else that makes for a good story-telling session for upper management.

Up next, I will share a few more tips that don’t fit into any of the previous categories and conclude this series. Please feel free to email me with any questions!

(Over)Communication With Your Security Champions

As mentioned in the previous article (Recognizing & Rewarding Your Security Champions), the most common reason for failure of a security champions program is the security team losing steam, and/or the champions losing interest. In this article, we will discuss a few ways to avoid this. The best way? Communication.

https://youtu.be/nm1MpTuSNyI

To start off with, pace yourself. Often when I speak to security teams who have a failed program, they tell me how they started off very strong. “We gave them 2 different trainings, 2 workshops, and 3 lunch and learns, all in the first three months. Then we were exhausted. We haven’t done anything with them in over a year.” This scenario is far too common.

To pace yourself, I suggest meeting with each champion once a month, for 30 minutes. Then hold one lunch & learn and send one email to the champions. This might not sound like much, but you must remember, they are already doing a full-time job for your organization.

In my 1:1 meetings I like to ask the following questions (adopted from Ray Leblanc’s Security Champions article on Hella Secure blog):

  • What are you working on?
  • What are you going to work on next?
  • Do you need any help?

Each of these questions is open-ended, with the hope that it will prompt a meaningful conversation. I usually take notes during the meeting, and then send them after to both of us, with any action items for either of us highlighted in bold. (Note: I’ve used this technique to get many of my previous bosses to do things for me. Set a reminder for a week from then, and then reply-all to that email chain and ask: “Any updates on these action items?” It works like a charm!)

In your lunch and learn (which does not need to be at lunch time, or involve food), teach them something you want them to know. Do not teach them things they do not need to know, unless they asked for that topic specifically. During this session you or a teammate can teach, or you can show them a training video you like, or even a recording of a conference talk that really hit home for you. If you show them something pre-recorded, ensure you watched it first, you don’t want to waste anyone’s time with death-by-powerpoint. The more fun you can make these sessions, the better. If you’re up for it, invite all of the developers and let everyone learn something new!

Woman running
Photo by Greg Rosenke on Unsplash

Ideas for lunch and learn topics:

  • The specifics on how to apply policies, standards and guidelines. This could be a secure coding workshop, or a threat modelling session.
  • Talks about the top vulnerabilities that you are seeing in your own products, including the risks they pose to your specific business model.
  • Workshops on how to use the tools that your team wants them to be responsible for. Especially how to configure them, how to validate results, and where to find information on how to fix what they find.
  • If they are responsible for design or architecture, give them secure design training.
  • Tell them about a security incident your team had, and how it could have been prevented (assuming you are allowed to share this information).
  • Hold a consultation on the new policy, standard, or guideline your team is considering publishing. Ask for their feedback, then adjust your documents accordingly.
  • Remember to take attendance (for metrics) and take notes of any questions for you to follow up.

The monthly email:

Sometimes you just don’t have time to do a lunch and learn event or hold 1:1s, but you still need to send a monthly email. The monthly email lets the security champions know what’s going on, and that they still matter to you. The program is still running, because you sent an email. If you don’t send this email, and you haven’t touched base in any other way, this leaves a space where your program may start to disappear.

The monthly email does not need to be fancy and doesn’t need to say a lot. Generally, the monthly email says:

  • What events are happening this month at your org (lunch and learn, all staff, any other meeting they should know about)
  • Any updates your team has (new policy, new tool, project updates, etc)
  • Anything interesting from the news that they may find valuable
  • Any local security events they may be interested in
  • Any podcasts, videos, blog posts or any other media that is relevant and you feel relates to them, about security (of course)

I live in Canada, and in Canada we are a country of immigrants. This means we have many, many different religions represented in most workplaces. In December, there’s Hannukah, Ramadan, Christmas, and more, and often people take time off for these special holidays. This means having a large meeting in December is darn-near impossible. This is the type of situation where you just send the monthly email! It could say something like the following:

Hello Security Champions!

As it is December and many of you will be off celebrating various holidays, we are not going to have any events this month. We also want to wish you happy holidays, and we hope you enjoy all the snow we got this past weekend!

In January we are going to boot the Champions program back up with a lunch and learn on XSS. As some of you are aware, we’ve found it in about 1/3 of our custom apps, and we want to stomp​ it out in the new year (with your help of course!) An invitation will arrive later this week.

In the meantime, please check out this XSS Deep Dive by Tanya Janca. We’re going to cover this topic a bit differently than she does, but it gives you a good idea of what we are up against.

Have a great December folks!

Sincerely,

The Security Team

My hope from this blog post is that you remember to continue to communicate with your champions. Don’t let your program slip, it will disappear faster than you think. When in doubt, send them an email and check in. Up next, we will discuss Metrics.

Recognizing & Rewarding Security Champions

If you’ve ever read the book The 5 Love Languages, or articles summarizing the 5 love languages, then you are aware that there are predictable patterns of how people respond to various acts of kindness. Someone’s “love language” is the specific type of kindness that they are most affected by. For example, someone for whom their love language is “words of affirmation” would respond very well to receiving a glowing performance review, a compliment on a new article of clothing, or accolades from their colleagues about a project they worked on.

The previous article in this series is Teaching Security Champions.

You may be wondering at this point if you accidentally clicked on an article from a women’s fashion magazine, not a technical article from We Hack Purple. But please have a bit more faith, and read on.

The 5 love languages are:

  1. Gifts
  2. Words of Affirmation
  3. Physical Affection
  4. Spending Quality Time
  5. Acts of Service

Two people sitting using laptops
Security Champions at work!

When we are creating a security champions program, it’s very important that we ensure they feel appreciated. We don’t want them to feel squished into doing two jobs, for only one paycheck. One of the biggest challenges that security team’s face when creating a champions program is having it fall apart after the first few months, either due to the security team losing steam, or champions losing interest. We need them to feel very aware of our gratitude, and interested in the program itself, for them to continue to want to serve the security team’s agenda.

As you likely already figured out, not all the love languages listed above are work appropriate. We can’t run around giving hugs or holding hands with other employees. That said, we can adopt most of them for work situations, so that we can show the champions they matter to us, in appropriate ways, that support our security program.

Below is a non-exhaustive list of several ideas to make your champions feel as valuable as you know they are for your program.

  1. (Security Related) Gifts
  • Physical or digital security-related gifts – books, videos, training, CTFs, perhaps a copy of Alice and Bob Learn AppSec?
  • Create a Certificate to put on their wall.
  • Stickers, posters or any other decoration that is security focused.
  • Tickets to a conference or training.
  1. Words of Affirmation
  • Make sure to put a note in their performance review about them being a champion.
  • Tell their boss every time they do something that makes a big difference.
  • Send them an email and tell them when they did something big, let them know that YOU saw.
  • Recognize them in front of their peers (special virtual background, star on their name is slack, etc.)
  • Digital badges for signature blocks.
  1. Physical Affection
  • High Fives are the only recommended form of physical affection that you should show another employee. High fives signal success, and your approval of whatever they just did.
    • *** And only do this if you are confident that the employee is comfortable. Please be mindful that some religions and cultures do not allow those of the opposite sex to touch each other and be respectful if this applies. Never push physical touching at work.
  1. Spending Quality Time
  • Giving them your time is a reward. When you do, give them your undivided attention (put your phone away), and turn your body towards them.
  • Let them see a new tool first, give them a “sneak preview” ahead of everyone else.
  • Let them help you make decisions. Ask for advice from them and feedback, then take it seriously.
  • Invite them to attend security events with you.
  • Whenever you meet with them, this is quality time. Ask them: What are you working on? What are you going to work on next? Do you need any help?
  1. Acts of Service
  • Help them with more than just security. Are you good at design? Help them with it! Are you great at presentations? Offer to let them practice in front of you. You don’t need to do this very often, just once can make a huge impression.
  • Make introductions, where appropriate. “Oh yeah, Chris from QA uses that tool, I’ll introduce you so you can learn.”
  • Find answers they need to security questions and problems. Never leave them hanging.

When people feel appreciated and valued at work, they work harder (many studies show this to be true). Your champions already have full time jobs on other teams, they are going above and beyond for you. Let them know that you are very aware of them, by always making them aware of it with your actions, not just your words.

In the next article we will discuss communication with your champions!

Teaching Security Champions

In the previous article, we talked about how to engage your champions. We want them interested, revved up and ready to go.

You are in a room full of brand-new security champions and they are itching to learn all about ‘cyber’, what do you do? What do you teach them? How do you impress them?

Only teach them what they need to know. Nothing more.

As someone who creates security training professionally, I have to say, I’ve seen a LOT of filler. Extra content that just does not need to be there. Software developers do not need to know the history of Diffie-Hellman, or the difference between symmetric and asymmetric encryption, unless they are building encryption software. So don’t try to teach it to them unless they have a keen interest and have asked about it.

What they really DO need to know is:

What you need, expect and want from them, as champions.

You should define the goals of your program and share them with your champions. Share your plans for them, as much as you can. Give them timelines, training information or anything else you have. You need to make clear what you are expecting, or you may not get it.

Technical topics for teaching your security champions:

  • Formal training on secure coding, with labs!
  • Threat modelling
  • Secure architecture (whiteboarding)
  • Code Review
  • How to fix the bugs they find
  • Repeat yearly as a minimum

Topics specific to your organization:

  • Which policies, standards and guidelines apply to them
  • Help them create missing guidelines
  • Teach them how to be compliant, help them get there
  • Their role during an incident
  • Job shadowing

Hold consultations to let them provide input on the policies that will affect them. Trust me, their feedback will be priceless AND it will make them feel heard.

The last topic you need to ensure they learn is tooling. If you expect them to use a tool you need to show them how, what the output means, how to validate the results, how to install and configure it. It is also your job to either help them pick excellent tools or involve them when you are choosing tools for them.

In the next article we are going to discuss how to Recognize Your Champions.

Engage Your Champions

In the previous article, ‘Recruiting Security Champions, we covered several ways to find, attract and recruit people to your cause. In this article, we will talk about how to get them revved up about security once you have found them.

Engage:

To occupy, attract, involve – in security activities!!!!

To participate or become involved – with your champs!!!!

If we want IT professionals to join our security champions programs, we must make it interesting and appealing to participate. We want to motivate them; to do extra work on top of their regular job, to care about security, to learn a lot of new things, to work with us. It needs to be good.

People discussing at a table.
Photo by Redd on Unsplash

Below are a few ideas of how you can make your champions feel engaged.

If possible, bring them on a security incident that has to do with software. Teach them what it’s like to respond, the consequences, and just how much damage insecure code can cause.

Share (appropriate) secrets with your champions. If you are going to share quite sensitive info, inform them of the concept of ‘need to know, then ‘Deputize’ them onto your team for that one meeting.  Being vulnerable and admitting mistakes is a great way to get buy-in, and interest.

Let your champions see everything first. New tools, documents, policies, changes, etc. And ask their opinions. First, because they will likely have great ideas, and second because it makes them feel like they matter.

Create a mailing list for your champions to tell them new security stuff. Send them links to podcasts, articles, events, or anything else that you think is relevant and they may find interesting.

Meet with them 1:1 once every month, and have a pre-set list of questions. Potential questions (thanks to my friend Ray at Hella Secure Blog): What are you working on? What are you going to be working on next? Do you need any help? These questions will spark conversation and led you down the right path. That said, when you ask questions like this brace yourself for potentially bad news so that you can play it cool if they reveal something that makes you cringe.

Hold team-building events, let them know each other. Having a friend on a team always makes it worth coming back.

Invite them to join security communities, such as OWASP or We Hack Purple Community (with of which are free to be part of!).

There are many, many ways you can make the champions feel engaged, and one of the best ones is to give them training, which is what we will talk about in the next article, Teaching Security Champions.

Discussion in a board room.
Photo by Leon on Unsplash

Recruiting Security Champions

In the previous article, ‘Building Security Champions, we covered what champions are, why you need them, and our plan to make an amazing program.

The #1 most important rule of recruiting security champions is that you must attract them. Do not “voluntell” someone to be a security champion. That person is not going to do their best for you, and they certainly won’t enjoy the experience. Attract the right people instead of forcing them.

How does one ‘attract’ champions?

Perform Outreach

Two people sitting using laptops
Security Champions at work!

  • Use lunch and learns to teach about security
  • Arrange security training
  • Anyone who asks questions or attends all the events is a potential champion
  • Use interesting titles for events if you can
  • Add a note to your email signature, saying you are looking for champions
  • Put a sign on the fridge in the kitchen
  • Talk about it at the all-staff meeting
  • Send an email to all of IT

Observe

Pay attention to who responds, attends events, asks questions, and who is ‘always there’. Those are the people you need.

Adjust Your Attitude

Change your team’s mantra to “I am here to serve you” and your team will attract even more candidates. Saying “you are my customers” to the rest of IT if you are a security professional, is basically the truth. Plus, you always get more bees with honey.

#2 most important rule of recruiting: ensure their manager is on board. You don’t want this person to have to fight to do work for you or feel conflicted. Ensuring their manager is comfortable.

In the next article, we will talk about how to engage your champions (which will result in you finding even more).