When I started working in DevSecOps, I hadn’t even done DevOps before (gasp!). I had been following the Waterfall software development methodology in the Canadian Public Service (with a tiny bit of agile thrown in) for over a decade, before I joined Microsoft and my entire world changed. I had been working on the OWASP DevSlop open-source project with my friend Nikky Becher at the time, muddling our way through, trying to figure out how to pentest in a DevOps environment, and suddenly I was supposed to be a DevOps expert. I thought “How can I catch up? And how can I do it FAST?”
I decided that I would create an app, using the Azure DevOps CI/CD, and I would ‘share’ my pipeline that I used to do it. Quickly I figured out there wasn’t a way in Azure DevOps (at the time) to open-source or ‘share’ a pipeline, so I decided I would film myself, live, as I built. I made a basic app, and a basic dev -> QA -> UAT -> Prod deployment, and I was off to the races. I started coding live on Twitch, and adding every security tool to my pipeline that I could get my hands on.
Not every episode of the OWASP DevSlop show went well, and neither did every tool. The DAST I chose threw a false positive while I was presenting at Microsoft in Ottawa, Canada (live on stage, naturally). There’s one episode with Nancy Gariché (one of the project leaders) and I just FAILED the whole time, and after 3 hours we never got the tool working at all. There was a 5 hour episode where I updated the .Net CORE framework, and all my dependencies that had vulnerabilities in them, and that was exhausting. But I LEARNED. And I learned fast.
In 2018 I joined a company called IANs Research, doing ‘Ask an Expert’ calls, helping clients with Azure and AppSec problems. As I learned more about DevOps and DevSecOps, I started helping clients with those topics as well. I would often have to do research to figure out the solution before a call, but I didn’t mind. All of their questions helped give direction to my learning. Over the years I have helped hundreds of AppSec teams crush their problems, and learned a LOT along the way.
In 2020 I started coaching two different companies with their AppSec programs, for a few hours a week, building out their DevSecOps programs. When contrasted with the short types of assistance I was giving to IANS clients, these long term relationships helped me see programs over extended periods of time. And I got my hands dirty on a weekly basis, trying many different types of tools (for better or worse).
I also attended numerous conference talks and read lots of articles, to see what others were doing, and learning best practices a long the way.
All of this learning was A LOT of work. One day an IANs client told me they were going to start in DevSecOps, and asked if I could make them a “do not do” list. A list of pitfalls that they should work to avoid. Dear reader, I dove deep down this rabbit hole. It had never occurred to me to start with “what not to do”, rather than “this is the way”. And how many headaches could be avoided…
WIth this in mind, I wrote a conference talk (video below), on this topic. And this blog series is going to explore each of the 15 ‘worst practices’ that I cover in the talk.
The 15 items I will present in this series are as follows:
- Breaking Builds on False Positives
- Untested Tools
- Artificial Gates
- Missing Test Results
- Runaway Tests
- Impossible SLAs
- Untrained Staff
- Forgotten Bugs
- No Positive Reinforcement
- Only Worrying About Your Part
- Multiple Bug Trackers
- Insecure SDLC
- Overly Permissive CI/CD
- Automation Only in the CI/CD
- Hiding Mistakes and Errors
I hope that by sharing mistakes that I have seen and made, all of us can avoid these issues going forward.
~ The next article, ‘The Boy Who Cried Wolf’, is here. ~