How REI built a DevSecOps culture and how Snyk helped

A few years ago, REI embarked on its digital transformation and cloud migration journey, moving on-prem development environments to AWS. But, as REI’s development teams began this transition, their security counterparts noticed that application security just wasn’t keeping up. As a result, REI began another journey: identifying the right security tooling and cultural shifts for AppSec success. 

Dan Ngo, Lead Security Engineer, Cybersecurity Engineering and Risk Management at REI, and Clinton Herget, Field CTO at Snyk, chatted at an AWS Re:Invent 2023 session about how the REI team built a strong security culture across development units. Read on to learn more about their story and watch the full video below. 

REI’s decision to establish an AppSec program

When Dan arrived at REI, the company was going through a digital transformation, including a cloud migration. At this point, security was mostly siloed out of developer workflows and focused on other items like endpoints or email security. There were some application security initiatives, but they mostly consisted of different development teams taking different approaches to securing their applications. These teams mainly relied on a few security champions to take the initiative. However, these security champions had varying experience with security practices, leading to inconsistency across the organization.

When Dan came in, he asked the developers what they were doing for security. They mentioned that they used a few tools to compile security reports, but they didn’t look at the results much. 

According to Dan, “There were some engineers and developers that had a security mindset and implemented security scans for compliance reasons. But they didn’t really get any benefit from it.”

Dan and his team had to establish a strong security foundation to grow alongside their app modernization efforts. They knew this foundation needed to include the right combination of tools, culture, and processes. Their ultimate goal was to shift left and invite developers and engineers into their security initiatives, leading to a DevSecOps approach across the organization. 

How REI uses Snyk tooling

As part of their initiative to build strong relationships with other teams, Dan’s team knew they needed to pick the right tools to find and fix vulnerabilities. After some trial runs with different tools, they turned to Snyk. The developers appreciated the visibility of the open source vulnerabilities with Snyk Open Source and the code fix suggestions with Snyk Code. The platform engineering team also favored Snyk because it integrated best with REI’s existing tooling and the underlying AWS cloud environment. 

REI’s tips for building a DevSecOps culture

As Dan’s team fostered this engineering/development partnership, they uncovered some tactics that worked and others that didn’t. They ultimately needed to go through a cultural transformation to get the buy-in and security results they were looking for. 

Here are three of the biggest takeaways from Dan’s experiences with building out this culture with Snyk’s tooling.

Seek to deeply understand other teams’ priorities and build trust

At the beginning of their security process, REI tried to rapidly implement a new static code analysis product without taking the time to understand the developers’ point of view. This implementation didn’t go over well. Dan mentioned that developers pushed back on the tool because it didn’t integrate with their existing pipelines. It created lots of friction between the two teams.

After gathering some feedback, Dan’s team realized they needed to work with — not against — the development and engineering teams’ existing workflows. They took a detailed look at REI’s existing platforms and teams and then aimed to adopt tools and processes that would work alongside them. 

To truly understand these other teams’ perspectives, Dan spent much of his time embedded within the site reliability engineering team, learning all about their existing processes and priorities.

“I helped contribute to the code, understand how microservices work, and understand how the infrastructure engineers did their work. And then from there, it was the building of trust, then coming and introducing more security concepts to them.”

Embrace continuous growth

Throughout the conversation, Dan mentioned several instances in which his team tried something new, made some missteps, and then learned from them. Rather than expecting perfection right off the bat, his team has taken the perspective of starting somewhere and then working to improve processes over time.

As an example of their continuous growth mindset, his team chose to open up office hours when they introduced Snyk. Through this open conversation and Snyk’s alerts, they uncovered that the developers themselves weren't able to fix many of the security scan findings. Indeed, those vulnerabilities were part of a library called CRAMPON, curated by their platform engineering team. Some developers were using outdated versions of this framework. As a result, Dan’s team was able to make improvements.

“We now have a deprecation policy,” Dan said. “We have an SLA."We're also working toward defining what a minimum CRAMPON framework version looks like and communicating that to developers so they understand that they need to be on a specific minimum version of our internal tooling.”

Identify and assign ownership to fixes

As part of their holistic and actionable approach to AppSec, Dan also wanted to identify ways to deal with legacy code. For them, the key was to clearly define which teams owned each piece of legacy software.

By breaking most of their monolith down, the team was able to understand internal ownership of most of the legacy code. Snyk helped them identify dependencies. Now, the security team can clearly see who owns which code and address fixes with the right groups. For the remaining pieces of code and services that are still in the legacy monolith and do not have clear ownership, the security team directly tackles any related security issues using Snyk, which builds respect with developers and nudges them to check issues themselves. 

REI’s application security successes 

Thanks to their process and tool choices, Dan and his team now see a much more collaborative relationship with the development and platform engineering teams. As they continue to mature the program, Dan said that his biggest goal is to ensure the developers can contextualize and understand application vulnerabilities and then prioritize and implement fixes within REI’s defined SLAs — especially mean time to fix (MTTF). Snyk helps him meet this goal by finding vulnerabilities early in the pipeline and giving the developers resources to fix them rapidly. 

“Seeing developers deploying a fix to our app because Snyk said there was a problem is a very good sign that we are on the right path to a robust process and a good trust between security and development teams." 

Watch the AWS Re:Invent presentation below to learn more about our partnership with REI. If you’d like to try Snyk’s developer-first security platform, you can also sign up for a free Snyk account directly from the AWS Marketplace or directly from the Snyk app. 

Video: AWS re:Invent 2023 - How REI built a DevSecOps culture from the start