How a 0-day event galvanized a developer-led security mindset at DISH

When a security incident happens, it’s one thing to reactively fix the issue, sweep it under the rug, and move on. It’s a whole other to respond to the situation with a proactive, forward-facing response — not only solving the existing issues but preparing the entire organization for the future. DISH Network did just that, responding to a significant security incident with new, shift-left initiatives that made their security and development teams stronger than ever.

Jim Armstrong, Leader of Product Marketing for Snyk, and Joe Farino, Head of DevSecOps at DISH, discussed this process at AWS Re:Invent. According to their conversation, a shift-left security approach takes strategic prioritization, shared developer-security responsibility, measurable success metrics, and repeatable training processes. 

DISH Network’s vast portfolio of applications

While DISH Network is known as a satellite TV offering, today, it’s expanded to become a connectivity company, retail wireless brand (Boost), streaming service (Sling), smart device provider (OnTech), and much more. Additionally, DISH is the first U.S. cloud-native nationwide 5G network builder and operator.

DISH has hundreds of in-house-developed applications to maintain these offerings. They employ a team of over 3,000 developers — roughly one developer for every five employees — to upkeep these apps. Several teams create and maintain these apps within on-prem, hybrid, and cloud environments. Every team runs independently, using a variety of programming languages, frameworks, and development IDEs. Each one also leverages a different DevOps configuration, making it challenging to maintain central control. 

A few years ago, DISH started pushing for a common thread of cloud transformation and more agile, automated processes across these varied development teams. One of their most successful initiatives was when DISH built the first cloud-native Open RAN nationwide 5G network in the U.S. with AWS.

Security and innovation weren’t going hand-in-hand

As they pushed for these digital transformation efforts, DISH’s development teams faced a significant roadblock: security. Although much of the organization focused on cloud innovation and agile practices, DISH’s security team still pushed for traditional security activities.

Joe said, “We didn't have our security checks in place until right before they went to push to production. So we’re telling them, ‘Hey, you have all these vulnerabilities you need to fix before you can do that and move on to your next topic.’ They're going to say no, and then we’re going to fight with them over it. So they're frustrated with security; we're frustrated with the app team.” 

Because developer velocity contributed to revenue more directly than security, it often won the battle, with developers circumventing deployment gating and leadership pushing back on fixing security issues before deployment. 

As with so many other organizations, the announcement of Log4Shell in 2021 was a wake-up call for Dish Network. They realized that there had to be a better way to resolve 0-day incidents than pausing development and parsing through hundreds of codebases to find instances. They worked with Snyk in a proof of concept to quickly identify which projects contained Log4J and remediate hundreds of affected applications in a short amount of time.

Then, in early 2023, DISH experienced the most significant security incident in the company’s history. To prevent similar events from happening in the future, DISH honed in on finding better ways to secure their applications. 

A new, developer-friendly approach to security

After their major incident, the DISH team settled on Snyk to secure their applications better and invite developers into day-to-day security activities. They chose Snyk, as our tools fulfilled their need for broad language coverage and supported plugins for several developer IDEs. In addition, DISH’s development teams gravitated towards Snyk’s out-of-the-box reporting and filtering capabilities, responsive support team, and easy deployment and maintenance.

As they started working with us to establish an application security program, DISH focused on a few key areas:

Prioritizing and securing critical apps first

After successfully using Snyk to remediate Log4J vulnerabilities in 2021, DISH leveraged our tools in a similar way in 2023, this time addressing pressing concerns from their more recent incident. They started by scanning their mission-critical applications for security issues.

They also paired Snyk tooling with dedicated security discussion channels to improve security education. Each app and infrastructure team whose applications contained vulnerabilities would get added to one of these channels. Then, management would work with these teams, coaching them through vulnerability fixes and answering questions. 

The team has continued to scan its vast application library. Today, many of DISH’s Wireless, Sling, OnTech, and IT development teams use Snyk to secure the many projects they deploy to AWS. As a result, one of the development teams found and fixed over 600 instances of nine unique critical issues across more than 200 APIs — all in less than 15 days. Those fixes remediated over 2,000 less critical vulnerabilities. Another team deployed fixes for over 800 critical and high-severity issues across 20-30 of their applications within four weeks.

Giving devs more security responsibility

DISH also prioritized putting more security responsibility into the developers’ hands with Snyk’s developer-first tools. Joe said, “This prevents it from leaving the developer's machine. When you're doing that, only one person has to be involved in fixing it — the developer. If you’re gating a development environment deployment instead, the entire team is going to be aware of that vulnerability when you hit that gate…which wastes resources.”

The team also increased development velocity by fixing vulnerabilities before the code even reached the repository. This new security efficiency enabled the team to better address device certification and FCC-mandated coverage requirement deadlines. It also led to a 70% reduction in new vulnerabilities infiltrating projects.

Establishing clear metrics for application security success

The DISH team also started focusing on establishing and measuring AppSec metrics over time. They began with coverage goals, gauging the percentage of mission-critical applications covered with Snyk Code. Next, they focused on the rate of new vulnerability introduction, measuring how well their developers prevent fixable vulnerabilities from getting deployed in the first place. 

Joe said, “Snyk is providing the developers with real-time feedback as they're writing code and saying, “Hey, that security issue you need to fix, it's easy to fix; here's how you do it. We find that the teams using that shift left tooling are introducing fewer defects overall — security or otherwise — by quite a bit.”

Creating repeatable processes for training development teams

DISH also focused on making these robust security processes repeatable and replicable across teams. They did so by integrating Snyk into the developers’ everyday IDEs and CLI, enabling the developers to use the tool with as few context shifts as possible. 

To encourage the continued use of Snyk tooling, they also built scorecards that show the percentage of developers on each team that actively use the tools and make these scorecards visible. This way, each development team could see how their usage stacked up to other teams. These metrics also help the security team identify and address usage gaps. They can contact the involved developers and leadership if usage dips below 80% on a critical application. 

In around six months, more than half of DISH’s licensed developers started using Snyk’s shift-left tooling as an integral part of their coding and code review efforts.

Looking to the future

As the DISH team continues to grow their application security initiatives, they will focus on a few areas. First, they want to expand coverage to all other development teams, leveraging the ROI they’ve seen with their critical app teams to prove value. In addition, they plan to improve patching and framework update processes, integrate Snyk into their CI/CD pipelines, add more automation to their security efforts, and tighten their DevOps pipelines’ access and permissions. DISH also wants to address container and infrastructure-as-code vulnerabilities down the road. 

Ultimately, DISH is aiming to proactively stay ahead of future executive orders and emerging regulations related to software supply chain security. A less reactive approach will enable them to keep business running as usual, even in the midst of regulatory changes or major incidents.

As they progress with their application security initiatives, Joe’s team stays grounded in shift-left security. He said, “We see the ROI is training the developers to write more secure code as they're writing the code. We want security to be integral — not an afterthought.”

To hear the whole DISH Network and Snyk story, tune into Jim Armstrong and Joe Farin’s conversation at AWS re:Invent.