The Evolution of Elevation: Threat Modeling in a Microsoft World
Published: January 17, 2012
Author: Dana Epp, Microsoft MVP - Enterprise and Developer Security
I’ve been asked to give a history of threat modeling from the perspective of an outsider looking at Microsoft. While I’m flattered by the idea, I’m also petrified. I have my own views of what has happened over the past 15 or so years, and I suspect I’ll contradict the views of some people I respect at Microsoft. That said, I welcome the opportunity for those people to email me at email@example.com and let me know what I might have missed. Or, let the world know and open the conversation on Twitter by sending a tweet to @danaepp. What follows is my view on the evolution of threat modeling, and where we have ended up in 2012.
The concept of threat modeling is not new—you can’t design a secure system until you understand the threats to it, and what weaknesses an adversary might exploit in the system. Thus, threat modeling.
I remember reading an interesting article on attack trees, written by Bruce Schneier and published in Dr. Dobbs Journal in 1999. Attack trees are a formal approach to model threats against computer systems that breaks down how a system can be attacked, node by node. It then assigns values on how probable a breach is. Attack trees help expose the attack surface of systems and software components, and the probability of attack. You can even assign a monetary value to each node to assist in quantitative risk analysis, allowing you to get to the heart of the areas on which you really need to work.
The problem is that attack trees quickly became rather complex. A full attack tree often has hundreds of different paths you can take, making it difficult to follow visually. Determining the classification of a threat from attack trees is also far too labor-intensive. I remember one day in the boardroom when we had more than 200 sheets of paper printed up for the attack tree of one of our products! It was simply too hard to manage. While the concept of attack trees is sound, the application of this approach is far from it.
Fast forward a few years. Around the time that Bill Gates sent out his well-known email about the Microsoft Trustworthy Computing initiative in January of 2002, there was some interesting work being published in the industry about writing more secure software. Ross Anderson had a great book called Security Engineering: A Guide to Building Dependable Distributed Systems. Another was Building Secure Software by John Viega and Gary McGraw. Both books covered how to avoid security issues by properly designing security into software right from the start. Each had different approaches, but ultimately they both focused on better application of safeguards in code and systems. These books should be required reading for anyone serious about building secure systems.
For me, there was one book that really stood out—Writing Secure Code by Michael Howard and David LeBlanc (Microsoft Press, 2002). This book contains a lot of real-world knowledge on how to apply security safeguards correctly to build safer systems. But what really made an impression me were the 10 pages on Security Design by Threat Modeling.
Several months earlier, I read an internal draft from Carnegie Mellon University on OCTAVE, a suite of tools and techniques for risk-based information security assessment and planning. (You can check out the final version at www.cert.org/octave/.) While interesting, the problem with OCTAVE was that it was designed for companies with more than 250 employees. Given that we worked in teams of less than 10 people, this just wasn’t a realistic process for us to adopt. (As a side note, several years later the authors of OCTAVE published OCTAVE-S, designed for organizations with less than 100 people.) While OCTAVE wasn’t the answer to moving away from attack trees and toward a more formalized method of modeling threats, those 10 pages in Writing Secure Code got me thinking.
The way I read it, Microsoft had built a new approach to threat modeling that was based around a simple premise: What if we could look at our systems differently and classify each type of risk into separate buckets? This is the basis for STRIDE threat modeling. (STRIDE is an acronym that stands for Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, and Elevation of privilege.) At the time, if you looked at the Common Vulnerabilities and Exposure (CVE) database, every vulnerability being reported could be classified into one (or more) STRIDE categories. And, the people thinking about security at Microsoft recognized that. Finally, I had found a formalized system that made sense to me, so I went on the hunt for more information.
Over the next few years, STRIDE evolved, mostly due to what Microsoft was learning as it went through the shift of training staff on secure programming. In 2004, Microsoft Press published Threat Modeling by Frank Swiderski and Window Snyder. Threat Modeling took the concepts of STRIDE to an entirely new level by showing how to approach the process within teams working on applications, and within the systems on which the applications run. Along with Writing Secure Code (now in its second edition), Threat Modeling became the de facto required reading for developers and quality assurance (QA) staff when they joined our team. Theory aside, I always encountered resistance because, at the time, developers and QA staff found the process too tedious. No one wanted to look at data-flow diagrams (DFDs) in Microsoft Visio and then write about them in Microsoft Word. Swiderski’s representation of threat modeling was too detailed and complex, and non-security people just glazed over. It was, to say the least, a challenge to get them to buy in.
And there was a bigger problem. As an information security professional, I was a fan of the common risk formula: Risk = Probability (chance) x Damage Potential (damage). By 2004, Microsoft was firmly entrenched with a formula called DREAD, an acronym for Damage, Reliability, Exploitability, Affected areas, and Discoverability. In his book, Swiderski went deeper into DREAD, but I really didn’t like the idea. It was just too subjective. Security-minded individuals tend to rank everything high, making most threats a 10. Developers who are not focused on security tend to rate threats low (usually a 1 or 0), creating a huge polarity in the reviewing process. Between skewed threat rankings and a large complexity in ranking threats with DREAD, I just couldn’t see this lasting long, and I was right. By the end of 2005, people like Dan Sellers at Microsoft were reporting that “DREAD was DEAD.” I was very happy.
I was also frustrated. Getting buy-in for threat modeling was difficult. Although the teams I worked directly with had “gotten it,” security awareness and education still had a long way to go. Microsoft started to document its Security Development Lifecycle (SDL) process publicly, and that helped, but it was still difficult to get people outside of Microsoft, especially those in small teams, to create DFDs and then go through STRIDE analysis for each element. This was particularly true when they had to write loads of documentation to explain the process. There had to be a better way.
And there was. It was a tool first introduced to me years later in a non-disclosure agreement (NDA) session at MVP Summit by Adam Shostack, a program manager on the Microsoft SDL team who has a deep passion for threat modeling. Jesper Johansson, Alun Jones, and I attended a closed session on how Microsoft conducted threat modeling with its own products using an internal tool called Wakidash, which melded a Visio DFD with broken-down elements you could work against for STRIDE categorization. In other words, it took a lot of the hard work in documenting threats from the visual representation of the drawing and simplified it into a single tool for quick documentation. It was an eye-opening experience, and as Adam demonstrated this wicked tool, I knew I had to have it. But I couldn’t. It wasn’t available for public viewing, and it was heavily dependent on internal Microsoft resources.
Without violating my NDA, I blogged about the experience, and pointed out that the tool needed to be merged with the public Threat Analysis and Modeling (TAM) tool from the Microsoft Application Consulting and Engineering (ACE) team. Jesper, Alun, and I started telling anyone at Microsoft who would listen that Wakidash should be made public. That was enough ammunition for Adam, and he took it to the leadership team. Of course, I have no idea what the conversations were like inside Microsoft, but, in the end, the community won. At the beginning of November of 2008, a public beta of a new iteration of Wakidash was made available as the SDL Threat Modeling Tool, and it’s been available to us ever since. Score one for Microsoft listening to MVPs! And, a special thanks to Adam for standing up for us to get it made available to the public, and not only to just a few MVPs.
Fast forward a few years. The SDL Threat Modeling Tool has been released, and there are several videos, podcasts, and webinars about using it. However, adoption is mixed. Some teams have grasped it and are running with it. But for others, the challenge of threat modeling is still there. With the pressure to ship software, the idea of threat modeling slipped for many organizations. While this is unfortunate, it is also a reality. Although some companies were mandating threat model documentation, far too many weren’t. Now comes an interesting twist to the story. People like Adam Shostack and Michael Howard saw this and decided to do something about it. Believing threat modeling could be more easily adopted if it were a bit more fun, they invented a card game called Elevation of Privilege. It’s a game of tricks and trumps, in which the suits follow STRIDE, and the goal is to review a DFD and associate cards within a suit in an effort to win the hand. Of course, this can be trumped with the Elevation of Privilege (EoP) cards. I got a chance to check out the game in the winter of 2009 in Adam’s office, and I brought Alun as well as Dan Griffin, another security-focused developer MVP, with me. After only a few hands, we were hooked. It really became a teaching aid for anyone, even if they had never seen or heard of STRIDE threat modeling. It was fun. It was useful. Above all, it helped teams adopt threat modeling more quickly, and with less resistance. In our office, we routinely make Friday afternoons EoP time; we rotate each week among product groups and play a few hands to explore where new threats might be lurking in updates of our codebase. No one thinks of it as work, and that’s okay by me. Everyone benefits.
Then Microsoft did the unimaginable. It released EoP under a creative commons license, and gave away decks to the public. I remember listening to Adam speak at RSA Conference 2010 while Microsoft handed out the coolest looking deck of cards I had ever seen. At last year’s RSA Conference 2011, Microsoft gave away more decks. They just hold that much interest in the security community.
Threat modeling has come a long way. From the hundreds of pages of attack trees I had to work with at the beginning of the decade to the simple 84 cards of the EoP deck, one thing is for sure: Threat modeling has evolved. No, that’s not saying enough. Actually, threat modeling has been elevated to new levels. Between EoP and the SDL Threat Modeling Tool, Microsoft has given the community a set of tools, techniques, and guidance to design safer systems. Although threat modeling is only a small part of the SDL, its impact is highly regarded as a critical component in the process. My understanding is that all products within Microsoft must now include a threat model. Although I doubt new hires sit and play EoP all day, I would bet that they go through the process of learning threat modeling through game play. And I think we could all learn from that.
About the Author
Dana Epp, Scorpion Software Corp's founder and CEO, researches software security and sets the vision in the convergence of information security principles and practices with digital information asset protection for small business. As a computer security software architect, Mr. Epp has spent the last 15 years focusing on computer programming with a particular emphasis on security engineering to offer a safer computing environment for business. His latest research has been on risk-based authentication, focusing on strong two-factor authentication for small business.