A Guide to Threat Modelling for Developers

How to simplify a complex problem

What are the security requirements for the software you are building? Finding a good answer is surprisingly complex. You want to prevent cyber losses over the lifetime of the system. But what are the concrete stories, acceptance criteria and technical scope that delivers that outcome? That is the puzzle addressed in this guide.

Somewhat unhelpfully, cyber specialists will often ask: 'What is your threat model?' This answer is very non-specific and uncertain, much like turning around and saying 'it depends'. Worse, 'threat model' is obscure technical jargon for most people adding unnecessary mystique. And if you research the topic of threat modelling the information can be overwhelming and hard to action. There is no agreed standard for a 'threat model' or anything like that.

So what are threat models and what is threat modelling? The core of the concept is very simple. It is about understanding causes in relation to cyber security losses. It is about using that understanding to protect your system in a risk-based way. It means starting from the potential threats in your particular case, rather than just following a checklist.

The reality of threats is that many causes combine. Cyber threats chain in unexpected, unpredictable and even chaotic ways.

Coming to understand the threat model for your system is not simple. There are an unlimited number of threats you can imagine to any system, and many of them could be likely. The reality of threats is that many causes combine. Cyber threats chain in unexpected, unpredictable and even chaotic ways. Factors to do with culture, process and technology all contribute. This complexity and uncertainty is at the root of the cyber security problem. This is why security requirements are so hard for software development teams to agree upon.

The stories behind real breaches show how complex threats and causality can be- often the details are astounding. The NotPetya story is a great example. Nation state malware was traded by a group called the "ShadowBrokers" and then weaponised. The eventual impact was major losses to organisations almost at random. Mearsk, the shipping firm, had to halt the progress of shipping. The confectioner Cadbury's had to stop making chocolate. What were their respective threat models? What development team could imagine such a complex chain of causality and collatoral damage? How long would it take your team to model this, and every other dangerous possibility?

Is threat modelling too complex to be of value? Should developers just follow a checklist, 'cross their fingers' and hope they get lucky? Skepticism can be healthy, but learning threat modelling is a key skill for developers, I believe. What we need is the right approach, and tools to tame the complexity. This guide has been written in that spirit, and begins with three ideas which make identifying good, risk-based security requirements much simpler.

Start from the technology

The first recommendation is to focus primarily on technical rather than broad threats, at least at first.

• Broad threats and threat sources include hacker groups, bad actors, disillusioned employees, human error or epidemics of new worm-like malware. These kinds of causes emerge from the world at large and are extremely various, uncertain and unpredictable. They are relative to the value of your system's data and services to your organisation and to others. These are the kinds of dramatic risks it is easy to talk about with non-technical folks.

Rabbit hole: What about nation states and 0-day?

In my experience, some developers enjoy talking about nation state adversaries, with mysterious capabilities such as 0-day exploits. Even quantum computers! Remember, thinking about risk is thinking about what is likely. Nation states are not the main risk for most systems. If you believe your system is an exception to this, I recommend escalating the threat within of your organisation to get the support and help you need with security.

• Technical threats and vulnerabilities are much more granular, such as particular weaknesses in software or missing security controls such as encryption or authorisation. These kinds of threats emerge from the structure and data-flow inherent in the system your team is building. Usually a bunch of technical threats combine together to allow a broad threat to impact your system.

By following this guide you will mainly focus on finding technical threats. This helps simplify the elaboration process, because the structure and data-flow of your system is something about which you can be certain. But it also means you can start from your existing strengths as a software developer, understanding technical stuff. This is a much stronger ground to start on than high-level risk analysis of threat sources, about which you may know little.

Don't forget about the bigger picture entirely though. A pragmatic and risk-based understanding of what broad threats are possible helps prioritise one technical threat over another. For example simple human error is usually much more likely than a nation state attack (see sidebar). That thinking can go into selecting what security scope to start examining first. When you focus on identifying technical threats first, it is then much easier to relate them back to broader threats that justify fixes and additional controls.

Take a collaborative approach

The second recommendation is to adopt a collaborative, team based approach. Identifying security requirements is not easy, and a diversity of perspectives will lead to better decision making. There will always be another vulnerability or technical threat to find, so bringing a wide variety of perspectives to the exercise makes brainstorming more robust. It also increases the likelihood you will identify the most important threats. Threat modelling in a group helps address risk holistically and helps the whole team to learn how to think and talk effectively about security.

Rabbit hole: Building the perfect threat model

Often developers seek some perfect final artifact: the definitive 'threat model'. This might take the form of a document or spreadsheet detailing the whole system alongside an exhaustive list of threats, vulnerabilities and attack trees.

The first problem with this, is that your system changes and you will never find all the threats. The second problem is that producing a separate artifact doesn't help with figuring out what steps are needed to combat the the threats you have found.

See how far you can get by following the steps in this guide, before you embark on building an elaborate and formal 'threat model'.

Getting product owners involved is a great oppotunity from a risk management perspective. Product owners have insights into user behaviour and business context that software developers simply lack. They should know about the value of particular services to the business and the impact if that data was exposed or lost. When cyber security losses occur they are business losses. If the worst does happen then the causes will likely be particular to your organisation and the technology you are using. The cyber security problem is not just about ticking technical boxes, its about making good investment decisions to protect the business.

Threat modelling 'little and often'

The third recommendation is to start threat modelling 'little and often'. Each threat modelling session with the team should be short and focussed enough to be quickly digested into something that can be delivered. Start by analysing the thinnest slice of your system possible; just what you are working on right now. Rather than trying to analyse your entire system upfront, build your team's muscle memory with threat modelling a little bit at a time.

Practises which require a completely specified software design do not match how agile teams work. There is no reason why threat modelling needs to be an exhaustive upfront analysis. Too often teams are overwhelmed by comprehensive and highly structured approaches to threat modelling[1]. I have seen teams try such approaches and run out of time and patience before any real threats were identified- let alone fixes delivered!

Rather than creating and maintaining an exhaustive 'threat model' document, do threat modelling 'little and often'. When you work this way each threat modelling session is tiny, having little impact. Yet the cumulative effect of doing them has a huge impact. When you know you'll be doing this again every iteration, there's less incentive to try to do everything at once and more incentive to prioritize the most important work right now.

Preparing to start

This section of the guide starts to make things more detailed and concrete so you can plan to start threat modelling with your team.

The three key questions

Understanding the simple structure of a threat modelling session, and doing a little bit of planning goes a long way to getting a great result.

The first thing to introduce is a simple and flexible structure for threat modelling [2]. This is based on three key questions. It helps to commit this structure to memory. You can use the three question structure as a guide whenever you need to assess threats. Like riding a bike, once you have mastered the basics you will be able to apply and grow those skills.

Activity	Question	Outcome
Explain and explore	What are you building?	A technical diagram
Brainstorm threats	What can go wrong?	A list of technical threats
Prioritise and fix	What are you going to do?	Priorised fixes added to backlog

This guide follows the three question structure. In each threat modelling session, you should spend about a third of your time answering each question. Then you will come out with a useful result. The rest of the guide will break this basic structure into more detailed steps, pointers and explanations to help you run successful threat modelling sessions.

Practical considerations

There are some things you need to get straight before you run a threat modelling session. The following pointers should help you plan.

Who should be involved?

If you are a security specialist, these sessions help you too. They provide the ideal forum to collaborate and learn about the deep technical structure of the systems the development teams are building. Also it is the perfect chance to share key insights with development teams around risk and security architecture.

Try to involve the whole delivery team in each session, which is to say both technical and non-technical roles. This brings more perspectives and ideas to the table, but also builds shared understanding. Excluding product owners, business analysts and delivery managers can mean the work to fix security flaws does not get done, as the value will not be widely understood.

You definitely do not need a security specialist to start threat modelling and discover valuable security scope. However, a threat modelling session is the perfect opportunity to collaborate with specialists, security architects or your risk management team. It will all depend on what the roles and expertise are like in your organisation.

Cadance and duration

To start, I recommmend a session length of 90 minutes. You need to give the team the time and space to learn the structure and security concepts involved. Things should get much faster once you get going, though. The most impactful threat modelling session I have ever participated in took less than 15 minutes. Short and snappy sessions are possible once everyone in the team has built 'muscle memory' with the practise.

I am often asked how frequent threat modelling sessions should be. I do not think there is any right answer, it depends on your team. I think of threat modelling just like any other team design session. I would not be so rigid as saying it has to be every single week. However, I have worked with many teams with a risk profile that would justify threat modelling every sprint. At the other extreme if it has been a few sprints without any threat modelling, the practise is clearly not continuous enough to be considered mature.

Running sessions face-to-face vs. running remotely

Here's what you need to prepare for a face-to-face session:

• Whiteboard or flipchart
• Coloured pens for diagrams
• Sticky notes for everyone
• Sharpie pens for everyone

If you are running remotely, in addition to a video conferencing tool, find a collaboration tool which offers the following:

• Collaborative editing
• Create, modify and delete cards/labels
• Draw block diagrams
• Use different colours
• Allow participants to vote

A face-to-face threat modelling session could happen in a meeting room, or more informally in the team's normal work area- if you have space. A typical session involves drawing a diagram to explain and explore the scope, brainstorming threats, then prioritising fixes for the backlog. However, a face-to-face session is not always possible.

When you run a session remotely you just need to plan a little differently so everyone can participate virtually. You will need video conferencing and collaboration tools. Agree and setup these tools ahead of time. Teams at ThoughtWorks have had success with a variety of tools, including Mural, Miro, Google Jamboard and Google Docs.

Get accustomed to your tools ahead of the session and get participants to test they have access. Whichever tool you choose, ensure you have security approval from your organisation to use the tool. Threat modelling outputs represent sensitive information for a number of reasons and must be protected.

Here are some more pointers to bear in mind when working remotely:

• It can help to create diagrams asyncronously before the exercise. This is because it can consumes a good amount of time to draw diagrams on virtual boards.
• Pay even more attention to creating a common understanding of the concepts and symbols you are using to illustrate the system. Explain diagram symbols, data flow arrows and colours of digital stickies.
• Be more intentional to ensure everyone is engaged in the exercise. Perhaps use some security related brain teasers as an ice-breaker. Refer to broader guides on remote facilitation.
• If you have a large group of people, it may make sense to create smaller groups and then consolidate the output. A couple of small sessions is better and more sustainable than one big session.
• You will need more breaks than required in a face-to-face session. Remote work is tiring.

Regardless if your session is remote or face-to-face, you should aim to finish on time. And with some concrete outcomes! This needs discipline - facilitating timings could be a role taken by a delivery manager or someone experienced making sure workshop sessions succeed.

Mona Fenzl and Sarah Schmid from ThoughtWorks Germany have had some success using a collaboration tool called Mural. They used it to create a threat modelling template to help other teams get started with that tool.

Scoping the session

Deciding the right focus and level of detail for your session is called 'identifying the scope'. Make sure you have decided this ahead of getting people together to perform the activity. Be guided by what has most value right now. Perhaps its simply the user stories you are working on this iteration?

Do not try and bite off too much scope at once! If you try threat modelling the entire system at once, either you will make no findings in the time available or you will overrun dramatically and there will be no appetite or budget to do it again. It is much better to timebox threat modelling into manageable chunks, performing the activity 'little and often'.

Here are some examples of scopes which have worked well:

• Scope in current iteration.
• An upcoming security sensitive feature, such as a new user registration flow.
• The continuous delivery pipeline and delivery infrastructure.
• A particular microservice and its collaborating services
• A high level overview of a system to identify security tech debt.

Whatever scope your team chooses, make sure it is not too big for you to cover in the time available.

A worked example: scoping

The rest of the guide uses a real feature to show the concrete steps involved in threat modelling. There is a development team at a retail organisation which is building a platform to sell groceries for home delivery. Here is the epic they have in the upcoming sprint:

As a customer, I need a page where I can see my customer details, So that I can confirm they are correct

If you have ever used an online store, you will be able to imagine a page which is used to update address details and perhaps view a loyalty card balance.

From experience, a feature of this size is a pretty reasonable scope for threat modelling session.

from Hacker News https://ift.tt/3e33Q65