> ## Content Index
> Fetch the complete content index at: https://thenexusofprivacy.net/llms.txt
> Use this file to discover other available public pages before exploring further.

# "Social threat modeling"
- URL: https://thenexusofprivacy.net/social-threat-modeling/
- Published: 2018-05-11T00:00:00.000Z
- Updated: 2026-04-15T22:38:31.000Z
- Description: Structured approaches to looking at threats -- not just technical shortcomings..
- Author: Jon
- Tags: social threat modeling, #Import 2026-05-21 02:50, #Import 2026-09-13 19:16

[](https://medium.com/plans?dimension=post%5Faudio%5Fbutton&postId=8dc330479a50&source=upgrade%5Fmembership---post%5Faudio%5Fbutton-----------------------------------------)

[Threat modeling](https://www.owasp.org/index.php/Category:Threat%5FModeling?ref=thenexusofprivacy.net) is a structured approach to looking at security threats — and what can be done in response. EFF’s [Assessing Your Risks](https://ssd.eff.org/en/module/assessing-your-risks?ref=thenexusofprivacy.net) describes how people wanting to keep their data safe online can do threat modeling, starting with questions like “what do I want to protect?” and “who do I want to protect it from?” Threat modeling is also an important software engineering technique, and it’s that aspect I’m going to focus on here.

When a company takes threat modeling seriously as part of an overall security development process, it can have a huge impact — the comments on Dan Kaminkski’s [Summer of Worms](https://www.facebook.com/dan.kaminsky.12/posts/10157344270504692) Facebook thread highlight how this played out at Microsoft in the early 2000s. Things have come a long way since then. Today there are books, checklists, tutorials, tools, and even games about how to do it well (although there are still plenty of companies who prefer to ignore the risks).

Even for companies that do practice it, threat modeling today generally has a rather selective focus. As [Amanda Levendowski](https://medium.com/u/190efea63f27?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) points out in [Conflict Modeling](https://www.levendowski.net/conflict-modeling/?ref=thenexusofprivacy.net):

> In the [security](https://www.amazon.com/Threat-Modeling-Designing-Adam-Shostack/dp/1118809998?ref=thenexusofprivacy.net) and [privacy](https://www.esat.kuleuven.be/cosic/publications/article-1412.pdf?ref=thenexusofprivacy.net) contexts, threat modeling developed as a predictable methodology to recognize and analyze technical shortcomings of software systems. And when compared with security and privacy threat modeling, systems have lagged in developing similarly consistent, robust approaches to online conflict.

Indeed. The Open Web Application Security Project’s [Application Threat Modeling page](https://www.owasp.org/index.php/Application%5FThreat%5FModeling?ref=thenexusofprivacy.net) discusses things like decomposing the application into components, identify the data that need to be protected, and focusing on trust boundaries between running processes. It doesn’t have much at all to say about the people who are in the system. There’s similarly no mention of important categories of other social and user harms like online conflict, harassment, computational propaganda, and [influencing elections](https://medium.com/a-change-is-coming/facebook-is-fucked-locking-the-barn-door-after-the-elections-were-stolen-a5541d5d658f?ref=thenexusofprivacy.net).

![At the top, a box labeled "Harass person".  Below, "Trigger them", "Images" and "Threats"; "Flood them with messages"; "Make them look bad" ](https://thenexusofprivacy.net/content/images/2025/03/simplified-threat-model.png)

**Simplified social threat model with different approaches to harassment*

To be clear: this isn’t a fundamental limitation of threat modeling, it’s just what’s been emphasized to date. And although the work’s still at a relatively early stage, several people are working on extending threat modeling or similar techniques to these social threats.

There isn’t yet a good name for this overall approach. I’m calling it “social threat modeling” for now, but as [Shireen Mitchell](https://medium.com/u/4b19e9d59e5e?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) of Stop Online Violence Against Women points out that’s only one aspect of it. It starts with people (not the technology), and involves looking at things from the perspective of different identities.

Whatever you call it, though, there’s a steadily-growing body of promising work here. A few examples:\*

- Susan Herring et. al.’s [Searching for Safety Online: Managing “Trolling” in a Feminist Forum](https://www.tandfonline.com/doi/abs/10.1080/01972240290108186?ref=thenexusofprivacy.net), in *The information society* (2002), analyzes the strategies that made a troller successful and the targeted group largely ineffectual in responding to his attack, as a means to understand how such behavior might be minimized and managed in general. Frances Shaw’s [Still ‘Searching for Safety Online’: collective strategies and discursive resistance to trolling and harassment in a feminist network,](http://twentytwo.fibreculturejournal.org/fcj-157-still-searching-for-safety-online-collective-strategies-and-discursive-resistance-to-trolling-and-harassment-in-a-feminist-network/?ref=thenexusofprivacy.net) in *The Fibreculture Journal,* from 2013, looks at similar dynamics in a network of blogs.
- Robert Meyer and Michel Cukier’s [Assessing the Attack Threat due to IRC Channels](https://ieeexplore.ieee.org/abstract/document/1633535/?ref=thenexusofprivacy.net), in *Dependable Systems and Networks*, 2006, uses a combination of bots and regular users in IRC chat, and the social structure of IRC channels, to investigate
- Borja Sanz et al.’s [A threat model approach to attacks and countermeasures in on-line social networks](http://paginaspersonales.deusto.es/claorden/publications/2010/sanz%5FRECSI10%5FA%20Threat%20Model%20Approach%20to%20Attacks%20and%20Countermeasures%20in%20OSN.pdf?ref=thenexusofprivacy.net), , in *Proceedings of the 11th Reunion Espanola de Criptografıa y Seguridad de la Información (RECSI),* focuses on identifying attacks against users of online social networks and possible countermeasures to mitigate the risks
- Leigh Honeywell’s [Another Six Weeks: Muting vs. Blocking and the Wolf Whistles of the Internet](https://modelviewculture.com/pieces/another-six-weeks-muting-vs-blocking-and-the-wolf-whistles-of-the-internet?ref=thenexusofprivacy.net) on *Model View Culture* analyzes different kinds of attackers and their motiviations in the context of somebadly-thought-out functionality. “In attempting to solve the problem of users being retaliated against for blocking, Twitter missed other ways that harassers operate on their service.”
- Mozilla’s Coral Project [applies a threat modeling perspective to online communities](http://guides.coralproject.net/threat-modeling-for-communities/?ref=thenexusofprivacy.net). [caroline sinders](https://medium.com/u/7bdeb357cfaa?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) of the Coral project briefly talks about threat modeling’s application to harassment in [SXSW canceled panels: Here is what happened](http://www.slate.com/articles/double%5Fx/doublex/2015/10/sxsw%5Fcanceled%5Fpanels%5Fhere%5Fis%5Fwhat%5Fhappened.html?ref=thenexusofprivacy.net), from 2016
- [Amanda Levendowski](https://medium.com/u/190efea63f27?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) describes Conflict Modeling as “a predictable framework to structure thinking around online conflict by suggesting a methodology for conflict modeling, defining a taxonomy of conflict — safety, comfort, usability, legal, privacy, and transparency (SCULPT) — and examining common mitigation techniques adopted by systems to reduce the risk of certain conflicts.” A draft was presented at the [2017 Privacy Law Scholars Conference,](https://www.law.berkeley.edu/research/bclt/upcoming-events/june-2017-10th-annual-privacy-law-scholars-conference-plsc/agenda-plsc-2017/?ref=thenexusofprivacy.net); as far as I know, the only public information is [on her web site](https://www.levendowski.net/conflict-modeling/?ref=thenexusofprivacy.net)
- [Shireen Mitchell](https://medium.com/u/4b19e9d59e5e?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) and I suggested applying threat modeling techniques to online harassment in our 2017 SXSW talk on [Diversity-friendly Software.](https://medium.com/a-change-is-coming/diversity-friendly-software-at-sxsw-2017-references-c0ca05a191a6?ref=thenexusofprivacy.net) I went into a little more detail in [Transforming Tech with Diversity-Friendly Software](https://medium.com/a-change-is-coming/transforming-tech-with-diversity-friendly-software-338f56d91df?ref=thenexusofprivacy.net) ([the slides have a short example](https://docs.google.com/presentation/d/1JB3bTbJvjEypKlPu1JKV20Oz9YlF5zRCl3vLIPdDTrA/edit?ref=thenexusofprivacy.net#slide=id.g2073602466%5F0%5F67)) and worked with [Kelly Ireland](https://medium.com/u/5787059db75?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) at [O.school](https://o.school/?ref=thenexusofprivacy.net) applying this approach to their pleasure education platform; Shireen is working with [Kaliya-IdentityWoman](https://medium.com/u/6991e9b5ea5f?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) on applying a generalized threat modeling approach to social and user harms in the self-sovereign ID world.
- Casey Fiesler’s “[speculative harm analysis](https://twitter.com/cfiesler/status/1273604933007192065?ref=thenexusofprivacy.net)” of Twitter’s audio tweets illustrates how just a few minutes of analysis can identify major opportunities for abuse in a new product feature.

One interesting aspect to this work is that it’s largely being presented outside of the mainstream security and software engineering world. In the more traditional “tech space”, it’s striking how little attention is getting paid to this issue. Twitter, Facebook, and Google spend zillions of dollars a year (and publish bunches of research papers) on AI; how much have they invested here? And the red-hot blockchain world has no shortage of money either, and a golden chance to get things right from much earlier on, but other than Kaliya, very few of the people I talked to at the recent Internet Identity Workshop were even thinking about this kind of approach.

And the results speak for themselves. [Twitter is toxic](https://www.amnesty.org/en/latest/research/2018/03/online-violence-against-women-chapter-1/?ref=thenexusofprivacy.net); their [latest attempt](https://slate.com/technology/2018/05/twitter-will-start-hiding-tweets-that-detract-from-the-conversation.html?ref=thenexusofprivacy.net) to deal with it is likely to fail as miserably as [all their previous ones.](https://www.buzzfeed.com/charliewarzel/a-honeypot-for-assholes-inside-twitters-10-year-failure-to-s?utm%5Fterm=.whYVrd3gK&ref=thenexusofprivacy.net#.mfN1wZeBr) Facebook is frantically trying to [close the barn door after the elections were stolen](https://medium.com/a-change-is-coming/facebook-is-fucked-locking-the-barn-door-after-the-elections-were-stolen-a5541d5d658f?ref=thenexusofprivacy.net) by making[ wildly implausible claims about how they’ll use AI to fix everything](https://www.wired.com/story/how-artificial-intelligence-canand-cantfix-facebook/?ref=thenexusofprivacy.net) in 5–10 years. As [Safiya Umoja Noble, Ph.D.](https://medium.com/u/b2173709989d?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) summed it up at the Data Society conference, “[if you’re designing technology for society, and you don’t know anything about society, you’re unqualified.](https://twitter.com/beccalew/status/996521144919429120?ref=thenexusofprivacy.net)”

Still, the winds of change are in the air. The UN is discussing [Facebook’s role in genocides](https://www.reuters.com/article/us-myanmar-rohingya-facebook/u-n-investigators-cite-facebook-role-in-myanmar-crisis-idUSKCN1GO2PN?ref=thenexusofprivacy.net), Amnesty International is reporting on [Toxic Twitter](https://www.amnesty.org/en/latest/research/2018/03/online-violence-against-women-chapter-1/?ref=thenexusofprivacy.net), and [Safiya Umoja Noble, Ph.D.](https://medium.com/u/b2173709989d?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------)’s outstanding [*Algorithms of Oppression*](https://nyupress.org/books/9781479837243/?ref=thenexusofprivacy.net) is getting excerpted [in *Time Magazine*](http://time.com/5209144/google-search-engine-algorithm-bias-racism/?ref=thenexusofprivacy.net)*.* More and more people are seeing [computer science as a social science,](http://www.achangeiscoming.net/docs/cssocsci.html?ref=thenexusofprivacy.net) and coming around to a point that [Zeynep Tufekci](https://medium.com/u/a5b491a8b18c?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------), [AnthroPunk, Ph.D.](https://medium.com/u/4a0c9dd58982?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------), and others have been making for quite a while: software companies need to get anthropologists, sociologists and other social scientists involved in the process. As [Window Snyder](https://medium.com/u/b9f75a7d027a?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) (co-author of a 2004 book on threat modeling and now chief security officer at Fastly) said at the recent OurSA conference, [“the industry changes when we change it.”](https://www.wired.com/story/oursa-security-conference-calls-out-lack-of-inclusion/?ref=thenexusofprivacy.net)

So I expect we’ll be seeing a lot more attention to this area over the next few months. It’ll be interesting to see which companies gets ahead of the curve.

\* If there’s other work that should be in this list, please let me know!

## Image credits

- Microsoft DREAD risk-ranking model from [OWASP’s Application Threat Modeling page](https://www.owasp.org/index.php/Application%5FThreat%5FModeling?ref=thenexusofprivacy.net)
- Simplified threat model for harassment originally by Shireen Mitchell and me on a napkin, refined with help from Kelly Ireland and presented at [Transforming Tech with Diversity-friendly software](https://medium.com/a-change-is-coming/transforming-tech-with-diversity-friendly-software-338f56d91df?ref=thenexusofprivacy.net)

## Update Log

May 15, 2018: clarifying that it’s anthropologists, sociologists, and other scientists who need to be involved  
May 17: some additional references and minor rephrasing.  
August 8: changed title, included paragraph on the name  
June 2020: added Casey Fiesler’s analysis of audio tweets  
March, 2025: cut-and-paste to The Nexus of Pricacy

*Originally published at* [*A Change Is Coming*](http://achangeiscoming.net/2018/05/11/social-threat-modeling/?ref=thenexusofprivacy.net) *with the overly-optimistic title* [*Social Threat Modeling: the Winds of Change are in the Air*](https://medium.com/a-change-is-coming/social-threat-modeling-the-winds-of-change-are-in-the-air-8dc330479a50?ref=thenexusofprivacy.net)*.. Thanks to* [*AnthroPunk, Ph.D.*](https://medium.com/u/4a0c9dd58982?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) *,* [*Shireen Mitchell,*](https://medium.com/u/4b19e9d59e5e?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------)[*Safiya Umoja Noble, Ph.D.,*](https://medium.com/u/b2173709989d?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) *and* [*caroline sinders*](https://medium.com/u/7bdeb357cfaa?source=post%5Fpage---user%5Fmention--8dc330479a50---------------------------------------) *for* [*feedback on an earlier version*](https://twitter.com/digitalsista/status/996859628179816448?ref=thenexusofprivacy.net)*.*