Designing and Applying an ‘Extremist Media Index’

by Donald Holbrook


Even though notions of extremism are subjective and relative, we can still be systematic in the way we approach them. The purpose of this paper is to illustrate how such a systematic mechanism can be compiled and visualised. I propose a set of definitions, in an ‘Extremist Media Index’, to grade ideological media material conveying religious-political sentiments according to some very basic criteria concerning stages of activism and the nature of religio-political discourse. I test the reliability of these criteria and apply the Index to publications by Anwar Al-Awlaki and associated with far-right extremism in order to display the range of sentiments conveyed and how these can be understood through systematic grading.

Keywords: Extremism, Media, Ideology, Islamism, Right-wing, Methodology


Counterterrorism is increasingly being framed as a battle of ideas. There are ideational components, it is argued, that underpin both terrorist organisations and processes of violent radicalisation. In early 2015, the British Prime Minister described efforts to tackle ‘extremist ideology’ as the ‘struggle of our generation’. The Home Secretary expressed the government’s determination to challenge ‘the impact that some people have in terms of the poisonous ideology that they are trying to implant in people’s minds’ [1]. In the United States, President Obama emphasised the need to ‘confront squarely and honestly the twisted ideologies that these terrorist groups use to incite people to violence’ [2].

These statements reflect a desire to weaken and undermine normative dimensions of violent political behaviour and target ideas that politicians and policymakers see as contributing to extremist behaviour. But what is ‘extremist ideology’? What does it look like and how do we define it, process it and code it? [3]

Setting aside complex—but important—questions about the role of ideology in radicalisation [4] or indeed the ‘essence’ of ideology itself [5], this Research Note explores ways in which the coding and grading of media content seen to convey extremist sentiment can be systematised and standardised. This, to reiterate, is not to assume that extremist ideology–or indeed non-extremist ideology–is a necessary or even central precursor to participation in terrorist activity [6]. The purpose here is merely to look at different ‘gradients’ of content.

The Research Note builds upon existing work with colleagues looking at tiers, expressions and variations of discourses associated with terrorism and political violence [7] and the aim here is to develop this thinking towards more systematic ways in which to approach such a body of content.

The focus is on ideology as beliefs and principles that guide political behaviour and define the goals and rationales of individual or collective political action [8]. The author is particularly interested in frames as interpretive schemata aimed at structuring political protest on the basis of fundamental ideas and beliefs [9] and how these are conveyed through different media [10].

Relativity and Subjectivity

Before setting out the proposed grading schema, we first need to revisit the central question, ‘what is extremist ideology’? ‘Extreme’ ideas are by definition relative. They allude to a set of more moderate beliefs and principles that diverge from a particular abstract centre or ‘mainstream’. But these frames of reference are not stable.

Central themes of what today is seen as right-wing extremism, such as anti-Semitism, white supremacism, racism and xenophobia were once broadly accepted by authors, composers, scientists, industrialists, politicians, academics and other members of a European or North American elites and intelligentsia [11]. More recently, issues such as gay rights and full gender equality have come to the fore with both tolerance and intolerance constituting a broadly-accepted mainstream, depending on cultural and political realities on the ground.

As Alan Fiske and Tage Rai noted in their construction of ‘virtual violence theory’, notions of just violence ‘that in one culture or in one historical epoch are moral may be immoral in another culture or in another historical epoch’ [12]. Author Orhan Pamuk, for instance, reflects how people of the ‘East’ would celebrate the conquest of Istanbul in 1453 whilst those from the ‘West’ lamented the fall of Constantinople [13].

The answer to ‘what is extremist ideology?’, therefore, is geographically, temporally and culturally relative and subjective. To appropriate Miles’s edict on bureaucratic biases: where you stand on the issue depends on where you sit.

Systematic Analysis

Yet, even though notions of extremism are subjective and relative, we can still be systematic in the way we approach them. The purpose of this Research Note is to illustrate how such a systematic mechanism can be compiled. The author proposes a set of definitions to grade ideological media material conveying religious-political sentiments according to some very basic criteria concerning stages of activism and the nature of religio-political discourse. This is, in part, inspired by Max Taylor’s work on COPINE [14], which categorised the severity of child sex abuse images. Ideological content is obviously profoundly different and more ambiguous, but perhaps one can arrive at some form of tiered conceptualisation that draws out distinctions in terms of discourse and, in particular, highlights references to violence, especially lethal violence, in pursuit of religio-political causes and the divergent ways in which such sentiments are framed.

At this point, it is worth stressing another caveat. We are not concerned with prescriptions, about what should or should not be said or articulated or what is or ought to be legal or illegal [15]. The focus is only on conceptual yardsticks: social scientific ways of organising discourse and language into categories in a systematic way.

Why might grading be important? Firstly, ‘terrorist’ statements and communiques from individuals clearly associated with terrorist movements differ more than one intuitively expects. Usama bin Ladin, for example, said nothing about violent activism in his early Open Letters, before endorsing violence publically against, initially, members of the armed forces and then combatants and civilians, before issuing much milder messages about the environment and global aid just before his death. If we want to understand such nuances we need to develop systematic, testable and repeatable tools of measurement.

Secondly, standardised and systematic grading may offer ways in which to compare discourses from different extremist cohorts, such as the far-right and Islamist-inspired extremism. Thirdly, whilst the role of ideology in ‘radicalisation’–a problematic and contested topic as note above–is beyond the scope of this Research Note, systematised grading offers opportunities to explore relationships between mere consumption of ideological content on the one hand and active engagement in terrorism on the other. At least it provides parameters for comparative analyses and structured case studies.

The goal here is, therefore, to develop a relatively approachable and testable framework that is applicable irrespective of subject or particular political/ideological disposition.

Framing and ‘Extremist’ Political Protest

Delineating different tiers of content with respect to political protest relates directly to the ways in which events, issues and realities are framed in order to mobilise support and encourage particular action. Robert Benford and David Snow defined action-orientated sets of beliefs and meanings that were intended to inspire and legitimate activities and campaigns of social movement organisations as ‘collective action frames’. These were divided into three core communicative tasks: diagnostic frames describing problems that need to be addressed, prognostic frames as solutions to these problems, and motivational frames providing activist indicators that are designed to turn observers into participants [16].

A funnel offers a helpful analogy to picture how this fits within the context of political violence: many people might agree that particular grievances need to be addressed, a smaller proportion might agree with particular alternatives or solutions offered and a much smaller proportion still accepts that violence could be used to address those grievances. An even smaller group, motivated by a host of factors, would then actually seek to participate or support terrorist violence to redress that grievance.

We can use the same funnel analogy to look at ideological discourses and political activism in this context where some similar tiers emerge (Figure 1): we have, very broadly speaking, (1) moderate material conveying ideological/political/religious content without advocating violence; (2) radical political content that is hostile, confrontational or isolationist; (3) material glorifying violence and perhaps dehumanising ‘enemy’ people and then different scales in terms of scope of violence and targeting and level of facilitating detail that is offered.

Developing this categorisation further, systematising with numerical indicators, we arrive at a more structured assessment of content termed, for sake of identification, the ‘Extremist Media Index’ (EMI).

Extremist Media Index

The system of grading devised for such an Index consists of two phases. First, titles or texts are coded according to a set of definitions that are intended to describe notionally ‘moderate’, ‘fringe’ and ‘extreme’ content. Second, the ‘extreme’ titles are coded in more detail to draw out ways in which violence is endorsed, the level of facilitating detail provided as well as the scope of proposed action. Does the author of the title (be that a video recording, audio speech, pamphlet or book, for example) clearly endorse violent tactics in pursuit of political aims in contemporary scenarios? If so, does s/he advocate targeted violence against soldiers only, or against non-combatants as well? If violence against non-combatants is celebrated or endorsed, does the author provide any details that would help a consumer of the particular title to carry out the prescribed act? [17]. The process of grading is described in Figure 2 and a more detailed set of definitions is provided in Appendix 1.

Testing and Reliability

The first task before applying the Index to a relevant batch of content is to test reliability. Complex ideological discourses, as noted, are hard to subject to rigid content analyses with a substantial amount of agreement between coders [18]. This is a subjective realm, as discussed above, not an exercise in measuring any specific message or language characteristics and although the boundaries above may seem intuitive to many, they can be hard to identify. Anything approaching ‘strong’ agreement would therefore be impossible.

Nevertheless, to get a sense of inter-coder reliability, the author assembled a research team consisting of himself and two other researchers. Three coders thus graded the sample batch blindly and reliability scores were calculated using established reliability tests [19].

First, we tested 145 titles involving known activist and militant leaders, groups and publishers as well as a random selection of material conveying religio-political content and distributed online to test the range of the definitions. The focus was on the Islamist and far-right dispositions.

Second, we graded 32 titles that were rated as ‘extreme’ in the first phase according to the tiers defined for the second phase of grading as noted in Figure 2, above.

The coding results from the first phase were fed into Deen Freelon’s ReCal online reliability calculator [20]. The results for the first phase of grading revealed a Fleiss’ Kappa of 0.665, Cohen’s Kappa of 0.669 and a Krippendorff’s Alpha of 0.666. Whilst hovering around the limit of acceptable agreement for Krippendorff’s more stringent Alpha, the results suggest an intermediate to good result from Fleiss and Cohen [21]. Agreement for two coders was understandably higher with Krippendorff’s Alpha of 0.69.

The coding results for the smaller cohort of ‘extreme’ texts in the second phase of grading were weaker. For three coders, Fleiss was 0.586, Cohen was 0.593 and Krippendorff was 0.59. For the first two coders, however, agreement was significantly higher with Cohen’s Kappa of 0.766 and Krippendorff’s Alpha of 0.796.

These inter-coder tests emphasise the challenges of establishing agreement between coders on such a subjective dataset and with such broad categories but nonetheless illustrate how tests designed for more structured content analyses can be applied to such a body of content. The coding can provide at least an indicator of content whilst highlighting the subjective nature of the process.

Applying the Index

To apply the Index to a more relevant sample of media output, I selected a number of titles authored by Anwar al-Awlaki (a famous Islamist ideologue who became involved with Al-Qaeda in the Arabian Peninsula) as well as a cohort of prominent white supremacists.

The purpose here is to illustrate how the Index can work in practice: by applying a theoretically grounded and empirically tested list of criteria to classify particular ideological content and visualise output according to the nature of the arguments conveyed.

We selected a collection of 118 Anwar al-Awlaki lectures and essays from Archive.org which had been downloaded over 80,000 times, according to the website, but also exist as bitTorrent files. We chose white supremacist publications from both the UK [Combat 18, Blood and Honour and White Nationalist Party/British People’s Party] and the US [David Lane, William Pierce, Creativity Movement, American Nazi Party]. All have publicly been associated with extremism and serve as an illustrative example of how the Index can be applied.

Together with the research assistants, the author coded the 118 Awlaki titles and 34 white supremacist titles, first, according to the initial phase of grading (‘moderate’, ‘fringe’ and ‘extreme’). We then re-coded the ‘extreme’ titles according to second phase of grading.

The results for the white supremacist titles are presented in figure 3.

Much of the white supremacist literature in the UK is cautious, partly because legal restrictions are known. Therefore, there is a tendency to focus on Aryan supremacy in an isolationist discourse, without endorsing violence or explicitly dehumanising others. Those texts that do endorse violence do so in a very ambiguous way. However, in the US scene we see authors going further. Ben Klassen of the World Church of the Creator, for example, endorsed all-out race wars against Jews and non-whites, without giving any details, which amounts to Extreme level 2 on the Index. Figure 3 has two peaks caused by the William Pierce novels Hunter and Turner Diaries, where the author, a physics professor, talks at length about devices, targets and attack scenarios, bringing it to Extreme level 3 on the Index [22].

Figure 4 shows grading results for 118 individual titles that Anwar al-Awlaki authored and disseminated over the years, both in audio lectures and essays.

The majority of Awlaki’s output is not extreme at all, but falls either in the moderate or fringe categories. These are titles that do not endorse violence in any way, being mostly religious tracts, but in many cases also isolationist or hostile towards non-Salafis, such as Sufis, and Shi’a and non-Muslims. There is also variation in his extremist output: plenty of titles legitimise mass-casualty violence but few offer details on how to bring this about. Most of this extremist output obviously came after Awlaki was released from prison in Yemen in 2007 and solidified his links with terrorism more openly, including with AQAP. But there are curious examples of earlier references to extremist ideas. In one part of an 18-CD audio lecture series on the Life of the Prophet Mohammed, published in 2000, for example, Awlaki drifted from the main topic of the otherwise moderate series to present justifications for suicide bombings against civilians in contemporary scenarios, which would be graded as Extreme level 3.

Variation, therefore, emerges in discourses associated with extremism, which is made explicit by the Index.

Conclusions and Limitations

It is necessary to end this Research Note by re-emphasising some caveats in relation to the Index. Again, this is a subjective process: It is inherently difficult to define objective parameters for something as value-laden as highly political ideological discourse. There are grey areas between categories and some boundaries are difficult to define, particularly between vague fringe material and vague level 1 extreme material therefore a degree of overlap between categories should be assumed. The red lines are not always that easy to spot. This is also a descriptive exercise: a method of organising discourse according to a pre-defined set of criteria, not an assessment concerning acceptability of particular language or disposition.

Nevertheless, the Index offers meaningful shortcuts to assessing content that are qualitative, theoretically grounded, empirically tested and repeatable. In designing such an Index, therefore, one ought to strike a balance between simplicity, practicality and accessibility for different coders on the one hand and comprehensiveness and something that can offer meaningful descriptions of different sets of discourses on the other.

About the Author: Donald Holbrook is a Senior Research Fellow at the Centre for the Study of Terrorism and Political Violence, University of St Andrews. Contact: [email protected] .


[1] Frances Perraudin (2015) ‘David Cameron: extremist ideology is “struggle of our generation”, in: The Guardian (20 July); URL: <http://www.theguardian.com/politics/2015/jul/20/extremist-ideology-struggle-generation-david-cameron > ;accessed September 2015 (emphasis added DH).

[2] The White House (2015) ‘Remarks by the President in Closing of the Summit on Countering Violent Extremism’; URL: <https://www.whitehouse.gov/the-press-office/2015/02/18/remarks-president-closing-summit-countering-violent-extremism> ;accessed September 2015 (emphasis added, DH).

[3] The British Government’s own exploration of definitions—and inclusion of ‘non-violent extremism’ has been controversial. See The Economist (2015) ‘A new front in the war on Islamist extremism David Cameron stretches the definition of “extreme” ideologies’ (20th July, online edition); Marsden, S. (2015) ‘Cameron’s counter extremism plan rests on shaky foundations’; URL: <http://blogs.lse.ac.uk/politicsandpolicy/camerons-counter-extremism-plan-rests-on-shaky-foundations/> ; accessed September 2015.

[4] E.g. Borum, R. (2011) ‘Radicalization into Violent Extremism I: A Review of Social Science Theories’ in Journal of Strategic Security ( Vol.4, No. 4, Winter, pp. 7–36).

[5] E.g. Freeden, M. (Ed.) (2007) The Meaning of Ideology: Cross-Disciplinary Perspectives, Abingdon: Routledge; Leader Maynard, J. (2014) ‘Rethinking the Role of Ideology in Mass Atrocities’, in: Terrorism and Political Violence (Vol. 26, pp. 821–841).

[6] Borum, R. (2011); Githens-Mazer, J. & Lambert, R. (2010) ‘Why conventional wisdom on radicalization fails: the persistence of a failed discourse’, International Affairs (86: 4, pp. 889–901).

[7] Holbrook, D & Taylor, M. (2014) ‘Developing Grading Processes for Ideological Content’, in: Journal of Policing, Intelligence and Counter Terrorism ( Vol. 9, No.1, pp. 32–47); Holbrook, D; Ramsay, G and Taylor, M. (2013). “Terroristic Content”–Towards a Grading Scale’, in Terrorism and Political Violence (Vol. 25, pp. 202–223); Ramsay, G. & Holbrook, D. (2015) ‘The representation of violence by insurgent political actors: the “violent” part of “violent extremism”?’, in Behavioral Sciences of Terrorism and Political Aggression ( Vol.7, No. 1, pp. 84–96). ‘Associated’ refers to material issued by, or in endorsement of, individuals and organisations linked to terrorism or conveying content that could be perceived as ‘terroristic’ in relation to the types of activism prescribed.

[8] Hegghammer, T. (2009) ‘The Ideological Hybridization of Jihadist Groups’, in: Current Trends in Islamist Ideology .Vol 9, Hudson Institute; Snow, D. & Scott B. (2007) ‘Ideology, Framing Processes, and Islamic Terrorist Movements’ in Mobilization (Vol. 12, No. 2 June).

[9] Wiktorowicz, Q. (2004) Introduction in Q. Wiktorowicz (Ed.) Islamic Activism: A Social Movement Theory Approach. Indiana University Press; Snow, D. & Robert Benford (1988) ‘Ideology, Frame Resonance, and Participant Mobilization’ in International Social Movement Research (Vol.1 pp. 197–217) JAI Press Inc; Gregg, H. S. (2010) ‘Fighting the Jihad of the Pen: Countering Revolutionary Islam’s Ideology’, in Terrorism and Political Violence (Vol. 22, No. 2, pp. 292–314); Tololyan, K. (2006) ‘Cultural Narrative and the Motivation of the Terrorist’, in: David Rapoport (Ed.) Critical Concepts in Political Science: Terrorism Volume IV: The Fourth or Religious Wave. Abingdon: Routledge.

[10] Paul Wilkinson’s (2006) definition is helpful: “Media is a generic term meaning all the methods or channels of information and entertainment. The mass media are taken to encompass newspapers, radio and television and other important forms of communications, including books, films, music, theatre and the visual arts. The late twentieth century has seen the globalisation of the mass media culture, but we should not overlook the fact that throughout history informal methods of communication such as the gossip of the taverns, streets and marketplace have been the standard local media for transmitting information, and these informal channels coexist with all the latest multimedia technology in contemporary societies.”–Terrorism versus Democracy: The Liberal State Response, Second Edition. London; Frank Cass, p. 144.

[11] Levitas, D. (2002) The Terrorist Next Door: The Militia Movement and the Radical Right, Thomas Dunne Books: NY, pp. 85–88.

[12] Fiske, A.P. & Rai, R. S. (2015) Virtuous Violence: Hurting and Killing to Create, Sustain, End and Honor Social Relationships, Cambridge: CUP, p. 141.

[13] Pamuk, O. (2005) Istanbul: Memories and the City, London: Faber & Faber, p.156 (emphasis added, DH)..

[14] Holbrook, D; Ramsay, G and Taylor, M. (2013).

[15] Even though plenty of examples exist, not just within the realm of terrorism legislation in the UK, but much more broadly in terms of laws against inciting racial or religious hatred or fomenting discrimination.

[16] Benford, R. D. & Snow, D. A. (2000) ‘Framing Processes and Social Movements: an Overview and Assessment’ in Annual Review of Sociology (Vol. 26, pp. 611-639): p. 614.

[17] Gilbert Ramsay’s definition of ‘enabling dimensions’ is relevant here,; see our paper Holbrook, D.; Ramsay, G. and Taylor, M. (2013). Many scholars have discerned variations of ‘extremities’; see esp. Hegghammer, T. (2010) Jihad in Saudi Arabia: Violence and Pan-Islamism since 1979 .New York: Cambridge University Press, on different violent Islamist groups (p. 6).

[18] ‘Agreement’, as Krippendorff (2004) notes, ‘is what we measure; reliability is what we wish to infer from it”, Content Analysis, an Introduction to Its Methodology (2nd ed) Thousand Oaks, CA: Sage Publications: 215.

[19] Krippendorff’s Alpha, see Krippendorff, K. (2011) ‘Computing Krippendorff ‘s Alpha-Reliability’; URL: <http://repository.upenn.edu/asc_papers/43> ; accessed September 2015; Krippendorff (2004) Content Analysis, an Introduction to Its Methodology (2nd ed) Thousand Oaks, CA: Sage Publications; Fleiss’ Kappa and Cohen’s Kappa, see Joyce, M. (2012) ‘Picking the Best Intercoder Reliability Statistic for Your Digital Activism Content Analysis’; URL: <http://digital-activism.org/2013/05/picking-the-best-intercoder-reliability-statistic-for-your-digital-activism-content-analysis/> ; URL:accessed September 2015.

[20] Freelon, D. (2010). ReCal: Intercoder reliability calculation as a web service. International Journal of Internet Science, 5(1), 20-33; see http://dfreelon.org/utils/recalfront/ ;accessed September 2015.

[21] See note 19 above.

[22] The fact that these are novels, fantasies, of course illustrates the complexities of coding. The author, however, had clear intentions of bridging his fantasy and reality, at least in some way.



Primary grading category


1 - Moderate

General religious, political, philosophical or historical material and news commentary containing no endorsement of violence or hatred towards identified communities with generally moderate content along the lines found in mainstream religious/political texts and news media output.

2 - Fringe

Content is religiously or ideologically conservative and isolationist, politically radical and confrontational, but without any justifications conveyed for violence in present-day scenarios. Anger and hostility might be expressed towards a given group of people, such as the ‘kuffar’ (unbelievers) or immigrants, without the added assumption that these people are somehow ‘subhuman’ and legitimate targets of violence.

3 - Extreme

Material that legitimises and/or glorifies the use of violence, especially serious and potentially fatal violence, to achieve particular goals, as well as the fighters and martyrs who die for the cause, with some allusion to the view that such prescriptions continue to be relevant for contemporary activists. Also included within this category is material that focuses on dehumanising particular communities, citing issues of race, sexuality, origin or other aspects that render such people ‘sub-human,’ thus undermining their right to life. This category thus captures both publications advocating ‘jihadi’ violence against combatants or civilians, as well some works of the extreme right-wing, for instance, that can be more opaque in terms of references to violence but with a focus on presenting people such as Jews and non-whites as sub-human in the context of an imagined or envisaged confrontation with these groups of people.

Secondary grading category


Extreme Level 1

Serious violence (i.e. potentially fatal) is only justified/promoted/welcomed with reference to combatants or is vague, without any detail, e.g. talk about the virtues of collective violence, glorification of insurgency warfare.

Extreme Level 2

Serious violence (i.e. potentially fatal) clearly justified/promoted/welcomed against non-combatants, but without any detail, e.g. “murder Muslims”, “kill the kuffar”

Extreme Level 3

Serious violence (i.e. potentially fatal) justified/promoted/welcomed against non-combatants and with some detail regarding facilitation, scope or direction: i.e. “do suicide attacks” (against non-combatants), “target the economy”.

Extreme Level 3b

Same as ‘3’ but specific and directly applicable details offered, e.g. bomb-making recipes

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.


Perspectives on Terrorism is  a journal of the Terrorism Research Initiative and the Center for Terrorism and Security Studies

ISSN  2334-3745 (Online)

Disclaimer, Terms and Conditions