Evidence-Based Counterterrorism or Flying Blind? How to Understand and Achieve What Works

Evidence-Based Counterterrorism or Flying Blind? How to Understand and Achieve What Works

by Rebecca Freese


Many counterterrorism efforts in last decades, and especially since 9/11, have been plagued with doubts as to whether or not they actually make us safer. Unfortunately, the terrorism research that is needed to better plan and evaluate counterterrorism efforts has suffered from both a lack of sufficient rigour and lack of influence on policy-making. This article reviews the state of terrorism research and evidence-based practice in counterterrorism, and the challenges to both. A framework is proposed for the kind of research that needs to be conducted in order to develop evidence-based counterterrorism programs as well as the subsequent evaluative research on existing programs as part of a wider quality improvement program. The argument is made that there is a need for a concrete infrastructure of evidence-based practice and quality improvement in counterterrorism to ensure the best outcomes for national security. The components of a full spectrum quality improvement program in counterterrorism are described. Insights are garnered from the field of medicine, which has recently delved full force into evidence-based practice and quality improvement.

Keywords: counterterrorism, research, evaluation


“Without big data, you are blind and deaf and in the middle of the freeway.”[1] This is how we need to think about the risks of not collecting and analyzing the extensive amount of data available in our modern world today. Data is the raw information that is transformed through analysis into “evidence.” And it is this evidence that paints the picture of what is truly happening in the environment so that one can make enlightened decisions to achieve the desired end results (and not get hit by that truck).

Evidence-based practice (EBP) refers to those practices, actions, and decisions that are grounded in objective evidence obtained from sound, scientific research and analysis. It works to eliminate the element of opinion, gut-instinct, guesswork, or emotion, which can be poisonous to the decision-making process. The goal of using EBP is simply to arrive at those decisions that can produce the most successful outcomes.

The concept of evidence-based practice originated in the field of medicine in the 1990s with what is known as “evidence-based medicine”[2], but its philosophical origins go back to at least the mid-19th century.[3] Evidence-based medicine is defined as the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”[4] Medical practitioners aim to treat patients with methods that have been shown to work and the concept and use of evidence-based medicine has become a deep part of the medical culture. Although there are still significant gaps within the medical field between the knowledge of evidence-based medicine and its universal use,[5] the momentum is always in the direction of using research to guide practice. Because of its importance, hospitals, training programs, and practitioners tout the use of evidence-based medicine as a badge of honor when communicating to their audiences. It has become accepted that in order to participate in the modern arena of medicine, practitioners must be focused on the application of available evidence as to what works in the daily care of their patients. This is not to say that individual patient scenarios and practitioner experience and judgment do not mix with the available evidence to ultimately lead to final decisions, but when possible, subjectivity and guesswork are removed from the equation.

Because of the benefits of this informed and structured approached to decision-making, EBP has extended to many other fields such as psychology, criminology, and education. Although each field, whether a physical or social science, has its own individual characteristics, inherent nature, and unique challenges (as is true when comparing counterterrorism and the practice of medicine), the underlying principles of EBP are the same. That is to say, in any field, one searches for and employs the “conscientious, explicit, and judicious use of current best evidence” in deciding the most appropriate course of action for the purpose of arriving at the best outcomes. With the concept of evidence-based practice in mind, this article explores the state of counterterrorism (CT) efforts in relation to research, evidence, and practice and proposes infrastructure modifications that assist in EBP and the achievement of best outcomes in counterterrorism.


Counterterrorism in Action

Terrorism is not a new phenomenon and one can see prototypes, albeit with differing characteristics and tactics, throughout history. Modern terrorism of the 1960s and 1970s, which was mostly hierarchical, secular, and more discriminate has evolved and enveloped in religious ideology, become more decentralised, and manifests a greater determination to create large-scale death and destruction to achieve its goals. Unfortunately, the future of terrorism could be even more worrisome. The wave of suicide terrorism that began in the 1980s has the potential to morph into ever more destructive chemical, biological, radiologic, nuclear (CBRN), and cyber attacks. Therefore, it is prudent to understand how terrorism threats can most effectively be countered.

Although national governments have always invested in protecting their citizens’ way of life, resources and energy devoted to counterterrorism have grown substantially since the attacks of 11 September 2001 on New York and Washington DC. In the wake of 9/11, the United States government created an entirely new agency, the Department of Homeland Security, to deal with the increasing threat. The range of counterterrorism efforts fall all along the spectrum from law enforcement, security of potential targets, emergency response systems, and treatment of victims to economic aid and sanctions, international resolutions, and media relations.[6] In the United States, one can see the results of counterterrorism policies everywhere, from the security checkpoints at airports to targeted killings via drone attacks in South Asia on the Al-Qaeda leadership to worldwide surveillance programs. Yet how can one really know whether or not these counter-measures are truly effective and achieve the desired outcome - enhanced security?

Recently in Foreign Affairs, two security studies experts debated whether drone attacks actually work in achieving counterterrorism goals. The argument for the use of drone attacks focused on the benefit of eliminating key Al Qaeda leadership figures, interruption of terrorist safe havens, and the fewer civilian deaths caused by drones in comparison to alternative modes of warfare.[7] The counterargument highlighted the damage the drone program causes to U.S. legitimacy, the loss of key intelligence during a strike, the enabling of jihadi recruitment, and the short-term perspective that hinders long-term strategy.[8] What was striking about this debate is that it was not based on much more than informed opinions. Frustratingly, it seems that much of our counterterrorism policy today is based on such hypotheses and speculation. How much more productive would the conversation be if determinations of when, where, and how to use targeted killings could be supported by actual evidence produced from objective scientific research? It is much easier to make an argument for or against something when you have evidence on your side.



Terrorism Research

Since research is the foundation for the evidence needed to make informed decisions when developing and evaluating counterterrorism programs, it is imperative to understand the state of research in the field of terrorism studies (for the purpose of this article, “terrorism research” refers to both research on terrorism and its associated counterterrorism response).

For research results to be valuable and useful to decision-makers who formulate policies that drive counterterrorism programs, the research must be of high quality, with conclusions that are informative, applicable, based on factual data, and have sufficiently high validity and reliability. Much of the research that is most useful in creating this type of evidence is empirical research. Empirical research is a form of rigorous research that is “based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief.”[9] The remarkable shortage of studies that are empirical in nature has been one of the most salient features of terrorism research.[10,11] Based on a comprehensive literature search of all peer-reviewed articles related to terrorism or political violence, Lum et al. found in 2008 that a mere 3% involved empirical analyses (either quantitative or qualitative); 1% were case studies, and 96% were what the authors described as “thought pieces.”[12]

Other shortcomings have been noted. Reviewing literature between 1995 and 2000, Silke found in 2001 that the use of statistical analyses in terrorism research in major journals was exceedingly low while the use of inferential statistics specifically never exceeded 4% of all articles [13]. The use of inferential statistics allows predictions to be made about the entire population based on the analysis of a sample studied [14] and therefore is important for justifying whether research conclusions should be used as evidence when formulating policy. Optimistically, the amount of terrorism research using inferential statistics has slowly been on the rise, with an almost quadrupling in the 7 years since 9/11, although still representing fewer than 12% of all articles examined.[15,16] Silke has also noted a conspicuous lack of methodological range being used in terrorism research, finding the counterterrorism field to be dominated by secondary data analysis using easily accessible sources of data (e.g. books, journals, and media) [17] with the creation of new knowledge only representing approximately 20 percent of the research conducted.[18] A follow-up analysis revealed that literature reviews continue to be the main mode of research in terrorism studies.[19] Schuurman and Eijkman also note the lack of primary research in the terrorism field as an impediment for progress, describing reliable primary data as “vital if we are to truly advance our understanding of terrorism: from its causes and precipitating dynamics to the best way to counter or prevent it.”[20] This is not to say that all research needs to involve primary data, be quantitative or involve sophisticated statistical analyses but it is crucial for published research to represent a range of methodologies. Such variation produces the most meaningful insights about terrorism and therefore can best guide practice.

The quality of terrorism research can also be examined as to whether or not it qualifies as “explanatory research.” Silke, expounding on work by Psychologist Colin Robson, describes research as progressing from exploratory (which helps start the process of building a foundation of knowledge and frequently consists of case studies) to descriptive and then finally to explanatory research, the later being the most scientifically rigorous, reliable, and applicable.[21] Silke notes that, “field areas which fail to make this final transition [to explanatory research] are left with constant gaps in their knowledge base and a fatal uncertainty over the causes of events and what are the truly significant factors at work”[22] and therefore cannot make the leap to producing results that are of predictive value.[23] Thus, without explanatory research, there is little upon which evidence-based counterterrorism practice can be formulated.

Another issue of concern in terrorism research has been the researchers themselves. In 2001, Silke found that 90 percent of all terrorism research was done by a single individual, making the creation of time-intensive statistics needed for more substantive research exceedingly difficult.[24] During the entire decade of the 1990s, he found that 80% of all the researchers who contributed to terrorism journals only wrote one article pertaining to the field during that time.[25] To a large degree this can be explained by the fact that many of the articles have been written by researchers whose primary concentration was in another field, such as psychology or sociology.[26,27,28] Although, terrorism research benefits greatly from a multidisciplinary approach, researchers are needed who research repeatedly and consistently in the field. Since 9/11, there has in fact been an upward trend in these types of dedicated researchers as well as a greater degree of collaborative research.[29,30]

These weaknesses in terrorism research may be why Sageman laments that although there has been an increase in energy and commitment to terrorism research since 9/11, there are still fundamental questions about terrorism for which we have no good answers, particularly why an individual turns to political violence.[31] He argues that one of the problems is that the United States government has not provided sufficient support for the methodical accumulation of detailed and comprehensive data that is critical for answering many of these questions. Without sufficiently extensive, and varied databases and empirical, explanatory research utilizing varied methodologies, along with numerous, dedicated researchers in the field, the evidence needed for the formulation of evidence-based counterterrorism programs will continue to falter.


Evaluative Research

Not only are there weaknesses in the research needed to formulate counterterrorism programs, but there is also a lack of research that evaluates the effectiveness of these programs once they are in effect. Most research in the field tends to focus on the explanations, causes, and sociology of terrorism.[32] Lum, et al. determined that of the already small percentage of counterterrorism publications that met the criteria for empirical research, only a fraction of these were actually evaluative in nature.[33] Even as research in terrorism ramps up, it appears that very little research is being done to scrutinize whether or not established counterterrorism efforts are in fact achieving their intended goals.

By way of illustration, the University of Maryland’s START (National Consortium for the Study of Terrorism and Responses to Terrorism) program, which is a Center of Excellence for the United States Department of Homeland Security, states on its website that their mission is to “advance science-based knowledge about the human causes and consequences of terrorism as a leading resource for homeland security policymakers and practitioners.”[34] Evaluation of the effectiveness of counterterrorism programs is not explicitly stated in this mission statement. Horgan and Stern also do not emphasize evaluative research when discussing the goals of terrorism research. They claim that, “our role as scholars is to discover and understand what influences the changing nature of terrorism and the impact it will have.”[35] Determining whether or not implemented counterterrorism programs actually work is conspicuously absent from this statement on the responsibility of academia. In their article they defend the slowly but steadily improving field of terrorism research by citing examples of the solid research that is occurring in understanding “who becomes a terrorist, why, and how many there are in the West.”[36] None of the examples provided by them referred to evaluative research. Lum and Kennedy suggest that “even the most liberal governments are resistant to self-evaluation,”[37] which is likely due to the clandestine nature of national security and may largely account for why research remains invested in the etiologies of terrorism and not the evaluation of how we respond to terrorism. What results is an environment with less focus and interest by researchers on evaluative research despite the crucial importance it plays in overall counterterrorism strategy and its success.


Evidence-Based Counterterrorism

The paucity of research articles that address the question of whether or not counterterrorism efforts are evidence-based is in itself evidence of the lack of prevalence of this important concept within the field. The same handful of authors write repeatedly on this topic and the overall consensus appears to be that many of our counterterrorism efforts are, in fact, not effective. Sageman describes the “disappointing state of the field, largely consisting of wild speculations without foundation.”[38]

Most research that studies whether counterterrorism programs are effective has focused predominately on the effectiveness of existing programs [39,40] and not whether policy-makers are using research on terrorism as evidence to drive counterterrorism program development and implementation. Furthermore, the evaluative research that has been performed in the field of counterterrorism seems to have focused mainly on outcomes and not on processes or resource use. Very little is known about the cost-effectiveness of our counterterrorism efforts and how they can be improved.[41] It is about moving beyond whether or not our efforts work to understanding how they work. The lack of evaluation on resources and processes could be due to the fact that the data needed for this research has not been easily attainable. It is also likely that these measurements have taken a backseat to the more important measurement of outcomes as states struggle to come to grips with terrorism.

When Lum et al. performed an analysis of the limited but available evaluative research that was of sufficient rigour, it was found that most interventions either had no effect or were harmful (i.e. terrorism increased)[42], leading the authors to conclude that “programs are being used without any knowledge, understanding, or even attempts to determine whether they are effective.”[43] Interestingly, no study showed that military strikes worked. This highlights our need to better understand what works since there has been a propensity by states to rely on military means and force to fight terrorism. Another finding by Lum et al. underscores an important point. The results of studies conducted on the success of metal detectors and security screening were evenly split between either working and being harmful.[44] This disparity was likely due to how outcomes were defined in the different studies. If the outcome was defined as the level of hijackings, then the intervention was considered beneficial. However, if the outcome was defined as the level of non-hijacking attacks, then terrorism was found to increase. Once these security measures were put in place, it is likely that terrorists just resorted to different avenues for attacks. This stresses both the importance of defining goals when developing counterterrorism programs so that effectiveness can be measured against those specific goals as well as taking into account how unintended consequences can result from implementation of counterterrorism programs.

Adams et al. make the point that counterterrorism initiatives since 9/11 have focused more on questions of morality and legality (e.g. enhanced interrogation and ethnic profiling) with less scrutiny into whether the initiatives actually work.[45] The authors also note that evaluation has tended to focus predominantly on the success of overarching government policies and strategies with less attention to individual counterterrorism programs or tactics, making it difficult to decipher more precisely what works and what does not. In their own evaluative research, the authors determined whether the counterterrorism method met its particular aim as well as whether it was aligned with, or counterproductive to, overarching counterterrorism strategies such as “gathering useful information [favorable signal: noise ratio], prioritizing and coordinating intelligence, promoting state legitimacy, encouraging community-generated tips and support of bystanders, and undermining terrorists’ narratives.”[46] This second dimension of measurement is a useful way to tackle the question of effectiveness without having to necessarily measure the end goal of an increase or decrease in terrorist plots or attacks, which is often hard due to the relative infrequency of these events as well as the difficulty in determining causal relationships between interventions and levels of terrorism. In examining specific U.S. counterterrorism activities since 9/11, Adams et al. determined that there was no credible evidence that the use of controversial counterterrorism tactics such as enhanced interrogation, preventative detention, and expanded search and surveillance powers have contributed to thwarting any terrorist plots since 9/11.[47] On the contrary, the most effective measures appear to be those that are least controversial like thwarting terrorism fundraising, denying safe havens, preventing access to port and border crossings, and bolstering state legitimacy.

After publishing their initial research, Adams, Nordhaus, and Shellenberger wrote in an article that indeed many of the controversial “War on Terror” tactics of the post-9/11 era were in fact “abandoned or dramatically scaled back based on overwhelming evidence”[48] after they were found not to be promoting the overall goal of greater U.S. security. The authors describe tactics like preventative detentions, pain-based interrogation and ethnic and religious profiling as being replaced with more “discerning and sophisticated practices.”[49] The authors appear somewhat optimistic that this transition occurred although recognize the long, messy and unstructured process upon which these conclusion were finally drawn, they noted that forgetting lessons from history contributed to many of the more impulsive and reckless tactics. However, when the authors state that overwhelming “evidence” led to the abandonment of many of the controversial tactics, one must question the nature of this evidence. Certainly it does not seem that it was objective, scientific evidence that led to these conclusions but more of an overall sense that things were not working out. The problem with not relying on scientific evidence is that even if it is determined that a tactic is not working as intended, an opportunity is missed to understand why the tactic was ineffective. This understanding is what helps guide informed decisions going forward. Interestingly, the 2011 article states that, “multiple NSA data-mining programs have been abandoned as independent reports, most notably from the National Academies of Sciences, concluded that they simply push terrorist activity further underground.”[50] In light of what was learned in June, 2013 about the widespread, clandestine phone and Internet surveillance programs run by the NSA, it serves as a reminder of the extent to which little is known or understood about the use of evidence-based practices in counterterrorism. Adams et al. state that “national security is still practiced more as a craft than a science” [51] and that the “House and Senate Intelligence committees…have never established any formal process to consistently evaluate and improve the effectiveness of U.S. counterterrorism measures.”[52]

The reason that development of counterterrorism programs has not been evidence-based may be due to the lack of research of sufficient rigour and applicability to drive practice and/or that a sufficiently large disconnect remains between academia and policy-makers. It may also be due to the continued emotion-driven response to terrorism and an overreliance on unfounded assumptions based on gut-feelings about what “should” work. Furthermore, the fact that many existing counterterrorism efforts have not retrospectively been found to be effective is problematic, but not the main concern. What is more important for evidence-based counterterrorism is whether or not these finding are driving change in how one approaches the fight against terrorism.

Challenges to Conducting Terrorism Research

There are many pitfalls to conducting research in both the social and physical sciences and although researchers develop strategies to avoid these pitfalls and try to obtain results that approach the truth to the greatest extent possible, results are never perfect. There has been much criticism as to the state of research in terrorism but in discussing its shortcomings and areas for improvement, it is important to understand the additional, unique challenges faced by terrorism research that makes the search for the truth particularly difficult.

One of the main challenges in conducting terrorism research is the clandestine and adversarial nature of the object of study that makes it difficult to collect reliable and systematic data.[53] Data collection through interviews or surveys, which is common in the social sciences, is extremely challenging in terrorism research. For those who do desire to work directly with terrorists, terrorist organisations, and local populations to gather information, it can be a risky and unpredictable endeavour. Another monumental challenge is that the nature of the field is extremely dynamic and diverse. The players and environment are in constant flux, especially as terrorist organisations work to continually adapt themselves to counterterrorism efforts.[54] Rigorous studies can often take a great deal of time to complete and it is possible that by the time results are published, the environment and therefore applicability may have changed. Other challenges for conducting research in terrorism is the lack of access by researchers to classified information,[55] the continued lack of a generally accepted (legal) definition of terrorism [56,57] and political obstacles that have prevented a full commitment, including financial funding and long-term perspectives, to terrorism studies.[58,59] Despite the challenges of terrorism research, it is argued that the field still lends itself to systematic, sound scientific methodology.[60]


Looking Forward

How do we get to Evidence-Based Counterterrorism?

Charles F. Kettering, the US electrical engineer and inventor once said, “A problem well stated is a problem half solved.” With a grasp on the deficiencies and challenges of terrorism research and the pitfalls of not having sufficient formal structures to connect research and practice, many of the solutions needed are better understood. In fact, some of the solutions have already begun to emerge.

First, it is clear that a commitment to conducting quality, rigorous, scientific research in the field of terrorism with university, government, and private program involvement is required. The START program at the University of Maryland is an example of progress on this front. The START program aims to tackle many of the shortcomings that have existed in this field of research by creating new databases and developing a range of rigorous methodologies. An increasing number of universities are also focusing on research in security studies such as CREATE (National Center for Risk and Economic Analysis of Terrorism Events) at the University of Southern California, and the Center for Terrorism and Security Studies (CTSS) at the University of Massachusetts Lowell.[61] In addition, the RAND Corporation is an institution that has created a significant body of knowledge in the field of terrorism to inform the public, first responders, and policy-makers.[62]

Research funding is also critical for facilitating the production of high quality research that will drive evidence-based counterterrorism. In 2011, the Department of Homeland Security awarded $3.6 million to Maryland’s START program [63] and the National Institute of Justice has also awarded a total of $3 million to 6 universities to conduct research on domestic radicalisation.[64] US federal financing of terrorism research is facilitated when academia can “demonstrate its relevance to policy and operational concerns”[65] and this connection between research and policy makers is another critical component of evidence-based counterterrorism. In order to prevent what Sageman calls “stagnation” in meaningful research on terrorism and its causes due to a disharmonious relationship between academia and government, an infrastructure of coordination, collaboration, and communication must be maintained between researchers, policy-makers, and practitioners.[66] In fact, many of the academic organisations engaging in terrorism research are aiming to influence policy with the results of their work. One of STARTS’s goals is to engage in outreach to policy and practitioner communities in counterterrorism. [67] Programs at Northeastern University and the University of Massachusetts at Lowell (CTSS) are also being developed to “bridge the academic-practitioner divide.”[68] Merari, in analysing the disconnect between academia and government, stresses the importance of promoting influence not just indirectly, but with direct, consistent contact as well.[69] He also suggests the bilateral movement of professionals between academia and government as a means by which the two worlds can be better melded.

Researchers and academics themselves will also help advance the field of terrorism studies in critical ways and ensure that the research is relevant and applicable for decision makers. Merari notes that, “before we complain that the client does not appreciate our merchandise, we must be sure that the goods are good…”[70] Therefore, to improve the product, not only are more dedicated researchers, primary data and empirical studies needed, but researchers need to explore new and innovative methodologies and approaches for conducting terrorism research that will deal with many of its unique challenges. For example, Porter, White, and Mazerolle have developed sophisticated models in which to measure terrorism, accounting for both the frequency and impact of a terrorist attack as well as for determining the effectiveness of a given counterterrorism intervention.[71] Other work has included development of models for assessing counterterrorism policies by applying political uncertainty and complexity theory[72] and advances in the databases needed for evaluating both terrorism and counterterrorism. [73] English advocates the use of new procedures and technology to overcome many of the challenges in data collection to ensure “Information Quality Management.”[74] In addition, Horgan and Stern have advocated interviewing former terrorists as a way to better understand why individuals turn to terrorism as opposed to inferring motivations from observed actions. They believe that this can be done both safely and effectively.[75] Having research associations and conferences,[76] especially international conferences, are a beneficial way for the research community to learn, collaborate and innovate to move the field forward.

Christakis takes a very interesting and important view on how to advance the social sciences that would have important implications for studying terrorism.[77] Christakis argues that it is time to move past the traditional social science subjects of sociology, economics, anthropology, psychology, and political science and “create new social science departments that reflect the breath and complexity of the problems we face as well as the novelty of the 21st century” - similar to the strides already taken in the physical sciences.[78] With what he calls changing “the basic DNA of the social sciences,” new fields like biosocial science, network science, behavioral genetics and computational social science need to be created.[79] These new fields hold great promise for helping tackle many of the challenges of understanding terrorism. One can only imagine what sort of counterterrorism programs can be developed for better understanding terrorism with the assistance of input from fields such as behavioral genetics.

In order to advance evidence-based counterterrorism, a push for more evaluative research is needed. A better balance must be struck between investing efforts in research to understand terrorism and evaluative research; both are necessary for evidence-based practice and better outcomes. Evaluative research would also be facilitated if national security agencies created an infrastructure that incorporates research and researchers into clandestine programs. Processes that would allow researchers to obtain security clearance more easily could help achieve this goal.[80] In general, there needs to be a greater receptivity to research by security agencies.[81]


The Future of Evidence-Based Practice

The trend towards evidence-based practice is not going away. If anything, it is intensifying. In the field of medicine, the US government has taken strides to firmly embed itself in the world of evidence-based practice. In the field of medicine, the Centers for Medicare and Medicaid Services (CMS) have begun to base reimbursements on the use of best practices developed from research evidence. The Physician Quality Reporting System (PQRS) is a program that uses both incentive payments and payment adjustments to promote reporting of quality information by healthcare practitioners.[82]

In a sign of things to come, the US Office of Management and Budget has now planned to put in place a process in which dispersal of funds to agencies will depend in part on how the agencies evaluate the effectiveness of their programs. Bornstein reports that, according to the office, approval for budget requests will be more likely when evidence-based practice is in force [83] and goes on to ask, “could this be the coming of age of ‘evidence-based policy making’?”[84] It seems clear that evidence-based counterterrorism is the way to go.


The Evidence-Based Counterterrorism Framework

Two stages of evidence-based practice (Figure 1) are proposed for studying and applying terrorism research. The first part (“Stage 1”) is the scientific research, both qualitative and quantitative, that builds the foundation of knowledge. This foundation answers questions such as “what,” “how” and “why” and helps provide the necessary insights about terrorism, its causes, and its impacts, that assist in decisions on how best to develop and implement specific counterterrorism (CT) programs that have the greatest chance of success in the short and long-term.

Figure 1. Evidence-Based Counterterrorism Framework

The second part (“Stage 2”) of evidence-based practice entails evaluative research. Evaluative research focuses on performing research on already existing programs to determine their overall effectiveness in practice so that new decisions can be made as to how to change, alter, or improve tactics and strategies going forward. Measurements in evaluative research focus on resources, processes and, most importantly, on outcomes (Figure 2).

Figure 2. Measurements in Evaluative Research

What evaluative research is essential for, and has not been fully appreciated or described as such in the counterterrorism literature, is “quality improvement.” Quality improvement is a concept being used with increasing fervour in healthcare but is also prominent in fields such as Industry and Administration. Duke University Medical Center defines quality improvement as “a formal approach to the analysis of performance and systemic efforts to improve it.”[85] It needs to be an ongoing effort. With changing environments and new information, the cycle of quality improvement keeps practices adapting and advancing to consistently attain the most advantageous results possible.

Lum, et al. point out that, similar to the physical sciences, evidence-based social policy needs to be based on the use of scientific studies not only to determine the relative benefit of implementation of particular programs (Stage 1 above) but also to evaluate the effectiveness of existing programs (Stage 2 above).[86] To take this one step further, quality improvement then uses the knowledge gained from evaluative research to make proactive changes to improve resource use, processes, and outcomes. Therefore, evaluative research is part of a cycle that continuously assesses and looks for opportunities for improvement and then works to implement them. Agencies involved in counterterrorism both at the State and Federal level should have Quality Improvement departments, with dedicated leadership and agency staff, that would help bring greater structure and focus into the achievement of best practices. An infrastructure of quality improvement within the four walls of an agency would drive the effort to connect research evidence and practice and ensure that resources, processes, and outcomes are appropriately evaluated. Given the human life, economic, social, and international stability that are at stake, it seems that evidence-based practice and quality improvement should become a standard part of counterterrorism efforts.


Quality Improvement 2.0: A Full Spectrum Approach for Counterterrorism
The focus on fact-based evidence to drive continuous improvement in resource utilization, processes and outcomes is at the core of quality improvement. However, there are also other fundamental quality improvement principles (Figure 3) that can help by building an organisational environment that fosters best results. In this respect, practices can be garnered from the field of healthcare, a field that strives to use quality improvement principles in its quest to achieve patient safety, efficiency, and best outcomes.

Figure 3. Quality Improvement Principles

Lessons Learned

The medical field puts a great deal of emphasis on learning lessons from events that have occurred. There has been a long tradition of conducting “M&M” (Morbidity and Mortality) conferences on a frequent basis, often weekly. During these conferences poor outcomes and near misses (often referred to as “sentinel events”) are reviewed in order to learn lessons and devise strategies for prevention and avoidance of similar events in the future. Although often difficult to foster, creating an open atmosphere to discuss errors actually serves to improve outcomes in the long run. The airline industry, also relentlessly focused on safety, has worked to create this type of environment. It is also important to note that successful efforts need to receive detailed reviews; equally important is to understand why they were successful as these evaluations assist in creating critically important best practices.

Governments have been known to form commissions and perform lengthy investigations into events where there have been bad outcomes (e.g. the 9/11 Commission), particularly those bad outcomes that are of a large scale and highly public. What is less clear is whether these types of reviews occur in the absence of such large and public failures, whether they occur on a regular basis, and whether near misses also receive thorough review. Near misses can point to faults in the system that if corrected can avoid errors and resulting failures in the future.

When it comes to counterterrorism efforts, learning contemporary lessons is important but not forgetting historical lessons learned when creating counterterrorism policy is also critically important. Silke notes that post-9/11, research with an historical focus decreased [87] and that, although understandable given the gravity of current terrorism threats, a diversion that lasts too long may lead to missed opportunities to spot important trends.[88] Historical cases applied to present-day contexts can offer valuable lessons that could drive astute strategic and tactical decisions on how to fight terrorism in the 21st century.


Focus from Blame to Systems

Quality improvement has also moved into an era that no longer focuses on blame. Where in the past a nurse or physician might be fired for a mistake that led to a patient death, a more sophisticated and nuanced understanding of errors, human nature, and systems has evolved. An environment not focused on blame is one that promotes open discussions and identification of problems and leads to productive actions for improvement. Short of malicious intent, prohibited behaviour, illegal activity, or a pattern of irresponsible behaviour, errors need to be looked at as opportunities for understanding what factors led to a mistake and how these types of errors can be avoided in the future. In the Hillary Clinton Senate Hearing on Benghazi to review the events surrounding the death of Ambassador Chris Stevens and three other Americans at the consulate in Benghazi, Libya by those believed to be radical Sunni Islamists, Senator Rand Paul stated:[89]

One of the things that disappointed me most about the original 9/11 was no one was fired. We spent trillions of dollars, but there were a lot of human errors. These are judgment errors and the people who make judgment errors need to be replaced, fired, and no longer in a position of making these judgment calls.… Had I been president at the time, and I found that you [Hillary Clinton] did not read the cables from Benghazi, you did not read the cables from Ambassador Stevens, I would have relieved you of your post. I think it's inexcusable.

Contemporary facets of quality improvement would argue this mentality is counterproductive to the goal of creating effective counterterrorism measures and that in this incident, focusing on system failures is more useful than a focus on human failures. In fact, Senator Rand Paul mentioned that in investigating the incident, a review board found 64 things that could be changed about the way things were done. One of the goals of quality improvement is to proactively identify cracks in the system and fix them. It is often said that, “every system is perfectly designed to get the results it gets.”[90] This is to say that changes must actually be made to the structure of the system and the way it functions in order to expect different results. Dr. Nielsen sums it up well by saying that humans are “an incredibly error-prone species. It’s very hard to change human nature. It’s really easy to change design, if you bother doing so.”[91] In fact, it is more often the system that is bad, not the people. What this means for agencies involved in national security is that tremendous efforts should be made to examine and design their systems for counterterrorism so that it is much less likely that mistakes can be made (e.g. improved communication systems).


Focus on the Customer

Another core tenant of quality improvement is a relentless focus on the customer. Customers are any persons affected by an industry’s activities and should always be kept in mind in decision-making to ensure best results. Although, the identity of the customer(s) is obvious in many fields (e.g. the patient in healthcare), customers can be identified in counterterrorism, as well.

Potential “customers” can be proposed for the counterterrorism field. The first are a nation’s citizens. Although many counterterrorism activities are clandestine, the public is aware of many activities (e.g. drone strikes and Guantanamo Bay) and many programs affect the public directly (e.g. 3 oz. bottles of liquid at the airport and domestic surveillance programs). For these reasons, public opinion has a large influence on how the government goes about fighting terror. A case in point is the public uproar over the detailed images of the body that airport screeners were producing that led to all scanners being removed and replaced with new ones using a different technology [92] as well as the discovery of the secret NSA phone and Internet surveillance programs that has lead to greater public discussion about how these sorts of programs should function. With consideration of the customer in decisions about security, implementation can often be better planned and potential issues dealt with up front.

In addition, it is believed that one of the main impediments to creating a more proactive (vs. the typical reactive) counterterrorism response is that it is difficult to garner public support (and, in the US context, bipartisan government support for that matter) to spend money on things the public is not worried about or on something that has not happened. It is the hope that with the use of evidence-based practices, the accumulated evidence can help sway stakeholders to the acceptance of appropriate proactive approaches. Lum and Kennedy support the notion of the public as a customer by noting that counterterrorism program effectiveness is not only important in terms of outcomes but “also as to how citizens view the legitimacy of government actions.”[93]

Other critical customers are the local population where terrorists and their organisation reside and those individuals susceptible to radicalisation. The battle against terrorism is often believed to be a battle of perception and legitimacy. With this is mind, it seems quite clear that these individuals are indeed “customers” in counterterrorism efforts. Close attention must be paid to how counterterrorism tactics affect and are perceived by this group of people. Focusing consistently on the customer(s) will help the overall success and effectiveness of counterterrorism efforts.


Leadership and Employee Empowerment

Lastly, in order for a quality improvement program to be successful, it must have full support of top leadership and direct involvement of key leadership officials. This can provide the much needed legitimacy to the program and its activities. Although strong leadership is needed, most of the more progressive organisations today are taking on a more flat (vs. hierarchical) organisational structure. This environment is conducive to quality improvement because it empowers employees. Members of the organisation at all levels can have a critical impact in identifying areas for improvement and this atmosphere drives innovation and practical solutions for many of the obstacles that will help drive the field forward. In many operating room environments, it is no longer only the surgeon who runs the show. Many hospitals are empowering everyone from medical students, to nurses, to surgical technicians to speak up if they believe something might be wrong or could be done better.



In 1988, Schmid and Jongman estimated that “perhaps as much as 80 per cent of the literature is not research-based in any rigorous sense.”[94] Almost two decades later, many experts have come to similar conclusions.[95,96] Although some of the more recent literature has shown some positive trends, it is hard to arrive at concrete conclusions about the most current state of counterterrorism research as little data is available regarding empirical and evaluative research that has been done in recent years. With the steep upward trajectory in counterterrorism research since 9/11, it is reasonable to conclude that analysis of research in the last few years may lead us to draw different conclusions. Further studies need to be conducted on the most current environment of terrorism research.

It is also important to note that one of the biggest pitfalls that seems to afflict counterterrorism tactics is that many tactics that are developed to achieve a particular aim, end up having other unintended consequences that have the potential to threaten the success of the overall effort. Much of this problem stems from only focusing on the connection between three points in decision-making. For example, since Al-Qaeda and its affiliates are responsible for a great deal of the terrorism threat today and targeted killings eliminate most of Al-Qaeda original leaders, then it is assumed that targeted killing will decrease the threat of terrorism. What is less clear, and what hopefully rigorous research can help elucidate in these circumstance, is whether the tactic is causing other results that are counterproductive, resulting in a net negative effect. These other results may be an increase in jihadi recruitment, greater sympathy for Al-Qaeda, or collateral civilian deaths that lead to decreased state legitimacy and an inability to recruit informants. The same argument can be made for the logic behind the decision to use enhanced interrogation techniques (i.e. information is needed to stop terrorist plots, enhanced interrogation leads to information, thus enhanced interrogation should help stop terrorist plots). By way of analogy, healthcare practitioners often prescribe medications to treat diseases, such as high blood sugar or blood pressure, expecting a beneficial end result because these physiological states are bad for the body. Yet often further research reveals that use of these medications are also associated with an increase in other ailments such as heart complications or even death, thus mitigating the assumed beneficial effect. This is why research and evidence is so important. When analysed in an objective and scientific manner, sometimes things that appear to work in an obvious way and are expected to lead to a certain result, in fact do not do so.

Another reason why governments should be investing in evidence-based practice and evaluative research, is that finding out what works is not just important for counterterrorism. In a world where money and resources are limited, it is important to compare effectiveness across all areas of spending. The current state of evaluative research in counterterrorism does not allow any discernable connection to be made between money spent and increases in national security. Pegors, pessimistic whether counterterrorism efforts are truly making an impact, concluded “if we are truly concerned with protecting the lives of US citizens, then we must reallocate our resources to areas that have a greater impact on a larger number of people.”[97] He compares the approximate 3,000 lives lost from the attacks on 9/11, to the greater than 500,000 lives lost to cancer, 40,000 from car accidents, and 700,00 from heart disease that same year.[98] In fact, an estimated 26,100 people between the ages of 25 and 64 died prematurely in the U.S. due to lack of health care coverage in 2010.[99] Therefore, evidence-based counterterrorism is essential not only to achieving optimal national security but in terms of resources, understanding how to get the best outcomes as a society.

In writing about evidence-based practice it is vitally important to recognise its limits. Evidence-based practice is not a panacea. Research results do not prove anything with 100 percent certainty nor are they always perfectly applicable. In addition, researchers often come to conflicting conclusions and research designs can be flawed. Anyone in the field of medicine can tell you the limits of evidence-based practice. All too often, a recommendation will come out as to how best to treat a patient or prevent a particular disease, only to have a different recommendation come out a few years later because of new or conflicting evidence. This often leaves medical practitioners at a loss for how best to proceed. Things can be even more complicated in a field like terrorism where situations change with extraordinary rapidity due to adaptations and evolution by the enemy as well as environments in the midst of political change and conflict as can be seen currently in the Middle East. Rigorous research studies often take time and can possibly be less relevant by the time they are published. Because of the dynamism and challenges of doing research in the field of terrorism, it is even more crucial to conduct evaluative research of those counterterrorism programs that have already been implemented. If programs must be put into place quickly or without scientific evidence of their ability to effectively address the current security risk, then, as the program is implemented, evaluative research should be begun to verify whether it is accomplishing its intended goals. It is a sort of retrospective safety check. And even this evaluative research is not perfect. However, despite its faults, having evidence that is not perfect is better than functioning without any evidence at all.

Although it may be difficult to push for evidence-based practice before sufficiently rigorous research is available to drive these practices, we should recognize that, just as in business, demand drives supply. If national security agencies prioritise evidence-based practice and express a genuine desire for empirical research and primary data, the research needed will more likely materialise (especially if government funding is forthcoming).

Overall a quality improvement infrastructure can help ensure that resources are being applied to the tactics and strategies that lead to the best outcomes while eliminating or altering counterterrorism programs that are found to be counterproductive or harmful. The determination of the effectiveness of counterterrorism efforts ultimately needs a long-term perspective and therefore an ongoing cycle of quality improvement is necessary. Creating a full spectrum quality improvement program incorporating many of the principles discussed is crucial for making the much-needed leap forward to achieving better outcomes in national security. It is an in-depth and lengthy endeavor that requires commitment and perseverance. The healthcare field has really only just begun to incorporate many of these principles into their practices and they still have a long way to go.

There are significant substantive differences between the fields of medicine and counterterrorism. Medicine (mutating bacteria that evolve to resist our antibiotics aside) mostly involves unilateral decision-making. Counterterrorism is decision-making in a game theory environment, where the terrorist side is trying to strategise, innovate, and outsmart the authorities at every move. Research in human biology and disease states and examining responses to interventions is inherently different than researching terrorism and evaluating effects of counterterrorism efforts. However, overarching strategies used in healthcare, as well as in other fields, in attempts to achieve the best outcomes, can indeed provide insights for fields like counterterrorism. Marrin and Clemente examined how practices in medicine could help improve intelligence analysis and described how looking at analogous professions, such as journalism, law and law enforcement can spark ideas that can prove beneficial to the field of intelligence.[100] In fact, important ideas can often come from seemingly unrelated fields. A.J. Jacobs, an author who read the Encyclopedia Britannica in an 18-month experiment, derived what he believed to be the six most important business lessons from all of history.[101] One of these was to “take ideas from far outside your field.” He writes how Isaac Newton came up with the idea that gravity worked at a distance and not like billiard balls colliding by reading about alchemy and magic and that Bill Gates often explores disparate fields to generate ideas. Hence, looking at analogous fields, and even some that do not appear so, may be a beneficial way to precipitate innovative and successful ideas and practices that will push the counterterrorism field forward, providing better end results.



The field of counterterrorism has suffered from a number of deficiencies in both Stage 1 and Stage 2 research as well as a lack of policy formation guided by objective, scientific evidence. The Evidence-Based Counterterrorism Framework presented here offers a promising approach for research, evidence, and quality improvement for entities involved in counterterrorism. Although there will never be one perfect answer as to how to best conduct counterterrorism operations, evidence will hopefully help guide practitioners as to which solutions are likely better than others.

What is needed is a greater awareness and a strong culture of evidence-based practice and quality improvement within the field of counterterrorism. It is time for the phrase "evidence-based counterterrorism" to be as familiar a concept to those in the field of counterterrorism as “evidence-based medicine” is to those in healthcare.


About the Author: Rebecca Freese, MD is an anesthesiologist and has worked on quality improvement in the healthcare field. She received her M.B.A. from Johns Hopkins University and her M.A. in Government with a specialization in Counterterrorism and Homeland Security from the Interdisciplinary Center in Herzliya, Israel.


[1] Quote by Geoffrey Moore, author and consultant.

[2] Hjorland, B. (2011). Evidence-based practice: an analysis based on the philosophy of science. Journal of the American Society for Information Science and Technology, 62 (7), p. 1301.

[3] Sacket, D.L., Rosenberg, W.M, Gray, J.A., Haynes, R.B., and Richardson, W.S. (1996). Evidence based medicine: what it is and what it isn’t. BMJ, 312(7023), p. 71.

[4] Ibid, p.71.

[5] Glasziou, P., Ogrinc, G., Goodman, S. (2011). Can evidence-based medicine and clinical quality improvement learn from each other? BMJ Qual Saf, 20 (Suppl 1), i13-
i17. doi:10.1136/bmjqs.2010.046524 .

[6] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), p. 35.

[7] Byman, D. (2013). Why drones work. Foreign Affairs. Retrieved from http://www.foreignaffairs.com/articles/139453/daniel-byman/why-drones-work

[8] Cronin, A.K. (2013). Why drones fail. Foreign Affairs. Retrieved from http://www.foreignaffairs.com/articles/139454/audrey-kurth-cronin/why-drones-fail .

[9] Penn State University Libraries. (n.d.). What is empirical research? Empirical Research. Retrieved from http://www.libraries.psu.edu/psul/researchguides/edupsych/empirical.html .

[10] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), p. 35-42.

[11] Schulze, F. (2004). Breaking the cycle: Empirical research and postgraduate studies on terrorism. In A. Silke (Ed.), Research on Terrorism: Trends, achievements, and
failures (pp. 161-184). London: Frank Cass.

[12] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), p. 35-42.

[13] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), 1-14.

[14] Crossman, A. (n.d.) Descriptive vs. inferential statistics. About.com. Retrieved from http://sociology.about.com/od/Statistics/a/Descriptive-inferential-statistics.htm

[15] Silke, A. (2007). The impact of 9/11 on research on terrorism. In M. Ranstorp (Ed.), Mapping Terrorism Research: State of the art, gaps and future direction (pp. 76 - 93). New York, NY: Routledge.

[16] Silke, A. (2009). Contemporary terrorism studies: Issues in research. In R. Jackson, M.B. Smyth, and J. Gunning (Eds.) Critical Terrorism Studies: A New Research
Agenda (pp. 34 - 48). New York, NY: Routledge.

[17] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp.1-14.

[18] Ibid, p. 8.

[19] Silke, A. (2007). The impact of 9/11 on research on terrorism. In M. Ranstorp (Ed.), Mapping Terrorism Research: State of the art, gaps and future direction (pp. 76 - 93). New York, NY: Routledge.

[20] Schuurman, B. and Eijkman Q. (2013). Moving terrorism research forward: the crucial role of primary sources. International Centre for Counter-Terrorism – The
Hague, p. 8 Retrieved from http://www.icct.nl/download/file/Schuurman-and-Eijkman-Moving-Terrorism-Research-Forward-June-2013.pdf

[21] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp. 1-14.

[22] Ibid, p. 1.

[23] Ibid, p. 2.

[24] Ibid.

[25] Ibid.

[26] Sageman, M. (2013). The stagnation of research on terrorism. The Chronicle of Higher Education. Retrieved May 12, 2013 from http://chronicle.com/blogs/conversation/2013/04/30/the-stagnation-of-research-on-terrorism/ .

[27] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp. 1-14

[28] Merari, A. (1991). Academic research and government policy on terrorism.  Terrorism and Political Violence, 3(1), pp. 88-102.

[29] Silke, A. (2007). The impact of 9/11 on research on terrorism. In M. Ranstorp (Ed.),  Mapping Terrorism Research: State of the art, gaps and future direction (pp. 76 - 93). New York, NY: Routledge.

[30] Silke, A. (2009). Contemporary terrorism studies: Issues in research. In R. Jackson,  M.B. Smyth, and J. Gunning (Eds.) Critical Terrorism Studies: A New Research Agenda (pp. 34 - 48). New York, NY: Routledge.

[31] Sageman, M. (2013). The stagnation of research on terrorism. The Chronicle of Higher Education. Retrieved May 12, 2013 from http://chronicle.com/blogs/conversation/2013/04/30/the-stagnation-of-research-on-terrorism/ .

[32] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), p. 38.

[33] Ibid.

[34] START website. (n.d.). National Consortium for the Study of Terrorism and Responses to Terrorism. A Center of Excellence of the U.S. Department of Homeland Security based at the University of Maryland. START. Retrieved from http://www.start.umd.edu/start/about/overview/mission/ and http://www.start.umd.edu/start/publications/START_Brochure_WEB.pdf

[35] Horgan, J. and Stern, J. (2013). Terrorism research has not stagnated. The Chronicle of Higher Education, para. 4 Retrieved from http://chronicle.com/blogs/conversation/2013/05/08/terrorism-research-has-not-stagnated/ .

[36] Ibid, para. 9.

[37] Lum, C. and Kennedy, L.W. (2012). Evidence-based counterterrorism policy. In C.  Lum and L.W. Kennedy (eds.), Evidence-Based Counterterrorism Policy (p. 5). New York, NY: Springer.

[38] Sageman, M. (2013). The stagnation of research on terrorism. The Chronicle of Higher Education, para. 5. Retrieved May 12, 2013 from http://chronicle.com/blogs/conversation/2013/04/30/the-stagnation-of-research-on-terrorism/ .

[39] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20(1), pp. 35 - 42.

[40] Adams, N. Nordhaus, T., Shellenberger, M. (2011). Counterterrorism since 9/11; Evaluating the efficacy of controversial tactics. Breakthrough Institute. pp.1- 64.

[41] Lum, C. and Kennedy, L.W. (2012). Evidence-based counterterrorism policy. In C. Lum and L.W. Kennedy (Eds.), Evidence-Based Counterterrorism Policy. New York, NY: Springer.

[42] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidencebased? What works, what harms, and what is unknown. Psicothema, 20 (1), pp. 35-42.

[43] Ibid, p. 41.

[44] Ibid.

[45] Adams, N. Nordhaus, T., Shellenberger, M. (2011). Counterterrorism since 9/11; Evaluating the efficacy of controversial tactics. Breakthrough Institute. pp. 1- 64.

[46] Ibid, p. 8.

[47] Ibid.

[48] Adams, N. Nordhaus, T., Shellenberger, M. (2011). Who killed the war on terror? The Atlantic, para. 3. Retrieved from
http://www.theatlantic.com/national/archive/2011/08/who-killed-the-war-on-terror/244273/?single_page=true .

[49] Ibid, para. 12.

[50] Ibid, para. 16.

[51] Ibid, para. 22.

[52] bid, para. 25.

[53] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp. 1-14.

[54] Brun, I. (2010). ‘While you’re busy making other plans’ – The ‘Other RMA.’ The Journal of Strategic Studies, 33(4), pp. 535-565.

[55] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidencebased? What works, what harms, and what is unknown. Psicothema, 20 (1), pp. 35 - 42.

[56] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13(4), pp. 1-14.

[57] Crenshaw, M. (1992). Current research on terrorism: The academic perspective. Studies in Conflict and Terrorism, 15(1), pp. 1-11.

[58] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp. 1-14

[59] Sageman, M. (2013). The stagnation of research on terrorism. The Chronicle of Higher Education, Retrieved May 12, 2013 from http://chronicle.com/blogs/conversation/2013/04/30/the-stagnation-of-research-on-terrorism/.

[60] Horgan, J. (1997). Issues in terrorism research. The Police Journal, 70, p. 193.

[61] Horgan, J. and Stern, J. (2013). Terrorism research has not stagnated. The Chronicle of Higher Education, Retrieved from http://chronicle.com/blogs/conversation/2013/05/08/terrorism-research-has-not-stagnated/ .

[62] RAND website. (n.d.). Counterterrorism. Rand Corporation. Retrieved from http://www.rand.org/topics/counterterrorism.html .

[63] Clark, M. (2011). Understanding the making of a terrorist: DHS funds vital research by MD University center of excellence. The Department of Homeland Security.  Retrieved from http://www.dhs.gov/blog/2011/12/21/understanding-making-terrorist-dhs-funds-vital-research-md-university-center .

[64] National Institute of Justice Website. (2012). Fiscal year 2012 awards. National Institute of Justice. Retrieved from http://www.nij.gov/nij/funding/awards/2012.htm

[65] Horgan, J. and Stern, J. (2013). Terrorism research has not stagnated. The Chronicle of Higher Education, para. 17 Retrieved from http://chronicle.com/blogs/conversation/2013/05/08/terrorism-research-has-not-stagnated/ .

[66] Sageman, M. (2013). The stagnation of research on terrorism. The Chronicle of Higher Education, Retrieved May 12, 2013 from http://chronicle.com/blogs/conversation/2013/04/30/the-stagnation-of-research-on-terrorism/ .

[67] START website. (n.d.). National Consortium for the Study of Terrorism and Responses to Terrorism. A Center of Excellence of the U.S. Department of Homeland Security based at the University of Maryland. START. Retrieved from http://www.start.umd.edu/start/about/overview/mission/ and http://www.start.umd.edu/start/publications/START_Brochure_WEB.pdf .

[68] Horgan, J. and Stern, J. (2013). Terrorism research has not stagnated. The Chronicle of Higher Education, para. 16. Retrieved from http://chronicle.com/blogs/conversation/2013/05/08/terrorism-research-has-not-stagnated/.

[69] Merari, A. (1991). Academic research and government policy on terrorism
Terrorism and Political Violence, 3(1), pp. 88 -102.

[70] Ibid, p. 99.

[71] Porter, M.D., White, G., Mazerolle, L. (2012). Innovative methods for terrorism and counterterrorism data. In C. Lum and L.W. Kennedy (Eds.), Evidence-Based Counterterrorism Policy (pp. 91-112). New York, NY: Springer

[72] Revilla, C.C. (2012). A complexity method for assessing counterterrorism policies.  In C. Lum and L.W. Kennedy (eds.), Evidence-Based Counterterrorism Policy (pp. 151-165). New York, NY: Springer .

[73] LaFree, G. (2012). Generating terrorism event databases: result rom the global terrorism database 1970 to 2008. In C. Lum and L.W. Kennedy (Eds.), Evidence-Based Counterterrorism Policy (pp. 41-64). New York, NY: Springer

[74] English, L.P. (2005). Information quality: Critical ingredient for national security.  Journal of Database Management, 16(1), pp. 18 - 32.

[75] Horgan, J. and Stern, J. (2013). Terrorism research has not stagnated. The Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/conversation/2013/05/08/terrorism-research-has-not-stagnated/

[76] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp. 1-14.

[77] Christakis, N.A. (2013). Let’s shake up the social sciences. The New York Times.  Retrieved from http://www.nytimes.com/2013/07/21/opinion/sunday/lets-shake-up-the-social-sciences.html?_r=0andpagewanted=print

[78] Ibid, para. 17.

[79] Ibid, para. 18.

[80] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20(1), pp. 35-42.

[81] Lum, C. and Kennedy, L.W. (2012). Evidence-based counterterrorism policy. In C.Lum and L.W. Kennedy (Eds.), Evidence-Based Counterterrorism Policy (p. 373).
New York, NY: Springer.

[82] CMS website. (n.d.). Physician Quality Reporting System. Centers for Medicare and Medicaid Services. Retrieved from http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/PQRS/index.html?redirect=/PQRS/

[83] Bornstein, D. (2012). The dawn of the evidence-based budget. The New York Times, para. 2. Retrieved from http://opinionator.blogs.nytimes.com/2012/05/30/worthy-of-government-funding-prove-it/?_r=0andpagewanted=print

[84] Ibid, para. 3.

[85] Duke University Medical Center. (2005). What is Quality Improvement? Patient Safety-Quality Improvement. Retrieved from http://patientsafetyed.duhs.duke.edu/module_a/introduction/introduction.html .

[86] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), pp. 35 - 42.

[87] Silke, A. (2007). The impact of 9/11 on research on terrorism. In M. Ranstorp (Ed.),  Mapping Terrorism Research: State of the art, gaps and future direction (pp. 76-93). New York, NY: Routledge.

[88] Ibid, pp. 88 - 89.

[89] CNN website. (2013). Transcripts: Hillary Clinton Senate hearing on Benghazi. CNN. Retrieved from http://transcripts.cnn.com/TRANSCRIPTS/1301/23/se.01.html .

[90] URC website. (n.d.). Quality Improvement. URC. Retrieved from www.urc-chs.com/quality_improvement .

[91] Chang, K. (2001). From ballots to cockpits, questions of design. The New York Times, para 33. Retrieved from http://www.nytimes.com/2001/01/23/science/from-ballots-to-cockpits-questions-of-design.html?src=pm .

[92] Unruh, B. (2013). Naked-image body scanners removed from airports. WND. Retrieved from http://www.wnd.com/2013/05/naked-image-body-scanners-removed-from-airports/ .

[93] Lum, C. & Kennedy, L.W. (2012). Evidence-based counterterrorism policy. In C. Lum and L.W. Kennedy (Eds.), Evidence-Based Counterterrorism Policy (p. 4). New York, NY: Springer .

[94] As cited in Horgan, J. (1997). Issues in terrorism research. The Police Journal, 70, p.193.

[95] Lum, C., Kennedy, L.W., Sherley, A. (2008). Is counter-terrorism policy evidence based? What works, what harms, and what is unknown. Psicothema, 20 (1), pp. 35 - 42.

[96] Silke, A. (2001). The devil you know: continuing problems with research on terrorism. Terrorism and Political Violence, 13 (4), pp.1-14.

[97] Pegors, T. (2012). Over defense: Taking another look at American counterterrorism spending. Columbia Political Review, para. 9. Retrieved from http://cpreview.org/2012/12/over-defense/ .

[98] Ibid, para. 9.

[99] FamilyUSA website. (2012). Dying for coverage: The deadly consequences of being uninsured. FamilyUSA. Retrieved from http://www.familiesusa.org/resources/publications/reports/dying-for-coverage-findings.html .

[100] Marrin, S. & Clemente, J.D. (2006). Improving intelligence by looking to the medical profession. International Journal of Intelligence and CounterIntelligence, 18 (4), p. 727.

[101] Jacobs, A.J. (2013). The six most important business lessons from all of history. LinkedIn. Retrieved from http://www.linkedin.com/today/post/article/20130911124325-4061630-the-six-most-important-business-lessons-from-all-of-history .


Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.


Perspectives on Terrorism is  a journal of the Terrorism Research Initiative and the Center for Terrorism and Security Studies

ISSN  2334-3745 (Online)

Disclaimer, Terms and Conditions