Sources and Information in Academic Research:  Avoiding Mistakes in Assessing Sources for Research and during Peer Review

Sources and Information in Academic Research:  Avoiding Mistakes in Assessing Sources for Research
and during Peer Review

by Tom Quiggin

The credibility of academic publications has come under attack in a variety of circumstances. 

Newspaper headlines such as “Scientific fraud is rife[1]” and “McGill University finds scientists published ‘falsified’ images[2]” are not helpful in maintaining the credibility of the academic community.  Additionally, a cottage industry appears to be growing in websites that specialize in identifying papers which publishers have been forced to retract.[3]  The website Retraction Watch has identified one case in which a publisher has retracted 172 papers from one author and may eventually retract 183 papers in total.  The website Copy Shake and Paste makes a series of references to PhD dissertations and professorial habilitations which have been questioned or rescinded due to plagiarism. [4] One of the PhDs in question was written by a German Minister of Education and Research.[5]

A responsibility comes with publishing papers in the academic community.  The papers can shape the views of the individuals that read them and will – on occasion – be used in official proceedings such as during court room trials or in commissions of inquiry.  Academic credibility also comes into play in the public realm when academics give media interviews or appear before governmental committees.

Preserving the credibility of research into issues related to terrorism and national security can be especially daunting for academics. In addition to all of the normal problems with source reliability, the various governmental agencies involved often have an obsession with secrecy, making information difficult to confirm. Interviews from those who have played a role in a terrorism or national security cases can often be tainted by their beliefs or actions.  Government agencies and groups may have agendas which they are willing to advance by providing ‘off the record’ briefings to the unsuspecting or willingly naive academic researcher.

Given these problems, how should  forthright academic researchers go about assessing the reliability of their sources and the information upon which they base their research product?  The problem can be especially difficult when faced with information coming from websites on the Internet.  Was the website you are considering using in a footnote created by a reliable source based on credible information, or was it written by www.14-year-old-Johnny-sitting-in-his-parents-basement.org? Worse still, was it created by an agency, group or person deliberately attempting to spread false information?

This research note will provide two tools that are potentially useful for those faced with the challenge of assessing the validity of sources and information for research or for doing a peer review exercise of a paper submitted to an academic journal for publication.  The first is the A1-E5 method for assessing the reliability of a source of information as well as the credibility of the information provided. The second is an example of a generic guide for assessing the validity of information found on websites. In both cases, the intent is to assist authors or peer reviewers in coming to their own conclusions about the reliability of sources and the credibility of information.

The “A1-E5” Methodology for Source Credibility and Information Reliability[6]

Merely quoting a source of information is not enough. The source has to be reliable and additionally the information provided has to be credible.  As such, it is at least a two step process to determine if you should include a particular source as a footnote or declare it to be valid in a peer review exercise.

The A1-E5 methodology[7] can be employed to assist in fulfilling these two requirements.  The source of the information is assigned a level of reliability based on a scale of A (high) to E (unknown).  The information that is being presented is rated on a scale of 1 (highly credible) to 5 (credibility unknown).  Once this is done, the material is assigned a rating such as A1 or B3.
 
Both parts of the method are important, in that it is frequently believed (sometimes wrongly) that if a source is deemed reliable, then the information must be credible.  Unfortunately, this is not the case.  All too often, a source that is reliable may be misinformed, out of date, subject to unconscious biases or – worst case scenario – using their perceived credibility to advance an agenda or support an ideology with information that is not credible.

Using the A1-E5 Method to Assess the Source

The first step is to assess the reliability of the source involved. This can be an individual, a group or an organization.  All of these entities provide an opportunity for the researcher or peer reviewer to assess their track record. In the case of a perceived conflict or disconnect (i.e. a normally reliable institutions but a doubtful author) then the lower of the two assessments levels  should be used. 

Authors are advised to remain in a critical frame of mind when assessing source reliability.  A well-known professor or researcher may have highly entertaining classes or provide great stimulation at a conference. Being well known does not necessarily infer being reliable. In addition, caution is always urged when using media sources. Some newspapers have a well- known and advertised bias which can be factored in when assessing reliability and credibility. Other papers may not. For instance, the Washington Times is owned by organizations related to the Unification Church of the Reverend Sun Moon of Korea. This does not mean you should or should not use the paper as a source, but rather you should ask yourself if there is a specific agenda in reporting on the particular story you are quoting.[8] The question should be asked for all such newspapers and magazines.

The rating scale for assessing reliability of a source is listed below. By asking the right questions, it should be possible to form a clearer picture of source reliability.

table 1

The Information Provided by the Source

In a separate exercise, the information provided must also be verified for consistency and internal logic.  The information must be analyzed to see it fits with a known body of knowledge or known patterns in the past.  If it does, then the credibility of the information may be considered as high.  If it does not, then questions must be raised and an assessment must be made to see if the inconsistencies represent a change in the situation or the discovery of new information.  Or perhaps it is a flawed analysis.  It is possible that a further examination of the credibility of the analysis must be made.

New or different does not mean wrong!  Peer review and research standards should not be seen as a means of enforcing orthodoxy or preventing challenges to status quo in a given field of inquiry.   But if the information provided is new or startling, then the researcher or peer reviewer should be especially alert to possible problems.

The rating scale for assessing the credibility of information is below. By asking the right questions, it should be possible to form a clearer picture of information credibility.

table 2

The Information Provided by the Source

In a separate exercise, the information provided must also be verified for consistency and internal logic.  The information must be analyzed to see it fits with a known body of knowledge or known patterns in the past.  If it does, then the credibility of the information may be considered as high.  If it does not, then questions must be raised and an assessment must be made to see if the inconsistencies represent a change in the situation or the discovery of new information.  Or perhaps it is a flawed analysis.  It is possible that a further examination of the credibility of the analysis must be made.

New or different does not mean wrong!  Peer review and research standards should not be seen as a means of enforcing orthodoxy or preventing challenges to status quo in a given field of inquiry.   But if the information provided is new or startling, then the researcher or peer reviewer should be especially alert to possible problems.

The rating scale for assessing the credibility of information is below. By asking the right questions, it should be possible to form a clearer picture of information credibility.

Evaluation of Web documents

How to interpret the basics

1. Accuracy of Web Documents

  • Who wrote the page and can you contact him or her?
  • What is the purpose of the document and why was it produced?
  • Is this person qualified to write this document?

Accuracy

  • Make sure author provides e-mail or a contact address/phone number.
  • Know the distinction between author and Webmaster.

2. Authority of Web Documents

  • Who published the document and is it separate from the "Webmaster?"
  • Check the domain of the document, what institution publishes this document?
  • Does the publisher list his or her qualifications?

Authority

  • What credentials are listed for the authors)?
  • Where is the document published? Check URL domain.

3. Objectivity of Web Documents

  • What goals/objectives does this page meet?
  • How detailed is the information?
  • What opinions (if any) are expressed by the author?

Objectivity

  • Determine if page is a mask for advertising; if so information might be biased.
  • View any Web page as you would an info-commercial on television. Ask yourself why was this written and for whom?

4. Currency of Web Documents

  • When was it produced?
  • When was it updated'
  • How up-to-date are the links (if any)?

Currency

  • How many dead links are on the page?
  • Are the links current or updated regularly?
  • Is the information on the page outdated?

5. Coverage of the Web Documents

  • Are the links (if any) evaluated and do they complement the documents' theme?
  • Is it all images or a balance of text and images?
  • Is the information presented cited correctly?

Coverage

  • If page requires special software to view the information, how much are you missing if you don't have the software?
  • Is it free or is there a fee, to obtain the information?
  • Is there an option for text only, or frames, or a suggested browser for better viewing?

Putting it all together

  • Accuracy. If your page lists the author and institution that published the page and provides a way of contacting him/her and . . .
  • Authority. If your page lists the author credentials and its domain is preferred (.edu, .gov, .org, or .net), and. . .
  • Objectivity. If your page provides accurate information with limited advertising and it is objective in presenting the information, and . . .
  • Currency. If your page is current and updated regularly (as stated on the page) and the links (if any) are also up-to-date, and . . .
  • Coverage. If you can view the information properly--not limited to fees, browser technology, or software requirement, then . . .

You may have a Web page that could be of value to your research!

Conclusion

Determining whether or not a source is reliable and whether or not the information is credible can sometimes be challenging to the researcher or peer reviewer.  However, it is possible to apply a simple set of methodologies for the purpose of focusing in on the questions of reliability and credibility.  Information derived from websites on the World Wide Web provides an extra layer of difficulty given the practice of anonymity or attempts to mislead.

For the younger researcher, caution is advised on the trap, noted above, of assessing information as credible simply because the source is well known or has a Wikipedia entry.  Unfortunately, a number of individuals who publish do so with a certain agenda or bias at work which may consciously or unconsciously shape their work.  Separating the source and the information as a separate exercise as noted in the A1-E5 methodology is critical in determining whether information from a given source should be included.

About the Author: Tom Quiggin, (M.A, C.D.) is a court qualified expert in “the reliability of intelligence as evidence in court”(Federal Court of Canada)  as well as the “structure, organization and evolution of the global jihadi movement” (Criminal Court and Federal Court).  He has 20 plus years of practical intelligence experience in a variety of positions.  These include the Royal Canadian Mounted Police, the Canadian Armed Forces, the United Nations Protection Force in Yugoslavia, Citizenship and Immigration Canada (War Crimes), the International War Crimes Tribunal for the former Yugoslavia (The Hague), and the Privy Council Office of Canada.  He was also a qualified arms control inspector for the Conventional Forces in Europe Treaty and the Vienna Document.  He holds a Masters Degree in International Relations and is a certified knowledge management practitioner.   He has a sole author book publication titled Seeing the Invisible:  National Security Intelligence Requirements in an Uncertain Age (Singapore: World Scientific, 2007) and has numerous publications and book chapters in Canada, the USA, the UK, Germany, Singapore and the Netherlands on his name.

Notes

[1]  Pete Etchells and Suzi Gage (University of Bristol) Scientific fraud is rife: it's time to stand up for good science, Friday 2 November 2012 07.00 GMT. The article is available online at:   http://www.guardian.co.uk/science/blog/2012/nov/02/scientific-fraud-good-science . (downloaded on 30 January 2013 at 1340 EST)
[2] Margaret Munro, McGill University finds scientists published ‘falsified’ images, Postmedia News, 29 January 2013 as published in the Ottawa Citizen. The article is available online at:  http://www.ottawacitizen.com/news/national/McGill+University+finds+scientists+published+falsified/7889807/story.html   (downloaded 31 January 2013 at 1348 EST).
[3] See, for example, Retraction Watch which is available at http://retractionwatch.wordpress.com/.  See also Copy, Shake and Paste at http://copy-shake-paste.blogspot.com.  (both sites downloaded on January 2013 at 1354 EST).
[4]See http://retractionwatch.wordpress.com/2013/01/15/retraction-record-broken-again-university-report-should-up-fujii-total-to-183/.  See also http://retractionwatch.wordpress.com/2012/07/02/does-anesthesiology-have-a-problem-final-version-of-report-suggests-fujii-will-take-retraction-record-with-172/ and http://www.anesth.or.jp/english/pdf/news20120629.pdf (all articles downloaded on 31 January 2013 from 1400 to 1405 EST)
[5] See the statement of the University of Düsseldorf titled Sitzung des Fakultätsrats der Philosophischen Fakultät.  It is available online at: http://www.uni-duesseldorf.de/home/startseite/news-detailansicht/article/sitzung-des-fakultaetsrats-der-philosophischen-fakultaet.html.  For an unofficial translation and further information on this situation see: http://copy-shake-paste.blogspot.ca/2013/01/dusseldorf-university-to-open-formal.html .
[6] The A1-E5 methodology was originally developed by the intelligence community. For more on this method see Thomas Quiggin, Seeing the Invisible - National Security Intelligence in an Uncertain Age. Singapore: World Scientific Press, 2007, pp.170-173.  This method has expanded into popular use and multiple variations exist.
[7]  Various versions and adaptations of this methodology exist.  See a slightly different version of this model in the brief article The Value of Information which is available online at: http://www.abchs.com/pdf/board/the_value_of_information.pdf .  This is the website of the American Board for Certification in Homeland Security.
[8] For an overview of issues raised by a newspaper having an agenda, see the article by Dante Chinni, a senior associate at the Project for Excellence in Journalism, Columbia University, Graduate School of Journalism titled The Other Paper as published by the Columbia Journalism Review, 1September 2002.
[9] These are the same basic topic areas that were addressed in the Federal Court of Canada when the court examined the reliability of information found in the World Wide Web.  See the decision in the 2009 case of Hassan Almrei at paragraph 340 where the court addresses the issues of authority, accuracy, objectivity, currency and coverage.  The decision is available online at http://web.archive.org/liveweb/http://decisions.fct-cf.gc.ca/en/2009/2009fc1263/2009fc1263.html (downloaded on 29 January 2013 at 1211 EST).
[10] For more about Jim Kapoun see http://antiochcollege.org/academics/faculty/jim-kapoun.html
[11] Among others, see the 2009 Federal Court of Canada case of Hassan Almrei which is available at http://web.archive.org/liveweb/http://decisions.fct-cf.gc.ca/en/2009/2009fc1263/2009fc1263.html (downloaded on 29 January 2013 at 1211 EST).  In particular, see the comments by the judge on the testimony of Dr Lisa Givens (library science) in paragraphs 336 to 348. 
[12]  Jim Kapoun,"Teaching undergrads WEB evaluation: A guide for library instruction." C&RL News (July/August 1998): 522-523.



Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

ENHANCING SECURITY THROUGH COLLABORATIVE RESEARCH

Perspectives on Terrorism is  a journal of the Terrorism Research Initiative and the Center for Terrorism and Security Studies

ISSN  2334-3745 (Online)

Disclaimer, Terms and Conditions