Assessing information quality: sorting the wheat from the CRAAP

The rapid expansion of the internet has enabled users to access unprecedented amounts of information.  However, not all of this information is valid, useful or accurate. In the world of ‘post-truth’ and ‘fake news’, the ability to critically assess information and its source is an essential skill.  Let’s consider what this means for public policy.

Growth of evidence-based policy

The need to assess research evidence is no longer limited to academics and scientists. The shift towards evidence-based decision-making means that policy makers and decision makers at every level now need to incorporate evidence into policy and practice.

The growth of the randomised controlled trials movement in public policy also reinforces the need for decision makers to be familiar with a range of research approaches.

In the UK, the What Works Network and the Alliance for Useful Evidence both work towards encouraging and improving the use of evidence to improve public services. Similarly, the Evidence Matters campaign seeks to promote the importance of evidence in policymaking, and tackle the misuse of research findings.

Publication doesn’t guarantee quality

While most of us are aware of the risks of encountering ‘fake news’ online, relying uncritically upon Google as an information source can leave one falling foul of ‘predatory’ open access journals, which masquerade as legitimate, peer-reviewed publications.

In recent years, there has been a boom in articles being published open access. There are now a vast number of good quality, open access publications in just about every subject imaginable.  Overall, this has been a positive development – who can argue with making more research free and easily accessible?

Open access not only has an ethical dimension – in many situations, it is also an obligation.  The UK government has already committed to ensuring that all publicly funded research is made available via open access.

However, the proliferation of open access material has led to a new problem – that of predatory open access journals. These journals operate using a business model that involves charging publication fees to authors, without providing the editorial and publishing services associated with legitimate journals. They may even include fake editors or members of the editorial board. Librarian and researcher, Jeffry Beall, has compiled a rather impressive list of ‘potential, possible, or probable predatory scholarly open-access publishers’.

The quality of articles published in predatory journals is therefore questionable. Recent (rather entertaining) examples of how unreliable such journals can be include the neuroscientist who managed to trick a number of scientific journals into publishing a nonsensical piece of research complete with a number of Star Wars references, including the authors Dr Lucas McGeorge and Dr Annette Kin, and the article reporting the case of a man who develops ‘uromycitisis poisoning’, inspired by a 1991 episode of Seinfeld (the actual article is still online).

Is this information CRAAP?

So how do you assess the quality of a piece of information?

One way to do this is to ask yourself – is this information CRAAP? The CRAAP test was developed by Meriam Library at California State University to help students think critically about the sources of information they had identified.  Some of the key questions to consider when evaluating information sources are:

  • Currency
    • When was the information published?
    • Has it been revised or updated?
  • Relevance
    • Who is the intended audience?
    • Is the information at an appropriate level (i.e. not too advanced/basic?)
    • How well does the information relate to your topic?
  • Authority
    • Who is the author/publisher/source/sponsor?
    • Is the author qualified to write on this subject?
  • Accuracy
    • Is the information supported by evidence?
    • Has the information between reviewed or refereed?
    • Are there any spelling/grammatical errors?
  • Purpose
    • Is the information fact, opinion or propaganda?
    • Is it objective and impartial?
    • Are there any religious/political/cultural/personal biases?
    • Is it trying to sell a product?

How we can help

While the increasing availability of information via the internet and the growth of open access content has undoubtedly been a positive development, it goes hand in hand with the need for users to critically assess information sources.

Here in the Idox Information Service, we take pride in our database of high quality resources. Our Researchers select only the best quality resources from hundreds of verified sources to populate our database. Our research database has been recognised by the Alliance for Useful Evidence as a key tool within the UK.

Each week, we support policy and decision makers by providing the latest research and evidence on a range of public policy issues – both through our current awareness services, and through bespoke literature reviews. In doing so, we hope to contribute in our own small way to the wider drive to improve the use of evidence in public policy decision making.

As we have seen, the use of inaccurate or misleading information can have significant real-world consequences.  The need for authoritative, accurate and relevant research has never been greater.


Find out more about the Idox Information Service and our subscription services here.

Follow us on Twitter to see what developments are interesting our research team.

Everything you wanted to know about Open Access … 5 great blogs from #OAweek

OA pic2

Image: Flickr user Meredith Khan, via CC BY-NC 2.0.

By Morwen Johnson

This week has been Open Access Week – an annual global event which is a chance for the academic and research community to discuss the benefits of Open Access, and inspire wider participation in helping to make Open Access a new norm in scholarship and research.

We’ve a particular interest in how the open access agenda is evolving. As an information service, one of our aims is to support evidence-based policymaking and increase the availability and uptake of research within practitioner communities. We also have a lot of librarians and information professionals in our research team – so understanding what’s happening in the publishing world is something we follow as part of our own CPD.

With this in mind, here’s the five blogs we’ve seen this week which our team found most interesting …

  • Your questions answered on Open Access

The Conversation blog is one of our favourites for pithy, accessible commentary and insight from academics on hot topics. They invited readers to submit questions on Open Access and this blog article gives a great intro to some of the key issues.

  • Open for Collaboration

In the UK, JISC is one of the main organisations driving the response by higher education to the Open Access agenda (along with other initiatives such as SHERPA).

They’ve published a few interesting resources this week, including a guest blog from the Coalition for Networked Information looking at progress towards openness in scholarship and research around the world. It reminds us that “The movement towards openness is about much more than publishing. It includes being open about methods, tools, software, and data.

  • When sharing isn’t as open as it might seem

Another favourite blog of ours is LSE’s Impact of Social Sciences blog. The article “What does Academia.edu’s success mean for Open Access? The data-driven world of search engines and social networking” looks at the rise of research sharing platforms. The author, Professor Gary Hall from Coventry University argues that “posting on Academia.edu is far from being ethically and politically equivalent to using an institutional open access repository”.

  • Research use by parliament

This blog by Caroline Kenny The impact of academia on Parliament: 45 % of Parliament-focused impact case studies were from social sciences caught our eye as it ties in nicely with what we try to do at the Knowledge Exchange. (*Don’t get us started on how much public money is spent on evaluations where the findings just sit in a report and never feed back into either policymaking or delivery!)

The use of research within Parliament (for example, in evidence to Select Committees) is good to hear about but academics need to do more to engage different stakeholders.

Another academic exploring this issue is Mark Evans – see his June blog Evidence-Based Policy Making: What Westminster Policy Officers Say They Do and Why for more on this.

  • The limits of “open”: Why knowledge is not a public good and what to do about it

Finally if you’ve got a bit more time to spare then this video from the Centre for Information Science at City University features a lecture from Dr Cameron Neylon (Professor of Research Communications at the Centre for Culture and Technology at Curtin University).

A couple of quotes which struck us – “As we deepen, and harden, the shared sense of what is excellent work within a discipline, we necessarily fortify precisely those boundaries where the web could bring us into contact with differing conceptions, precisely those that might bring the most benefits.” And we have the opportunity to see “Openness as a process or practice, not a binary state of an object, something to work towards rather than something that we fail at achieving.

(* As it’s 58 minutes, you may prefer to read the draft transcript available on his own blog.)


Follow us on Twitter to see what developments in policy and practice are interesting our research team.