The need for reviewing evidence
The increasing prominence of evidence-based principles has led to a need for Government departments and agencies to have ways of accessing, harnessing and using the best available research evidence for effective policy making – and the first step in this is to review what is already known. While existing evidence is clearly not the only source of information available to policy-makers, it is an obvious and vitally important strategy to determine what is already known about the issue in question.
This ‘knowledge economy’ that has developed in recent years, however, has created a number of problems in terms of potential volume and quality of existing work. As Davies (2003) notes:
The sheer amount of potential research evidence in most substantive areas of social science and public policy, coupled with the rapid growth of access to knowledge and information as a result of information technology, make it almost impossible to keep abreast of the research literature in any one area.
This problem of interpreting the results of a multitude of separate but similar studies has led to the creation of a method of knowledge management called reviewing. The ‘science of reviewing’ is not, however, a new phenomenon, and has been developing over the past 100 years or so. Nevertheless, notable developments towards a more systematic approach have occurred in the past 30 years.
Reviewing evidence properly takes time, but there are various techniques that can be used. This toolkit will help researchers to identify whether a Rapid Evidence Assessment is best for their needs, and help with the process of planning and carrying out a review.
Methods for reviewing evidence
There are a number of different methods for reviewing existing evidence. This section discusses six main types of review to help you decide which one is most appropriate for your research needs. Below is an overview of these methods and what they aim to do. The timings are only a rough guide and each project will be different depending on the resources available and how the method is implemented.
Time taken: 1 week to 2 months
Literature reviews collate studies that are relevant to a particular topic and summarise and appraise the research in order to draw conclusions from it. However, literature reviews do not explicitly set out how the studies will be found, included and analysed; their findings should therefore be treated with a great deal of caution.
When is it good to do a literature review?
Literature reviews can be useful for providing information on a specific topic in a very short period of time. It is feasible to carry out a literature review alone and is therefore a suitable method when time, money and access to external support are limited.
What are the disadvantages?
Literature reviews are prone to selection and publication bias; that is, they only tend to review evidence that is readily available, and they can be over reliant on sources that disproportionately report studies with positive outcomes. They are also not clear on methodology in that they do not state their inclusion or exclusion criteria (which specify the studies to be included and excluded in the REA); how they have appraised the research; or how conclusions have been reached.
The literature review was the dominant model until the late 1980s, when, in the health field, a number of studies were published which showed inadequacies and bias in the process. One study, for example, showed that successful treatment for heart attacks would have been identified earlier if systematic review methods had been used to summarise knowledge (Antman et al 1992). Illustrating that literature reviews are weak when the review question aims to synthesis findings on the outcomes from specific interventions.
Time taken: 1 week to 2 months
A scoping review can be used to determine the range of studies that are available on a specific topic.
This ‘map’ of the existing literature is undertaken with limited resources (particularly time); constrained by all or some of the following:
- Question: a delimited narrow focus (if a broad question then need to further limit search)
- Search: use few search sources (e.g. just one or two bibliographic database); use only key terms rather than extensive search of all variants; if there are many existing recent reviews, then consider a map of research in those reviews
- Screen: use only electronically available abstracts and texts
- Map: use only easily available sources; provide only simple description with limited analysis
Time taken: 2 to 6 months
REAs provide a balanced assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research. They aim to be rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process.
The speed at which this is undertaken will depend on how quickly the evidence is needed, the available resource to carry out the REA and the extent to which reviewers are prepared to limit the systematic review process.
How will it be rapid?
There are a number of aspects of the systematic review process that can be limited in an REA to shorten the timescale. There is no requirement for all stages to be limited.
- The REA question – if the question is broad the search needs to be further limited.
- Searching – consider using less developed search strings rather than extensive search of all variants; if there are many existing recent reviews, then consider a review of reviews rather than of primary studies.
- Screening stage – REAs can use ‘grey’ and print sources but less exhaustively than systematic reviews. It is possible to only use electronically available abstracts and texts but this is unadvisable because of the increased risk of bias.
- Mapping stage – if included at all; often has to be limited in terms of the breadth of the initial evidence map.
- Data extract only on results and key data for simple quality assessment.
- Simple quality appraisal and/or synthesis of studies.
When is it good to use an REA?
Rapid Evidence Assessments can be undertaken in the following circumstances:
- When there is uncertainty about the effectiveness of a policy or service and there has been some previous research.
- When a policy decision is required within months and policy makers/researchers want to make decisions based on the best available evidence within that time.
- At policy development stage, when evidence of the likely effects of an intervention is required.
- When it is known that there is a wide range of research on a subject but questions still remain unanswered.
- When a map of evidence in a topic area is required to determine whether there is any existing evidence and to direct future research needs.
- As a starting point. Ideally, one is undertaken to answer a particularly pressing policy concern, and once the immediate question is answered it can form the basis of a more detailed full systematic review. In such cases, a Rapid Evidence Assessment could be better described as an ‘interim evidence assessment’.
In these situations an REA can provide a quick synthesis of the available evidence by shortening the traditional systematic review process.
What are the disadvantages?
In shortening the traditional systematic review process REAs risk introducing bias. Systematic reviews also suffer from biases but limiting the process increases the risk of them occurring. For example, limiting the search to published literature may introduce bias into the REA because of the exclusion of unpublished material. Therefore, the need for the evidence to be provided rapidly should outweigh the risk of increased bias. REAs (and all other review methods) should record how they have been less comprehensive than a full systematic review and discuss the likely levels of bias this has caused so that those taking decisions are aware of limitations of the evidence.
All review methods, including REAs, risk generating inconclusive findings that provide a weak answer to the original question. For example, there may not be studies of sufficient methodological quality to address the question. The tight timescales in an REA mean that if findings are inconclusive there is less time than in a systematic review to go back and reformulate the question or inclusion criteria.
Time taken: 8 to 12 months minimum
A full systematic review is the combination of a full evidence map and synthesis. Systematic reviews of evidence have explicit objectives and studies are chosen on explicit criteria. A thorough search for studies is conducted using electronic and print sources and the grey literature (unpublished/work in progress) as well as hand searching journals and textbooks, searching of specialist websites, and use of personal contacts. Each study found is screened according to uniform criteria and the reasons for excluding studies clearly documented (Cochrane Collaboration 2007).
If you have already undertaken a systematic listing of existing research (full analytic map) a post map synthesis can be undertaken in about 4 months.
When is it good to use a systematic review?
Systematic reviews are the most robust method for reviewing evidence because they reduce the bias in the way studies are found, included and synthesised. Transparency allows future studies to be added to the review which enables a cumulative body of sound evidence to be developed on a subject area over time. Furthermore, it should be possible for anyone else to conduct the same review and come to the same conclusions.
There is a myth that systematic reviews are only suitable for experimental studies. While it is true that many of the existing systematic reviews have concentrated on large scale quantitative studies, systematic review methods can be applied to any type of question. Where quantitative studies are included it may be possible to undertake a meta-analysis – a method of analysis which combines statistically the results of a number of studies. If more qualitative data are used then a qualitative synthesis can summarise these results. Some reviews do both, and integrate a meta-analysis of data from controlled trials with a synthesis of findings from qualitative studies (Gough & Elbourne, 2002). This is also true for Rapid Evidence Assessments.
What are the disadvantages?
The strength of systematic reviews is that they aim to be as comprehensive as possible in the studies they find, include and synthesise. However, this leads to their principle disadvantage, which is that they are time consuming. They also require a team of researchers (either ‘in-house’ or contracted to carry out the work) and are therefore resource intensive (although likely to be less so than undertaking a piece of primary research).
Time taken: 12 months minimum
A multi arm systematic review involves undertaking the full synthesis as in a systematic review but with different sub questions looking at different types of evidence.
For example, a review of healthy eating that contained (i) a statistical meta analysis of quantitative experimental research on the efficacy of health promotion interventions aiming to increase fruit and vegetable intake by young people; (ii) a synthesis of qualitative data from research on young people’s views about eating and health; (iii) a synthesis combining the results from (i) and (ii).
Time taken: Often quicker than other types of full systematic review.
This method only includes existing reviews, preferably systematic, rather than primary studies.
It can be used where there has already been considerable research and a number of research reviews undertaken in a particular area. The limitation is that it will obviously not pick up research outside of existing reviews. Also, reviews are of variable quality and each will still need to be screened to assess how systematic and comprehensive it is.