A ‘question led’ approach to REAs means that the REA question:
- Is the driver for all REA processes
- Is a statement that can be investigated rather than a subject area of interest
- Should be clear and answerable
- Should need an answer and be worth answering
An REA is capable of answering many different types of question; for the purposes of this Toolkit these have been split into ‘impact’ and ‘non-impact’ questions. This distinction is not ideal but reflects the fact that the most common REA question (and hence the one that methods are most developed for) is currently the impact or “what works?” question.
An REA can address more than one type of question, particularly in combining impact questions (what works) with implementation (what is required to make it work) and economic questions (what are the costs and benefits).
- Impact questions
- Non-impact questions
Impact questions
Reviewing methods are most developed for “What Works?” questions: for example, ‘Do Teen Courts reduce rates of juvenile re-offending’? Methods for finding, coding, quality appraising and synthesising such studies are well developed and available ‘off the shelf’ for you to use.
Methods for undertaking REAs are most developed for impact questions but an REA may still not be suitable for your specific impact question.
Answering impact questions through an REA relies on finding studies that have:
- investigated the population you are interested in;
- investigated the intervention that you are interested in;
- used a suitably rigorous method (i.e. they have at least used a control group); and
- measured (quantitatively) the outcomes that you are interested in.
If these types of studies do not exist in the area that you are interested in the findings from the REA are likely to be inconclusive.
The narrower the question (for example, in terms of how specific the population or intervention is defined) the more you may limit the available evidence. However, there is a trade off to be made because making your question broader is likely to take up more resources.
Non-impact questions
REA methods for answering non-impact questions are less developed than for impact questions. This is largely because systematic reviews have focused on synthesising evidence from experimental and quasi-experimental research, and because there is a greater consensus on the hierarchy of available evidence on “what works?” (i.e. Randomised controlled trials at the top and simple before and after studies at the bottom).
REAs, however, are still suited to answering a range of other questions, which can be grouped as:
- ‘Needs’ questions
What do people want or need? - Process questions
Why/how does it work? - Implementation questions
What is required to make it work? - Correlation questions
What relationships are seen between phenomena? - Attitude questions
What do people think? What are their experiences? - Economic questions
How much does it cost and with what benefit/harm?
PICO
The first stage in devising either impact or non-impact questions is to think through the question that you want to ask. One method for doing this is to use the PICO acronym:
- Population
- Intervention
- Comparison
- Outcome
Before conclusions can be drawn from the studies that have been selected for inclusion in an REA, they need to be critically appraised to ensure that they are both relevant and that their findings are reliable.
The quality of the studies is likely to vary considerably and therefore the REA team must decide on clear explicit criteria for critically appraising the studies to separate those of higher quality from the weaker ones. There are three main dimensions considered in quality and relevance appraisal of studies (Gough 2007). These are:
- the methodological quality of the study being considered;
- the relevance of that research design for answering the REA question; and
- the relevance of the study focus for answering the REA question.
For REA questions that are not looking for answers on “what works?” the first thing to consider is whether all three criteria have been considered.
Often methodological quality (A) is assessed but not whether the method is actually relevant to answering the REA question. It is important to remember that non-impact questions can be answered using quantitative and qualitative data (for example, implementation questions can use data from surveys)
The following are the key questions to ask under each dimension.
- The particular issues that are considered under each dimension. For example, how is methodological quality going to be defined?
- The criteria used to make judgements about these.
- The cut off points for making decisions based on these judgements.
- Whether the judgement is for exclusion or weighting: 1) studies are excluded from the REA if they fail the quality appraisal, 2) Studies are ‘weighted’ according to their quality and relevance The stage of the REA at which the decision is made: quality appraisal can be carried out at different stages of the REA: screening; data extraction; and as a part of synthesis.
Those studies that remain in the REA after the critical appraisal stage form a critically appraised map of evidence – a map of all the studies relevant to your question whose quality has been assessed.
Common to all types of REA questions is that there will be many assumptions within them that need to be made explicit. These form the conceptual framework for the REA and underpin the:
- criteria for including or excluding studies;
- terms for searching for relevant literature;
- terms for sorting/describing literature;
- detailed questions to ask of each research report;
- framework for the synthesis stage of REA.
The REA team, ideally including users, would start unpacking the question by defining each of the key concepts.
The process of doing this is specific to each REA question but the example illustrates how this can work in practice.
Conceptual framework example
REA question: “How do parents influence the impact of homework on attainment in primary school children?”
The REA team would need to consider:
- The meaning and nature of homework, attainment, parental involvement, students and theories about relationships between them
- ‘Measures’ of all of these concepts
- The study types that can help address these questions
They would need to consider further sub questions:
- What are the effects on student attainment of initiatives that teach parents how to help with homework?
- How do different kinds of parental involvement in homework influence student attainment?
- Through what mechanisms does parental involvement in students’ homework affect student attainment?
To do so, the researchers would start by defining what kinds of homework they were interested in, and which kinds of parent and child. They might also specify a focus on research done in the UK and say that they were only going to look at homework for children in primary schools.
They might also state, at the start of their work, which kinds of research study would be most appropriate for answering this question. This could, for example, specify that both written questionnaire and more in-depth interview studies were relevant as long as parents, or children of a given age, were asked to describe and/or reflect upon the homework they had come into contact with.
In setting out their conceptual framework, the researchers might be very specific about what aspects of homework are of interest, maybe taking a particular stance about how children are likely to learn and saying that they are interested primarily in the ways in which parents and children talk about homework in terms of problem solving. In contrast, the researchers might say they were planning to use a more flexible, open frame and state that they aimed to see which ways of looking at homework are voiced by the parents and children themselves.
Understanding of the key concepts within the REA question may vary according to the experience, views and values of individuals and groups involved in the REA process. These different preconceptions, concerns and interests will influence the methods and conclusions of the REA. For that reason, it is fundamental to include REA ‘users’ in the construction and development of the REA.
A priori reviews
A priori reviews use a protocol at the beginning of the project to set out how each stage of the project will be addressed. These stages are set out in the protocol and then followed in a logical sequence.
Key features of a priori reviews
- Tend to be quicker than iterative reviews because they do not involve making changes to and/or redoing certain stages of the review.
- The process for answering the review question is clearly stated and allows other researchers and research users to suggest improvements to the methods at the beginning of the project.
- Reviewers are less likely to be influenced by, for example, their knowledge of study authors or by study findings if the review’s methods are defined explicitly at the start of the review.
- A priori reviews are less suited to broad questions or unclear conceptual framework because there is less scope for revisiting the questions and its assumptions once the review is underway.
The fact that a priori reviews tend to be faster than iterative reviews makes them attractive for use in REAs. However, REAs may need to make adjustments and refinements at different stages of the project in order to ensure they are still meeting users needs. Regular communication between all those involved is, therefore, vital to ensure the original approach is still valid.
Iterative reviews
An iterative review may have a general plan for how it will be carried out, but the detail is only worked out as the REA is being done. Particular stages may be repeated and developed as the review proceeds.
Key features of iterative reviews
- Able to repeat some stages of review (e.g. synthesis of findings) to explore different approaches to the evidence.
- Better at dealing with broad questions or reviews where the conceptual framework is unclear than a priori reviews because the questions and its assumptions can be revisited once the review is underway.
- Stages are less well defined and therefore may need to be repeated or developed. It is inadvisable in an REA to allow the process to be too iterative because this will limit the control that you have over how long it will take. On the other hand, users (e.g. policy officials) are given more of chance to ensure that each stage is still meeting their needs as the REA progresses.
- A protocol is still important but this will be adapted as the review progresses.
Therefore, iterative reviews are particularly useful in checking and refining the different stages of an REA to ensure it is delivering what is wanted. However, as they tend to be slower than a priori reviews, they so should be carried out by an experienced team who can balance the need to repeat or develop stages against the need to ensure the REA is delivered on time.
Examples of inclusion criteria
The inclusion criteria listed below are typical for REAs. The justification for why the criteria have been chosen should be included under each of the headings.
- Nature of what’s being studied
specific practices/approaches/outcomes/ other phenomena? Why? - Setting and population
e.g. school type, student age, adults, countries, why? - Date of research
ever, since 1920, since 1990, Why? - Research methods
all methods, only empirical, only certain designs? Why? - Language of report
e.g. English only, Norwegian only, both? Why?
How will it be rapid?
REAs are carried out more speedily than systematic reviews but the example inclusion criteria demonstrate that they need be no less rigorous when it comes to determining conceptual boundaries – which, in turn, determine which studies they will contain. In order to ensure the process is rapid constraints are imposed on the inclusion criteria.
- Limiting the REA to English language only studies
- Focusing on more recent literature only.
- Check with policy colleagues which issues are of urgent concern and ensure the REA only includes studies relevant to these. This can be useful in highlighting limitations that need to be made to the setting and population, and to what is being studied (e.g. outcomes). It also places an emphasis on the importance of identifying the policy need and involving users in the REA.
- Limit the REA to look at existing systematic reviews only. Clearly, this approach will only be effective if the areas have already been reviewed systematically, but for issues likely to result in a very large number of studies – particularly likely if the question crosses multiple topic areas – this may be the only way of obtaining an answer quickly.
This stage of the project will benefit from the help of a librarian or information specialist because they can advise on devising a search strategy. They should be able to provide guidance on defining search terms that will help balance sensitivity (the amount of literature that is found) and specificity (the amount of relevant versus non-relevant literature found).
The search strategy should include the following four elements. Under each element is an example of how the strategy was used by the Home Office to carry out an REA on evidence based approaches to gang violence.
1) What is being searched for (defined by the inclusion criteria)
The Home Office defined the inclusion criteria as follows.
- Nature of what’s being studied
The REA question was ‘What is effective in preventing or reducing young people’s involvement in gang and gun related activity, as victims or offenders’ - Setting and population
Young people - Date of research
Not studies before 1980 - Research methods
Only studies that were rated (by the REA team) to be point 3 or above on the “Maryland Scale. - Language of study reports
English only
2) The sources that will be searched (for example, books, journals, websites and electronic databases)
The Home Office used the following databases.
- Criminal Justice Abstracts
- ASSIA
- Acompline
- IBSS
- Psychological Abstracts
- Sociological Abstracts
- PAIS
- SIGLE
- ERIC
- Social Sciences – Full Text
- Inside Web
- Web of Knowledge
- Urbaline
- Planex
3) How these will be searched and the search terms that will be used within these.
The detail of the search terms that the Home Office used in their REA on evidence based approaches to gang violence is available on pages 40-41 of the final report (PDF).
This stage of the project is one of the most important to involve an information specialist or librarian who will be able to work with you to define the search terms in relation to your question and conceptual framework. They can also advise on important practical considerations for writing search strategies, such as, the use of ‘Ors’ and ‘Ands’, and special characters such as ‘*’. The Toolkit provides some guidance on this as well in ‘Searching help’.
4) The actual detailed methods of this strategy are written up in the final technical report so that others can see how the search was undertaken.
Downloads
Comprehensive
Comprehensive, or exhaustive searches aim to identify as much of the literature as possible that meets the inclusion criteria. The resulting set of studies should not be biased in any systematic way (any studies that were inadvertently missed should essentially be coming from the same pool of research as those that were found). Searches that are successful in identifying most of the literature are highly sensitive; those that identify mostly relevant studies and few not relevant ones are highly specific.
In order to identify as much relevant research as possible, systematic reviews emphasise sensitive searching. However, there is a trade-off between sensitivity and specificity with specificity decreasing as sensitivity increases. The more sensitive the search, the more time needs to be spent sifting out the irrelevant studies.
Given the time-sensitive nature of REAs, some sensitivity may have to be sacrificed in the knowledge that some potentially relevant research may be missed. However, the principle that the missing studies should not differ systematically from those included remains (i.e. it should remain representative of all possible research).
“What works?” questions
Comprehensive searches should be used for REAs that attempt to answer “What works?” questions. The model that underpins these REAs posits that it contains a random sample of all possible research in the area under investigation. Thus, while not all research may be identified, the sample is representative of the total population of potentially available research. The directions, and sizes, of observed effects are therefore taken to be a valid estimate of ‘real world’ phenomena.
Purposive
In full systematic reviews a purposive search strategy aims to to collect examples of research across a variety of types – for example across several disciplines. It is thus different in its theoretical model to exhaustive searching. The aim is not, for example, to gain an overall picture of the balance of benefit and harm of a particular intervention, but rather to build up a ‘mosaic’ of different perspectives or points of view. No one perspective is taken to be more important than another, simply because it is reported in more than one study, and therefore ‘conceptual saturation’ is a more helpful principle to bear in mind, than simply aiming to retrieve every study – if additional studies do not add any more detail to the mosaic. This approach can also be used in REAs.
However, purposive REAs are also used as a pragmatic response to providing evidence to policy makers within a limited timeframe. Where an REA does not have time or resources to search exhaustively decisions may have to be taken about which sources to prioritise to identify as much of the available literature as possible. The risks of this approach are substantial because of the likely bias introduced and the potential for missing important studies. Purposive REAs require both good subject knowledge, so informed decisions can be made about which sources to priorities and experience of conducting REAs. All these decisions need to be recorded and stated explicitly in the final report; acknowledging any biases.
Once you have defined your search strategy the next stage of the REA is to start searching for relevant literature. Many REAs will do these two processes iteratively; developing a search strategy and then refining it as they search depending on the quantity and relevance of studies found.
Where can you search?
The chart above lists the different sources that are available for searching as part of REAs. The three categories of sources: Electronic, print and ‘grey’ literature can all form part of an REA but concessions may have to be made in order to meet the timescales.
How will it be rapid?
The time pressures of REAs mean that the searching process needs to be carried out quickly. You should consider the factors below to get the most out of the time available and to recognise the limitations caused by searching rapidly.
Make searching achievable
Searching is an activity that can go on indefinitely. One can never be 100% certain that every available reference has been retrieved. And, for currency, it is always necessary to keep searching for new studies. However, in an REA, it is especially important to be aware of time limitations. Searches should be designed so that they are achievable within the time available. They should only be allowed to overrun in the event of it becoming clear that the validity of the REA would be threatened if they were to be curtailed. Any weaknesses in the searching should be made explicit, acknowledging that the timescales have made the searching less than comprehensive (and hence possibly biased).
Seek expert advice
The skills of the REA team are a potential limitation – both in terms of being able to search quickly and in terms of their capacity to operationalise complex questions. If you are conducting the REA in-house it is worth using the skills of specialists in information management, the subject area and REA/systematic reviews at the searching stage. Searching is a skilled and resource intensive process and drawing in outside help will assist the quality and speed with which you can retrieve relevant studies. The type of advice that you can get from, for example librarians, is:
- available sources to search;
- understanding a specific database’s coverage and controlled vocabulary (indexing); and
- learning how to manipulate different databases.
Limit the search
There are a number of ways in which the search can be limited (as stated in the inclusion criteria), shortening the time required for searching and for screening studies that were found.
- Limit the searching of grey literature. Systematic reviews will typically search a range of electronic, print and unpublished sources to get a comprehensive coverage of relevant studies. REAs do not have time to search as comprehensively but may need evidence from documents not recorded on electronic databases. Therefore, if resources are available, team members should work in parallel to search grey and electronic literature. Some grey literature should also be quickly available from policy officials and researchers. REAs can be undertaken using only electronic databases but this risks missing important evidence contained in other sources.
- Only search for studies written in English.
- Only search for newer studies. Limiting how far back in time you search for studies will reduce the volume of material that needs to be screened. There should be some justification for the time period chosen though (e.g. a particular political or policy timeframe that is relevant to your REA).
- The risk in limiting the search is the introduction of bias
Be aware of bias
Limiting the search may introduce bias into the REA. Certain studies are harder to find and will therefore be less likely to be included in the REA if the searching is limited. ‘Positive’ results (i.e. findings that report statistically significant impacts of an intervention) are more likely to be:
- published;
- published quickly;
- published in English;
- published more than once; and
- cited by others.(Egger M et al 1997)
Essentially the less comprehensive the search the more risk there is of bias. REAs sacrifice the comprehensive nature of searching in return for more rapid findings. The likelihood of an REA being biased and the causes of this should be acknowledged and discussed alongside the findings.
Make searching achievable
Searching is an activity that can go on indefinitely. One can never be 100% certain that every available reference has been retrieved. And, for currency, it is always necessary to keep searching for new studies. However, in an REA, it is especially important to be aware of time limitations. Searches should be designed so that they are achievable within the time available. They should only be allowed to overrun in the event of it becoming clear that the validity of the REA would be threatened if they were to be curtailed. Any weaknesses in the searching should be made explicit, acknowledging that the timescales have made the searching less than comprehensive (and hence possibly biased).
Seek expert advice
The skills of the REA team are a potential limitation – both in terms of being able to search quickly and in terms of their capacity to operationalise complex questions. If you are conducting the REA in-house it is worth using the skills of specialists in information management, the subject area and REA/systematic reviews at the searching stage. Searching is a skilled and resource intensive process and drawing in outside help will assist the quality and speed with which you can retrieve relevant studies. The type of advice that you can get from, for example librarians, is:
- available sources to search;
- understanding a specific database’s coverage and controlled vocabulary (indexing); and
- learning how to manipulate different databases.
Limit the search
There are a number of ways in which the search can be limited (as stated in the inclusion criteria), shortening the time required for searching and for screening studies that were found.
- Limit the searching of grey literature. Systematic reviews will typically search a range of electronic, print and unpublished sources to get a comprehensive coverage of relevant studies. REAs do not have time to search as comprehensively but may need evidence from documents not recorded on electronic databases. Therefore, if resources are available, team members should work in parallel to search grey and electronic literature. Some grey literature should also be quickly available from policy officials and researchers. REAs can be undertaken using only electronic databases but this risks missing important evidence contained in other sources.
- Only search for studies written in English.
- Only search for newer studies. Limiting how far back in time you search for studies will reduce the volume of material that needs to be screened. There should be some justification for the time period chosen though (e.g. a particular political or policy timeframe that is relevant to your REA).
- The risk in limiting the search is the introduction of bias
Be aware of bias
Limiting the search may introduce bias into the REA. Certain studies are harder to find and will therefore be less likely to be included in the REA if the searching is limited. ‘Positive’ results (i.e. findings that report statistically significant impacts of an intervention) are more likely to be:
- published;
- published quickly;
- published in English;
- published more than once; and
- cited by others.(Egger M et al 1997)
Essentially the less comprehensive the search the more risk there is of bias. REAs sacrifice the comprehensive nature of searching in return for more rapid findings. The likelihood of an REA being biased and the causes of this should be acknowledged and discussed alongside the findings.
Data collection
There are two main components of data collection in an REA: the location of studies and the description of them.
Locating studies
Electronic databases for locating studies include the Applied Social Sciences Index and Abstracts (ASSIA), Sociofile and the International Bibliography of the Social Sciences (IBSS). Deciding upon the most appropriate databases to search for a given REA is one of the tasks that are done as part of writing the protocol. The eight databases available on GSR desktops provide a good starting point for searching electronically available documents. Depending on your question and topic area, you will probably need to search more widely using additional databases.
Access to good library facilities is essential for the completion of a successful REA. Not only will the team need to access electronic databases that have expensive subscriptions (if their searches are not contracted out), but they will need to retrieve a large number of papers quickly. Quick inter-library loans make an REA proceed smoothly and, now that papers are increasingly available in electronic form, access to a secure document service – such as the one provided by the British Library – is also very helpful. It would be difficult to conduct an REA without these services and, thus, a good institutional base for the REA can be considered a pre-requisite for this work.
- GSR databases
- Searching databases effectively requires technical knowledge and skill. Searching help (below) provides guidance on how to get the most out of databases.
- Searching help
- A Quick Reference Card (PDF, CSA Illumina website) containing search tips and aids to understanding search terms.
- Online tutorials (CSA Illumina website) on quick and advanced searching.
- Not all databases operate in the same way and it is important to check the specific conventions of each database that you search. This applies to search terms used, the structure and content of Thesauruses, and symbols used for truncation. The advice of an information specialist should be sought when using different databases.
- Cochrane handbook
- The Cochrane Handbook (Cochrane website), aimed at a healthcare audience, contains detailed guidance on searching for systematic reviews. Many of the principles in the Cochrane Handbook can be applied to social science REAs.
Recording studies
Once studies have been located the references of these studies need to be recorded. The failure to maintain good records of searches is a common mistake. Since an REA needs to be able to account for every study it contains, it needs to record the timing, search strings used and outcome of every search. These records can be maintained using standard word-processing and spreadsheet applications: they simply require planning and consistent use.
Once hundreds (sometimes thousands) of titles and abstracts have been downloaded, however, reference management software is usually used. This software, such as EndNote and Reference Manager, can often import and format references downloaded from many bibliographic databases. These references can then be viewed one by one and decisions made regarding their suitability for inclusion in the REA. Reference management packages usually have the facility for researchers to record the outcome of screening decisions and other REA information, such as whether the full text of a reference is available.
Screening and selecting studies
Once studies have been identified through searching, the next stage is to screen them to check that they meet the inclusion criteria.
Screening is usually a two stage process, the first involves reviewing the abstracts and the second, reviewing the full studies.
Review abstracts
This involves reading the abstracts that have been found through the searching. Each abstract should be compared against the inclusion criteria and if the abstract meets the criteria then the full study should be read.
Not all studies will have abstracts and some will not contain information on all the inclusion criteria (this is particularly a problem with electronic searching). In these cases, decisions need to be made on whether or not to include the study on the information available.
In systematic reviews all references are typically screened by two people working independently, this is also desirable in an REA but where resources are particularly limited it can be carried out by a single person.
Review full studies
This involves reading the full report or article and comparing it against the inclusion criteria. The key information from each study needs to be coded using the data extraction form.
Searching electronically for studies can produce many non relevant studies (meaning the search is low on specificity). Screening out these non-relevant studies using abstracts can be very labour intensive. Once this process is complete, screening full reports take a considerable length of time.
Tips for keeping screening rapid
- Invest some time in piloting the screening criteria to ensure the validity and reliability of their application and to ensure that members of the REA team are clear about how to apply it. This will save time in the long run and reduce bias.
- Work closely with your librarian/information specialist. Obtaining full studies can be time consuming and the earlier they are involved in your REA and aware of the type of information you will need the better. It can take substantial resources for librarians to order papers, particularly if you are requesting large numbers, so agreeing the process for doing this in advance is essential.
- Include a time limit in your inclusion criteria for receiving full articles. Any that you receive after this time should not be included in the study.
- Consider using one person to screen abstracts (rather than two people screening each reference). If you adopt this approach it is important to ensure, to begin with, that the team will be including and excluding studies in the same way – and thus, screening should be carried out as a team activity until it has demonstrated the necessary degree of consistency across individuals to begin to screen independently.
[accordion title="Describing studies"]
Describing studies involves systematically asking questions of each full study using the data extraction form, and then ‘coding’ the information collected. This allows:
For mapping: providing a description of each study to build up a map of the research field. This will provide an initial map of the available studies, prior to the critical appraisal stage.
For synthesis: providing information to enable synthesis such as how studies were undertaken for quality and relevance appraisal; the study findings; and reporting on aspects of individual studies (coding information for synthesis is sometimes called data extraction).
The coding strategy may specify that information is to be recorded using open text boxes or through pre-set categories to be ‘ticked’. In order to ensure rigour in extracting information for mapping and synthesis the process often involves two people working through the strategies independently. Differences between the two researchers are then resolved, resulting in a final, agreed version.
The information from each study needs to be recorded in a data extraction form. Studies are described in a standardised way in the data collection phase of the REA. Rather than developing a standardised strategy for each REA, it is more usual to adopt an existing one and make minor adjustments to it for the specific needs of the REA.
Below is an example of the data extraction form used by the EPPI-Centre for systematic reviews. The tool allows researchers to identify, extract and code information about each individual research study. The complete tool is too detailed to be used in an REA but can be adapted to suit your needs. You will need to decide which questions in the data extraction form are most important to collect for your specific REA. There is also an additional form with questions that can be used for describing economic studies. The questions can be put into an Excel spreadsheet to keep all the data from the studies in one document.
The quality of the studies is likely to vary considerably and therefore the REA team must decide on clear explicit criteria for critically appraising the studies to separate those of higher quality from the weaker ones. There are three main dimensions considered in quality and relevance appraisal of studies (Gough 2007). These are:
- the methodological quality of the study being considered;
- the relevance of that research design for answering the REA question; and
- the relevance of the study focus for answering the REA question.
Most impact questions will have incorporated these three dimensions as part of the inclusion criteria and so only studies of the specified quality and relevance will have been included.
Where this is the case a decision still needs to be made about whether to exclude or weight studies that fail the quality appraisal. If you decide to exclude studies then all those below the quality standard will simply be thrown out of the REA. The alternative is that studies are ‘weighted’ according to their quality and relevance but still included in the REA.
It is often dimension (A) on which studies are weighted according to their methodological quality (for example whether they used a control group that was randomly assigned). It is important to consider how successfully the method was applied and reported in assessing methodological quality. For example, studies that have reported using Randomised Controlled Trials may have implemented them poorly or provided too little information on key parts of the method (such as the randomisations process) to be able to appraise the quality as high.
Those studies that remain in the REA after the critical appraisal stage form a critically appraised map of evidence – a map of all the studies relevant to your question whose quality has been assessed.
The quality of the studies is likely to vary considerably and therefore the REA team must decide on clear explicit criteria for critically appraising the studies to separate those of higher quality from the weaker ones. There are three main dimensions considered in quality and relevance appraisal of studies (Gough 2007). These are:
- the methodological quality of the study being considered;
- the relevance of that research design for answering the REA question; and
- the relevance of the study focus for answering the REA question.
For REA questions that are not looking for answers on “what works?” the first thing to consider is whether all three criteria have been considered.
Often methodological quality (A) is assessed but not whether the method is actually relevant to answering the REA question. It is important to remember that non-impact questions can be answered using quantitative and qualitative data (for example, implementation questions can use data from surveys)
The following are the key questions to ask under each dimension.
- The particular issues that are considered under each dimension. For example, how is methodological quality going to be defined?
- The criteria used to make judgements about these.
- The cut off points for making decisions based on these judgements.
- Whether the judgement is for exclusion or weighting: 1) studies are excluded from the REA if they fail the quality appraisal, 2) Studies are ‘weighted’ according to their quality and relevance The stage of the REA at which the decision is made: quality appraisal can be carried out at different stages of the REA: screening; data extraction; and as a part of synthesis.
Those studies that remain in the REA after the critical appraisal stage form a critically appraised map of evidence – a map of all the studies relevant to your question whose quality has been assessed.
The EPPI-Centre organises the three dimensions of quality and relevance into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. These judgements are combined into dimension D which signifies the overall WoE judgement. Then either the findings of lower quality studies are excluded, or given less weight in the synthesis. The WoE framework is questions N13-N16 in the data extraction form on the left.
The Maryland Scale of Scientific Methods (MSSM) was developed by Sherman and colleagues (1997) at the University of Maryland for their review of crime prevention interventions. It is a five-point scale for classifying the strength of methodologies used in “what works?” studies.
Although developed for the criminology field it has wider application; the five levels of methodological quality are generic and so can be applied to other areas of social science. Sherman and colleagues (1997) argue that only studies with a robust comparison group design can provide evidence that a programme has caused the reported impact. This equates to level three and above in the Maryland Scale. To see the scale itself click on the symbol on the left. For more background information on the scale see Sherman et al, 1997 (PDF, NCJRS website).
Maryland scale
| Level 1 | Observed correlation between an intervention and outcomes at a single point in time. A study that only measured the impact of the service using a questionnaire at the end of the intervention would fall into this level. |
| Level 2 | Temporal sequence between the intervention and the outcome clearly observed; or the presence of a comparison group that cannot be demonstrated to be comparable. A study that measured the outcomes of people who used a service before it was set up and after it finished would fit into this level. |
| Level 3 | A comparison between two or more comparable units of analysis, one with and one without the intervention. A matched-area design using two locations in the UK would fit into this category if the individuals in the research and the areas themselves were comparable. |
| Level 4 | Comparison between multiple units with and without the intervention, controlling for other factors or using comparison units that evidence only minor differences. A method such as propensity score matching, that used statistical techniques to ensure that the programme and comparison groups were similar would fall into this category. |
| Level 5 | Random assignment and analysis of comparable units to intervention and control groups. A well conducted Randomised Controlled Trial fits into this category. |
The Agency for Healthcare Research and Quality is the health services research arm of the U.S. Department of Health and Human Services (HHS). They have nearly 40 scales (each with 3 to 34 items) for appraising Randomised Controlled Trials.
Resources for appraising economic studies
Although there has been considerable development of economic evaluations in the health field, there is a lack of integration of economic with other social science evaluations and a lack of research based ‘best-practices’ for appraising such studies . The Weight of Evidence questions in the template below should help with appraising economic studies and it can be incorporated with the appraisal forms for qualitative and quantitative studies.
Weight of evidence
The EPPI-Centre organises the three dimensions into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. There are different ways of combining these judgements. Whichever is used, dimension D is used to signify the overall WoE judgement. Then either the findings of lower quality studies are excluded, or they are given less weight in the synthesis. The WoE framework for economic evaluation is questions H3-6 in the data extraction form.
Resources for appraising qualitative studies
Appraising the quality of qualitative research is a contested field, not least because there is little agreement as to the difference the use of different methods make and whether the nature of qualitative enquiry is itself amenable to rigid quality framework.
Weight of evidence
The EPPI-Centre organises the three dimensions of quality and relevance into a framework called ‘Weight of Evidence’ (WoE). Each study is weighted according to dimensions A, B and C in conjunction with each other. These judgements are combined into dimension D which signifies the overall WoE judgement. Then either the findings of lower quality studies are excluded, or given less weight in the synthesis. The WoE framework is questions N13-N16 in the data extraction form.
- Quality in qualitative evaluation (PDF, 748KB) - a framework created for GSR for assessing the credibility, rigour and relevance of individual qualitative research studies. It comprises 18 questions to assess studies and depending on the length of time you have to appraise studies this framework could be shortened further to meet your needs.
- 10 questions framework (PDF, PHRU website) - a framework of ten questions developed by Glasgow University to appraise the quality of studies that use a qualitative methodology. This is primarily aimed at the health field but the questions apply to other subject areas.
The synthesis stage will generate the findings to answer the REA question and enable conclusions to be drawn. This is likely to be the stage that users are most interested in, particularly policy officials, and so you should ensure that the output from the synthesis is going to be in a format that meet their needs. For example, drawing out the implications for policy could be important.
1:3:25
This approach seeks to balance the need to present the findings of the research in a form that is quick and accessible to read, with a requirement to demonstrate that the research has been conducted in a robust and reliable manner. Thus, short (sometimes in bullet-point form) presentations of results are written on a maximum of one page to provide the high-level implications of the research for government policy. A maximum of the three pages can be used for a more traditional Executive Summary with some information on the research itself but a focus on the findings and implications of the research. The aim of the 25 page (or less) report is to present the results in more context than is possible in the shorter formats, while still maintaining an accessible, jargon-free style. Detailed methods and tools (such as search and coding strategies) are sometimes appended as annexes to the 25 page report.
More than just publication
REAs are often designed to inform policy or operational development and simply producing a report may not be the best method for ensuring that the findings are used. Other strategies such as presentations and workshops should be considered to bring users back together to discuss the implications of the REAs findings. Events such as these can be used to create discussion about the next steps and action points to put the findings into practice.
Users can also be involved in communicating findings directly through the production of reports or talks following the REA.
- A rapid evidence assessment of the impact of mentoring on re-offending – Home Office (PDF, Home Office)
- Evidence Based Approaches to Reducing Gang Violence: A Rapid Evidence Assessment for Aston and Handsworth Operational Group – Home Office (PDF, 612KB)
- The evidence of harm to adults relating to exposure to extreme pornographic material – (Ministry of Justice)
- Rapid Evidence Assessment of Interventions that Promote Employment for Offenders (PDF, Department for Education and Skills)
- Interventions to improve the co-ordination of service delivery for High Cost High Harm Household Units – HM Treasury (EPPI-Centre)
- The effectiveness of interventions for people with common mental health problems on employment outcomes: a systematic rapid evidence assessment – HM Treasury (EPPI-Centre)