Planning a REA

This section discusses forming a team to carry out an REA. Drawing together people with the right skills to carry out an REA is vital. As well as the actual researchers who will be doing the REA there are also a wider group of users that need to be involved in the project if it is to be a success. The project manager is responsible for bringing together the REA team.

Expert Advice

If you have not undertaken an REA before, or are uncertain about applying the method to your policy question, it is important to get advice from someone with experience of doing REAs. This Toolkit will help you consider whether an REA is suitable and provide tools to complete the different stages but if you are new to the method the best way to learn how to do one is to undertake it with guidance from someone with experience of REAs.

Project manager

Given that an REA needs to be completed to a tight timetable, and there is usually a great deal of work to accomplish, good project management is essential. The manager of an REA should ideally have significant experience of working on previous REAs and/or other types of review. An REA is not a good project on which to learn how to manage any type of review because of the emphasis on efficient completion.

Role of the project manager

Generic good practice in project management, such as creating project plans and milestones, applies equally to REAs as it does to other research projects. With an REA, however, there is an added emphasis on getting off to a good start. The project manager needs to ensure that those involved have a clear idea of the direction of the REA. Good engagement with users will ensure that the various stakeholders of the REA have a common understanding of its purposes and this should be part of the process of writing the protocol for the REA. If the REA is being contracted out it is likely that the project manager will still need to identify and begin to engage users (rather than leave it to the contractor) in order to save time later on in the project and because of their research and policy knowledge in the area. They will also need to ensure that there is a dedicated project manager in the contracted out team whom they can work closely and regularly with.

The protocol is important because it not only describes the scope of the REA, but will usually also contain its anticipated timeline and significant milestones. Having this as a shared statement of intent means that those working on the REA, and those with an interest in it, understand their roles, what they are required to contribute, and when. It also helps to determine the scope and size of the initial searches: for example, if the search strategy is retrieving 10,000 titles and abstracts, and the project timeline states that the REA team only has capacity to screen 3,000, then the scope or sensitivity of the searches will need to be adjusted.

Given the importance of completing the REA rapidly, the manager of an REA needs to be in regular contact with those doing the work (if they are not, for example, conducting the searches themselves) so that they can be aware of emerging risks that might cause delays.

The first stage of an REA involves clarifying who is undertaking the work, for whom and to what purpose. There are four groups of people who are involved in the REA process:

• Researchers with expertise in conducting REAs, including an information specialist/librarian (the REA team)
• Users
• Commissioners (who may be a sub group of users)
• People with knowledge of the topic area in question (may or may not be part of the REA team)

The Team carrying out the REA will need to contain, or have access to, individuals with skills and knowledge in:

• Searching for research
• Designing a coding tool
• Access to topic area knowledge
• A variety of research methods
• Understanding / interpreting research
• Project management
• Information retrieval
• Information management
• Assessing the quality of research studies
• Research analysis / synthesis
• Writing-up / communicating findings to users

Training

There are a number of courses that provide systematic reviewing and critical appraisal training that would be transferable to working REAs.

Why involve users?

Different user communities will have different needs and priorities when asking research questions. In order for an REA to meet the needs of its stakeholders, it is vital that they are involved in its process. The questions policy officials would like answered by research may be different from those posed by teachers. Even though the tight timetable of an REA presents real challenges to meaningful involvement, the consequences of failing to take account of its users can lead to the production of an essentially irrelevant REA. User involvement therefore needs to be planned carefully from the outset but choices may need to be made about who the key users are and to tailor the REA to their needs.

Users play an important role in widening the range of perspectives and viewpoints that are incorporated into the REA question and the way an issue is interrogated. These can include:

• users playing an active role in developing and carrying out the day to day work on the REA;
• users acting as members of an REA group, advisory panel or focus group;
• users acting as peer referees for key REA outputs (i.e. protocol and draft final report).

How to involve users

Identifying who the key users are of an REA is the first step towards involving them. Some groups will be easy to identify: funders of the work are clearly a key user group. However, there may be other stakeholders that require investigation and time needs to be given both to find out who these people are, and to inviting them to become involved.

The level of involvement of users can vary according to:

• the type of user involved (e.g. policy official, practitioner, service user);
• the proactive or reactive involvement of users (e.g. whether taking an active role in developing the protocol or providing feedback on a protocol);
• and the degree of involvement: consultation, collaboration, user control. (Oliver et al 2004)

1. Developing the protocol

The beginning of the project is the most important time to involve users. Planning the scope of the REA in conjunction with its users is key because there is usually not enough time to change the scope once work has commenced. The scope of the work covers everything from the population, interventions and outcomes (if any) of interest, to the type of research that will be included. Users are, therefore, key in helping to formulate the question. Also important is agreement on the way the reliability and validity of studies will be assessed.

While some consultation can be carried out virtually (by email, teleconference etc), a face-to-face meeting at the beginning is essential to build up working relationships and understandings of common purposes.

Once the initial consultation has taken place, the outcomes of this process should be written up and placed in a protocol. As well as ensuring that the researchers are clear about their questions and research to be included, they can also be important devices for ensuring that both research teams and users have common understandings of the scope and purposes of the work, and of the type of outcomes expected from it.

How to do an rea: writing a protocol

2. After the ‘mapping’ stage

To some extent, all REAs are operating in the dark with regard to their likely outcomes. They are dependent on research that has already been carried out and the extent to which it is able to answer the REA question is unknown to begin with. It is essential to timetable a meeting during the course of the REA to take stock, and look at the research that has been identified. The most appropriate time for this to take place is after the field of research has been ‘mapped’. By this stage the broad scope of research that has been identified can be taken to a user group and form the basis of a discussion about what the priorities should be for the remainder of the REA.

‘Describing studies

3. Interpreting the findings

Once the research has been synthesised it can be useful to involve users again in interpreting its findings, pulling out the most significant conclusions and drawing up recommendations. Also, while some aspects of user involvement can be carried out virtually (such as feeding back on a protocol), interpreting the results of an REA is probably best carried out through face-to-face discussion.

Synthesis of findings – stages

4. Communicating the findings

Involving users in communicating the findings may lead to the REA’s findings being more relevant to them and other users of the research. User involvement in the development or use of communication materials may also increase the channels for disseminating the findings of the REA, giving them greater impact and reaching the individuals and groups to whom the findings are most relevant.

One method for communicating the REA’s findings is to develop user summaries written by the users. For example, in a series of reviews on educational research, teachers, parents, and school students wrote summaries of what the review findings meant for them (Harlen et al, 2002).

Communicating findings

Time is often the key factor that dictates the resources needed in an REA but the quicker it needs to be completed the more people and money it is likely will be needed. This section discusses these three resource elements and also provides guidance on the information management requirement for REAs

People

How many people are needed?

The number of people needed in the REA team and their roles will depend on the type of REA that you want to carry out and how quickly you want to complete it. Some teams will expect members to be involved in all stages of the REA whereas others will have people who specialise in different aspects (e.g. searching).

Who should be in the REA team?

The section on forming the team covers the types of skills people need to carry out an REA, including training for staff. However, these skills may not be available within your department or agency and there may not be time to carry out training prior to the REA starting. In this case parts of the REA can be carried out externally.

This often involves the main REA team specifying the scope of the REA and writing the protocol.

Stages of the REAs that might benefit from specialist expertise are:

• searching (in communication with the REA team).
• retrieval of references – once a list of citations has been arrived at, asking a library (or other information specialists) to retrieve the paper or electronic documents is the most efficient way to access the references.

Bringing in external expertise should be planned as early as possible in the project. Since the time between commissioning and commencing REAs is often short – days or weeks, rather than the months that are more common with large research projects – it can be difficult to recruit researchers quickly to work on an REA. For this reason, organisations which undertake this work, will either need to have staff in place who can be pulled into working on an REA at short notice, or a pool of more casual staff who are available to work as and when they are needed.

Time

It is not possible to say how long each stage of an REA will take, since there are so many possible variables. However, if an REA is given four months from its start to the completion of a draft report, the following timeline might be useful to consider:

Task	Time
Forming the team, consulting users, writing protocol	Month 1
Searching screening (and writing map in two-stage REAs)	Month 2
Data extraction and quality assessment	Month 3
Synthesis and writing the draft report	Month 4

Within this basic framework, the resources available will dictate the amount of work it is possible to complete in each stage. For example, with very clear definitions of inclusion/exclusion criteria and an experienced team, it is possible for one researcher to screen up to 1000 titles and abstracts per day. However, with less experienced researchers, or more complex/less well defined concepts, this can fall to less than 300. Therefore, the search strategy needs to take account not only of the number of references that can be managed, but also with regard to the ease with which the team is able to agree on whether or not a particular study meets the inclusion/exclusion criteria of the REA.

One other important area that can lead to delay is waiting for libraries to obtain the full studies that you have been screened for inclusion in the REA. Some papers can take a considerable time to arrive and so it is important to ensure you have a librarian/information specialist in the project as early as possible to manage the retrieval of information effectively (see screening and selecting studies)

Finance

The costs of an REA will depend on the type of REA that you are undertaking and the length of time that it needs to be completed in. Typical elements for a budget for an REA will include:

• Staffing: management, researchers and information specialist costs.
• Inter-library loan costs: these can be significant, since hundreds of papers can be needed at short notice.
• External advice: consultancy from a specialist in REA methods to ensure the work proceeds according to plan.
• Other communication costs: conference presentations
• Publication costs: copy editing, formatting, typesetting, printing.
• Software: costs of specialist reviewing software; software to run statistical meta-analyses (where necessary).
• Office and IT costs: the costs of maintaining and running the offices in which the team works.
• Topic specialist
• User involvement

Contracting out an REA

The cost of contracting out an REA will depend on the volume of literature, how dispersed it is, how easy it is to locate, how easily defined the intervention is, how far back the search needs to go, how quickly the REA needs to be done, how much input from experienced researchers is needed and how experienced the reviewer is. As an example, the average cost of an REA is around £40K. However the range is large: from £20K up to £70K.

Information management

Good information systems are essential for an REA to be completed successfully. There are two ways of organising information for use in an REA: using either a sequence of generic software packages, or specialist reviewing software.

Standard office software

Standard office software can be used to manage data generated from an REA, for example, using Excel to record searches. This option will cost less than specialist software since it is pre-installed on most PCs. However, the purchase of more specialised software can save time by making the REA more efficient.

Specialist software

There are specialist software packages for systematic reviews that are also suitable for REAs. Some are designed to manage the data through the entire project, whereas others perform specific tasks (such as synthesis). ‘Chaining’ a variety of software applications that perform specific tasks is a common means of managing REA data. It is effective, since it uses the strengths of each application, though it does require good documentation. On the right is a list of software applications and the tasks that they are able to perform.

• EPPI-Reviewer: web-based software that manages review data from recording search logs and screening references through to data extraction and a range of synthesis methods including meta-analysis and qualitative ‘thematic’ synthesis.
• SRS: web-based software that manages review data from screening through to data extraction. It has a comprehensive suite of administration functions for managing review productivity.
• Comprehensive Meta-analysis: as the name suggests, a tool for conducting meta-analyses with probably the widest range of statistical methods available from the software listed here. The Joanna Briggs Institute in Australia has a range of web-based software that can be used to manage qualitative, quantitative and economic data.
• RevMan: the Cochrane Collaboration’s free reviewing tool. Potentially useful if the REA will contain a meta-analysis. However, it is intended for limited data extraction and meta-analysis only.

Skills and resources

As with other types of research the advantages of contracting out the REA to external researchers relate to the specialist skills that can be bought in and the staff time that is saved.

There are also economy of scale advantages to contracting out work as specialist review researchers are already likely to have relevant software and access to journals and databases. Review software, in particular, may be costly and time consuming for government researchers to purchase.

Contract out stages of the REA

REAs need not be commissioned out entirely and it may be possible to just use external researchers to carry out stages such as searching. This usually involves the main REA team in specifying the scope of the REA and writing the protocol. The searches might then be conducted by a separate organisation in communication with the REA team. This allows the review team to utilise specialists with experience of searching databases and inputting search terms and saves their time in finding and logging thousands of search results.

The retrieval of references is also a task that does not need to be undertaken by the core team. Once a list of citations has been arrived at, a library – or other information specialists – can retrieve the documents.

Review or subject specialists

The REA team may have a choice of contracting out the work to specialist reviewers or to subject specialists. The advantage in commissioning specialist reviewers is there familiarity with managing REAs. While REAs are conceptually simple, they are difficult projects to manage successfully. They are also less likely to have vested interest in advancing a particular approach or cause bias because of preconceptions about authors or studies in the area.

However, specialist reviewers can lack subject specialists’ knowledge of the area being studied and the broader context that it sits within. Having this understanding is particularly important when formulating the question, unpacking the assumptions behind the question to develop the conceptual framework and setting inclusion/exclusion criteria. Specialist knowledge can be introduced through an advisory group to work with review specialists. These groups can provide invaluable advice at each key stage of the REA.

Not meeting the policy need

The key risk that needs to be controlled for when contracting out an REA is ensuring that it is able to meet the initial policy need and answer the REA question. You will be relying on external reviewers to search, screen, assess the quality of and then synthesise studies and communicate the findings.

This places a lot of emphasis on the start of the project clearly defining the question and the concepts that underpin it because these will be used to set the criteria for determining which studies are included, how they are coded and then synthesised. The contractor is also likely to be responsible for writing the final report and it is important to clearly specify the format, style and content of the report so that it meets the needs of the REA users.

These risks at each stage of the project may result in significant amounts of time spent reviewing the studies included, requesting clarification of coding and synthesis, and editing reports. This toolkit contains guidance on all the stages of the REA and having an understanding of these is important when commissioning out the REA.

Contract out stages of the REA

If you decide to only contract out the searching stage of the REA there are a number of risks to the project that need to be controlled. First, the process of searching and screening for literature within a team can help its members to become familiar with the literature and definitions of concepts. This familiarisation will need to take place at some point during the REA, and will therefore mean that misunderstandings are more likely when the team first ‘meets’ the literature – during data extraction.

Second, searching is often an iterative process, with search strategies being tested and refined before they are finalised. If searching is contracted out, there will need to be mechanisms in place for the REA team to review the searches and ensure that they are retrieving the correct literature, and are not inadvertently injecting bias into the REA.

As with other research projects the consideration of whether or not to contract out the research depends on a number of factors, including the availability of resources and skills to carry out the REA. If you are considering contracting out the REA to external researchers this section provides guidance on doing this effectively. You will also have departmental procurement rules that need to be followed.

Selling the benefits to contractors

REAs are conducted to tight timescales and in order for contractors to deliver against these roles and objectives should be clear from the outset. The specification for an REA will need to specify the outputs and purpose of the REA.

The work also needs to engage the interest of research organisations and contractors who you might want to undertake the work. They will often have to juggle existing workloads and priorities in order to undertake the REA, so they will need to be convinced of its value to them. For example, a not infrequent complaint of academics is the difficulty of getting permission to publish their findings. Since publications are of great importance to an academic’s career, they might be encouraged to bid for a piece of work if the commissioning brief states that they will be able to publish their work. You will need to consult your departmental rules on publication first, however.

Tender document

The tender document needs to clearly state the policy or operational need for the work, the context and timescale. Depending on your department’s procurement rules, you may also be able to give an estimate of the available funding and other resources to complete the REA. Contractors are then able to compete on what they are able to offer, given these constraints.

The amount of detail in the tender will depend on your needs but it should clearly state what stages of the REA you want the contractor to undertake (e.g. all of it, just the searching) and which will be done internally. You will also need to clearly specify the depth, breadth and type of review required. You can leave open those aspects of the work that may require further discussion between contractor and commissioner, such as the conceptual framework, but it is important to get agreement from your procurement team.

Involving users

The tender should outline the degree of involvement that you and other users expect to have in the project, including specifying how this should happen (such as, meetings at key stages). For example, reviewing the REA protocol developed by the contractor is one stage at which the REA question and methods might be fine-tuned. This will need to be planned and will require contractors to allocate resources, so needs to be described in the tender specification, or negotiated after commissioning.

The ‘User involvement’ section of this Toolkit offers a framework to help manage the process of stakeholder involvement but the process can be time consuming so needs to be specified in the tender.

• User involvement

Monitoring progress

There are various stages during an REA at which progress can be monitored. The most obvious point is when the studies in the REA have been described, and the team is able to take stock and think about which studies will be included in the final synthesis. It may be that the REA has not found any studies that are relevant. In this case, the decision may need to be taken to stop the work at this point – or for more sensitive searching to be carried out, in the knowledge that the final report may not be ready, or able to answer the question for which it was commissioned. In order for work progress and monitoring to run smoothly, agreement on what to do in particular circumstances should be reached at the beginning of the REA.

It is important that you obtain all the information that the contractor finds at the different stages of the REA (e.g. the completed data extraction forms that describe the individual studies in the REA). This not only provides a audit trail but is useful for future research and can be used to builds up a database of information in the area.

Review clauses

Review clauses in the REA contract set specific times in the project where the work can be stopped. The likelihood of ending up with an REA that finds a very limited number of relevant studies will vary from project to project. In cases in which it is possible, a two-stage contracting process might be advisable, with a ‘map’ commissioned in the first instance – to ensure that relevant research exists – and then a synthesis, once it has been demonstrated that there will be studies to synthesise. This helps to mitigate against the risk of preceding with an REA where there are very few relevant studies to include.

This Toolkit provides guidance on the full REA process but you will need to decide whether you have the skills and resources to undertake one in house. The key general advantages and disadvantages to doing this are listed below.

Advantages

Greater control over the project. Undertaking the REA directly allows greater control over the day to day work in the REA. However, even when contracting out work you will need to maintain good overall control of the project.

Familiarisation with the literature. Undertaking the REA, particularly screening and critically appraising the studies internally gives a deeper understanding of the evidence that is available in the area.

Subject knowledge. It is likely that you have a good understanding of the subject area you work in and whilst this may also be true of the contractor it is not certain. Subject knowledge is important, for example, it can be used to check whether the search strategy has picked up key texts.

Saving money by not having to pay contractors to do the work. This saving should be balanced against the cost of the staff hours needed to undertake the REA.

Developing a pool of expertise. The more REAs that are undertaken by GSR members the greater the expertise in government. As well as being able to advise other GSR members on how to do REAs, this should also strengthen the management of contracted out REAs.

Disadvantages

Resource intensive. REAs require significant staff time to meet short timescales. You will also need to be able to draw on a flexible pool of people who can do work at busier periods in an REA, such as screening all the abstracts that have been found.

Skills and experience. The tight timescales in REAs make it important to for someone in the team to have the necessary skills and experience undertake them. If you have never done an REA before it is useful to speak to an expert before starting the project. Even when you have the necessary skills and experience in house it is still necessary to be able to draw on other specialists such as librarians or information specialists.

Lack of specialist software. It is not essential to purchase specialist software to manage the REA process but if you do not you will need to spend time developing your own systems using standard packages (e.g. Excel).