Peer-reviewed Article About Research in a Healthcare Organization
PLoS Med. 2017 Aug; 14(8): e1002370.
Assessing the touch of healthcare inquiry: A systematic review of methodological frameworks
Samantha Cruz Rivera
Centre for Patient Reported Outcomes Research, Institute of Applied Health Inquiry, College of Medical and Dental Sciences, University of Birmingham, Birmingham, U.k.
Derek G. Kyte
Heart for Patient Reported Outcomes Research, Plant of Applied Health Research, College of Medical and Dental Sciences, Academy of Birmingham, Birmingham, Uk
Olalekan Lee Aiyegbusi
Centre for Patient Reported Outcomes Research, Establish of Practical Wellness Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, Britain
Thomas J. Keeley
Middle for Patient Reported Outcomes Research, Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom
Melanie J. Calvert
Middle for Patient Reported Outcomes Research, Plant of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, Great britain
Mike Clarke, Academic Editor
Received 2017 February 28; Accepted 2017 Jul seven.
- Supplementary Materials
-
S1 Appendix: Search strategy. (TIF)
GUID: 23CE9FD3-E6A4-495C-90E2-1F9B5F6E8431
S1 PRISMA Checklist: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. (PDF)
GUID: 5C9B41FC-D6BE-4850-B885-0DC6360154B1
- Information Availability Statement
-
All relevant data are within the newspaper and supporting files.
Abstruse
Groundwork
Increasingly, researchers need to demonstrate the touch on of their research to their sponsors, funders, and beau academics. However, the near appropriate way of measuring the impact of healthcare research is bailiwick to argue. We aimed to place the existing methodological frameworks used to mensurate healthcare research impact and to summarise the mutual themes and metrics in an impact matrix.
Methods and findings
Two contained investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the mutual concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Xx-4 unique methodological frameworks were identified, addressing five broad categories of impact: (1) 'primary research-related impact', (two) 'influence on policy making', (3) 'health and health systems bear upon', (iv) 'health-related and societal impact', and (v) 'broader economical bear on'. These categories were subdivided into sixteen common touch subgroups. Authors of the included publications proposed 80 dissimilar metrics aimed at measuring bear on in these areas. The chief limitation of the study was the potential exclusion of relevant articles, as a issue of the poor indexing of the databases searched.
Conclusions
The measurement of research impact is an essential exercise to assist direct the resource allotment of limited research resources, to maximise research do good, and to help minimise enquiry waste product. This review provides a collective summary of existing methodological frameworks for enquiry bear upon, which funders may use to inform the measurement of enquiry impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their inquiry.
Author summary
Why was this written report done?
-
There is a growing interest in demonstrating the touch on of research in order to minimise enquiry waste material, allocate resources efficiently, and maximise the do good of enquiry. However, there is no consensus on which is the most advisable tool to measure the impact of research.
-
To our knowledge, this review is the first to synthesise existing methodological frameworks for healthcare enquiry impact, and the associated bear upon metrics by which various authors have proposed touch should be measured, into a unified matrix.
What did the researchers exercise and detect?
-
We conducted a systematic review identifying 24 existing methodological research touch on frameworks.
-
Nosotros scrutinised the sample, identifying and summarising 5 proposed impact categories, 16 impact subcategories, and over eighty metrics into an impact matrix and methodological framework.
What do these findings mean?
-
This simplified consolidated methodological framework volition help researchers to sympathise how a enquiry study may give rise to differing forms of impact, likewise as in what ways and at which time points these potential impacts might be measured.
-
Incorporating these insights into the design of a study could enhance bear upon, optimizing the employ of research resources.
Introduction
In 2010, approximately U.s.$240 billion was invested in healthcare research worldwide [1]. Such research is utilised by policy makers, healthcare providers, and clinicians to make important prove-based decisions aimed at maximising patient benefit, whilst ensuring that express healthcare resources are used as efficiently equally possible to facilitate constructive and sustainable service delivery. It is therefore essential that this research is of high quality and that it is impactful—i.e., it delivers demonstrable benefits to society and the wider economy whilst minimising research waste [one,ii]. Enquiry impact tin be defined as 'whatsoever identifiable 'do good to, or positive influence on the economic system, society, public policy or services, health, the environs, quality of life or academia' (p. 26) [3].
There are many purported benefits associated with the measurement of inquiry touch, including the ability to (1) appraise the quality of the research and its subsequent benefits to society; (two) inform and influence optimal policy and funding allocation; (3) demonstrate accountability, the value of enquiry in terms of efficiency and effectiveness to the government, stakeholders, and society; and (iv) maximise impact through better agreement the concept and pathways to bear on [iv–seven].
Measuring and monitoring the impact of healthcare research has become increasingly common in the Great britain [5], Australia [five], and Canada [viii], as governments, organisations, and higher didactics institutions seek a framework to allocate funds to projects that are more probable to bring the about benefit to order and the economy [five]. For instance, in the UK, the 2014 Research Excellence Framework (REF) has recently been used to assess the quality and touch of research in higher didactics institutions, through the assessment of affect cases studies and selected qualitative touch metrics [nine]. This is the first initiative to classify research funding based on the economic, societal, and cultural impact of research, although information technology should exist noted that research impact only drives a proportion of this resource allotment (approximately xx%) [9].
In the UK REF, the measurement of research impact is seen as increasingly important. Notwithstanding, the impact chemical element of the REF has been criticised in some quarters [10,11]. Critics deride the fact that REF impact is determined in a relatively simplistic style, utilising researcher-generated case studies, which commonly attempt to link a detail inquiry upshot to an associated policy or wellness improvement despite the fact that the wider literature highlights great diverseness in the fashion research impact may exist demonstrated [12,13]. This led to the current debate about the optimal method of measuring impact in the futurity REF [10,xiv]. The Stern review suggested that enquiry impact should not merely focus on socioeconomic affect only should as well include bear upon on government policy, public appointment, bookish impacts outside the field, and pedagogy to showcase interdisciplinary collaborative impact [10,11]. The Higher Education Funding Council for England (HEFCE) has recently set out the proposals for the REF 2021 exercise, confirming that the measurement of such bear on will continue to form an of import function of the procedure [15].
With increasing pressure for healthcare research to lead to demonstrable health, economic, and societal bear on, there is a need for researchers to understand existing methodological impact frameworks and the means past which touch on may be quantified (i.e., impact metrics; see Box 1, 'Definitions') to better inform research activities and funding decisions. From a researcher'due south perspective, agreement the optimal pathways to impact can aid inform report blueprint aimed at maximising the impact of the projection. At the same time, funders need to understand which aspects of impact they should focus on when allocating awards and so they can brand the most of their investment and bring the greatest benefit to patients and lodge [ii,four,5,16,17].
Whilst previous researchers have summarised existing methodological frameworks and bear upon case studies [4,22–27], they accept not summarised the metrics for use by researchers, funders, and policy makers. The aim of this review was therefore to (1) identify the methodological frameworks used to measure healthcare research affect using systematic methods, (2) summarise common impact themes and metrics in an impact matrix, and (3) provide a simplified consolidated resource for employ past funders, researchers, and policy makers.
Methods
Search strategy and selection criteria
Initially, a search strategy was adult to identify the available literature regarding the different methods to mensurate research bear upon. The following keywords: 'Impact', 'Framework', and 'Research', and their synonyms, were used during the search of the Medical Literature Assay and Retrieval System Online (MEDLINE; Ovid) database, the Excerpta Medica Database (EMBASE), the Wellness Management Information Consortium (HMIC) database, and the Cumulative Alphabetize to Nursing and Centrolineal Health Literature (CINAHL+) database (inception to May 2017; see S1 Appendix for the total search strategy). Additionally, the nonindexed Periodical of Research Evaluation was paw searched during the same timeframe using the keyword 'Impact'. Other relevant articles were identified through 3 Internet search engines (Google, Google Scholar, and Google Images) using the keywords 'Impact', 'Framework', and 'Inquiry', with the offset 50 results screened. Google Images was searched considering different methodological frameworks are summarised in a single epitome and tin can hands be identified through this search engine. Finally, additional publications were sought through advice with experts.
Post-obit Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see S1 PRISMA Checklist), 2 contained investigators systematically screened for publications describing, evaluating, or utilising a methodological enquiry impact framework within the context of healthcare research [28]. Papers were eligible if they included full or fractional methodological frameworks or pathways to research affect; both master research and systematic reviews fitting these criteria were included. We included any methodological framework identified (original or modified versions) at the point of first occurrence. In addition, methodological frameworks were included if they were applicable to the healthcare bailiwick with no need of modification within their structure. Nosotros divers 'methodological framework' as 'a torso of methods, rules and postulates employed by a particular procedure or set of procedures (i.due east., framework characteristics and development)' [xviii], whereas nosotros divers 'pathway' as 'a way of achieving a specified result; a grade of action' [19]. Studies were excluded if they presented an existing (unmodified) methodological framework previously available elsewhere, did not explicitly describe a methodological framework only rather focused on a single metric (e.one thousand., bibliometric analysis), focused on the impact or effectiveness of interventions rather than that of the research, or presented example study data just. There were no language restrictions.
Data screening
Records were downloaded into Endnote (version X7.3.1), and duplicates were removed. Ii independent investigators (SCR and OLA) conducted all screening following a pilot aimed at refining the process. The records were screened past title and abstract before full-text articles of potentially eligible publications were retrieved for evaluation. A total-text screening identified the publications included for data extraction. Discrepancies were resolved through give-and-take, with the involvement of a 3rd reviewer (MJC, DGK, and TJK) when necessary.
Data extraction and assay
Data extraction occurred after the terminal pick of included articles. SCR and OLA independently extracted details of affect methodological frameworks, the country of origin, and the twelvemonth of publication, too as the source, the framework clarification, and the methodology used to develop the framework. Information regarding the methodology used to develop each methodological framework was besides extracted from framework webpages where bachelor. Investigators also extracted details regarding each framework'due south impact categories and subgroups, along with their proposed time to impact ('brusk-term', 'mid-term', or 'long-term') and the details of whatsoever metrics that had been proposed to measure impact, which are depicted in an touch on matrix. The structure of the matrix was informed by the work of Yard. Buxton and S. Hanney [2], P. Buykx et al. [5], Southward. Kuruvila et al. [29], and A. Weiss [30], with the intention of mapping metrics presented in previous methodological frameworks in a concise way. A consensus meeting with MJC, DGK, and TJK was held to solve disagreements and finalise the data extraction procedure.
Results
Included studies
Our original search strategy identified 359 citations from MEDLINE (Ovid), EMBASE, CINAHL+, HMIC, and the Journal of Research Evaluation, and 101 citations were returned using other sources (Google, Google Images, Google Scholar, and expert advice) (see Fig i) [28]. In total, we retrieved 54 total-text articles for review. At this phase, 39 manufactures were excluded, as they did not propose new or modified methodological frameworks. An additional xv articles were included following the backward and forward citation method. A total of 31 relevant manufactures were included in the final analysis, of which 24 were manufactures presenting unique frameworks and the remaining vii were systematic reviews [4,22–27]. The search strategy was rerun on xv May 2017. A further 19 publications were screened, and two were taken forward to full-text screening but were ineligible for inclusion.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.
Methodological framework characteristics
The characteristics of the 24 included methodological frameworks are summarised in Table one, 'Methodological framework characteristics'. Fourteen publications proposed academic-orientated frameworks, which focused on measuring academic, societal, economic, and cultural bear upon using narrative and quantitative metrics [2,3,v,8,29,31–39]. Five publications focused on assessing the touch on of enquiry by focusing on the interaction process between stakeholders and researchers ('productive interactions'), which is a requirement to reach research impact. This arroyo tries to address the outcome of attributing enquiry impact to metrics [7,twoscore–43]. Ii frameworks focused on the importance of partnerships between researchers and policy makers, as a core element to attain research impact [44,45]. An additional 2 frameworks focused on evaluating the pathways to impact, i.due east., linking processes between enquiry and touch on [30,46]. One framework assessed the ability of health technology to influence efficiency of healthcare systems [47]. 8 frameworks were developed in the UK [2,3,29,37,39,42,43,45], 6 in Canada [viii,33,34,44,46,47], iv in Commonwealth of australia [v,31,35,38], three in the Netherlands [7,40,41], and ii in the United States [thirty,36], with 1 model developed with input from diverse countries [32].
Table ane
Methodological framework characteristics.
| Framework | Original source for modified framework | Framework description | Impact categories | Methodology followed to develop the framework | |||||
|---|---|---|---|---|---|---|---|---|---|
| Literature review | Stakeholder involvement | Methods to incorporate stakeholder views | Pilot phase | ||||||
| Buxton and Hanney Payback Framework [two] Uk, 1996 | • The archetype/purist/noesis-driven model • The trouble-solving/engineering model • The political model • The enlightenment/percolation/limestone model • The interactive/social interaction model [48–54] | Framework used to assess health research touch through academic outputs and wider societal benefits | • Knowledge benefits • Benefits to future enquiry and research utilize • Political and administrative benefits • Wellness sector benefits • Broader economic benefits | Yes, but no systematic approach reported | None reported | None reported | A case study arroyo was undertaken to determine the effectiveness of the model and exemplify the affect categories | ||
| Canadian Institutes of Health Inquiry (CIHR) [8] Canada, 2005 | • Payback Framework [2] | Framework designed to judge the impact of health enquiry and the benefits of investing in health research | • Cognition product • Enquiry targeting and chapters • Informing policy • Health and health sector benefits • Economic impacts | Yes, merely no systematic arroyo reported | Meetings, presentations, and console discussions with the Canadian regime, academics, health research stakeholders, and international funding agencies | Not specified | No information reported in the main newspaper or on the webpage | ||
| Canadian University of Wellness Sciences (CAHS) [33] Canada, 2009 | • Payback Framework [2] • CIHR framework [8] | Framework focused on evaluating how research activity influences determination making in gild to improve wellness, the economy, and social benefits | • Advancing noesis • Building capacity • Informing decision making • Health • Broad socioeconomic touch on | Yep, just no systematic approach reported | Semistructured interviews were conducted to collect stakeholder, sponsor, and external skillful feedback | Conceptual cluster analysis was used to analyse the data collected, which helped to capture the perspective of different stakeholders involved in the health system | Half-dozen papers were commissioned to examination the applicability of the framework. After modifications, the project was sent to external review and changes were made until its approval | ||
| K. Graham et al. Alberta Innovates—Wellness Solutions (AIHS) touch framework [34] Canada, 2012 | • Payback Framework [2] CAHS model [33] | A framework to measure, assess, and illustrate the human relationship between research investment and affect. The ultimate goal is to contribute to the nigh societal benefit among the people | • Advancing noesis • Building capacity • Informing conclusion making • Health • Broad socioeconomic bear on • Organisational performance | No systematic arroyo reported | None reported | None reported | The CAHS model was assessed for applicability and feasibility through retrospective and prospective studies and other tools to map pathways to impact. The findings of the assessment provided feedback to improve the AIHS model | ||
| Yard. Cohen et al. [35] Australia, 2014 | • Payback Framework [2] • AIHS framework [34] | To evaluate the policy and practicebenefits of enquiry outputs, which take identify later and across the enquiry setting | • Scholarly outputs • Translational outputs • Policy or practice impacts • Long-term population outcomes | No systematic approach reported | Two online surveys and semistructured interviews were conducted among the primary chief investigators of the research grants included | Non specified | Case studies were summarised and presented to an feel panel, which scored the relevant impact categories of this framework | ||
| S. Kuruvilla et al. Research Impact Framework (RIF) [29] United kingdom, 2006 | • Payback Framework [2] | Conceptual framework to draw the possible impacts of health research outcomes | • Research-related impacts • Policy impacts • Service impacts: health and intersectorial • Societal impacts | No systematic arroyo reported | Semistructured interviews with principal investigators of selected projects | Thematic analysis was adopted to analyse the data. The categories of the framework were used equally themes | The RIF was validated through consistency with available wellness enquiry literature and empirical assay of research projects | ||
| L. Kalucy et al. Primary Health Intendance Inquiry & Data Service (PHC RIS) [31,55] Australia, 2007 | • Payback Framework [two] | Methodology designed to assess main healthcare enquiry. Potent collaboration, personal relationships and the participation of practitioners, health intendance managers, and policy makers in the definition of the research questions and in the inquiry procedure were identified as the strongest pathways to impact | • Knowledge production • Enquiry targeting, capacity, edifice, and absorption • Informing policy and product evolution • Health and health sector benefits • Broader economic benefits • Inquiry transfer | No systematic approach reported | Part ane: Phone interviews were conducted among the 4 chief investigators of the included projects funded past the National Health and Medical Enquiry Quango | Data provided by the interviewees were analysed following the Payback Framework and thematic analysis approach | Parts 2–17: chief investigators completed an online questionnaire to refine the methodology to measure enquiry impact | ||
| J. Guinea et al. Affect Oriented Monitoring (IOM) [32] Various countries, 2015 | • Payback Framework [2] | Methodology used to identify and appraise the impacts of health projects through a set of predefined categories | • Advancing knowledge • Capacity building and enquiry targeting • Informing decision making, practice, and policy • Population health and health sector benefits • Broadcasting and cognition transfer | No systematic approach reported | A project coordinators' survey was conducted to determine and collect all the possible public health research benefits. An finish users' survey was conducted to make up one's mind the usefulness and practicality of the results of the coordinators' survey to measure research impact. Additionally, a scoring matrix was developed to assess projection impacts | Not specified | A small sample of research projects was used to test some of the methods incorporated to measure inquiry impact | ||
| J. Lavis et al. Commutation model [44] Canada, 2003 | – | Assessment tool to mensurate decision-making impact of health enquiry. Impact measures are categorised according to the level of impact to be measured and the machinery of research uptake: producer-push, user-pull, and exchange measures | • Producer-push button process • User-pull process • Exchange procedure | No systematic approach reported | None reported | None reported | Ii examples were used to demonstrate how the assessment tool can be used | ||
| L. Meagher et al. [43] Uk, 2008 | • Linkage and exchange model [56] | Methodology for assessing research touch of policy and practice | • Primary knowledge producers • Noesis users, beneficiaries, brokers, and intermediaries • Impacts or research (outcomes) • Inquiry impact processes • Lessons learned and recommendations • Methods for identifying and assessing nonacademic enquiry impact | Yes, but no systematic approach reported | Laurels holders of the UK's Economic and Social Research Council (ESRC) inside the psychology field, heads of departments, ESRC-funded chief researchers, and research users were recruited to conduct a questionnaire and survey; 2 focus group interviews, semistructured interviews, media-related searches, and example studies to determine the level of engagement with research users, impact and processes, activities, and roles leading to impact | Non specified | None | ||
| P. Buykx et al. The Wellness Services Inquiry Touch on Framework [5] Australia, 2012 | • PHC RIS [31] • RIF [29] • Substitution Model [44] | Framework for recording and monitoring the impact of wellness research. This framework consolidates the most relevant elements of the PHS RIS, RIF, and exchange model | • Knowledge generation and communication • Capacity building, training, and leadership • Informing policy • Improving wellness and health systems bear upon • Social and economic do good affect | No systematic approach reported | None reported | None | Not available in the main paper | ||
| Higher Education Funding Council for England (HEFCE) Enquiry Assessment Practice (RAE) [39] Britain, 2005 | – | The aim of the RAE is to assess the quality of research conducted by academic institutions in the UK, in club to inform funding decisions by higher education funding bodies. | • RAE1: Staff information • RAE2: Enquiry output • RAE3: Inquiry scholarships • RAE4: Attractiveness for external funding • RAE5: Farther information on groups of research | None reported | Focus groups, workshops, and meetings with HEFCE institutions, funding body officers, external experts, and stakeholder groups | Non specified | None | ||
| C. Donovan Australian Research Quality Framework (RQF) [38] Substituted in 2015 past the Excellence in Research for Australia (ERA) framework Australia, 2008 | – | A five-signal rating calibration to evaluate enquiry excellence and societal returns of publicly funded inquiry. This framework highlights the importance of end-users' interactions to enhance the utilize of research | The bear upon rating calibration: • Research has produced an outstanding social, economic, ecology, or cultural benefit. • Enquiry has produced a significant social, economic, environmental, or cultural benefit. • Research has produced new policies, products, attitudes, behaviours, or outlooks in the end-user communities. • Research has engaged with the end-user community to address a social, economical, environmental, or cultural effect. • Enquiry has had express or no identifiable social, economic, environmental, or cultural outcome. | None reported | The technical working group on research bear upon (senior university managers, representatives from business and industry, experts in impact evaluation, and members of the development advisory group) and the Australian higher instruction sector participated during different phases of the development of the framework. The Australian college education sector was consulted. The technical working group was in charge of further evolution of the framework characteristics | Non specified | None | ||
| HEFCE Research Excellence Framework (REF 2014) [iii] United kingdom, 2011 | • RQF [38,39] | Framework for assessing the quality of enquiry of UK higher teaching institutions | • REF1: Staff details • REF2: Inquiry outputs • REF3: Impact template and case studies • REF4: Environmental data • REF5: Environmental template | Scoping written report, no systematic approach reported | An initial consultation on the REF was conducted amid HEFCE institutions and other stakeholders to determine the potential of introducing bibliometric indicators in the REF. A 2d consultation was conducted based on the results of the first initiative, which included proposals on how to assess the impact of inquiry | Non specified | A pilot practise was conducted to test and develop bibliometric indicators to measure research quality. A 2nd pilot was conducted to test the proposals to assess research touch on | ||
| R. Jacob and M. McGregor Health Technology Cess (HTA) System Assessment Framework [47] Canada, 2007 | – | The impact of wellness technology assessment is measured by the ability to influence the efficiency of the healthcare organization | Levels of importance according to the type of policy involved: • Level ane: General statements of ministerial policy • Level two: Planning guidelines for wellness services arrangement • Level 3: Do norms prescribed by the professional corporation • Level 4: Ministry rules apropos coverage of health services • Level 5: Hospital rules apropos utilisation of services • Level 6: Ministry building decisions on the system of specific wellness services • Level vii: Ministry decisions concerning the allocation of resource | No, a case written report approach was undertaken to determine the impact of health technology assessment on policy conclusion | Interviews and questionnaires were conducted with stakeholders affiliated with the Canadian Ministry building of Health; those who were involved in defining policy were contacted to corroborate the supporting documentation collected | Not specified | None | ||
| R. Landry et al. Research Utilisation Ladder [46] Canada, 2001 | • Knott and Wildavsky Model [57] | Assessment of the pathway in which research progresses towards its utilisation by decision makers and practitioners | • Transmission • Cognition • Reference • Endeavour • Influence • Awarding | Yeah, only no systematic approach reported | A postal service survey and phone calls were used to place potential participants. Once the participants were recruited, a questionnaire focused on utilisation cognition was distributed. The stakeholders involved were faculty members of different Canadian universities | Not specified | A modified version of the Knott and Wildavsky model was used to measure knowledge utilization from the information collected. A quantitative arroyo was adopted to analyse the data of the model | ||
| C. Sarli et al. The Becker Medical Library Model for Assessment of Research Impact [36] US, 2010 | • West. K. Kellogg Foundation Model [58] | Methodology beyond citation counts, to assess enquiry impact as a result of research output. This model allows the interaction between researchers and institutions to document and quantify the bear on of enquiry | • Research output • Noesis transfer • Clinical implementations • Customs benefit | Yes, but no systematic arroyo reported | Authors consulted good stance, researchers, clinicians, and librarians to identify indicators to measure research impact | Non specified | The evidence available to measure the touch on of the Ocular Hypertension Treatment Study was analysed using the preliminary framework. The outcomes refined the framework leading to a tool to assess research touch on | ||
| Five. C. Brueton et al. [37] Uk, 2014 | • The Becker Model [36] | Framework used to better study and assess touch of methodological research | • Advancement of knowledge • Implementation | No systematic approach reported | 2 semistructured interviews and email queries to the methodologists for the included projects were used to examine other indicators and analyse bear witness of research implementation | Non specified | None | ||
| Imperial Netherlands Academy of Arts and Sciences The social impact of applied research [41] Holland, 2002 | – | Measuring the societal bear upon of applied inquiry, as an incentive for researchers to better their performance within this field | • Science and certified knowledge • Teaching and training • Innovation and professionalism • Public policy • Collaboration and visibility | Yes, only not systematic approach reported | None | None | None | ||
| A. Weiss Logic Model for Medical Research [30] US, 2007 | • United Manner Model [59] | Measurement outcomes (sensation, implementation, and patient do good) provide the footing to assess the quality of the investment in research | • Initial upshot: awareness • Intermediate consequence: implementation • Long-term outcome: patient do good | No systematic approach reported | None | None | Qualitative methods and a case study were conducted to assess the bear on of the model adult | ||
| J. Canavan et al. [45] UK, 2009 | • Models of research impact: A cross sector review of literature and do [sixty] | This approach proposes 6 recommendations that can increment the likelihood of an effective and practical measurement of research impact policy | Recommendations: • Commissioner-researcher partnerships • Embedded researcher model • Planning for impact from the outset • Adopting facilitative research methodological strategies • Following expert practice towards inquiry impact • Connecting research measurement to performance management | Yes, but no systematic approach reported | None | None | A case study was developed to exemplify the framework proposed | ||
| J. Spaapen and L. van Drooge Social Impact Assessment Methods for research and funding instruments through the written report of Productive Interactions between science and society (SIAMPI) [xl] The netherlands, 2011 | – | Learning tool to better understand how research interactions lead to social affect. This approach focuses on productive interactions, especially exchanges betwixt researchers and stakeholders | Types of productive interactions: • Direct interactions • Indirect interactions • Financial interactions | Yes, only no systematic arroyo reported | The European committee, research organisations, science policy makers, inquiry councils, academies, and other bodies involved in research were involved in meetings and discussions | A instance study approach was used to test the framework. The feedback of the example report representatives was used to refine the framework [61] | Not specified | ||
| Chiliad. O. Kok and A. Schuit Contribution Mapping [7] The Netherlands, 2012 | – | An arroyo to monitor and evaluate contributions to determine how the utilisation of inquiry can contribute to better action for wellness. The method focuses on processes, actors, and efforts to heighten contributions and enable alignment of efforts. | Research related contributions categories: • Changes in abilities and deportment of involved and linked actors • Contributed cognition products • Contributions through linked utilisation • Indications of utilisation at a distance | Yes, but no systematic approach reported | Investigators of the projects were included. Potential key users and other potential informants were interviewed to typhoon the model and empathize contributions. Stakeholders were consulted to refine the model | Preliminary results were shared with stakeholders for feedback and validation. Once discrepancies were solved, results were shared with stakeholders | Not specified | ||
| Southward. Morton Inquiry Contribution Framework [42] Uk, 2015 | – | Assessment of research impact using contribution analysis to explicate the influence in policy and practice | • Concluding outcome • Policy or practice change • Chapters, noesis and skill • Sensation, reaction • Engagement, participation • Activities and outputs • Inputs | No systematic approach reported | Semistructured interviews were conducted among project partners that took part in the process of the research (i.e., conferences and workshops) and research users (i.east., practitioners) | A thematic analysis approach was used to analyse the information collected and refine the framework. The themes that emerged were tested against other sources | A example study was used to illustrate how the Inquiry Contribution Framework can appraise touch on | ||
Methodological framework development
The included methodological frameworks varied in their development procedure, simply there were some mutual approaches employed. Near included a literature review [2,5,7,8,31,33,36,37,twoscore–46], although none of them used a recognised systematic method. Most also consulted with various stakeholders [3,8,29,31,33,35–38,43,44,46,47] but used differing methods to incorporate their views, including quantitative surveys [32,35,43,46], face-to-face interviews [7,29,33,35,37,42,43], telephone interviews [31,46], consultation [three,seven,36], and focus groups [39,43]. A range of stakeholder groups were approached beyond the sample, including principal investigators [vii,29,43], research end users [7,42,43], academics [3,8,39,forty,43,46], award holders [43], experts [33,38,39], sponsors [33,39], projection coordinators [32,42], and primary investigators [31,35]. Withal, some authors failed to identify the stakeholders involved in the development of their frameworks [2,v,34,41,45], making it difficult to assess their appropriateness. In add-on, only 4 of the included papers reported using formal analytic methods to interpret stakeholder responses. These included the Canadian Academy of Health Sciences framework, which used conceptual cluster analysis [33]. The Research Contribution [42], Enquiry Impact [29], and Primary Health Care & Data Service [31] used a thematic analysis approach. Finally, some authors went on to airplane pilot their framework, which shaped refinements on the methodological frameworks until approval. Methods used to pilot the frameworks included a case written report approach [2,3,30,32,33,36,forty,42,44,45], contrasting results against available literature [29], the utilize of stakeholders' feedback [7], and assessment tools [35,46].
Major impact categories
1. Primary research-related impact
A number of methodological frameworks advocated the evaluation of 'inquiry-related bear on'. This encompassed content related to the generation of new noesis, knowledge broadcasting, capacity edifice, grooming, leadership, and the development of research networks. These outcomes were considered the direct or primary impacts of a enquiry project, every bit these are oftentimes the starting time evidenced returns [30,62].
A number of subgroups were identified within this category, with frameworks supporting the collection of impact data across the following constructs: 'research and innovation outcomes'; 'dissemination and knowledge transfer'; 'capacity building, grooming, and leadership'; and 'academic collaborations, research networks, and information sharing'.
one.1. Research and innovation outcomes. Twenty of the 24 frameworks advocated the evaluation of 'research and innovation outcomes' [2,3,five,7,eight,29–39,41,43,44,46]. This subgroup included the post-obit metrics: number of publications; number of peer-reviewed articles (including journal impact factor); commendation rates; requests for reprints, number of reviews, and meta-analysis; and new or changes in existing products (interventions or technology), patents, and research. Additionally, some frameworks also sought to gather information regarding 'methods/methodological contributions'. These advocated the collection of systematic reviews and appraisals in club to identify gaps in cognition and make up one's mind whether the knowledge generated had been assessed earlier being put into practice [29].
1.2. Dissemination and knowledge transfer. Nineteen of the 24 frameworks advocated the cess of 'dissemination and knowledge transfer' [2,three,5,7,29–32,34–43,46]. This comprised collection of the following information: number of conferences, seminars, workshops, and presentations; teaching output (i.e., number of lectures given to disseminate the research findings); number of reads for published manufactures; article download charge per unit and number of journal webpage visits; and citations rates in nonjournal media such as newspapers and mass and social media (i.e., Twitter and blogs). Furthermore, this impact subgroup considered the measurement of research uptake and translatability and the adoption of research findings in technological and clinical applications and by different fields. These can be measured through patents, clinical trials, and partnerships between manufacture and business, government and nongovernmental organisations, and university research units and researchers [29].
ane.3. Capacity edifice, training, and leadership. Fourteen of 24 frameworks suggested the evaluation of 'capacity building, preparation, and leadership' [2,3,v,8,29,31–35,39–41,43]. This involved collecting information regarding the number of doctoral and postdoctoral studentships (including those generated as a upshot of the research findings and those appointed to conduct the research), as well as the number of researchers and research-related staff involved in the research projects. In addition, authors advocated the drove of 'leadership' metrics, including the number of enquiry projects managed and coordinated and the membership of boards and funding bodies, periodical editorial boards, and advisory committees [29]. Additional metrics in this category included public recognition (number of fellowships and awards for significant research achievements), academic career advancement, and subsequent grants received. Lastly, the impact metric 'research system management' comprised the drove of information that can lead to preserving the wellness of the population, such as modifying enquiry priorities, resource allocation strategies, and linking health enquiry to other disciplines to maximise benefits [29].
ane.4. Bookish collaborations, inquiry networks, and data sharing. Lastly, 10 of the 24 frameworks advocated the collection of touch data regarding 'academic collaborations (internal and external collaborations to complete a research projection), research networks, and data sharing' [2,three,v,seven,29,34,37,39,41,43].
2. Influence on policy making
Methodological frameworks addressing this major impact category focused on measurable improvements within a given knowledge base and on interactions between academics and policy makers, which may influence policy-making development and implementation. The returns generated in this impact category are mostly considered as intermediate or midterm (1 to 3 years). These represent an of import interim phase in the procedure towards the last expected impacts, such as quantifiable health improvements and economic benefits, without which policy change may not occur [30,62]. The following touch on subgroups were identified inside this category: 'type and nature of policy impact', 'level of policy making', and 'policy networks'.
2.one. Type and nature of policy bear on. The most common impact subgroup, mentioned in 18 of the 24 frameworks, was 'type and nature of policy impact' [2,vii,29–38,41–43,45–47]. Methodological frameworks addressing this subgroup stressed the importance of collecting information regarding the influence of inquiry on policy (i.e., changes in practice or terminology). For instance, a projection looking at trafficked adolescents and women (2003) influenced the WHO guidelines (2003) on ideals regarding this particular grouping [17,21,63].
2.2. Level of policy bear upon. Thirteen of 24 frameworks addressed aspects surrounding the need to record the 'level of policy impact' (international, national, or local) and the organisations within a level that were influenced (local policy makers, clinical commissioning groups, and health and wellbeing trusts) [2,5,viii,29,31,34,38,41,43–47]. Authors considered it important to measure the 'level of policy impact' to provide evidence of collaboration, coordination, and efficiency inside health organisations and between researchers and health organisations [29,31].
2.three. Policy networks. Five methodological frameworks highlighted the need to collect information regarding collaborative research with manufacture and staff move between academia and industry [five,seven,29,41,43]. A policy network emphasises the relationship between policy communities, researchers, and policy makers. This relationship can influence and lead to incremental changes in policy processes [62].
3. Wellness and health systems impact
A number of methodological frameworks advocated the measurement of impacts on wellness and healthcare systems across the following impact subgroups: 'quality of care and service delivering', 'evidence-based practice', 'improved data and health data management', 'cost containment and effectiveness', 'resources allocation', and 'wellness workforce'.
3.1. Quality of intendance and service commitment. Twelve of the 24 frameworks highlighted the importance of evaluating 'quality of care and service delivery' [2,five,viii,29–31,33–36,41,47]. There were a number of suggested metrics that could be potentially used for this purpose, including wellness outcomes such as quality-adjusted life years (QALYs), patient-reported outcome measures (PROMs), patient satisfaction and feel surveys, and qualitative data on waiting times and service accessibility.
three.ii. Evidence-based practice. 'Evidence-based practice', mentioned in 5 of the 24 frameworks, refers to making changes in clinical diagnosis, clinical exercise, treatment decisions, or determination making based on research evidence [5,8,29,31,33]. The suggested metrics to demonstrate show-based exercise were adoption of health technologies and research outcomes to ameliorate the healthcare systems and inform policies and guidelines [29].
3.three. Improved information and health information management. This touch subcategory, mentioned in 5 of the 24 frameworks, refers to the influence of research on the provision of health services and management of the health system to prevent additional costs [5,29,33,34,38]. Methodological frameworks advocated the collection of wellness organization financial, nonfinancial (i.due east., send and sociopolitical implications), and insurance information in society to determine constraints within a health system.
3.iv. Cost containment and cost-effectiveness. Six of the 24 frameworks advocated the subcategory 'toll containment and toll-effectiveness' [2,5,viii,17,33,36]. 'Cost containment' comprised the collection of information regarding how research has influenced the provision and direction of health services and its implication in healthcare resource resource allotment and use [29]. 'Toll-effectiveness' refers to data concerning economic evaluations to assess improvements in effectiveness and health outcomes—for instance, the toll-effectiveness (cost and wellness outcome benefits) assessment of introducing a new health technology to supervene upon an older one [29,31,64].
3.v. Resource allotment. 'Resources allotment', mentioned in 6frameworks, can be measured through ii impact metrics: new funding attributed to the intervention in question and equity while allocating resources, such equally improved allocation of resource at an surface area level; better targeting, accessibility, and utilisation; and coverage of wellness services [2,5,29,31,45,47]. The allotment of resources and targeting can be measured through wellness services research reports, with the utilisation of health services measured by the probability of providing an intervention when needed, the probability of requiring information technology once more in the hereafter, and the probability of receiving an intervention based on previous feel [29,31].
3.6. Health workforce. Lastly, 'health workforce', present in 3 methodological frameworks, refers to the reduction in the days of work lost because of a particular disease [2,5,31].
four. Health-related and societal touch
Three subgroups were included in this category: 'health literacy'; 'health knowledge, attitudes, and behaviours'; and 'improved social equity, inclusion, or cohesion'.
4.1. Wellness cognition, attitudes, and behaviours. Eight of the 24 frameworks suggested the assessment of 'health noesis, attitudes, behaviours, and outcomes', which could exist measured through the evaluation of levels of public date with science and enquiry (e.m., National Health Service (NHS) Choices stop-user visit rate) or by using focus groups to analyse changes in knowledge, attitudes, and behaviour amongst guild [two,5,29,33–35,38,43].
four.2. Improved disinterestedness, inclusion, or cohesion and human being rights. Other methodological frameworks, 4 of the 24, suggested capturing improvements in disinterestedness, inclusion, or cohesion and human being rights. Authors suggested these could be using a resource like the United Nations Millennium Development Goals (MDGs) (superseded by Sustainable Development Goals [SDGs] in 2015) and human being rights [29,33,34,38]. For example, a cluster-randomised controlled trial in Nepal, which had female participants, has demonstrated the reduction of neonatal bloodshed through the introduction of maternity health care, distribution of commitment kits, and home visits. This illustrates how research tin target vulnerable and disadvantaged groups. Additionally, this research has been introduced by the World Wellness Organisation to achieve the MDG 'improve maternal health' [16,29,65].
iv.iii. Health literacy. Some methodological frameworks, 3 of the 24, focused on tracking changes in the ability of patients to make informed healthcare decisions, reduce health risks, and better quality of life, which were demonstrably linked to a detail programme of enquiry [5,29,43]. For example, a systematic review showed that when HIV health literacy/knowledge is spread amidst people living with the condition, antiretroviral adherence and quality of life amend [66].
5. Broader economic impacts
Some methodological frameworks, 9 of 24, included aspects related to the broader economic impacts of health research—for instance, the economic benefits emerging from the commercialisation of research outputs [2,5,29,31,33,35,36,38,67]. Suggested metrics included the amount of funding for research and development (R&D) that was competitively awarded by the NHS, medical charities, and overseas companies. Additional metrics were income from intellectual property, spillover effects (any secondary benefit gained as a repercussion of investing direct in a chief activity, i.e., the social and economic returns of investing on R&D) [33], patents granted, licences awarded and brought to the market, the development and sales of spinout companies, research contracts, and income from industry.
The benefits contained within the categories 'health and health systems bear upon', 'health-related and societal impact', and 'broader economical impacts' are considered the expected and last returns of the resources allocated in healthcare research [30,62]. These benefits commonly arise in the long term, across 5 years according to some authors, but there was a recognition that this could differ depending on the project and its associated research area [4].
Data synthesis
Five major impact categories were identified across the 24 included methodological frameworks: (1) 'master inquiry-related bear on', (two) 'influence on policy making', (3) 'health and wellness systems touch', (4) 'wellness-related and societal impact', and (v) 'broader economic impact'. These major touch categories were further subdivided into 16 impact subgroups. The included publications proposed 80 different metrics to measure research impact. This impact typology synthesis is depicted in 'the impact matrix' (Fig 2 and Fig three).
The impact matrix (1).
CIHR, Canadian Institutes of Health Enquiry; HTA, Health Technology Cess; PHC RIS, Primary Wellness Intendance Research & Information Service; RAE, Research Assessment Exercise; RQF, Research Quality Framework.
The impact matrix (2).
AIHS, Alberta Innovates—Wellness Solutions; CAHS, Canadian Institutes of Health Inquiry; IOM, Touch Oriented Monitoring; REF, Research Excellence Framework; SIAMPI, Social Impact Assessment Methods for inquiry and funding instruments through the study of Productive Interactions between science and club.
Commonality and differences across frameworks
The 'Research Impact Framework' and the 'Health Services Research Impact Framework' were the models that encompassed the largest number of the metrics extracted. The most ascendant methodological framework was the Payback Framework; seven other methodological framework models used the Payback Framework as a starting point for development [8,29,31–35]. Additional methodological frameworks that were commonly incorporated into other tools included the CIHR framework, the CAHS model, the AIHS framework, and the Substitution model [eight,33,34,44]. The capture of 'research-related impact' was the most widely advocated concept across methodological frameworks, illustrating the importance with which primary short-term impact outcomes were viewed by the included papers. Thus, measurement of impact via number of publications, citations, and peer-reviewed articles was the most common. 'Influence on policy making' was the predominant midterm bear upon category, specifically the subgroup 'type and nature of policy touch', in which frameworks advocated the measurement of (i) changes to legislation, regulations, and government policy; (ii) influence and involvement in decision-making processes; and (iii) changes to clinical or healthcare training, practice, or guidelines. Inside more long-term touch on measurement, the evaluations of changes in the 'quality of care and service commitment' were commonly advocated.
In light of the commonalities and differences among the methodological frameworks, the 'pathways to research impact' diagram (Fig iv) was developed to provide researchers, funders, and policy makers a more comprehensive and exhaustive way to measure healthcare research impact. The diagram has the reward of assorting all the impact metrics proposed by previous frameworks and group them into dissimilar impact subgroups and categories. Prospectively, this global picture will aid researchers, funders, and policy makers plan strategies to attain multiple pathways to impact before conveying the research out. The analysis of the data extraction and construction of the impact matrix led to the development of the 'pathways to research impact' diagram (Fig 4). The diagram aims to provide an exhaustive and comprehensive way of tracing research impact past combining all the bear upon metrics presented by the different 24 frameworks, grouping those metrics into different impact subgroups, and grouping these into broader affect categories.
Pathways to research touch.
NHS, National Health Service; PROM, patient-reported upshot measure; QALY, quality-adjusted life year; R&D, research and development.
Discussion
This review has summarised existing methodological bear upon frameworks together for the get-go time using systematic methods (Fig 4). It allows researchers and funders to consider pathways to affect at the design stage of a study and to understand the elements and metrics that need to exist considered to facilitate prospective cess of impact. Users practice non necessarily demand to cover all the aspects of the methodological framework, equally every research projection tin affect on different categories and subgroups. This review provides data that can assist researchers to better demonstrate bear upon, potentially increasing the likelihood of conducting impactful research and reducing research waste material. Existing reviews have not presented a methodological framework that includes different pathways to impact, wellness bear on categories, subgroups, and metrics in a unmarried methodological framework.
Academic-orientated frameworks included in this review advocated the measurement of touch predominantly using so-chosen 'quantitative' metrics—for example, the number of peer-reviewed articles, journal impact gene, and citation rates. This may be because they are well-established measures, relatively piece of cake to capture and objective, and are supported by research funding systems. However, these metrics primarily measure the dissemination of research finding rather than its touch on [xxx,68]. Whilst information technology is truthful that wider dissemination, particularly when delivered via world-leading international journals, may well lead eventually to changes in healthcare, this is by no means certain. For instance, case studies evaluated by Flinders University of Australia demonstrated that some enquiry projects with non-peer-reviewed publications led to meaning changes in health policy, whilst the studies with peer-reviewed publications did not consequence in any type of impact [68]. Every bit a result, gimmicky literature has tended to advocate the collection of information regarding a diverseness of dissimilar potential forms of touch on alongside publication/citations metrics [ii,3,5,7,eight,29–47], as outlined in this review.
The 2014 REF exercise adjusted Great britain university research funding allocation based on evidence of the wider impact of research (through example narrative studies and quantitative metrics), rather than only co-ordinate to the quality of enquiry [12]. The intention was to ensure funds were directed to high-quality research that could demonstrate actual realised benefit. The inclusion of a mixed-method arroyo to the measurement of impact in the REF (narrative and quantitative metrics) reflects a widespread conventionalities—expressed by the majority of authors of the included methodological frameworks in the review—that private quantitative impact metrics (e.thousand., number of citations and publications) practice not necessary capture the complication of the relationships involved in a research project and may exclude measurement of specific aspects of the research pathway [10,12].
Many of the frameworks included in this review advocated the collection of a range of academic, societal, economical, and cultural impact metrics; this is consistent with recent recommendations from the Stern review [x]. Yet, a number of these metrics run into inquiry 'lag': i.e., the time between the point at which the research is conducted and when the bodily benefits arise [69]. For instance, some cardiovascular enquiry has taken up to 25 years to generate touch [70]. Likewise, the bear on may not arise exclusively from a single piece of research. Unlike processes (such as networking interactions and noesis and research translation) and multiple individuals and organisations are often involved [4,71]. Therefore, attributing the contribution made by each of the different actors involved in the procedure can be a challenge [4]. An additional problem associated to attribution is the lack of evidence to link research and affect. The outcomes of inquiry may emerge slowly and be absorbed gradually. Consequently, it is difficult to determine the influence of research in the development of a new policy, practice, or guidelines [four,23].
A farther problem is that touch evaluation is conducted 'ex post', afterward the research has ended. Collecting information retrospectively tin can exist an issue, as the information required might not be bachelor. 'ex ante' cess is vital for funding allocation, as it is necessary to determine the potential forthcoming affect before enquiry is carried out [69]. Additionally, ex dues evaluation of potential benefit tin overcome the issues regarding identifying and capturing evidence, which can be used in the futurity [4]. In order to acquit ex ante evaluation of potential benefit, some authors propose the early on involvement of policy makers in a research project coupled with a well-designed strategy of broadcasting [40,69].
Providing an alternate view, the authors of methodological frameworks such as the SIAMPI, Contribution Mapping, Enquiry Contribution, and the Exchange model suggest that the problems of attribution are a result of assigning the bear on of research to a particular impact metric [7,40,42,44]. To address these issues, these authors propose focusing on the contribution of research through assessing the processes and interactions between stakeholders and researchers, which arguably accept into consideration all the processes and actors involved in a research projection [7,xl,42,43]. Additionally, contributions highlight the importance of the interactions between stakeholders and researchers from an early stage in the research process, leading to a successful ex ante and ex post evaluation by setting expected impacts and determining how the inquiry outcomes have been utilised, respectively [7,40,42,43]. However, contribution metrics are generally harder to measure in comparison to academic-orientated indicators [72].
Currently, in that location is a debate surrounding the optimal methodological impact framework, and no tool has proven superior to another. The about appropriate methodological framework for a given study volition probable depend on stakeholder needs, equally each employs dissimilar methodologies to appraise research bear upon [4,37,41]. This review allows researchers to select individual existing methodological framework components to create a bespoke tool with which to facilitate optimal study design and maximise the potential for touch depending on the feature of their study (Fig two and Fig 3). For instance, if researchers are interested in assessing how influential their research is on policy making, perhaps considering a suite of the appropriate metrics drawn from multiple methodological frameworks may provide a more comprehensive method than adopting a single methodological framework. In addition, research teams may wish to apply a multidimensional approach to methodological framework development, adopting existing narratives and quantitative metrics, as well equally elements from contribution frameworks. This arroyo would arguably present a more than comprehensive method of impact cess; however, further research is warranted to make up one's mind its effectiveness [4,69,72,73].
Finally, it became clear during this review that the included methodological frameworks had been constructed using varied methodological processes. At present, there are no guidelines or consensus around the optimal pathway that should be followed to develop a robust methodological framework. The authors believe this is an area that should be addressed by the research customs, to ensure future frameworks are developed using best-practice methodology.
For case, the Payback Framework drew upon a literature review and was refined through a case study approach. Arguably, this arroyo could exist considered inferior to other methods that involved all-encompassing stakeholder interest, such every bit the CIHR framework [8]. Nonetheless, seven methodological frameworks were developed based upon the Payback Framework [8,29,31–35].
Limitations
The present review is the first to summarise systematically existing touch on methodological frameworks and metrics. The main limitation is that 50% of the included publications were found through methods other than bibliographic databases searching, indicating poor indexing. Therefore, some relevant articles may not have been included in this review if they failed to indicate the inclusion of a methodological impact framework in their championship/abstruse. We did, however, make every effort to endeavor to detect these potentially hard-to-reach publications, e.g., through forward/backwards commendation searching, mitt searching reference lists, and expert communication. Additionally, this review merely extracted information regarding the methodology followed to develop each framework from the main publication source or framework webpage. Therefore, further evaluations may non have been included, every bit they are across the telescopic of the current newspaper. A further limitation was that although our search strategy did not include language restrictions, we did not specifically search non-English language linguistic communication databases. Thus, we may take failed to identify potentially relevant methodological frameworks that were developed in a non-English linguistic communication setting.
Conclusion
In conclusion, the measurement of research impact is an essential practice to help direct the allotment of limited research resource, to maximise benefit, and to help minimise research waste. This review provides a collective summary of existing methodological impact frameworks and metrics, which funders may apply to inform the measurement of enquiry impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.
Supporting information
S1 Appendix
Search strategy.
(TIF)
S1 PRISMA Checklist
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.
(PDF)
Acknowledgments
We would also like to thank Mrs Susan Bayliss, Information Specialist, University of Birmingham, and Mrs Karen Biddle, Enquiry Secretary, University of Birmingham.
Abbreviations
| AIHS | Alberta Innovates—Health Solutions |
| CAHS | Canadian Academy of Health Sciences |
| CIHR | Canadian Institutes of Wellness Research |
| CINAHL+ | Cumulative Index to Nursing and Centrolineal Health Literature |
| EMBASE | Excerpta Medica Database |
| ERA | Excellence in Inquiry for Australia |
| HEFCE | College Educational activity Funding Quango for England |
| HMIC | Health Management Information Consortium |
| HTA | Wellness Technology Cess |
| IOM | Impact Oriented Monitoring |
| MDG | Millennium Development Goal |
| NHS | National Health Service |
| MEDLINE | Medical Literature Assay and Retrieval System Online |
| PHC RIS | Principal Health Care Research & Information Service |
| PRISMA | Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
| PROM | patient-reported outcome measures |
| QALY | quality-adjusted life twelvemonth |
| R&D | research and development |
| RAE | Research Assessment Exercise |
| REF | Research Excellence Framework |
| RIF | Research Impact Framework |
| RQF | Research Quality Framework |
| SDG | Sustainable Development Goal |
| SIAMPI | Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society |
Funding Argument
Funding was received from Consejo Nacional de Ciencia y Tecnología (CONACYT). The funders had no role in study blueprint, information drove and analysis, decision to publish, or preparation of the manuscript (http://www.conacyt.mx/).
Data Availability
All relevant data are within the paper and supporting files.
References
1. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis J, et al. Biomedical enquiry: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. doi: 10.1016/S0140-6736(thirteen)62329-half dozen [PubMed] [Google Scholar]
ii. Buxton M, Hanney S. How can payback from health services enquiry be assessed? J Health Serv Res Policy. 1996;1(ane):35–43. . [PubMed] [Google Scholar]
four. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: A review. Res Eval. 2014;23(1):21–32. doi: 10.1093/reseval/rvt021 [Google Scholar]
5. Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al. Making evidence count: A framework to monitor the impact of wellness services research. Aust J Rural Health. 2012;20(2):51–8. doi: 10.1111/j.1440-1584.2012.01256.x [PubMed] [Google Scholar]
6. Martin BR. The Research Excellence Framework and the 'impact agenda': are nosotros creating a Frankenstein monster? Res Eval. 2011;twenty(3):247–54. doi: 10.3152/095820211x13118583635693 [Google Scholar]
xiii. Bainbridge R, Tsey K, McCalman J, Kinchin I, Saunders 5, Lui FW, et al. No one's discussing the elephant in the room: contemplating questions of enquiry impact and benefit in Ancient and Torres Strait Islander Australian health enquiry. BMC Public Wellness. 2015;xv doi: 10.1186/s12889-015-2052-iii [PMC gratis article] [PubMed] [Google Scholar]
15. HEFCE. Consultation on the 2d Research Excellence Framework. 2016.
16. Ovseiko PV, Oancea A, Buchan AM. Assessing inquiry impact in academic clinical medicine: a report using Inquiry Excellence Framework pilot impact indicators. BMC Health Serv Res. 2012;12 doi: 10.1186/1472-6963-12-478 [PMC complimentary article] [PubMed] [Google Scholar]
17. Kuruvilla S, Mays North, Walt G. Describing the impact of health services and policy research. J Wellness Serv Res Policy. 2007;12 doi: ten.1258/135581907780318374 [PubMed] [Google Scholar]
23. Banzi R, Moja L, Pistotti Five, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to appraise the impact of health enquiry: an overview of reviews. Health Res Policy Syst. 2011;nine doi: 10.1186/1478-4505-9-26 [PMC free article] [PubMed] [Google Scholar]
26. Bornmann Fifty. What is societal impact of research and how can it be assessed? a literature survey. American Order for Data Science and Technology. 2013;64(2):217–33. doi: 10.1002/asi.22803 [Google Scholar]
27. Patel VM, Ashrafian H, Ahmed K, Arora Due south, Jiwan South, Nicholson JK, et al. How has healthcare inquiry performance been assessed? A systematic review. J R Soc Med. 2011;104(vi):251–61. doi: 10.1258/jrsm.2011.110005 [PMC free article] [PubMed] [Google Scholar]
28. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–ix. [PubMed] [Google Scholar]
29. Kuruvilla Due south, Mays North, Pleasant A, Walt G. Describing the impact of health research: a Research Impact Framework. BMC Wellness Serv Res. 2006;vi:134 ; PubMed Central PMCID: PMCPMC1635046. doi: 10.1186/1472-6963-six-134 [PMC costless article] [PubMed] [Google Scholar]
32. Republic of guinea J, Sela E, Gómez-Núñez AJ, Mangwende T, Ambali A, Ngum N, et al. Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects. Res Eval. 2015;24(2):131–45. doi: 10.1093/reseval/rvu034 [Google Scholar]
34. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health inquiry impact: Development and implementation of the Alberta Innovates–Wellness Solutions bear on framework. Res Eval. 2012;21(five):354–67. doi: 10.1093/reseval/rvs027 [Google Scholar]
35. Cohen G, Schroeder J, Newson R, Rex L, Rychetnik Fifty, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new bear on cess tool. Health Res Policy Syst. 2014;13(1):3–1p. doi: ten.1186/1478-4505-xiii-three . Language: English. Entry Date: 20150923. Revision Date: 20150923. Publication Type: Periodical Article. [PMC free commodity] [PubMed] [Google Scholar]
37. Brueton VC, Vale CL, Choodari-Oskooei B, Jinks R, Tierney JF. Measuring the impact of methodological research: a framework and methods to identify bear witness of bear on. Trials. 2014;15(ane):464–1p. doi: 10.1186/1745-6215-xv-464 . Linguistic communication: English. Entry Date: 20150313. Revision Date: 20150923. Publication Blazon: Journal Article. [PMC gratis article] [PubMed] [Google Scholar]
38. Donovan C. The Australian Research Quality Framework: A live experiment in capturing the social, economical, environmental, and cultural returns of publicly funded research. New Directions for Evaluation. 2008;2008(118):47–60. doi: 10.1002/ev.260 [Google Scholar]
42. Morton S. Progressing research impact assessment: A 'contributions' arroyo. Res Eval. 2015;24(four):405–19. doi: 10.1093/reseval/rvv016 [Google Scholar]
43. Meagher L, Lyall C, Nutley S. Flows of noesis, expertise and influence: a method for assessing policy and do impacts from social science inquiry. Res Eval. 2008;17(3):163–73. doi: 10.3152/095820208x331720 [Google Scholar]
44. Lavis J, Ross Southward, McLeod C, Gildiner A. Measuring the impact of wellness enquiry. J Wellness Serv Res Policy. 2003;8(three):165–70. doi: 10.1258/135581903322029520 . [PubMed] [Google Scholar]
45. Canavan J, Gillen A, Shaw A. Measuring research affect: developing applied and cost-constructive approaches. Evidence & Policy. 2009;5(2):167–77. [Google Scholar]
46. Landry R, Amara N, Lamari M. Climbing the ladder of research utilization—Evidence from social science inquiry. Scientific discipline Communication. 2001;22(4):396–422. doi: x.1177/1075547001022004003 [Google Scholar]
47. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997;13(1):68–80. [PubMed] [Google Scholar]
48. Weiss CH. Using social research in public policy making: Lexington Books; 1977. [Google Scholar]
49. Weiss CH. The Many Meanings of Inquiry Utilization. Public Adm Rev. 1979;39(5):426–31. doi: ten.2307/3109916 [Google Scholar]
50. Kogan M, Henkel M. Government and research: the Rothschild experiment in a government section: Heinemann Educational Books; 1983. [Google Scholar]
51. Thomas P. The Aims and Outcomes of Social Policy Research. Croom Helm; 1985. [Google Scholar]
52. Bulmer M. Social Science Research and Government: Comparative Essays on Great britain and the U.s.a.: Cambridge University Press; 2010. [Google Scholar]
53. Berth T. Developing Policy Enquiry. Aldershot, Gower1988.
54. Yin R, Moore G. Lessons on the utilization of inquiry from nine case experiences in the natural hazards field. Knowledge in Guild. 1988;1. [Google Scholar]
57. Knott J, Wildavsky A. If dissemination is the answer, what is the trouble? Knowledge: Cosmos, Diffusion, Utilization. 1980;ane. [Google Scholar]
lx. Nutley S, Percy-Smith J and Solesbury W. Models of research touch on: a cross sector review of literature and exercise. London: Learning and Skills Research Centre; 2003. [Google Scholar]
62. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(one):1–28. doi: x.1186/1478-4505-ane-2 [PMC free article] [PubMed] [Google Scholar]
64. Drummond M, Cooke J, Walley T. Economical evaluation under managed contest: evidence from the U.K. Soc Sci Med. 1997;45. [PubMed] [Google Scholar]
65. Manandhar DS, Osrin D, Shrestha BP, Mesko N, Morrison J, Tumbahangphe KM, et al. Effect of a participatory intervention with women'due south groups on birth outcomes in Nepal: cluster-randomised controlled trial. Lancet. 364(9438):970–9. doi: x.1016/S0140-6736(04)17021-9 [PubMed] [Google Scholar]
67. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research touch: Development and implementation of the Alberta Innovates–Wellness Solutions impact framework. Res Eval. 2012;21 doi: 10.1093/reseval/rvs027 [Google Scholar]
68. van Driel ML, Maier M, De Maeseneer J. Measuring the impact of family medicine research: scientific citations or societal impact? Fam Pract. 2007;24(5):401–2. doi: 10.1093/fampra/cmm061 [PubMed] [Google Scholar]
69. Hansen J, Muscat N, Keskimaki Fifty. Measuring and improving the societal impact of health care research. Eurohealth. 2013;19(3):32–5. [Google Scholar]
71. Cohen G, Schroeder J, Newson R, Male monarch 50, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practise impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;thirteen doi: ten.1186/1478-4505-13-3 [PMC gratuitous article] [PubMed] [Google Scholar]
72. Bornmann L. Measuring the societal impact of research. Research is less and less assessed on scientific affect alone-we should aim to quantify the increasingly important contributions of science to society. EMBO Reports. 2012;13(viii):673–6. doi: 10.1038/embor.2012.99 [PMC gratuitous commodity] [PubMed] [Google Scholar]
73. Erno-Kjolhede E, Hansson F. Measuring research performance during a changing relationship between scientific discipline and club. Res Eval. 2011;20(2):131–43. doi: 10.3152/095820211x12941371876544 [Google Scholar]
Articles from PLoS Medicine are provided hither courtesy of Public Library of Scientific discipline
Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5549933/
0 Response to "Peer-reviewed Article About Research in a Healthcare Organization"
Publicar un comentario