How we monitor and evaluate HRB grants
Researchfish - grant evaluation
Researchfish is a web-based platform designed to systematically track evaluation metrics in the form of output, outcome and impact data arising from HRB funded awards. Examples of these outputs and outcomes include:
- Publications supported by the HRB award
- Further funding leveraged by the award
- New collaborations and partnerships established as a result of the award
- Impact on policy and practice etc.
Researchfish will better enable the longitudinal tracking of Health Research outcome and impact information, from the time an award is approved up to five years after it has ended.
Researchfish is being used by HRB to:
- better track important outcome and impact information
- enable evidence-informed decision making
- balance resource allocation and
- communicate and demonstrate the impact of HRB funded projects.
Researchfish does not collect any information on grant deliverables or funding spend.
Difference between Annual Reports and Researchfish?
- Annual reporting supports due dilligence / corporate governance of HRB grants.
- Researchfish supports evaluation of outcomes and impacts of HRB grants.
Annual reports are a mechanism to provide updates on the scientific progress of the grant with respect to the original application, and to provide a financial update on spending. They constitute corporate governance and due diligence processes that monitor research grants awarded by HRB.
The Annual Reports process takes place HRB’s Grant Management System, GEMS.
Researchfish does not collect this type of governance data. It is solely concerned with outputs and outcomes, such as publications, leveraged funding, policy impacts, etc.
It is important to note that there is no overlap between annual reporting and Researchfish data collection.
HRB End of Grant Outcome Tracker Survey – replaced by Researchfish
Researchfish has replaced the End of Grant (EOG) Outcome Tracker that was previously used by the HRB.
The EOG Outcome Tracker Survey differs from Researchfish in that the former was a survey completed by grant holders after the conclusion of their award, whereas with Researchfish we now ask grant holders to provide annual updates on their projects' outputs, outcomes, and impacts.
Researchfish facilitates more timely and comprehensive reporting instead of reporting in bulk at the conclusion of the award.
Researchfish webinars
The HRB hosted two online webinars, one for Host Institutions and a second for researchers, to explain the new platform, how it will be used by HRB, the benefits for researchers and host institutions, and key dates for the roll out of the system.
Read more about the HRB's new grant monitoring process.
Webinar for researchers
Presented by Veda Muppavarapu (HRB), Euan Mackay (HRB), Claire Wooding (Researchfish), Katy Elliott (Researchfish).
Running order
- Welcome, introductions and housekeeping
- HRB introduction and aims
- Introduction to Researchfish
- HRB Researchfish implementation and data collection updates
- Researcher Researchfish account set up and available support
- Recap and Questions and Answers
Researchfish - Frequently asked questions
Researchfish is an online outcomes and impact tracking platform operated by Elsevier publishing. Funders such as the Wellcome Trust, UKRI and British Heart Foundation use Researchfish to assess the impact of the research projects they fund.
From January 2024, the Health Research Board will use Researchfish to track outputs, outcomes and impact over time for all HRB-funded projects. Researchfish will allow Principal Investigators (PI’s) to easily report on outcomes, and to link their portfolio of research projects.
CloseThe value of the HRB’s current research funding commitment is in the region of €200 million. As this is public money, there is an onus on the HRB to account to government and other stakeholders, including the public, for the funds it allocates and the returns on the research that it supports. Therefore, it is imperative that the HRB measures the extent to which this portfolio of funding is achieving the HRB mission and delivering the intended benefits. This is achieved through the HRB's Monitoring and Evaluation (M&E) activities.
A new Monitoring and Evaluation Implementation Plan, which was approved by the HRB Board in May 2023, pointed to the need for an evaluation tool that would enable HRB to collect longitudinal output, outcome and impact data from our investments to learn from and communicate about the impact of our funding. The move to Researchfish is a part of a larger plan at the HRB to streamline reporting for grants and lower the burden for grant holders whilst discharging our responsibility as a public body to effectively demonstrate accountability for public money.
CloseThe HRB will require award recipients to submit outcomes information to the Researchfish platform on an annual basis, during a submission period.
To do this, PI’s will need to create a Researchfish account and complete an outcomes report.
While Researchfish allows for real-time monitoring of outputs and outcomes and researchers are encouraged to update their report throughout the year, there will be a specific six-week window every year where researchers will need to submit their information on the platform. PIs will need to report their outputs and outcomes on Researchfish on an annual basis, up to five years after a HRB award has concluded.
CloseThe next Researchfish submission period is 24 January 2024 to 11 March 2024.
Researchers will need to log on to Researchfish and submit their report not later than 12 noon, Monday 11 March 2024.
All awards that have ended since 1 January 2019 and those that are active on or before 1 September 2023 are required to submit a report via Researchfish during the January to March 2024 submission period.
CloseThe HRB and Researchfish will send emails alerting PIs of the upcoming submission period between 27 November 2023 and 8 December 2023. The emails will be sent to the address associated with your HRB award and will include instructions for setting up your Researchfish account.
CloseTo create a Researchfish user account, you must receive an invitation email from Researchfish to set up your account.
If you are required to submit a report in the January 2024 submission period, you will receive an email from Researchfish inviting you to set up your account between 27 November 2023 and 8 December 2023.
If you cannot find your original invitation email, a new invitation can be requested by following the steps in Invitation Help in the Help and Support section of the Researchfish website.
CloseA video guide to submitting a report on Researchfish is available at https://eval.researchfish.com/help-videos.
CloseGenerally, yes, but please see the HRB’s communications about your specific award(s).The HRB will tell you whether a submission is mandatory or optional for each award that you hold.
CloseThe HRB will commit to collecting data on outcomes and impacts generated from awards for up to five years after an award has ended. This is because the HRB recognises that many outcomes and impacts materialise years after an award has ended and not capturing this data would mean that researchers do not have the opportunity to present a full picture of the impact generated through their HRB investment.
CloseAll researchers will need to submit a report annually. However, where there are no new outcomes to submit you can select the “not applicable this year” for any mandatory questions.
If you feel it is unlikely that there will be further outcomes from your award, please email your HRB point of contact to discuss closing out your award.
CloseThe HRB may grant a one-year exemption from completing a Researchfish return for PIs who are unable to complete a return. Common reasons for exemptions include:
- Parental leave
- Long term sick leave
- Maternity Leave
- Fieldwork where no internet access will be available
If this applies to you, please notify your HRB contact as soon as possible.
Exemptions will only be applied if you are unable to complete for the full duration of the submission period. Please note that Researchfish allows for PIs to delegate completion of the report to members of their team. While a team member may enter information to Researchfish, it is only the PI who is authorised to finally submit the information.
CloseAwards are added to Researchfish by the HRB on a rolling basis. As such, a new award may not appear in the system immediately. Alternatively, your award may be listed against a different individual (e.g. a previous PI).
Contact email: evaluation@hrb.ie if you cannot see an award that you expect to see among your Researchfish records.
Similariily, if, the details relating to an award/grant associated with your account are incorrect, you should contact email evaluation@hrb.ie in the first instance, as Researchfish are unable to change award details.
If you have issues with the Researchfish system, please contact support@researchfish.com
CloseHRB will use data provided through the Researchfish platform to facilitate greater oversight and governance of HRB investments and report upstream and outwards on the impact of HRB investment. Researchfish data will be used to report at a scheme, cohort or portfolio level but will not be used to monitor individual awards. Researchfish impact data will be used for activities such as:
Demonstrating accountability for public money by identifying and communicating the value of HRB’s contribution to the Health Research ecosystem in Ireland.
Allowing the HRB to advocate for more funding in health research in areas that need it the most.
Allocating resources to schemes and portfolios in an evidence informed manner, generating positive societal and economic outcomes.
Yes, HRB award holders will need to submit an Annual Report in addition to the Researchfish report. A more streamlined version of the HRB Annual Report is being developed for 2024 to ensure there is no duplication with Researchfish. Annual Reports 2024 will be followed as per the usual process, including the submission of a financial report. While the Annual Report focuses on monitoring the delivery of an award in relation to scientific progress, Researchfish collects important evaluation information on outputs, outcomes and impacts generated from the HRB award.
Close
HRB is hosting two virtual information sessions for Host Institutions and PIs to provide more information on the Researchfish process and the next submission period. Please sign up at the links below to attend an information session relevant to you-
Health Research Board (HRB) - Researchfish Webinar for Host Institutions | Researchfish (28th November 2023, 11am)
Health Research Board (HRB) - Researchfish Webinar for Researchers | Researchfish (4th December 2023, 10am)
CloseAward holders who have completed their HRB award and have submitted an End of Grant report may still need to submit a Researchfish report. (If your award ended on or after 01/01/2019). However, in these cases, we are only asking you to report on any additional outputs, outcomes and impacts that have generated from your HRB award since your End of Grant submission. You do not need to re-submit information already submitted to the HRB via an End of Grant Survey. We are aware that outputs, outcomes and impacts from Health Research projects take time to materialise and therefore, this is an opportunity to capture any development on your award since your last submission to the HRB.
CloseThe HRB End of Grant report was completed by all award holders who had an HRB award finish in the preceding year. Questions in the report were built based on the Buxton and Hanney Payback framework. Award holders were asked to complete this End of Grant report just once, capturing a snapshot of outputs, outcomes and impact generated by the HRB award at the point of reporting. Researchfish now replaces the HRB End of Grant report. Award holders will be asked to submit information on outputs, outcomes and impacts generated from their award every year and up to five years after their award has ended. Reporting every year will ensure that award holders can record information as and when an output or outcome has occurred, instead of reporting in bulk after the award has ended. Reporting for up to 5 years after an award has ended will ensure that any important outcomes and impacts generated from the HRB award are captured for longitudinal analysis of impact. Further, Researchfish integrates with external data sources to allow data to be imported or re-used if it exists elsewhere, to reduce the amount of manual entry and where possible enhance the data, which wasn’t possible with the previous End of Grant Report.
CloseThe following awards are not required to submit a Researchfish report within the first submission period-
Awards that ended before 01/01/2019
Awards that started after 01/09/2023
Summer Scholarship Awards
Conference and Events Scheme Awards
Researchfish is GDPR compliant. Please click on the links below for more information related to data processing-
Privacy Policy | Elsevier Legal
Cookie Notice | Elsevier Legal
Accessibility Statement | Researchfish
CloseHow we monitor and evaluate
As the HRB is a publicly funded organisation, there is an onus on us to account to government and the public, for the funds we allocate and the returns on the research that we supports. Therefore, it is imperative that the HRB measures the extent to which our funding is achieving the HRB mission and delivering the intended benefits. The HRB monitoring and evaluation processes are delivered in three phases, guided by the Payback Framework and the HRB Evaluation Strategy for Funded Research 2022-2025.
Phase I: Ongoing monitoring and review
All HRB grants are categorised by the HRB Post-Award and Evaluation team using a range of different classification dimensions depending on the required output.
Dimension | Purpose of information collection | Examples | Classification system, based on: |
Strategic Focus Area | Map the distribution across HRB strategic focus areas – for reporting to the HRB Board and Department of Health | Focus Area 1; Enabler A | HRB Strategy |
Broad research area | Map spend across broad research areas for internal planning and evaluation | Clinical Research; Health Services Research | Australian and New Zealand Standard Research Classification (ANZSRC) |
Health field (‘In what’) | Map investment across disciplinary areas for strategic mapping and to aid external queries | Cardiovascular; Neurology | Australian and New Zealand Standard Research Classification (ANZSRC) |
Area of Focus (‘In what)
| Identify specific disease areas for internal interest and to aid external queries | Prostate cancer; Cystic Fibrosis; Type 2 Diabetes |
|
Therapy/Potential Therapy (‘Why’) | Identify potential applications and impacts of research for internal interest and to aid external queries, PQs etc | Personalisied medicine; Medical device | - |
HRCS Health Categories (‘In what’) | Map investment across specific health categories for strategic information purposes and for external queries | Cancer; Cardiovascular. | UK Health Research Classification System (HRCS) |
HRCS Research Activity (What approach/type’) | Map investment by type of activity on spectrum from underpinning to health services research | Underpinning research; Development of treatments. | UK Health Research Classification System (HRCS) |
Field of application linked to national priorities | Map the distribution of HRB awards across NRPE priority research areas | Medical Devices; Underpinning. | NRPE priority areas |
Site of Research (‘Where’) | Map the distribution of research investment across institution types | Hospital; University. | - |
Annual Reports are required for the majority of active HRB grants. Requests are issued at the end of January of each year for reports to be submitted by the end of March each year. There are two parts of the report, Part A relates to the grant management and scientific progress while Part B relates to the financial progress of the grant and they are reviewed by both the HRB Post-Award and Evaluation team and HRB Finance team. Additional reporting mechanisms associated with grant-specific Key Performance Indicators (KPIs) exist for selected large-scale investments e.g. Clinical Research Facilities.
CloseIndependent reviews of selected large-scale or strategically important HRB investments are undertaken typically half way through the grant to assess scientific and strategic progress, and to ensure all milestones are being met. An independent panel of international experts is selected to carry out the interim review. The outcome of the review can lead to management action and/or budget changes if necessary. These might include changes to the work programme, or adjustments to the governance of the award such as the establishment of independent advisory boards to oversee progress and performance.
CloseWhen the current contract of a large-scale or strategically important HRB infrastructure investment is coming to an end, a Renewal Review will take place to establish whether there is justification for the HRB to continue to support the infrastructure, and at what level. The grantholders will submit a report on progress to date and plans for a future phase of funding. An independent panel of international experts will be convened to assess the progress to date and make a recommendation about future funding. The outcome of the Renewal review can lead to altered objectives and budget changes if necessary.
ClosePhase II: Collecting outputs and outcomes
The HRB Post-Award and Evaluation team tracks and collates a wide variety of evaluation metrics from its funded research. These metrics are gathered at the end of a project and can be grouped into six broad categories. The HRB uses this information to evaluate the benefits and impact of its’ investment in health research.
1. Knowledge Production
- Peer reviewed publications and citations
- Other non-peer reviewed publications such as, books, book chapters, editorials or bulletins
- Research reports and grey literature
- Scientific presentations at national and international conferences
- Cochrane systematic reviews produced or findings included in a review
2. Research collaborations, partnerships and networks
- New national/international collaborations or strategic partnerships formed or strengthened with other research teams, industrial partners or health agencies
- Internationalisation of research: Involvement of HRB-funded researchers with EU and global health research initiatives
3. Research capacity-building
- Professional background of personnel such as clinicians, health professionals and scientists
- Number of higher degrees, such as PhD, obtained by research personnel
- Development and use of novel research techniques and methodologies
- Establishment of new datasets, databases or research data lodged in national/international database
- Achievement of peer recognition and awards
4. Informing policy and the public
- Dissemination and knowledge-transfer events or networks established with research 'users', such as policy-makers and health professionals
- Advisory roles of HRB-funded researchers to the government and/or policy-makers
- Commissioned reports or projects from government departments or health agencies
- Policy briefing papers, practical handbooks and other grey material produced and disseminated to research users such as policy-makers and health professionals
- Evidence of public outreach and dissemination through media and other public fora
- Evidence of meaningful public and patient involvement and engagement
5. Healthcare innovations and interventions
- Development, testing, evaluation or implementation of products (e.g. diagnostics, drugs, devices), preventative and therapeutic interventions, health IT systems, clinical decision support tools, protocols, care models
- Contribution of research to clinical treatment, best practice guidelines and health promotion initiatives
- Randomised control trials completed and new interventions established as a result
- Numbers of patients enrolled on clinical trials or engaged with studies undertaken in clinical research facilities
- Contribution of research to actual health benefits within Irish population
- Savings to the health system through gains in health service efficiency, improved primary care or introduction of preventative health measures, where research and evidence generated by researchers contributed to this
6. Wider economic benefits
- Improved international reputation of Ireland for health and medical research (e.g. by attracting pharma industry R&D and collaborative partnerships with HRB-funded researchers invited keynote addresses to international conferences, involvement of HRB-funded researchers in international research programmes)
- Success of HRB-funded personnel in leveraging national and international research funding, for example though the EU's Framework Programmes
- Patents and other IP applications and award of commercialisation support grants to develop marketable products or devices
- Licence agreements and revenues generated as a result
- Spin-out companies or formal collaborative partnerships between researchers and industry
Following the collection of outputs and outcomes in the Outcome Tracker questionnaire, all data is consolidated in internal databases for review and analysis to inform future strategic decisions and generate reports.
CloseResearchers are invited to provide narratives about their research grants to capture the story of their progress including their career journey, lessons learned and problems encountered along the way. These qualitative narratives may feature as Success Stories or News items on the HRB website etc, where appropriate.
ClosePhase III: Impact assessment
The HRB commissions periodic bibliometric analysis of HRB-supported peer-reviewed publications. The analysis is used by the HRB to assess the scientific impact of the research it funds and to gain strategic insights in terms of trends in output and impact, and areas of strength and weakness. In addition, this type of analysis allows the HRB to examine the extent of internationally and domestically co-authored research linked to HRB-funded publications.
CloseThe HRB carries out and commission field and scheme review periodically to assess the HRB's investment in areas of strategic importance. During the current Strategy 2016-2020, the HRB is planning the following reviews, the results of which will be available to view in the Evaluation reports list.
- Clinical research infrastructure
- Research leaders
- Career tracking & mobility of HRB-funded researchers
- Policy and Practice impact of HRB investment
- Innovation/enterprise outputs and impacts
- Assessment of the impacts and value of co-funding/partnerships developed through EU participation