How we monitor and evaluate

As the HRB is a publicly funded organisation, there is an onus on us to account to government and the public, for the funds we allocate and the returns on the research that we supports. Therefore, it is imperative that the HRB measures the extent to which our funding is achieving the HRB mission and delivering the intended benefits. The HRB monitoring and evaluation processes are delivered in three phases, guided by the Payback Framework and the HRB Evaluation Strategy for Funded Research 2022-2025.

Phase I: Ongoing monitoring and review

All HRB grants are categorised by the HRB Post-Award and Evaluation team using a range of different classification dimensions depending on the required output.

Dimension

Purpose of information collection

Examples

Classification system, based on:

Strategic Focus Area

Map the distribution across HRB strategic focus areas – for reporting to the HRB Board and Department of Health

Focus Area 1;

Enabler A

HRB Strategy

Broad research area

Map spend across  broad research  areas for internal planning and evaluation

Clinical Research; Health Services Research

Australian and New Zealand Standard Research Classification (ANZSRC)

Health field

(‘In what’)

Map investment across disciplinary areas for strategic mapping and to aid external queries

Cardiovascular;

Neurology

Australian and New Zealand Standard Research Classification (ANZSRC)

Area of  Focus

(‘In what)

 

Identify specific disease areas for internal interest and to aid external queries

Prostate cancer; Cystic Fibrosis; Type 2 Diabetes

 

Therapy/Potential Therapy

(‘Why’)

Identify potential applications and impacts of research for internal interest and to aid external queries, PQs etc

Personalisied medicine; Medical device

-

HRCS Health Categories

(‘In what’)

Map investment  across specific health categories for strategic information purposes and for external queries

Cancer;

Cardiovascular.

UK Health Research Classification System (HRCS)

HRCS Research Activity (What approach/type’)

Map investment by type of activity on spectrum from underpinning to health services research

Underpinning research;

Development of treatments.

UK Health Research Classification System (HRCS)

Field of application linked to national priorities

Map the distribution of HRB awards across NRPE priority research areas

Medical Devices;

Underpinning.

NRPE priority areas

Site of Research

(‘Where’)

Map the distribution of research investment across institution types

Hospital;

University.

-

Close

Annual Reports are required for the majority of active HRB grants. Requests are issued at the end of January of each year for reports to be submitted by the end of March each year. There are two parts of the report, Part A relates to the grant management and scientific progress while Part B relates to the financial progress of the grant and they are reviewed by both the HRB Post-Award and Evaluation team and HRB Finance team. Additional reporting mechanisms associated with grant-specific Key Performance Indicators (KPIs) exist for selected large-scale investments e.g. Clinical Research Facilities.

Close

Independent reviews of selected large-scale or strategically important HRB investments are undertaken typically half way through the grant to assess scientific and strategic progress, and to ensure all milestones are being met. An independent panel of international experts is selected to carry out the interim review. The outcome of the review can lead to management action and/or budget changes if necessary.  These might include changes to the work programme, or adjustments to the governance of the award such as the establishment of independent advisory boards to oversee progress and performance.

Close

When the current contract of a large-scale or strategically important HRB infrastructure investment is coming to an end, a Renewal Review will take place to establish whether there is justification for the HRB to continue to support the infrastructure, and at what level. The grantholders will submit a report on progress to date and plans for a future phase of funding. An independent panel of international experts will be convened to assess the progress to date and make a recommendation about future funding. The outcome of the Renewal review can lead to altered objectives and budget changes if necessary.

Close

Phase II: Collecting outputs and outcomes

The HRB Post-Award and Evaluation team tracks and collates a wide variety of evaluation metrics from its funded research. These metrics are gathered at the end of a project and can be grouped into six broad categories. The HRB uses this information to evaluate the benefits and impact of its’ investment in health research.

1. Knowledge Production

  • Peer reviewed publications and citations
  • Other non-peer reviewed publications such as, books, book chapters, editorials or bulletins
  • Research reports and grey literature
  • Scientific presentations at national and international conferences
  • Cochrane systematic reviews produced or findings included in a review

2. Research collaborations, partnerships and networks

  • New national/international collaborations or strategic partnerships formed or strengthened with other research teams, industrial partners or health agencies
  • Internationalisation of research: Involvement of HRB-funded researchers with EU and global health research initiatives

3. Research capacity-building

  • Professional background of personnel such as clinicians, health professionals and scientists
  • Number of higher degrees, such as PhD, obtained by research personnel
  • Development and use of novel research techniques and methodologies
  • Establishment of new datasets, databases or research data lodged in national/international database
  • Achievement of peer recognition and awards

4. Informing policy and the public

  • Dissemination and knowledge-transfer events or networks established with research 'users', such as policy-makers and health professionals
  • Advisory roles of HRB-funded researchers to the government and/or policy-makers
  • Commissioned reports or projects from government departments or health agencies
  • Policy briefing papers, practical handbooks and other grey material produced and disseminated to research users such as policy-makers and health professionals
  • Evidence of public outreach and dissemination through media and other public fora
  • Evidence of meaningful public and patient involvement and engagement

5. Healthcare innovations and interventions

  • Development, testing, evaluation or implementation of products (e.g. diagnostics, drugs, devices), preventative and therapeutic interventions, health IT systems, clinical decision support tools, protocols, care models
  • Contribution of research to clinical treatment, best practice guidelines and health promotion initiatives
  • Randomised control trials completed and new interventions established as a result
  • Numbers of patients enrolled on clinical trials or engaged with studies undertaken in clinical research facilities
  • Contribution of research to actual health benefits within Irish population
  • Savings to the health system through gains in health service efficiency, improved primary care or introduction of preventative health measures, where research and evidence generated by researchers contributed to this

6. Wider economic benefits

  • Improved international reputation of Ireland for health and medical research (e.g. by attracting pharma industry R&D and collaborative partnerships with HRB-funded researchers invited keynote addresses to international conferences, involvement of HRB-funded researchers in international research programmes)
  • Success of HRB-funded personnel in leveraging national and international research funding, for example though the EU's Framework Programmes
  • Patents and other IP applications and award of commercialisation support grants to develop marketable products or devices
  • Licence agreements and revenues generated as a result
  • Spin-out companies or formal collaborative partnerships between researchers and industry
Close

Following the collection of outputs and outcomes in the Outcome Tracker questionnaire, all data is consolidated in internal databases for review and analysis to inform future strategic decisions and generate reports.

Close

Researchers are invited to provide narratives about their research grants to capture the story of their progress including their career journey, lessons learned and problems encountered along the way. These qualitative narratives may feature as Success Stories or News items on the HRB website etc, where appropriate.

Close

Phase III: Impact assessment

The HRB commissions periodic bibliometric analysis of HRB-supported peer-reviewed publications. The analysis is used by the HRB to assess the scientific impact of the research it funds and to gain strategic insights in terms of trends in output and impact, and areas of strength and weakness. In addition, this type of analysis allows the HRB to examine the extent of internationally and domestically co-authored research linked to HRB-funded publications.

Close

The HRB carries out and commission field and scheme review periodically to assess the HRB's investment in areas of strategic importance. During the current Strategy 2016-2020, the HRB is planning the following reviews, the results of which will be available to view in the Evaluation reports list.

  • Clinical research infrastructure
  • Research leaders
  • Career tracking & mobility of HRB-funded researchers
  • Policy and Practice impact of HRB investment
  • Innovation/enterprise outputs and impacts
  • Assessment of the impacts and value of co-funding/partnerships developed through EU participation 
Close