Summary

This is the Final Report of the Evaluation of the Hellenic Foundation for Research and Innovation (HFRI). Technopolis Group, supported by a panel of four experts, carried out this evaluation in the period September – December 2021. The study was commissioned by the European Investment Advisory Hub (EIAH) of the European Investment Bank (EIB), upon request by the HFRI.
The overall objective of the evaluation was to carry out an assessment and evaluation of HFRI’s activities, operational processes, and impacts during its first period of operation, in relation to the funding received. The aim was to produce recommendations for further development of the Foundation. The findings are based on a mix of quantitative and qualitative methods, encompassing desk research, data review (HFRI portfolio and proposal/project data analysis), bibliometric data analysis, a stakeholder survey (successful and non-successful applicants), an expert panel site visit, stakeholder interviews, and an HFRI self-assessment exercise.

Attainment of the objectives

HFRI was founded in 2016 with the objective to “promote Research and Innovation in Greece and more specifically to support and assist unrestricted research without any thematic or geographical limitations, having Quality and Excellence as a unique criterion.”
In the first four years of its operations (2017-2021), HFRI has succeeded in reaching close to all its operational objectives, setting up its governing and operational bodies, ensuring additional financial resources, and organising the funding distribution to the research community, based upon excellence and meritocracy. An adequate staff employment has been its major challenge.
The high-level commitment of the research community, investing considerable time and effort, as well as the commitment and capacities of HFRI’s staff have been the major drivers for this success. As such, it testifies to the importance of HFRI’s activities for the Greek research system.
HFRI has succeeded in attaining its strategic objectives. To date, the HFRI has distributed 50% of its total budget. Its funding decisions, based on excellence and merit, provide important opportunities for doing unique/original research. It provides support for the personal development of Greek researchers, at all stages in their career and in all fields of research. From a gender equality perspective, overall 35% of the total approved funding in 2016-2020 was to the benefit of female researchers. This included slightly more than 50% of the budget for PhD scholarships but only about 20% of the budget for the Faculty Member and Equipment calls. These data, however, broadly reflect the male/female distribution in applications.
HFRI contributed to constraining the brain drain phenomenon, facilitated the return of earlycareer researchers, enabled the renewal or acquisition of high value research equipment, and supported participation of Greek researchers in Europe-wide research infrastructures in the field of Social Sciences and Humanities. Through the science & society calls, the HFRI aimed at responding to societal needs and disseminating scientific knowledge in society.
An important added value is the continuity in funding that the Foundation ensures, allowing for strategic planning of the research activities, at a personal and institutional level.
Bibliometric data show that HFRI funding has allowed for a significant sharing of the knowledge gained by the researchers funded. The high quality of the research results in the fields of physical sciences and communication science, most of them published in top ranking journals, suggests a signification contribution of HFRI to the strengthening of research competitiveness in the Greek system.

Operational efficacy as an independent research funding organization

Based upon data related to its first two years of full operation (2019/2020), HFRI’s cost of administration is 5% of its grants budget, which is within the international norm and indicates good operational management.
HFRI has the legal status of a private non-profit organisation. It has autonomous decisionmaking power on strategy and its implementation that is standard in international practice. However, de facto HFRI has the status of a public administration body. For administrative, financial, and human resources management matters, it is expected to apply the legislation for the public administration sector. This arrangement has significant repercussions on its operational functioning. It causes delays in the implementation of its operational decisions and causes HFRI’s current understaffing.
The government’s recent decision to abolish the deputy director position deprives HFRI of its internal capacity for administrative oversight. Internationally, basic research funding organisations have a dual leadership structure, consisting of a scientific director or president and an administrative director. Both skills (scientific background as well as management expertise) are needed to run such an organisation.
Overall, HFRI applications for funding had a success rate of 15% in 2016-2020. This is low. Internationally, the scientific community tends to regard a success rate between 20 and 33% as optimal and a guarantee of fair competition. Many research councils, however, operate with lower rates. Success rates below 10%, however, are generally considered unacceptable since they distort the balance between the costs and benefits of proposal writing.
There are significant differences among the different instruments. Proposals for PhD scholarships and ERC grants reached good to excellent success rates (27% and 60%); the “1821” Science and Society call had a low success rate (13%). For all other instruments and programmes, the 2016-2020 calls reached very low success rates (10% and under). This is worrying especially for the Faculty Member and Postdoc instruments. The success rates of the high-quality proposals confirm that these programmes were underfunded: only about 25% of high-quality proposals was retained for funding. Particularly underfunded were the thematic programmes, specifically the Research, Innovation and Dissemination Hubs programme were only 7% of the high-quality proposals were funded.
The underfunding of HFRI compared to the needs is even more apparent when comparing HFRI’s 2019 budget to the budget of other research councils in countries of a similar size in population. Both in terms of number of researchers in the country and size of the country’s population, HFRI’s budget was considerably lower than the budgets in the other countries.
HFRI has only recently started its monitoring operations. As it is currently defined, the project monitoring framework is good practice, even though there are signs of unnecessary rigidity in the administrative and financial processing. Nevertheless, the monitoring framework rightfully goes beyond a mere auditing approach and the involvement of external evaluators is intended to provide insightful and quality feedback for the projects’ evaluation.
What seems to be lacking is a results-based evaluation framework at the institutional level that guides the collection and analysis of monitoring data, going beyond the immediate outputs and outcomes. Such an evaluation framework would provide HFRI with the needed ‘strategic intelligence’ to guide and eventually adjust the scope of its funding and its operations, and to ensure its accountability, i.e., its reporting on the value of its activities and the relevance of the investment – to national policy makers and society at large.
HFRI has dedicated significant efforts in ensuring quality communication and service delivery to its targeted beneficiaries, with very good results. Interviewed and surveyed beneficiaries were highly satisfied with the clarity of the information provided and the user-friendliness of the communication channels, as well as the availability and competences of HFRI staff in their service delivery. The communication strategy goes beyond the mere delivery of information and includes the creation of an online community to develop a dialogue on research matters and activities. While the close to exclusive focus on the research community is understandable in this first stage of HFRI’s activities, based on international practice one would expect a stronger focus in HFRI’s communication strategy on enhancing public understanding of science.

HFRI set-up and operations in the international context

The structure of HFRI’s governance bodies and their mandates are largely in line with good international practice. The role of the Advisory Committee is unclear and the proportion of women in the Scientific Council extremely low. The HFRI adheres to international practice by establishing a clear separation between the Scientific Council and the administration in terms of responsibilities and tasks.
As in other research councils internationally, the HFRI instrument portfolio includes the normal repertoire of ‘basic research’ funding instruments. The funding streams for PhD students and early-career researchers are in line with the international trend among research councils to use non-thematic funding instruments, which aim to support and develop the structure of the national research community. Seeing the national context and its fragmented R&D landscape, a bottom-up and non-thematic centre-of-excellence funding instrument to build up sustainable research groups with critical mass, across universities and research institutions and as an incentive for defragmentation would be an appropriate next step for the HFRI.
There are, however, considerable limits to HFRI’s autonomy compared with international practice. The detailed and exhaustive description of HFRI’s tasks, structure, and processes de facto implies that little to no space is given to the HFRI General Assembly and Scientific Council for strategic decisions and the introduction of change.
In addition, there is a fundamental discrepancy in the description of HFRI’s mandate between the Annex to the EIB loan agreement and the HFRI Founding Law (and the expectations set based on it). The description in the EIB Loan Agreement aligns the HFRI with the internationally normal tasks of a research council, funding investigator-driven basic research and ensuring the diffusion of scientific knowledge in society. This also reflects the ‘division of labour’ that has de facto been established in the Greek R&I governance system for the public funding of research with the HFRI as research council providing support for individual researchers and non-thematic research projects (without geographical criteria), and the GSRI as an innovation agency being in charge of (predominantly ESIF-funded) support for applied research and industry-oriented innovation. The Founding Law, instead, words HFRI’s mandate in more general terms (“the promotion of research and innovation”) and assigns tasks to the HFRI that are typically competence of an innovation agency (“to support, through lump-sum funding, the creation and operation of start-ups to capitalise on research results” and to cover “costs for the protection of intellectual property rights”).

Quality of the project selection and assessment processes

HFRI, like almost all independent research funding organisations, uses a peer-review system to assess applications. Its current approach reflects international good practice and uses the ERC as a model. The HFRI shows a high level of transparency, implementing all standard measures. The quality of the evaluation reports is an issue to address.
HFRI ensures fairness through its processes and structures aimed at avoiding ‘scholarly bias’, such as the rotation system to reach the appropriate balance between continuity and renewal of panel membership and the demand-driven distribution of the call budgets over the disciplinary areas. Other components are the right to object to specific reviewers and the right to appeal panel decisions (‘redress’). The volume of appeals submitted and accepted is surprisingly high. While they predominantly relate to matters of detail and have little or no effect on funding decisions, they also considerably influence HFRI’s time-to-grant performance. To strengthen consistency in the appraisal processes over time, lessons learned from the experience gained should now result in the publication of process descriptions and criteria definitions. Finally, there is room for improvement in the definition of the PI assessment criteria.
Peer review-based evaluation systems typically set up a hierarchical system of panels and sub-panels. HFRI has nine panels which we consider too many, considering the limited size of the country and the budget. Data also show a considerable imbalance between the panels in terms of the number of proposals to handle and sub-panels to set up. A revision of the current categorisation of the fields, more in line with the ‘original’ OECD FORD classification, would allow for an improved spreading of the assessment work over these panels and the reduction of their number. The proportion of female evaluators/experts is extremely low according to all international standards and the involvement of international reviewers/experts very limited.
HFRI’s proposal assessment processes are very closely aligned with the ones adopted in the ERC. However, what works for the ERC may not always be the ideal solution for national research councils, especially smaller ones as the HFRI. The current two-stage evaluation procedure is over-complex for the current types of instruments that the HFRI funds, and too lengthy. It therefore contributes to the major criticism of HFRI, i.e., its time-to-grant. In addition, a clearer division of labour between remote reviewers and panel members would be appropriate.
Various challenges lay at the basis of HFRI’s particularly long time-to-grants. The HFRI has made substantial efforts to alleviate some of these problems but should continue its efforts and consider implementing additional measures.

Image of the HFRI in the national and international R&I community

The most prominent achievement of the HFRI is that it established its legitimacy in the eyes of the research community, based on the trust that its assessment processes are fair, and its funding decisions based on merit. The image is one of a research council funding high quality research and adopting funding schemes that are of high importance and value for the national research system and society at large. HFRI has strengthened research capacities in the country and improved researchers’ career growth prospects. HFRI funding also supported the creation of critical mass, in scientific areas of competitive advantage, and allowed for the conduct of interdisciplinary research.
HFRI’s relationship with the international R&I community has predominantly been with the Greek diaspora, seeking the possibility for closer collaboration.

Alignment of HFRI financing with the broader Greek Science, Technology, and Innovation (STI) priorities

In 2016/2017, the Greek national innovation system was confronted with a major challenge: the loss of skilled human capital, i.e., brain drain. It was recognised as a key challenge by national policymakers and various policy measures were designed to refrain and reverse the brain drain.
HFRI funding successfully tackled the primary drivers for the brain drain: the lack of career prospects and, closely related, the lack of funding for research. Recognising the urgency of the situation, HFRI took multiple measures not only to be able to fund the highest possible number of researchers at risk of emigrating (PhD graduates and Postdoctoral researchers), but also to ensure adequate funding opportunities for younger faculty members. Various elements of evidence emerged during our study on the positive effects of HFRI’s funding to constrain the brain drain and ensure the growth of research career opportunities.
A primary tool for HFRI to support the national Smart Specialisation Strategy ‘Reinforcing research activities’ intervention area was its funding – seeing the ‘division of labour’ with the GSRI, this also included the focus of its funding on fundamental research. Not only did HFRI fund high quality research, leading to a considerable volume of publications, often published in top quality journals, but by providing an arena where researchers and research proposals compete, HFRI set a high standard for research quality – not only for the funding it provides, but also a standard against which research-performing organisations judge quality, and it therefore tends to quality-assure the national basic research effort.
The value of HFRI funding goes well beyond the benefits for the research system, though. It enhances capacity in fundamental research that meets national needs. HFRI makes it possible for researchers to obtain external funding for research in the social sciences and humanities, which are important for social development, and which play a growing role in work addressing the so-called societal challenges.
A primary objective of the national policy in 2014-2020 was to strengthen the competitiveness of the Greek economy. In this context, a funder like HFRI is an essential component of any effective national innovation system, and one whose importance increases with economic and social development, during which production and consumption become increasingly knowledge-based, and the scope to remain competitive while relying on imported knowledge declines. Typically, therefore, countries increase their basic research effort when they move from technology catch-up to looking for ways to get ahead of competitors in the more advanced countries.

Conclusions and recommendations

HFRI is the first organisation of any size to fill the ‘basic’ research funding gap in Greece. HFRI is nonetheless a small organisation with a small budget, not yet sufficient to meet the needs of the current Greek system, let alone the growing needs for research as Greece continues its economic recovery and along the path of development. The current underfunding, and the accompanying low success rates, may cause serious problems for the whole system and reduce the quality of HFRI’s review process, lead to distrust and demotivation of applicants and ultimately, undermine HFRI’s legitimacy. An increase in its funding budget is critical for HFRI to maintain its international standards, continue supporting the Greek research system, and prevent brain drain,
We recommend the Greek government significantly to increase its contribution to HFRI from national sources compared to the amount granted in the 2016-2021 period.
In the national R&I governance system, HFRI has the function of basic research funding organisation. There is de facto a clear division of roles with the GSRI which acts as the country’s innovation agency. The current description of HFRI’s mandate in the Founding Law, however, undermines the coherence and complementarity that has been established between these two research funding bodies, attributing tasks to the HFRI which in international practice, are typically in the competence of an innovation agency. In addition, it fails to make an explicit reference to HFRI’s function as a research council, focused on investigator-driven basic research and science communication to society.
HFRI’s Founding Law appears to be unfit for purpose, specifying processes, routines, and structures in such a level of detail that it de facto deprives the HFRI of the right to determine its own organisation chart and institutional framework, thus considerably limiting its autonomy.
The recent amendment to the Law, abolishing the Deputy Director function and setting the profile of the (new) Director in predominantly scientific terms, appears to be based upon an underestimate of the need in a research council for a dual leadership (scientific and administrative).
Further, despite its legal status as private non-profit organisation, HFRI is obliged to apply the public sector administrative, financial, and human resources regulations. This causes inflexibility and long delays, and most important, is responsible for HFRI’s current understaffing. The latter is a matter of the highest urgency.
We recommend the Greek government formally to recognise HFRI’s function in the Greek R&I system as the public funding organisation responsible for investigator-driven basic research and the communication of scientific knowledge to society, complementing the research funding tasks of the GSRI, in line with international practice. In the mid-term, a permanent position for the HFRI should be envisaged, funded by the Greek state.
The Founding Law needs revision, bringing it more in line with international practice and foreseeing the use of dialogue-based performance agreements. It should give HFRI the status of an independent agency, tie it to achieving a small number of high-level goals, and otherwise make it autonomous in day-to-day practice. HFRI’s task should nonetheless be tightly enough defined to prevent it from moving from researcher-initiated research into other areas, and sufficiently protected to make it hard for other interests to raid its already limited resources.
In the very short-term, an amendment to the Law is needed that excludes HFRI from the provisions of staff hiring, promotion, salaries etc that apply to the public administration sector.
The recent abolition of the Deputy Director function should be re-considered.
HFRI’s governance structure, funding and non-funding processes are in line with international practice, with due attention to the key principles of transparency and fairness. HFRI uses ERC as a model for its operational processes, which is widely seen as the ‘gold standard’ in the international community. HFRI has succeeded in gaining legitimacy in the eyes of the community while ensuring continuity in the highly needed basic research funding and good operational management. It is a remarkable achievement.
There have been some inevitable teething problems and lessons to be learned. A refinement of HFRI’s strategy and processes is therefore appropriate. The adaptation of some processes to the specific context of the Greek research community and its current state of development would also be beneficial.
In relation to its non-funding activities, we recommend HFRI to devote more attention and effort to collecting strategic intelligence on the research system that is needed for its own strategy and in support of national R&I policymaking. A closer contact to overall R&I policy and a more intense collaboration with the bodies in the R&I governance system that collect and have access to strategic intelligence would be beneficial from this perspective.
We highly recommend HFRI to develop an institutional evaluation framework that is resultoriented and can serve its needs for accountability towards the government and the Greek citizens. This evaluation framework should inform the (projects) monitoring framework and its implementation. We also encourage HFRI to improve its communication to the public, to enhance the public understanding of science.
In relation to the funding instruments, we recommend the HFRI to take up its ‘broader’ role of a research council in the Greek R&I system, like research councils internationally, and aim to support and develop the structure of the national research system. We suggest starting with a centres-of-excellence programme as a tool for capacity-building and de-fragmentation of the Greek (basic) research system.
In relation to its peer review-based evaluation processes, we recommend the HFRI to ensure higher quality of the evaluation reports and improve the descriptions of the evaluation criteria so as to increase transparency and reduce the high number of appeals on matters of detail.
The proportion of international experts as well as women among both reviewers and panel members should be increased. Accompanying measures that can be taken against the low success rates, such as proposal bans for persons who submitted proposals with exceptional low quality, should be discussed with researchers and experts. We also suggest reconsidering the current categorisation of the disciplines, spreading the assessment work more equally over the panels and reducing their number.
We recommend the HFRI to adopt a one-stage procedure for all its instruments. In particular, for the Post-Doc and Faculty Member calls, there should be a clearer division of labour between remote reviewers and panel members, with remote reviewers taking care of the assessments of the scientific quality while the panel members assess the proposal in its entirety and set it in context. We recommend the HFRI to make more use of scientifically qualified administrative staff in the selection of the panel members and external experts, under the authority of the Scientific Council. We also suggest revising the reviewers’ remuneration policy to keep the review process costs within sustainable limits.
Various of the above-mentioned recommendations and suggestions aim at addressing (also) the major criticism to HFRI, i.e. its time-to-grant. We recommend installing additional measures such as launching the search for panel members/reviewers prior to the call deadlines and a better spread of the ‘large’ calls across the year, in order to avoid peaks in the workload for both administration and applicants. We highly recommend the HFRI to establish a stronger connection and exchange of experience with other research councils in Europe, in particular the smaller research funders. Most of these organisations are also members of Science Europe, which offers various international learning opportunities.