Home CV pdfresearchpublications conferencesconsultancies service
   LINDA BUTLERbibliometrics ... research evaluation      

Research Interests

Quantitative measures of research performance

There is a significant body of work in the research literature on the use of quantitative indicators to assess research performance, but it has a number of major gaps. Most of the literature focuses on bibliometric indicators, with little attention paid to other measures beyond a simple listing. In addition, while many indicators of esteem are routinely mentioned in researchers' "wish lists" of how they would prefer to be judged, and these are often used in the assessment of individual performance, little work has been undertaken to determine whether these can be operationalised for use at higher levels of aggregation. Linda's interest in this domain is across two main areas : firstly, a critical assessment of existing performance measures - their validity, fairness, transparency and impact on research, and the cost of implementation; and secondly, determining whether measures more commonly used at the individual level can be applied to departments or whole institutions. Her initial research into these topics was funded by an ARC Linkage grant.

Eminent scholars - the impact on their hosting institution

Many countries have established competitive programs to recruit high profile, high performing researchers to their universities or research institutes. For example, a current program of the Australian Research Council is the Australian Laureate Fellowships scheme (successor to the Federation Fellows scheme) which “reflects the Commonwealth’s commitment to support excellence in research by attracting world-class researchers and research leaders to key positions, and creating new rewards and incentives for the application of their talents in Australia.” Recipients receive significant funding (it can surpass $3mill dollars over the 5 years of the grant). Such schemes are expected to have a significant impact on the institution hosting the award recipients and, in some cases, a discernible impact for the country. As yet little has been done to study whether these schemes have been successful in achieving their stated aims. New research in this area has been started in collaboration with the Institute for Research Information and Quality Assurance (iFQ) in Germany and the Georgia Institute of Technology.

Assessment of Impact

The assessment of 'impact' in its broadest sense - incorporating outcomes beyond academe - has dropped under the policy radar in Australia since the abandonment of the Research Quality Framework (RQF) at the end of 2007. While its successor, the Excellence in Research for Australia initiative, is focussed squarely on academic performance, the developments and discussions around the RQF nevertheless achieved a heightened awareness of broader impact of research. Significant developmental work had been undertaken, much of it based on the work of researchers at Brunel University and RAND Europe, specifically their 'Payback' methodology.While it might be in hibernation in Australia, elsewhere 'impact' is rising in importance. In the context of changes to the UK's system of research assessment, the Higher Education Funding Council of England (HEFCE) is now seeking to assess the social and economic impact of university research alongside quality. Canada too has devloped a keen interest in this area of research, with the Canadian Academy of Health Sciences funding a major project on research evaluation in its broadest sense through 2007 and 2008. Research in this area was concentrated on a multi-national study of the returns from heart and stroke research (Project Retrosight), from which the first results were published in late 2011.

Performance measures for the Humanities, Arts and Social Sciences (HASS)

Research on indicators for HASS disciplines follows two paths. One strand investigates the development of novel bibliometric indicators. Bibliometric analysis in these fields has been insufficiently geared to their distinctive nature, and has been limited by attempts to replicate techniques that rest more comfortably with the natural sciences. Using experience obtained through previous studies, including collaboration with the Centre for Science and Technology Studies (CWTS) at Leiden University, one line of interest is the degree to which discussions in all modes of publications in these fields are filtered through journals; and hence the degree to which the expansion of bibliometric studies to encompass citations to all forms of literature can result in valid performance indicators. In parallel, a second line of interest is the use of non-bibliometric measures, particularly esteem measures. These are often proposed as alternatives to overcome the deficencies of bibliometric indicators, but few have been operationalised.

Organisational structure of Australia's research landscape
Research continues on developing advanced quantitative techniques to analyse Australia's scientific performance and the organisational structure of its research landscape.In the past this work has been facilitated by REPP's extensive database of all Australian publications appearing in the international journal literature has been established. A number of evaluative enquiries into the research performance of sectors and institutions within the Australian research community were to support science policy decision-making. Many studies were undertaken on behalf of government agencies, national research organisations and universities. Current lines of enquiry rely more heavily on the on-line resources of Elsevier's Scopus and Thomson Reuter's Web of Science.