Steering Group Meeting
|Chair||Sir Gareth Roberts||Wolfson College, Oxford|
| ||Sir Leszek Borysiewicz||Imperial College|
|Professor Vicki Bruce||University of Edinburgh|
|Professor David Eastwood||University of East Anglia|
|Professor Roderick Floud||London Metropolitan University|
|Professor Georgina Follett||Dundee University|
|Professor Alan Jackson||University of Southampton|
|Dr John Kemp||EVOTEC NeuroSciences GmbH|
|Sir Paul Nurse||Cancer Research UK|
|Professor Fabian Monds||Invest Northern Ireland|
|Professor Teresa Rees||University of Cardiff|
|Mr Phil Ruffles||Rolls Royce plc|
|Sir David Watson||University of Brighton|
|Observer||David McAuley||DEL NI|
|Technical Review Team||Dr Siân Thomas||HEFCE|
Monday 16 December
Wolfson College, Oxford
1. The Chair welcomed Members to Wolfson College and outlined the programme for the residential meeting. He then invited each Member to present feedback to the Group on discussions held with colleagues in institutions, subject areas or other areas about Research Assessment since the last meeting. For ease of reference, we have summarised Members’ comments under subject areas.
2. Many colleagues in medicine denied that the RAE as a method of assessment could be much improved. Nevertheless a potent sense of injustice with the current system flowed from the size of the 2001 RAE financial settlement and its apparent inconsistent application across the subject spectrum.
3. Metrics were largely regarded as a poor basis for assessing research. Research income was a particularly poor metric in life sciences since it would skew assessment and funding in favour of resource-intensive and fashionable research areas, such as biomedicine.
- International benchmarking was not in favour with the NHS because much practice-based medical research is focussed on exceptional national or local challenges.
4. Multi-authorship of research papers in medicine was felt to be a problem. Colleagues argued that the new system must recognise the contribution made by individuals and individual groups to large-scale collaborative projects.
5. Colleagues in preventative medicine felt that the RAE placed too much emphasis on curative medicine.
Arts and Humanities
6. Colleagues in the arts and humanities powerfully defended peer review as the principal method of research assessment, although it was accepted that metrics could play a useful supporting role in some circumstances.
7. The importance of QR funding to arts and humanities researchers was repeatedly emphasised. QR forms by far the majority of research funding in this area – no SRIF funding is forthcoming, for example. Yet in absolute terms there are more researchers in the arts and humanities than researchers in science.
8. There was felt to be some scope for reducing the number of UoAs in arts and humanities, although not too much.
9. Again, there was felt to be scope for reducing the number of UoAs in some areas, although many social science disciplines that currently fall between UoAs may have a sound case for a new unit.
10. In the physical sciences bibliometrics seemed to correlate closely with the quality of pure research and so could be a potentially valid metric.
11. Colleagues expressed support for capacity building in certain areas.
12. It was felt that the boundaries of each current UoA heavily influenced the scope of research in the sector, rather than the other way round.
13. In industry, the assessment of individuals against agreed research goals is common practice, and there was some suggestion that this approach could work in HE. However, other colleagues argued that this approach was more suited to applied, near-market research, and that HE should continue to focus on pure research.
Art and design
14. External research income was felt to be a poor metric for research assessment since most external non-QR funding flows to individuals rather than institutions.
15. Members conceded that metrics already played a significant role in the RAE, and that other external metrics that tend to predict the outcome of the RAE could be usefully brought within the process. However, Members agreed that any new metrics must be thoroughly tested before adoption to guard against any unintended consequences.
16. Several Members agreed that the RAE mitigates against career breaks, and this could have a negative impact on researchers with childcare or other similar commitments.
17. It was generally agreed that the RAE had enhanced the management of research at institutional level.
Tuesday 17 December
Wolfson College, Oxford
18. Apologies were received from Sir Leszek Borysiewicz. Two members, Professor Alan Jackson and Sir Paul Nurse, not present at the first meeting, were welcomed to the group. The chair explained the format for the day’s meeting. The group would receive and consider three main strands of evidence, and a report on the informal meetings held in the summer and autumn, outcomes of the facilitated workshops and the analysis of responses to the invitation to contribute. The afternoon session would debate the emerging issues and commission the technical review team to undertake the next stage of work.
19. The chair briefed members on the outcomes of the informal meetings with various interested groups. Two models of assessment had emerged from these discussions, which were presented in outline to the group.
20. Jonathan Grant and Steven Wooding of RAND Europe presented to the group their interim report of the workshops conducted across the sector. Members discussed the emerging issues and their impact on the group’s work. It was agreed that the workshops had yielded some useful information and speculated as to whether it might be interesting to conduct a further workshop with an established RAE panel.
21. The group discussed the possibility of commissioning a further study into the uses made of the RAE and its outcomes. By measuring the many current uses of the RAE, the group hoped to identify and mitigate against its susceptability to mis-use.
22. Sian Thomas and Tom Sastry presented the headline themes from the invitation to contribute responses received thus far. The group agreed to commission a full independent analysis of the responses and also requested that the responses be divided amongst the members to ensure every contribution was considered by the group.
23. Will Naylor updated the group on the international comparisons study commissioned. SPRU were commissioned to update a previous similar study. The executive summary of that report was circulated along with a short note on the work to be completed. Members noted an interest in the study exploring the international perceptions of the UK system and highlighted some additional countries that might be of interest including Japan, the Republic of Ireland, and changes being made to the Canadian system. The USA was also identified, though members noted that this system was extensively reported elsewhere.
Technical review – identification of preferred approaches
24. The group re-visited the purpose of the review and research assessment generally. These are three-fold; for the treasury, as the ultimate funders of research to have confidence and accountability in their expenditure, to provide the academic community with evidence of their standing nationally and internationally, and to inform funding allocations.
25. Sian Thomas addressed some the key questions to the group based on the evidence gathered thus far. The group discussed issues around the research base, subject areas, process issues, framework for assessment and approaches for further development.
26. The group explored the possible approaches now open to them; the current RAE model, one based on the current expert review model but with substantive changes; an institutional assessment model. Members felt that in its present form the current model was only an option with substantive changes and improvements. The group commissioned the technical review team to develop in detail the two possible models of assessment for consideration, one based on expert review and the one based on an institution assessment approach and a third hybrid model incorporating elements from both. These would be detailed with a range of options including advantages and disadvantages.
Any other business
27. In view of the progress made at this meeting, the group asked the review team to draft an update note to be published on the review website.
28. Members agreed to convene an additional meeting in February to discuss the developed approaches. The next steering group meeting will on Thursday 13 February 2003 at The Royal Society, 6 - 9 Carlton House Terrace, London SW1Y from 1.00pm. There will be a buffet lunch from 12.00pm.