Calling time on REF
Calling time on REF
(Universities are currently enjoying the magic few months between the submission of their REF data and the dreaded judgement day in May 2022. REF is the Research Excellence Framework – which sits like a threatening monkey on every research academic’s back. Prove your worth; justify your research aims; chase new grants; demonstrate your impact; publish, publish, publish.)
Sorting through old papers the other day I found the letter of appointment for my first job at Keele University in 1978. I was offered a salary of £3,600, which seemed to me a princely sum. In addition, I had the option to live, with my young family, in a two-bedroom staff flat on campus for a very low rent taken from my salary before tax. My teaching timetable was reasonable, with ample time to fulfill composition commissions (I am a composer). These would be expected of me as my contribution to the university’s research profile, and constitute 40% of my workload. That letter was my contract. At that time the university was very pleased to be associated with the performances and broadcasts I reported, and asked annually for a list (no more) of any new publications.
Though Keele and York, where I had studied as a student, were new universities, they still clung, in many respects, to the Oxbridge model. There were senior tutors and senior common-rooms and a general feeling of affluence and privilege; not just a feeling, but actual privilege. In 1978 there were professors at Keele living in detached four-bedroom campus houses, for which they paid rent lower than the rent I paid on my two-bedroom flat. The rent had stayed the same since their appointment 15 years earlier. Some of those same professors could routinely be found on the steps of the library, or in the campus bars, always available for a chat. There were some, I was told, whose niche option courses regularly failed to recruit students, and who supervised very few dissertations, and who had hardly any research students. Some of them published their ongoing research in the occasional journal article without ever seeming to get ‘the book’ finished. There seemed to be little notion of ‘accountability’.
On the other hand, there was a sense that academics could be trusted, and there was complete academic freedom. No-one, in the Humanities at least, was going to tell you what research you should be doing; that was up to you. The message was: you have earned the right to be here, you are valued, and we want you to share your work with the community through teaching and research presentations. I don’t remember the term ‘blue sky thinking’ in those days, probably because the term wasn’t needed. It was taken for granted that an academic could launch into uncharted territory, and of course it was recognized that the culmination of investigation (the book) might take ten years to materialize. Scientists applied to research councils for funding for their projects, and those in the humanities applied to the British Academy or the Arts Council or their university for time off from teaching to undertake their research in libraries or at home. But through the 1980s everything began to change.
The ‘Research Selectivity’ exercise in 1986 was introduced to address the perceived need to allocate government research funding more efficiently. Though I remember well the howls of protest from departments suddenly faced with the onerous challenge of itemizing their five (5!) most important publications from the last five years, the exercise was in truth undemanding. It was also rather ineffective, and according to its champion, Sir Peter Swinnerton Dyer, some of the panels allocated funding even before the submissions had come in.[1] Subsequent exercises, now called ‘RAE’ (Research Assessment Exercise) became progressively more onerous; every member of academic staff had to be submitted; details of research income, and statements covering the ‘research environment’ and evidence of ‘esteem’ were also required. Upwards of 60 expert assessment panels were assembled to evaluate and score the research of a number of ‘cost centres’ comprising a collection of ‘units of assessment’ (subjects); each subsequent exercise revised the number and make-up of these panels. By 2001 the exercise had become a dominant part of academic life. The financial stakes were high. Universities set up administrative teams dedicated to advising departments on their submissions; ‘mock’ exercises were introduced; senior academics from other universities were recruited as ‘critical friends’ to help their rivals formulate their strongest submission. The RAE, now called REF (Research Excellence Framework), hung as a sword of Damocles throughout the assessment period, not just in the months leading to submission. The cost of the exercise is now enormous. In 2015 it was estimated that universities spent £4000 on REF preparation for each academic submitted.[2] For REF 2021 it will doubtless be far higher.
RAE/REF has completely changed the landscape in universities, and transformed the job description to which new lecturers now sign up. REF is uppermost in the minds of appointment panels, and applicants for new posts are well aware that, however much they have to offer as a teacher, a strong record of research publication is key to success. Departments are quizzed relentlessly by senior administrators on plans for increasing their research funding, for this is how non-specialists can most easily assess the worth of researchers, and because university finance officers want to cream off the ‘overheads’ which come with large research grants. Thus, pressure is put onto academics to chase funding for their projects from the research councils, and this too has become an elaborate and costly process. Large grants require proposals to set up networks, seminars, conferences, websites and plans for public engagement. The research councils make allocations according to criteria which they set, through their advisory panels of academics with their own preferences and self-interests. Getting a grant means framing an application in a way which addresses the themes set for each round, and also demonstrates how the proposed project will reach beyond narrow specialism to deliver ‘impact’ on the wider community. The application process can take months. Applications are peer reviewed, sent back for revision, re-reviewed and as often as not rejected. Thus, academics need to have more than one iron in the fire at a time; it can often seem that working on grant applications takes up more time than doing the research. For many researchers, more time is all they need for their work, and they resent having to spend time fighting for it in this way. Worse, it can often seem that the project one is proposing is not really the one that one should, or would want to be, doing, but rather a project that is likely to be funded. This is a complete reversal of the academic freedom which Swinnerton Dyer set out to support more fairly in 1986.
If there was little accountability in 1978, there is certainly plenty now. Accountability is a good thing. Universities and central government should know where research funding goes.
The situation which pertained when I was starting out was probably untenable, especially as universities began to expand rapidly. Government may have preferred to see more ‘selectivity’ in the sense that universities would deliver research in some subjects and not others, or that universities would concentrate their funding on a few individual academics and remove the research element from the contracts of others. In general, however, universities did not want to do that, priding themselves on departments which ‘teach from research’, believing that students deserve to be taught by academics with a direct involvement in current thinking. RAE developed as a way of supporting this notion through the allocation of funds to departments (via their university) according to the success and relevance of their work. The process of evaluating publications and PhD supervision for every member of a department’s team encouraged a greater sense of urgency (making sure there were publications within each RAE cycle: three years to begin with, increasing to seven years most recently). It also pressed them to clarify their aims and defend the originality of their research; and it encouraged departments to check that they were supporting their researchers properly.
These are all good outcomes of research assessment, but I would argue that the level of scrutiny achieved by the 1996 RAE was probably sufficient. That exercise felt tiresome but manageable. At that stage a small departmental team could take on the task of compiling the department’s return, requiring basic co-operation from other colleagues through the submission of factual data connected to their individual achievements. The essential criteria of Originality, Significance and Rigour were firmly embedded.
But because the exercise is in essence a competition, with financial prizes and bragging rights, the bar has been continually raised to the point where, now, no research academic is spared the misery of reading, understanding and attempting to interpret the opaque criteria and rubrics set out for each successive REF, and adapting their working methods to comply. In its latest incarnation, for example, the REF’s emphasis on ‘impact’ requires that research at the highest level is nonetheless conducted with a view to its value to non-academic stakeholders. It is often not obvious to researchers how this is to be achieved. If your academic life has been devoted to editing medieval manuscripts, for example, constant harassment from a research committee to demonstrate public participation or links with industry can seem irritating at best, and close to bullying at worst. Management committees plead that their incessant cajoling is designed to encourage and support academics towards achieving their most effective outcomes; but on the ground it doesn’t always feel like that. It can seem that one is being pushed into devoting inordinate amounts of time to activity whose only aim is achieving REF compliance. Further frustration arises from the fact that for most of the assessment period advisors are unable to answer some of the basic questions about the exercise, because aspects of the rules are under review. Thus, there is a constant sense of working in the dark and second guessing the format that will be required for certain elements of the submission. Having acted as a ‘critical friend’ myself, and read the reports of others, I know for certain that the lack of clarity and changes from one exercise to the next, mean that pre-assessments (mock exercises) can never offer more than informed speculation based on past experience. Even the basic definitions for grading principal research outputs (‘internationally significant’ ‘internationally recognised’ etc) are so open to interpretation that there can be no certainty.
If this is the name of the game, how surprising is it if some universities are tempted to gradually shift support away from well-established and formerly successful departments who have difficulty in meeting the demands of the new fashion? At the same time who could blame a professor who was enticed away from a successful career in industry with the promise of freedom to pursue research in a free-thinking environment set up to support innovation and experiment, for feeling that universities are actually less conducive environments than the share-holder driven companies they left? And who could blame a student for feeling that their lecturers, their thesis supervisor and their academic advisor, don’t seem to have enough time for them? They don’t. The combination of rising student numbers and the increasing pressure of REF has made their weekly timetables unmanageable. The leafy campuses and gleaming spires are still there, but those who work in them hardly notice any more.
Is it time to call time on REF? Is it time to row back the overelaborate and increasingly costly industry which grew from the desire to allocate research funding fairly and efficiently, but which no longer seems either fair or efficient? Could we let university lecturers get back to enjoying the research they are capable of doing and want to be doing, without the sense of having to negotiate an artificial, ever-changing, time and energy-sapping obstacle course? The aim of ensuring accountability has been achieved, surely? In my experience most academics are dedicated, articulate, professional people who know the importance of being able to communicate their research to peers and public. Is it necessary to bombard them constantly with demands to do more, do it faster, do it a different way – with the attendant sub-text: you are not doing enough? By 2015 Swinnerton-Dyer was already asking if REF had gone too far. Speaking specifically about the introduction of ‘impact assessment’ he commented: “The one question a modern civil servant fails to ask is ‘is it worth the extra effort?’”[3]
I would say it’s not. My sense is that the cost of the exercise itself, taking in all the costs associated with assessment as well as all the costs to universities themselves in preparing their submissions, is far too high. Far from ensuring the efficient use of public money, REF is squandering it. In addition, I would argue that it has encouraged an artificial research environment often led not by the pursuit of knowledge, but by the pursuit of funding.
[1] https://www.timeshighereducation.com/features/evolution-of-the-ref/2008100.article
[2] https://www.timeshighereducation.com/news/ref-2014-cost-250-million
[3] https://www.timeshighereducation.com/features/evolution-of-the-ref/2008100.article