Teaching Excellence Framework

The TEF ratings are based on statistics such as dropout rates, student satisfaction survey results and graduate employment rates.^[1] These are assessed by experts in teaching and learning who make a recommendation to a TEF panel, which includes academics and students, that will make the final award. Universities are measured across three areas: teaching quality, learning environment, and student outcomes and learning gain.^[8] For 2017, all institutions meeting basic standards will be allowed to raise fees.^[2] The first ratings were to have been announced on 14 June 2017, publication having been delayed from May due to the UK general election,^[9] but were postponed until 22 June after the election resulted in a hung parliament.^[10] It was expected that 20–30% of the institutions would be rated gold, 50–60% silver, and 20% bronze.^[11] The actual distribution, across all rated institutions including further education and alternative providers, was 26% gold, 50% silver, 24% bronze.^[12]

The ratings are described by the Department for Education as:^[13]

• Gold: "provision is consistently outstanding and of the highest quality found in the UK Higher Education sector"
• Silver: "provision is of high quality, and significantly and consistently exceeds the baseline quality threshold expected of UK Higher Education"
• Bronze: "provision is of satisfactory quality"

Institutions that do not enter the TEF or that do not meet the minimum quality threshold will not receive an award.^[14] Institutions with insufficient data for a full assessment but which meet the quality standards can receive an unrated provisional award.^[15]

The TEF ratings do not measure absolute performance, like traditional university league tables, but rather performance against benchmarks based on their student intake.^[16] A university with a low absolute dropout rate of 2% and a benchmark of 2% would thus be rated worse on this measure than a university with a much higher absolute dropout rate of 8% but a benchmark of 11%. The ratings are thus a measure of whether a university exceeds, meets or falls short of expectations based on the profile of students admitted and subjects taught.^[17]

The "initial hypothesis" for the ratings is based on six core metrics, for which institutions receive a double-positive flag, a positive flag, no flag, a negative flag or a double-negative flag, depending on whether they exceed or fall short of their benchmark by certain thresholds. These are:^[18]

• Teaching on my course (from the National Student Survey)
• Assessment and feedback (from the National Student Survey)
• Academic support (from the National Student Survey)
• Non-continuation (from the Higher Education Statistics Agency and Individualised Learner Record data)
• Employment or further study (from the Destination of Leavers from Higher Education survey)
• Highly skilled-employment or further study (from the Destination of Leavers from Higher Education survey)

Institutions received three or more positive flags and no negative flags are initially considered Gold; institutions with two or more negative flags are initially considered Bronze; all other institutions are initially considered Silver. This initial hypothesis can then be modified by the panel based on the written submissions and 'split' metrics (a breakdown of the core metrics by gender, ethnicity, age, disability, etc.). While the extent of these modifications was expected to be limited, changes were made to the initial hypothesis in 22% of cases. Among higher education institutions and alternative providers, three were downgraded, 17 were upgraded from Bronze to Silver, 15 were upgraded from Silver to Gold, and one was upgraded from Bronze to Gold.^[19]

Following the publication of the 2017 "trial year" results, the TEF is to undergo a "lessons learned exercise" that will feed into the 2018 exercise as well as a full independent review on its use of statistics by 2020.^[5][6][20]

After the publication of the results, the acting director of the Russell Group said that "TEF does not measure absolute quality and we have raised concerns that the current approach to flags and benchmarking could have a significant unintended impact",^[21] while the vice-chancellor of the University of Southampton, which was rated bronze, said "There is no logic in our result at all", and that he had "deep concerns about its subjective assessment, its lack of transparency, and with different benchmarks for each institution removing any sense of equity and equality of assessment". He also pointed out that exceeding the benchmark by what the TEF considered a significant margin was much easier for institutions with lower benchmarks – to beat its benchmark on drop-out rate of 4.5% by the required two percentage points, Southampton would have to have achieved a drop-out rate of only 2.5% – leading him to conclude that "the benchmarking is fundamentally flawed".^[22]

Analysis of the results and the panel statements by higher education policy thinktank Wonkhe noted that the University of Nottingham, which had a positive flag for highly skilled employment and a negative flag for student satisfaction, was awarded gold, "the presumption that a negative flag would rule out Gold hav[ing] been overturned by the panel, perhaps because the TEF guidance also steered the panel away from over-reliance on NSS scores." Similarly, the University of Bristol overcame two negative flags – both in NSS-related categories – to be awarded silver, but the University of Liverpool, with the same number of negative flags, received bronze, "perhaps because one was not in an NSS-derived category". Wonkhe further noted that "it seems perverse that an institution – in Bristol’s case – which was ‘notably’ below benchmark should receive a higher outcome than Liverpool for which the statement is softer" and that "for institutions with a similar data pattern to Bristol’s, such as Southampton (with two negative flags in the same categories, but which wasn’t upgraded to Silver) there could be some well-deserved anger. And if you look to Durham, with its one positive flag, and no negatives, it only has a Silver result when compared to Nottingham’s Gold."^[23][22][24]

Nick Hillman, the director of the Higher Education Policy Institute (HEPI), said after the results were released that "the fact that some of the results seem surprising suggests it is working", as it was designed to be different from other league tables. He added, however, that "in this early guise, the TEF is far from a perfect assessment of teaching and learning. While it tells us a lot of useful things, none of them accurately reflects precisely what goes on in lecture halls."^[25]

In response to some of these points, Chris Husbands, the chair of the TEF panel said that the TEF was not supposed to be a "direct measure of teaching" but rather "a measure based on some of the outcomes of teaching", that different outcomes for institutions with similar metrics was to be directed as "The TEF is metric-led, not metric determined" and that the TEF was "a relative, rather than absolute measure of university and college performance". He also noted that "whilst universities have been impressive at widening participation they have been less assiduous in combatting the impact of disadvantage after students enroll", and said that the TEF was with doing as it had "raised the profile of teaching" and "focused attention on things which need to be done better".^[26]

At a conference held in late June, Imperial College London's Vice Provost, Simone Buitendijk, stated that the TEF was a 'godsend' for higher education. She stated: "For people like me, a vice-provost, TEF exercises are actually a godsend because what happens is, for the first time, the president and the provost start paying close attention to the quality of teaching...It’s not a bad thing if there is very close attention being paid to teaching at research-intensive universities." University College London's President, Michael Arthur, suggested that the TEF would bring benefits to universities in the long-run.^[27]

Eighteen institutions chose to appeal their 2017 ratings, including at least four from the Russell Group. Of eleven institutions that said they were appealing, three were awarded Silver and eight Bronze. Appeals must demonstrate a "significant procedural irregularity" and cannot challenge the academic judgment of the TEF panels. At least one institution, Swansea University, submitted an intent to appeal but had its case ruled inadmissible.^[28] Only the University of East Anglia was re-graded on appeal, moving from silver to gold, and the only other change made was a revision of the statement of findings for Durham University; all other appeals were rejected. In addition, four institutions appealed their eligibility for provisional TEF awards, of which three were successful.^[29]

Prior to their publication, the TEF results were expected to be significantly different from the usual rankings of universities in the United Kingdom. The Guardian reported in May 2017 that a number of "world-renowned" universities were at risk of receiving a bronze rating; particularly London institutions, which normally have lower student satisfaction scores. The Times Higher Education also reported in early June 2017 that the members of the Russell Group (two in London) were in danger of being rated bronze, while post-1992 universities were expected to do well. However, universities also submitted additional written information to the TEF to clarify their institutional context: the head of King's College London said that he hoped this would raise the institution from a bronze to a silver rating, while SOAS noted that the financial cost of living in London meant that the student retention rate in the city was lower than the national average. The director of HEPI, Nick Hillman, said that there might not be any gold-rated universities in London, but that for institutions such as the London School of Economics this would not have a significant impact as "[i]ts name and reputation for research excellence will trump any negative press it gets from the TEF".^[9][30] Similarly, a "Mock TEF" carried out by the data analytics team at Times Higher Education in 2016 showed that while Russell Group institutions did well on absolute results, once results were adjusted for student intake only Cambridge, Durham, Birmingham, Exeter and Newcastle (in order of their ranking) were definitely rated as gold, although this did not include any adjustments that may be made for the qualitative submissions from institutions.^[31]

The link between the TEF and tuition fees has been criticised, with the National Union of Students (NUS) voting in 2016 to boycott the National Student Survey (NSS), the results of which feed into the TEF, unless the link was broken.^[32] There were suggestions that the boycott may have backfired as participation levels in the National Student Survey rose nationally with some commentators linking this to the additional publicity from the boycott.^[33] However 12 institutions, including Cambridge, Oxford and several other Russell Group universities, were omitted from the NSS results in 2017 due to having less than the required 50% of final year students complete the survey; which is seen as evidence that the boycott was successful in at least some of those institutions. However, with the boycott coming too late to prevent the use of the NSS in the 2017 TEF, it would be necessary to sustain it for a further two years in order to have any effect and the national NUS conference voted in 2017 not to debate a continued boycott or send it to their National Executive Committee. The boycott may also benefit universities such as Bristol for which student satisfaction has traditionally been low.^[34] Whether metrics such as student satisfaction and employability data are valid measures of teaching quality has also been questioned.^[35]

Some top universities threatened to boycott the TEF, fearing that reputational damage might outweigh potential gain.^[36] However, in January 2017, just prior to the deadline for signing up, the higher education minister said that "almost all" universities would, after all, take part, and Times Higher Education was able to confirm that all but five of the English Russell Group universities had committed to participating, with the others not yet decided.^[37]

• About the TEF, Office for Students