Challenges and Realisations from the

Higher Education Academy/JISC

Benchmarking and Pathfinder Programme

An End of Programme Review by the Higher Education Academy,

Evaluation and Dissemination Support Team

September 2008
 Foreword
Derek Morrison

Associate Head of e-Learning and Benchmarking & Pathfinder Programme Leader

Higher Education Academy
September 2008

In August 2005 the Higher Education Academy invited me to consider a secondment from
my University of Bath role to the then relatively new agency in order to assume leadership of
a benchmarking of e-learning exercise which would also be the gateway to an associated
'pathfinder' programme. I have undertaken many such secondments during my working life
with the best experiences and outcomes arising from contexts and environments where there
are big challenges to overcome but also the opportunities to apply creative thinking and the
freedom to build a strong team to help overcome said challenges. The Benchmarking &
Pathfinder Programme probably represents one of the most rewarding initiatives with which I
have been involved. The report by Terry Mayes, Jane Plenderleith, and Veronica Adamson
provides the necessary detail but in this foreward I outline the programme leader's view of
what has enabled the initiative to realise a significant part of its overall potential although I
readily acknowledge that there is much more still be done.

Several things stand out for me. First, as the report points out, there was the need to consult
with a sector that was deeply suspicious of concepts like benchmarking and possible misuse
of the results of the exercise. Second, in an immature area of development it was essential
that the Academy was not seen to confuse particular benchmarking tools with processes and
instruments that could facilitate organisational insights and development, i.e. we wanted
more than an introspective or technical exercise based on populating an online tool with data
which would then attempt to offer an auto comparison and relative positioning in what is an
extremely diverse UK HE sector. Third, the programme needed to be progressive so that the
analytical and reflective opportunities offered by benchmarking fed into actual institutional
decision-making to be realised in the Pathfinder Programme in the form of activities which
could be sustained beyond the initial funded period. Fourth, the programme needed to be
inclusive, so that institutions of all profiles and at all stages of e-learning development felt a
sense of ownership and control but were also prepared to contribute to programme activities
and, collaborate/share with each other. Finally, we wanted to introduce a model in which
agencies like the Academy are not just perceived to be funders but also make valued
practical contributions to helping participating institutions realise their goals. We did this by
embedding an expandable team of benchmarking advisers and ‘critical friends’ whose
guidance and benign interventions were much appreciated by many of the 77 participating
institutions.

Although we can always improve, and there is still much left to do, I think the programme has
achieved a great deal since January 2006. If I were asked to identify the major outcomes or
insights I would assert that:
1. The programme has been very inclusive to a point where the "team" supporting and
realising the programme now includes many of the participating institutions
themselves. The CAMEL cohort self-support groupings, and the Pathfinder Network
Projects, are of particular note in this regard.
2. We needed to readjust our sector-level Pathfinder concept to reflect the reality that
the insights gained from the benchmarking exercise meant that it was more useful for
the projects to focus on being 'Pathfinders' at institution level rather than seek to be
an elite sector-level exemplar. As the report highlights, the inclusive nature of the
 programme meant that the journey for each institution was unique, but all gained
something valuable from their involvement.
3. Many of the Benchmarking & Pathfinder institutions themselves now offer a valuable
resource for the sector; and relatively modest funding has facilitated cross-
institutional support and sharing of experiences.

As the report you are about to read highlights, the Benchmarking and Pathfinder Programme
was not really about technology at all, it was about people and the richly diverse
organisations they work in. Any successes achieved by this programme have been because
of these people "on the ground" and their ability to successfully negotiate the challenges they
faced. Particular plaudits must go, therefore, to the project teams in each of the institutions
who responded so positively to what, for many, was the new experience of having either a
specialist adviser and – or – critical friend embedded into the infrastructure of the programme
rather than the more conventional model of formal and limited interactions between agency
funders and institution projects. I also believe that we have all benefited from the magnificent
contributions of the uniquely experienced, knowledgeable and talented team of advisers and
critical friends. This team have all made contributions far beyond their contracted hours
because of their commitment and belief in what we were trying to achieve. For this reason in
this foreword and in no particular order I wish to applaud the much-valued contributions of:

• Professor Paul Bacsich (benchmarking adviser, Pathfinder critical friend, and

benchmarking "Wiki Meister")
• Professor Peter Chatterton (benchmarking adviser and Pathfinder critical friend)
• Dick Hill (benchmarking adviser)
• Bruce Carter (benchmarking adviser)
• Svava Bjarnason (the then Director of ACU/OBHE - benchmarking adviser)
• John Fielden (ACU/OBHE - benchmarking adviser)
• Allan Schofield (ACU/OBHE - benchmarking adviser)
• Cliff Wragg (ACU/OBHE - benchmarking adviser and Pathfinder critical friend)
• Professor Betty Collis (Pathfinder critical friend)
• Professor Grainne Conole (Pathfinder critical friend)
• Andrew Comrie (Pathfinder critical friend)
• Dr Jane Plenderleith (HE Academy Evaluation & Dissemination Support Team)
• Veronica Adamson (HE Academy Evaluation & Dissemination Support Team)
• Professor Terry Mayes (Team leader HE Academy Evaluation & Dissemination
Support Team)

It was the synergy achieved by combining the efforts and "benign interventions" of the above
list of advisers and critical friends, the energy and co-operation of the individual project
teams, plus the community-building activities of the five Pathfinder Network Projects that I
believe offers the sector such a useful reference model for future developments of this type.

There are already a wealth of reports and briefings currently available on both the
Benchmarking and Pathfinder weblogs with more pending. We will also provide a portal page
to all of the aggregated reports on the Academy web site.

Finally, I think it is important also to applaud the role of HEFCE, particularly Dr Liz Beaty, the
then HEFCE Director of Learning and Teaching, who charged the Academy with leading this
demanding initiative and then trusted all of us to deliver. I believe this has been as successful
a programme as it could be in the time available to us; but yet there is so much left to do.
Seventy-seven institutions took part. That means there are still many institutions that did not.
But, as the presentations at ALT-C 2008 and other recent national and international events
have proved, there is now a very high level of interest in what has been achieved, and in
how it has been achieved. The challenge for us all now is in how to maintain the impetus.
Contents
Purpose and Structure of this Report .......................................................................................1 
Executive Summary .................................................................................................................3 
1  The Programme’s Rationale and Strategic Background ...................................................5 
2  Approach to Evaluation .....................................................................................................6 
3  The Benchmarking Programme ........................................................................................7 
4  Benchmarking Methodologies...........................................................................................9 
5  The Benchmarking Process ............................................................................................12 
6  Commentary on the Benchmarking of e-Learning Exercise............................................14 
7  The Pathfinder Programme.............................................................................................19 
8  The Design and Operation of the Programme ................................................................20 
9.  Reporting.........................................................................................................................23 
10.  Critical Friends.............................................................................................................24 
11.  The Pathfinder Project Themes...................................................................................25 
12.  Key Messages from the Programme ...........................................................................26 
Appendix A: Benchmarking Institutions by Phase and Methodology ..................................... 29
Appendix B: Pathfinder Projects by Institution ....................................................................... 31
Purpose and Structure of this Report
This report attempts to highlight the main lessons that have been, or may be, drawn from the
Academy’s e-Learning Benchmarking and Pathfinder Programme, calling on the authors’
experience of providing close support for the programme itself, and extracting key messages
from the participants’ own evaluation reports. In that sense it can be regarded as having
elements of a synthesis, as well as both formative evaluation and summative review.
Much has already been written and reported on the programme. The Academy maintained a
weblog for both the Benchmarking Exercise1 and the Pathfinder Programme2 with links to
relevant reports and documentation, and encouraged regular postings on general issues of
interest in the domain of e-learning as well as operational information about the programme.
Each of the participating institutions also maintained a weblog of their benchmarking and,
where appropriate, Pathfinder projects. An evaluation of the Pilot Phase was produced by
Professor Terry Mayes in August 2006. Consultants responsible for the various
benchmarking methodologies that have been supported in the programme have produced
reports at the end of each of the three phases (Pilot, Phase 1 and Phase 2). The Phase 2
reports from the teams of consultants involved included discussion of some of the key points
of development in both the methodological approaches and the institutional findings about
e-learning in the course of the programme. The Evaluation and Dissemination Support Team
(EDSuT) produced summary reports at the end of Phase 1 and Phase 2 of benchmarking. All
the Pathfinder projects have made available their reflective accounts of the Pathfinder
‘journey’, and almost all have produced briefing papers, summarising results and key
messages from their project for a defined readership. EDSuT also reported on the
experiences of the Critical Friends at various points throughout the Pathfinder programme.
This report draws on all of those sources of information. It starts by outlining the three
Phases of the benchmarking exercise in terms of participating institutions, the methodologies
used and support activities. It presents a brief summary of the five benchmarking
methodologies and their development through the Programme. It summarises the
experiences and key messages from institutions participating in the benchmarking process,
and offers a commentary on some of the issues, questions and challenges arising from the
benchmarking exercise in general. The second half of the report describes the Pathfinder
programme. It discusses Pathfinder’s relation to benchmarking, its approach to evaluation,
and the innovations it introduced. Finally, the report summarises the main outcomes of the
Pathfinder programme and considers its key messages for future development. Where
appropriate, this report points the reader to other documents, in which more comprehensive
descriptions may be located. Most of these are available to download from the Academy’s
weblogs.
The title of this report echoes the title of an event held in June 2008 at the conclusion of the
Benchmarking and Pathfinder Programme. Like that event, this report is intended as a
celebration of the achievements and outcomes of the Programme, a review of these
achievements summarising a range of institutional and agency perspectives, and an initial
consolidation of programme outputs and outcomes for wider dissemination and impact in the
sector.

1
http//elearning/heacademy.ac.uk/weblogs/benchmarking/
2
http//elearning/heacademy.ac.uk/weblogs/pathfinder/

Page 1 of 50
 Undertaking our Pathfinder project has been a sharp affirmation of the value to the sector
of creating a non-threatening context in which to share ideas and problems, where
difficulties can be discussed openly and treated sensitively, where new ideas can be
generated and confidences respected.

Extract from University College Plymouth St Mark & St John Pathfinder Journey report.

Almost imperceptibly it appears that institutional maturity for the uptake of e-learning has
moved beyond the assurance that all modules are using an institutional VLE for baseline
communication and information-sharing with students, to a position where innovation in
approaches and technologies can address enhancement strategies. This is evidenced by
the large numbers of academics attending our dissemination events and feeding back on
intentions to change practice. As a result Pathfinder ought to demonstrate to the funding
councils that trust, rather than measurement and metrics, leads to contextually-significant,
continuous improvement.

Extract from the De Montfort University Pathfinder Project Report

Pathfinder funding allowed us a period of reflection and action, focusing attention and
resources on e-learning. We have learnt a lot.

Extract from the University of Warwick Pathfinder Journey Report

Acknowledgements

We record here our gratitude to all those who have participated in the programme, ensuring
not just a successful outcome for the projects, but the building and strengthening of a real
community. Special thanks must go to Derek Morrison, for his leadership and drive; to Liz
Pearce and Ellie Spilman for their thoughtful and effective support; to the Benchmarking
Consultants and Critical Friends for their enthusiasm and expertise, and finally to the project
teams themselves who have responded to all pressures with good humour and much
commitment.

Terry Mayes, Veronica Adamson and Jane Plenderleith

Page 2 of 50
Executive Summary
1. The programme has achieved its primary aim of building e-learning capacity and
embedding good practice into mainstream provision. All of the institutions report real
benefits from participating, and the project reports contain a message of significant
progress in understanding e-learning processes, practice and provision. There is a
sense of growing confidence about the use of technology for learning and teaching,
and a much deeper understanding of the issues for institutional policy.
2. The programme has achieved a wide penetration across the sector, with 77 institutions
participating in at least one of the benchmarking phases. The level of engagement
achieved for the wider programme from the individual projects has been very high.
Institutions have responded well to the opportunity to take responsibility for their own
approach, with the lead agency taking a facilitative rather than a supervisory role.
3. The programme has been characterised by trust. The benchmarking exercise
depended on the institutions trusting the benchmarking consultants with their
confidential analyses, and trust between institutions willing to share their analyses
across a cohort of peers. The Pathfinder institutions have been trusted to devise
projects based on their own individual strategic review and analysis, the projects have
been trusted to conduct their projects effectively and reflectively through building trust
relationships with Critical Friends and with each other, and through sharing their
reflections on the challenges. The direction from the programme, and its administrative
style, has been light-touch, in keeping with HEFCE’s aim to achieve institutional
ownership of the programme. The programme’s overall approach has been widely
appreciated by the projects themselves.
4. Institutions were using benchmarking in one of two ways: to ‘prove’ that they are on the
right track, or to ‘improve’ their current practice through a rigorous and sometimes
painful process of self-review. This distinction between ‘proving’ and ‘improving’ is
relevant to definitions of formative and summative evaluation and reflects the
essentially evaluative nature of benchmarking.
5. The benefits of expert consultant facilitation, support and brokerage were consistently
identified throughout the programme.
6. The Academy’s policy of ‘constrained diversity’ served to stimulate developing
understanding of benchmarking methodologies and their application.
7. The programme has achieved the unusual goal of building capacity primarily in areas
of acknowledged weakness, rather than strength. This is a consequence of awarding
Pathfinder funding to an institution only after the benchmarking stage had been
satisfactorily completed.
8. Although the programme did not require benchmarking outcomes to become public
(though several institutions chose to place their reports in the public domain) the
programme, through the Pathfinder ‘journeys’, has furthered HEFCE’s goal of
presenting a rich picture of where much of the sector is positioned with regard to the
wider, and evolving, e-learning landscape.
9. Pathfinder has introduced three innovations for a programme of this scope. Each has
been reported by the projects themselves as highly effective. These are:
• An ‘Evaluation and Dissemination Support Team’ that has worked formatively inside
the programme, yet has remained distinct from the programme’s management. This
team has offered programme-wide support for the projects’ own evaluation
methods. It also presented reflective views and insights, and summary reviews and
reports, from an external, objective standpoint.
• The clustering of projects, in phase 1 of Pathfinder, into groups of four, with the
encouragement (through additional funding) to each cluster to learn from each other
in an open and sharing way, following the ‘CAMEL’ model (see section 4.5 below) .

Page 3 of 50
 This has been reported by the projects, without exception, to have added a very
successful collaborative dimension, and has enhanced the outcomes across the
board.
• The appointment of a Critical Friend to the institutions in each cluster, with a role of
‘insider’ support, yet offering guidance and expertise often not available to an
individual project from within the institution. This, too, has been widely appreciated
by the projects.
10. The Academy and JISC have collaborated in running the programme, though the
Academy has played the lead role. In many of the projects both agencies have a joint
interest – their specific areas of organisational focus coming together in projects that
are essentially concerned with the interaction of technology and pedagogy. The
programme has also drawn interest from the QAA, and the need for the Leadership
Foundation to play a role has been expressed. The benefits of a CAMEL approach
might usefully be achieved by all these agencies working together in that way.
11. Many participating institutions have stated their intention to repeat the benchmarking
exercise in some form at regular intervals. It is a small step from that intention to the
formal inclusion of benchmarking in institutional review procedures.
12. There is an emerging interest across the programme as a whole in a subject-based
approach to sector-level benchmarking, with particular reference to the Subject Centres
and some CETLs as both a focus for benchmarking activities and as a forum for
reflecting and acting on some programme outputs and recommendations.
13. Five of the original pilot Pathfinders were awarded funding to take their approach to the
wider sector and all have done so to great effect. Consequently, there are now special
interest groups (SIGs) emerging around key issues (quality and e-learning, course
redesign, podcasting and pedagogy, researching the student experience). It is too early
to comment on the sustainability of the communities that have been drawn to these
SIGs, although initial signs are promising that these will continue to play a sector-wide
role. Modest ongoing support from the agencies for these emerging communities is
likely to produce significant leverage.
14. If the key outcome of the Benchmarking and Pathfinder programme were to be
captured in a single phrase, this could be ‘building relationships’. The transforming
theme of the programme has been in directly empowering people to use technology to
enhance practice, processes and provision. The programme has helped to develop
powerful and potentially lasting relationships between institutional departments and
services, between institutions with often vastly different profiles and cultures, and
between the sector and support agencies.
15. Institutions still struggle with the challenge of positioning these change projects in a
way that allows them to be optimally effective. In some ways this reflects tensions in a
culture in the sector that accords significant autonomy to academic disciplines, but
seems to hold in lower esteem areas of provision that are designated as ‘support’.
Despite its many successes, Pathfinder has encountered difficulties in gaining a truly
institutional commitment to a significant minority of its projects.
16. The programme has focused on e-learning, but has achieved a broader significance for
quality enhancement and continuous improvement in all aspects of provision. There is
increasing strategic recognition that technology-based solutions are integral to all
aspects of the sector’s business, with particular reference to the core activities of
learning, teaching and assessment.
17. The overall programme has been a significant success and it seems important that the
progress made should be built on through further enhancement-led initiatives. The
programme’s ethos and processes provide a reference model.

Page 4 of 50
1 The Programme’s Rationale and Strategic Background
1.1 In response to publication of the HEFCE Strategy for e-Learning (March 2005)3, the
Higher Education Academy and JISC submitted a proposal and draft project plan to the
HEFCE for a Benchmarking Exercise and related Pathfinder Programme in June 2005.
The Benchmarking Exercise was intended to help institutions establish where they
were in regard to the embedding of e-learning. The Pathfinder Programme was
designed to help selected institutions, on behalf of the sector, to identify, implement
and evaluate different approaches to the embedding of technology-enhanced learning
in ways that result in positive institutional change. The proposal was accepted by
HEFCE, and the Academy was invited to take the lead role in partnership with JISC in
planning and implementing the proposals.
1.2 An important element in the policy context that influenced this strategy had been the
restructuring of the UKeU, an episode which many in the sector interpreted as a failure
of the Funding Councils’ attempts to exploit the potential of technology-based learning
through centrally-driven initiatives. It was the reallocation of funds from that initiative
that provided the resource for the Benchmarking and Pathfinder Programme. The
strategic background to the programme is the emphasis in the HEFCE strategy on
placing the responsibility for, and ownership of, e-learning development clearly with the
individual institution.
1.3 In the strategy, HEFCE asked the Academy and JISC to identify and implement a
benchmarking tool for HEIs which might also provide information, at a sector-wide
anonymised level, to help HEFCE and its partners daw conclusions on the state of
e-learning, progress towards embedding it, and the impact of the 2005 strategy:
... we should know more about the present state of all forms of e-learning in HE.
This is essential to provide a baseline to judge the success of this strategy.
However, understanding HE e-learning is not just a matter for HEFCE. Possibly
more important is for us to help individual institutions understand their own
positions on e-learning, to set their aspirations and goals for embedding e-
learning – and then to benchmark themselves and their progress against
institutions with similar goals, and across the sector. (Our emphasis)
In taking forward this project, the Academy was encouraged to take a ‘light touch’
approach and to be responsive to the needs and expectations of institutions.
1.4 A Town Meeting was held at Academy York on 9 November 2005 in order to consult
with the sector about the proposals for a national Benchmarking Exercise and
Pathfinder Programme. As a result of the feedback from this meeting the original
project plan was amended during November 2005 and resubmitted to the HEFCE
which accepted the revisions in December 2005. Not surprisingly, at first some
regarded the Benchmarking/Pathfinder programme call as representing another
attempt to steer e-learning development centrally and at the Town Meeting there was
widespread opposition to the proposal that all participants should undertake a common
benchmarking methodology. Partly this seemed to reflect the English sector’s attitude
towards quality assurance, shaped by the experience of QAA audit. It also reflected a
determination not to engage in any sector-wide activity that might allow the publication
of league tables.
1.5 The HEFCE strategy placed emphasis on the embedding of technology applications
into all aspects of institutional activity. An internal Academy discussion paper
questioned whether placing an emphasis on embedding across the board tended to

3
http://www.hefce.ac.uk/pubs/hefce/2005/05_12/

Page 5 of 50
 obscure the need to evaluate where the added value of the technology really lies.
Throughout the duration of the programme, discussions returned frequently to the
really challenging issue of how to assess the extent of the enhancement that was
obtained through technology, and how to connect the enhancement directly to the
student experience, including ultimately the depth of student learning. Some of the
debate appears on the Academy Pathfinder weblog4, but in the main the difficulty of
this issue is acknowledged implicitly in several of the project reports. Each project was
encouraged to find ways of evaluating its own impact.
1.6 The HEFCE strategy for e-learning was developed with an intended timeframe of ten
years, and included eight Measures of Success for the embedding of e-learning over
this period. It was anticipated that interim reviews and refinement of the strategy would
be informed by the benchmarking exercise and Pathfinder Programme. These
attempted to describe outcomes that would confirm that e-learning had been effectively
embedded. Even these, however, rather avoided the issue of the extent to which such
embedding would bring about enhancement of student learning. In the terms that the
issue was discussed on the weblog these are still ‘input’ measures (of provision, rather
than ‘output’ measures of learning achieved).

2 Approach to Evaluation
2.1 The rationale for evaluation was set out in the document dated 05.10.06 by Derek
Morrison and Liz Pearce entitled ‘A Capacity Enhancement Approach to institutional
evaluation and dissemination for the HE sector Benchmarking Exercise and the
Pathfinder programme’. The evaluation emphasis was formative.
Rather than evaluate the projects per se the primary purpose of the Evaluation &
Dissemination Support Team is to build evaluation, reflection, and dissemination
capacity in the institutions taking part in the Benchmarking and Pathfinder
initiatives. Put more simply, rather than the Academy evaluating projects the aim
is to help projects evaluate themselves; with, in the case of the Pathfinder
Programme, the input of peers.
2.2 Support for the projects to undertake evaluation of their own effectiveness was
provided at the programme level. As the programme proceeded from the pilot stage to
phase 1, the support at the programme level was strengthened. During the pilot phase
of Pathfinder and phase 1 of Benchmarking an evaluation and dissemination support
team (EDSuT) was recruited. This was significantly strengthened for phase 1 of
Pathfinder by recruiting a further six Critical Friends.
2.3 An innovative approach to evaluation was attempted through an emphasis on
continuous reflective input to a weblog that would be maintained by the participating
institution. The reporting requirements were light touch, but were expected to make
appropriate links to the project’s blog, where a reflective elaboration would be
accessible.
2.4 Each of the benchmarking institutions was asked to complete a short summary report
to enable the Academy to gather views and lessons from the benchmarking exercise in
a consistent way across all of the methodologies. The EDSuT produced summary
reviews of the views and experiences of participating institutions at the end of Phase 15
and Phase 26.

4
http://elearning.heacademy.ac.uk/weblogs/pathfinder
5
http://elearning.heacademy.ac.uk/weblogs/benchmarking/?p=315
6
http://elearning.heacademy.ac.uk/weblogs/benchmarking/?p=371

Page 6 of 50
2.5 Pathfinder projects were asked to approach evaluation by identifying stakeholders and
using a simple framework to extract the key evaluation questions and then to track the
impact of the interventions. A mini-guide to evaluation for projects in this area was
produced by Glenaffric Ltd from the Evaluation Handbook previously developed for
JISC innovation programmes7, and distributed to the Phase 1 projects.
2.6 The conceptual framework for capacity enhancement presented two strands:
'Evidenced by' and 'Supported by'. All of the activities in the evidence strand were
intended to be undertaken by the institution taking part in the programme, and the
evidence gathered and recorded by the project itself. The support strand was seen as
providing a set of resources that would offer direct guidance and encouragement. It
was anticipated that the evidence for capacity enhancement would constitute a very
diverse set of outputs, including qualitative assessments and reflective accounts.
These would be summarised in the reports from each project, and a synthesis provided
by the EDSuT. The EDSuT also conducted face-to-face or telephone interviews with
the Pathfinder pilots, and recorded some that were downloadable as audio files from
the weblog site. For the Pathfinder phase 1 projects an extra layer of support was
provided, in the form of Critical Friends. By this stage the emphasis had moved very
much away from external evaluation and the Critical Friends were positioned as
formative support, working from within each project. The EDSuT interviewed each
Critical Friend on at least two occasions and reports were produced.

3 The Benchmarking Programme

Overview
3.1 The Pilot phase of the benchmarking exercise ran from January to July 2006. Phase 1
started in October 2006, and Phase 2 in May 2007. By the end of the programme in
July 2008, a total of 77 HEIs had undertaken an internal e-learning benchmarking
exercise supported by the Academy, including some Welsh and Scottish institutions
(see Appendix A). The following table illustrates the number of institutions using the
different methodologies in each Phase:
Methodology Pilot Phase 1 Phase 2 Total
ELTI 3 6 0 9
eMM 1 0 7 8
Pick&Mix 3 7 10 20
MIT90s 1 4 0 5
OBHE/ACU 4 21 10 35
Phase Total 12 38 27 77

3.2 Complementing the HEFCE-funded benchmarking activity, the Academy has also
supported a self-organised grouping of Scottish Universities who formed a Scottish
Benchmarking Group (SBG) and were monitoring the exercise in order to inform their
own projects.
3.3 The benchmarking exercise was managed by the HE Academy e-Learning team, and
supported by the EDSuT led by Professor Terry Mayes with Veronica Adamson and
Jane Plenderleith of Glenaffric Ltd. Professor Mayes supported the Pilot phase; the
extended team support Phases 1 and 2. The main role of the EDSuT was to help
institutions to enhance their own capacity to evaluate and disseminate, complementing

7
http://www.jisc.ac.uk/media/documents/programmes/elearningcapital/evaluationhandbook.pdf

Page 7 of 50
 the work of the methodology support consultants who have a defined operational role.
This included identifying issues, potential synergies, and sources of information across
benchmarking methodology clusters, providing support for further development and
innovation opportunities in relation to action planning arising from benchmarking, and
synthesising key themes and messages arising from the exercise for the sector.
3.4 For each Phase of the exercise, the Academy organised a Town Meeting for interested
institutions to introduce the benchmarking exercise and provide guidance on the
various methodologies. Subsequent start-up meetings were also organised for each
Phase, providing an opportunity to discuss and reach a shared understanding of
anticipated outcomes and surface some of the key areas of interest and expertise in
the participating institutions. Start-up meetings also offered a forum for discussing
approaches, methods and expectations with the various consultancy teams, and
introduced the programme-level support provided by the EDSuT.
3.5 In April 2007, the Academy organised a Programme-level symposium and workshop
entitled ‘Taking Stock’ for participants in the Benchmarking of e-Learning Exercise and
the Pathfinder Programme. The aims of the event were to take stock of the progress
currently made in both the Benchmarking and Pathfinder Programmes, to shape the
stages to come by reflecting on the experience of the current participants, and to
expose all participants to multiple perspectives on current trends in institutional e-
learning development across the UK.
3.6 A final programme-level benchmarking meeting at the end of Phase 2 in January 2008
provided a forum for a review of the Phase 2 and the benchmarking exercise in
general, and an exploration of next steps for participating institutions. It also presented
an opportunity to take stock of the implications of and outcomes from benchmarking in
relation to the development of national e-learning strategies and other sector
developments in e-learning.
3.7 The Academy was also concerned to support the development of a community with a
theoretical and academic interest in benchmarking e-learning as well as a community
of practitioners engaged in the process of benchmarking e-learning. The development
of a concordance exploring of the similarities and differences and mapping points
between different approaches was undertaken as a separate, but complementary,
activity during the benchmarking pilot phase. This work was continued and extended
during the later Phases through the development and maintenance by Professor Paul
Bacsich of a Benchmarking wiki8.
3.8 A further important element of support for benchmarking was the Academy’s
benchmarking weblog9. The weblog was used as a focus for reflection on issues of
interest pertaining to benchmarking or arising from the process, and to communicate
programme-level information and guidance (for example about reporting and
programme meetings). Consultants also made weblog postings about general
methodological issues and operational information. For Phase 2, the EDSuT also
developed Helga10, a social networking website based on the open source Elgg
system11 for benchmarking institutions to establish their own communities and groups
for private online reflections, team discussions, and sharing resources.

8
http://elearning.heacademy.ac.uk/wiki/index.php/BenchmarkWiki:About
The concordance and wiki were developed and maintained on behalf of the Academy by Professor
Paul Bacsich, benchmarking consultant and leader of the Benchmarking e-Learning Associated
(BELA) group.
9
http://elearning.heacademy.ac.uk/weblogs/benchmarking/
10
http://www.glenaffric.co.uk/helga/
11
http://elgg.org/

Page 8 of 50
4 Benchmarking Methodologies
4 Introduction
4.1 It was recognised at the outset of the Programme, informed by feedback at the
inaugural Town Meeting in November 2005, that there was no simple solution or
definitive methodology for benchmarking e-learning. The sector is diverse, institutions
are protective of their autonomy, and methodologies specifically designed for
benchmarking e-learning in HE were limited in number. Most were still in the early
stages of development, had only been exercised in other contexts, and some just with
very limited pilots. Hence all the potential benchmarking e-learning methodologies
required adaptation for implementation in the UK HE context. The Academy believed
there was no single benchmarking method that would suit the needs of all institutions,
and that the final arbiters of what was appropriate should be the institutions involved in
the process themselves. However, it was also acknowledged that offering and
supporting an infinite variety of benchmarking approaches was neither feasible nor
desirable. Taking all these factors into consideration, the Academy offered to support
five benchmarking approaches that had been identified by the institutions that had
responded to the initial expression of interest.
4.2 All five methodologies were implemented during the Pilot phase. The Pilot was not,
however, designed as a controlled trial of the methodologies with a view to a
consensus approach or recommended single approach. For Phases 1 and 2, the
Academy implemented a system of ‘constrained diversity’, supporting methodologies
which were proposed for use by four or more institutions. This system was introduced
following the evaluation of the Pilot phase which concluded that while there was no
methodological reason to include or exclude a particular benchmarking approach,
participating institutions had drawn value from coming together in groups at key points
in the benchmarking process, and from the general opportunities for networking and
collaboration presented by cohort groupings. This resulted in four methodologies being
offered in Phase 1 (ELTI, MIT90s, OBHE/ACU and Pick&Mix) and three in Phase 2
(eMM, OBHE/ACU and Pick&Mix).

The Methodologies
4.3 ELTI (Embedding Learning Technologies Institutionally) was originally developed as
part a JISC project which completed in 2003. ELTI focuses on three key areas: Culture,
Infrastructure and Expertise. The approach was designed to inform the process of
embedding learning technologies, assist in developing appropriate institutional
structures, culture and expertise and encourage collaboration.
4.4 eMM (e-Learning Maturity Model) was developed by Professor Stephen Marshall of the
Victoria University of Wellington12 for application in the context of the New Zealand
tertiary education sector. eMM is based on the principle that an organisation's
processes mature along a five step model of capability in e-learning moving from 'ad
hoc' processes and decision-making to an informed, engaged and reflective culture of
continuous improvement.
4.5 MIT90s is a conceptual framework for understanding the effect of evolutionary and
revolutionary change through the development of the use of technology in educational
organisations leading to a range of increasing potential benefits. The framework was
developed at the Massachusetts Institute of Technology (MIT) in the 1990s for planning
and monitoring strategic change in relation to e-learning, and represents an
organisation as comprising of five elements, all in interaction with each other – its

12
http://www.utdc.vuw.ac.nz/research/emm/

Page 9 of 50
 strategies (for technology use), its organisational structures, individuals in roles,
management processes and technologies
4.6 OBHE/ACU (Observatory for Borderless Higher Education/Association of
Commonwealth Universities) is a collaborative benchmarking methodology which offers
institutions the opportunity for comparison at the process level and promotes the
development of shared good practice statements. The approach is informed by the
development of an Institutional Review Document by each participating institutions,
structured on eight themes: Strategy Development, Management of e-Learning, e-
Learning Delivery, Resources for e-Learning & Value for Money, e-Learning and
Students, e-Learning and Staff, Collaboration & Partnerships, and Communications
Evaluations and Review.
4.7 Pick&Mix was developed by the Programme benchmarking consultant Professor Paul
Bacsich, Pick&Mix is based on a systematic review of other approaches to
benchmarking e-learning, looking for commonalities of approach but also taking a fresh
start. It includes a set of core and supplementary criteria, with an option for the
inclusion of local criteria specific to the needs of a participating institution.

Methodology Developments and Refinement

4.8 The OBHE/ACU benchmarking approach was managed and supported by a team of
consultants based at the ACU. This methodology involves a process of institutional
self-review facilitated by consultant visits and resulting in the production by the
institution of an Institutional Review Document (IRD). A draft report identifying good
practice across all participating institutions was then produced by the consultants on
the basis of the collective IRDs for each cohort. This draft report was discussed and
analysed during a two-day residential workshop, where examples of institutional good
practice were shared, and key high-level management issues considered. Following
the workshop, the consultancy team prepared a final report, containing the agreed
elements of good practice. Institutions were then invited to rate their performance
against each of the good practice statements using defined criteria. A summary of all
ratings was then provided, with institutional identities protected, enabling individual
institutions to compare their own ratings with those of others without infringing
confidentiality. Each institution was then offered further consultant support to help with
action planning in the light of the benchmarking process.
4.9 The other methodologies were managed and supported by the Benchmarking e-
Learning Associates (BELA) group led by Professor Paul Bacsich. Much work was
done through the programme to refine and develop existing benchmarking
methodologies to enhance their relevance and effectiveness for the UK HE context. In
Phase 1, efforts were made to provide an intellectual underpinning to MIT90s and to
reconceptualise MIT90s in terms of criteria formulation and group working. The ELTI
conceptual framework was refined and revised, and new contextualisations with
relevance to the UK HE sector were added by the institutions involved in Phase 1.
Resources produced by the institutions for the implementation of ELTI were made
available for wider use following the completion of the exercise. Pick&Mix has evolved
through the three Phases of the programme, reflecting both refinements in the
implementation of the methodology, and changes in the scope of e-learning for
participating institutions. The expertise and resources developed for eMM in the Pilot
phase were accessed and further developed when this methodology was implemented
with a cohort of institutions during Phase 2.
4.10 The BELA consultants report at the end of Phase 213 highlights some developing
commonalities between the benchmarking methodologies. Pick&Mix was further

13
http://elearning.heacademy.ac.uk/weblogs/benchmarking/?p=358

Page 10 of 50
 adapted to incorporate some of the key concepts from MIT90s. Some Pick&Mix
institutions reported against MIT90s categories and envisioned their change
management in such terms. Through the programme, an increasing set of
commonalities also developed in the ways that the methodologies were used, for
example in the organisation of cohort meetings and their format, scoring meetings and
the use of ‘slices’ to explore and benchmark a particular service, organisational
structure, academic provision or other institutional area of interest. The development of
common features in the approaches to benchmarking in the programme included
influences on, and from, the OBHE methodology: appreciation expressed by
participants in the Pilot phase for the workshop approach that is central to the OBHE
approach was instrumental in the provision of additional cohort meetings for other
methodologies in later Phases, and later iterations of OBHE in the programme
incorporated some of the vocabulary and concepts of slicing. However, consultants
have also noted that there are almost no commonalities between the various
methodologies at a detailed level.
4.11 Increasing attention was paid in successive Phases of the exercise to the role in
benchmarking of the Measures of Success outlined in the 2005 HEFCE Strategy for e-
Learning. Some institutions addressed the HEFCE measures in their internal
institutional reports and in the summary reports they produced for the Academy. In
Phase 2, effort on the part of the consultants went into mapping both Pick&Mix and
eMM to the HEFCE measures14. The requirement to produce an anonymised carpet of
scores and the desire to produce a carpet across the sector drew attention to the need
for standard indicators. Towards the end of Phase 2, work was also undertaken to map
both Pick&Mix and eMM to the Indicators of Success that HEFCW was developing for
its 2008 Enhancing Learning and Teaching through Technology Strategy15. HEFCW
has asked the Academy to support benchmarking in institutions in Wales that had not
participated in the three Phases of the benchmarking exercise, in preparation for
enhancement activities in support of the implementation of this strategy. The
refinement of benchmarking methodologies continues in this context through the
development from Pick&Mix of the ELDDA benchmarking methodology, currently being
used in the Gwella Enhancing Learning and Teaching through Technology Programme
for HE in Wales16.
4.12 The OBHE methodology was originally developed for application in an international HE
context. eMM has an increasingly international user base, an attribute that was of
particular interest to some participating institutions in the UK, some of which expressed
the intention to take benchmarking forward in the context of the Worldwide Universities
Network group17. The UK benchmarking exercise has enhanced the development of
these methodologies in this international context, in the case of eMM leading to a
number of refinements with wider applicability. Opportunities for international
benchmarking of e-learning in HE remain, with some caveats about differences in the
methodological approach (consultants have noted that the application of eMM in the
UK is more in the style of lightly moderated self-audit and hence possibly susceptible to
grade inflation).

14
http://elearning.heacademy.ac.uk/wiki/index.php/Measures_of_Success
15
http://www.hefcw.ac.uk/Publications/circulars_5137.htm
16
http://elearning.heacademy.ac.uk/weblogs/gwella/
17
http://www.wun.ac.uk/

Page 11 of 50
5 The Benchmarking Process
Rationale for Engaging in Benchmarking
5.1 In all phases of the Programme, the opportunity for active reflection on e-learning
processes, provision and practice in institutions was a major element in the rationale
for benchmarking. Participants were keen to explore the extent to which e-learning
approaches are embedded in systems and processes across the institution. The role of
the exercise as a support for strategic planning was identified as increasingly important
through the Programme. In Phase 2, several institutions noted that the exercise was
taking place at a particularly appropriate time for their institution to instigate a more
strategic approach to the use of technology.
5.2 A further key driver for participation was the desire to test institutional perceptions and
measure performance in the e-learning domain in relation to the rest of the sector
through rigorous and recognised processes, including both internal review and a
consideration of the institution’s position alongside similar HEIs.

Methodology Choice
5.3 As experience and expertise in benchmarking grew in the sector through the
Programme, institutions were increasingly informed in their choice of a methodology to
suit their particular needs and context. Documentation about the different
methodologies was refined and more widely available with each Phase. Institutional
representatives who had been through the benchmarking process attended Town
Meetings and were able to offer insights on particular approaches to their peers. Some
institutions explicitly wanted to join the same cohort group as other comparable
institutions in order to capitalise on existing collaborative relationships or forge new
ones. The reputation and status of individual consultants also played a part in the
choice of methodology for some institutions.
5.4 The sense of belonging with a particular group of institutions was a mostly tacit but
significant factor in the choice of methodology. No Russell Group institutions chose
Pick&Mix, mostly opting for OBHE/ACU or eMM. The former was noted as a well-
established methodology with an international reputation; the latter was attractive to
research-led institutions on account of its underpinning evidence base. MIT90s
appealed to institutions interested in establishing a bespoke framework for strategic
change. ELTI was attractive to institutions that were particularly interested from the
outset in an inclusive approach to e-learning benchmarking involving various staff
groups and institutional processes.
5.5 Broadly, institutions were using benchmarking in one of two ways: to ‘prove’ that they
are on the right track, doing the right things in accordance with established or common
practice, or to ‘improve’ their current practice through a rigorous and sometimes painful
process of self-review. This distinction between ‘proving’ and ‘improving’ is relevant to
definitions of formative and summative evaluation and in some ways therefore also
reflects the essentially evaluative nature of benchmarking. There were indications that
older, established institutions used benchmarking to confirm the appropriateness of
their processes and their position in the sector, while newer institutions were more
concerned with learning how to do things better.

Scope of Benchmarking
5.6 Most institutions entered the benchmarking exercise with the intention of attempting to
benchmark e-learning provision across the entire institution, and some were successful
in carrying out some sort of evidence gathering, analysis and reporting across all
faculties, schools, departments and academic services. However, due to constraints of
staffing, resources and time, several institutions narrowed the scope of their

Page 12 of 50
 benchmarking activities in practice to focus on selected faculties (departments,
schools) or programmes in depth.

Affordances and Benefits

5.7 For the majority of those with responsibility for managing the benchmarking exercise on
behalf of their institutions, the main benefit of participation in the exercise was its role in
raising senior management awareness of e-learning. Not only had the process of active
reflection stimulated strategic thinking, but had also contributed to high-level
championing of technology-enhanced learning and teaching developments.
Furthermore, the outcomes of the exercise provided evidence to inform further strategic
planning and operational decision-making for the use of technology.
5.8 Benchmarking also served to raise the profile of e-learning among academic staff, and
helped to focus reflective thinking on the changing nature of the learning and teaching
process. The process also facilitated intra-institutional communication by bringing key
stakeholders together, forging relations between previously disparate groups and
services.
5.9 Participating in benchmarking highlighted the concept of a continuous cycle of review
and improvement which could be embedded in institutional review processes. In this
context, the adoption of a set of ‘measures of success’ based on those in the HEFCE
e-Learning Strategy was identified as a clear benefit from the process, particularly in
establishing a commonly agreed benchmark of good practice for all institutions.
5.10 The benefits of expert consultant input were consistently identified throughout the
programme, particularly in raising senior managers’ awareness of the scope of
institutional e-learning activities, and the potential benefits of further development.
Opportunities for networking and collaboration with colleagues from other institutions
through e-learning benchmarking events were also highlighted as a clear benefit from
participating in the exercise. Participants valued peer support through cluster meetings
and workshops, including the opportunity to share experiences with other institutions in
an open and honest way, without vulnerability to inspection-oriented judgment. A
number of subsequent collaborative initiatives have been initiated, including
institutional visits to explore areas of good practice in particular services and academic
areas, publications and collaborative research projects. Benchmarking itself is also a
focus for collaboration, as participants highlighted the benefits of belonging to a
network of UK institutions that have taken part in this programme, and also the
opportunity to become a participant in an international community of institutions that
have used or are still using a particular methodology.

Constraints and Challenges

5.11 In all the phases of the exercise, participants consistently identified time and timescales
as a major challenge. This was particularly problematic in Phase 2, where already
constrained timescales were compounded by a start-up phase that coincided with the
summer months. However, it was generally appreciated that the rigorous timetable that
was required at programme level (the administrative need for completion of all phases
of benchmarking and Pathfinder by July 2008), and supported operationally by the
consultants implemented the various methodologies, served to sharpen the focus of
the benchmarking activities and clarify the nature of the activities in hand.
5.12 In some cases, changing staff roles and responsibilities and organisational
restructuring initiatives impacted on staff availability and motivation to engage with
benchmarking. Logistical and/or cultural constraints were also identified in terms of
securing sufficient representation from all the relevant departments or schools, and
attracting a wide range of staff and students to contribute to focus groups and other
evidence gathering processes. Some institutions commented on the practical challenge

Page 13 of 50
 of extracting data specifically relating to e-learning from more general reports and
sources of information.
5.13 Some methodology-related issues and constraints were also identified, including the
need to adapt some of the language and terminology for the UK HE context. The value
of an aggregated results grid which can hide wide disparity between different
programmes or departments was also questioned. There was a request for a tighter
definition of some of the core criteria, and for some kind of handbook or user manual to
support the benchmarking process with this methodology. The extent to which discrete
benchmarking tools contain implicit assumptions about good practice in e-learning has
been a subject of discussion at methodology cohort workshops and Programme
meetings.

6 Commentary on the Benchmarking of e-Learning Exercise

What is Benchmarking?
6.1 There was a developing consensus across the whole programme, drawing on evidence
and emerging maturity of understanding through the various phases, that the term
benchmarking in this context is something of a misnomer inherited from the terms of
the 2005 HEFCE Strategy for e-Learning. What has been taking place through the
programme is in fact a process of institutional self-review and facilitated reflection. The
exercise provided an opportunity for analysis and reflection on e-learning processes,
provision and practice. While the key focus was on e-learning, the scope of the review
had a broader significance for quality enhancement and continuous improvement in all
aspects of learning, teaching and assessment provision. The benchmarking exercise
has been widely appreciated as having been of significant benefit to the participating
institutions in a number of ways.
6.2 There is a need to balance the needs and interests of individual institutions with the
national comparative value of a benchmark standard. Despite continued reassurances
to the contrary, some participating institutions expressed a persistent concern that the
object of the exercise is to produce some kind of a league table of e-learning
proficiency for the public comparison of institutions. On the other hand, some
disappointment has also been expressed at the lack of a coherent set of standards or
agreed good practice to which institutions may aspire or against which they can confirm
their own position. It is important therefore to clarify that a league table is not a
benchmark. A benchmark is a line based on aggregated scores for an agreed set of
indicators, against which institutions can position themselves. This positioning can
serve to identify both development needs and areas of good practice for celebration
and to facilitate forward planning and strategy development. However, the
establishment of the line (benchmark) requires the agreement of a set of indicators and
the submission of (anonymised) data. This has not been possible through the
benchmarking exercise for a number of reasons, not least the use of multiple
methodologies, and variations in the criteria not just between methodologies but also
from phase to phase and cohort to cohort.
6.3 That said, a significant element in the value of the exercise for participating institutions
was the benchmarking that occurred at a cohort level where trust relationships were
established and areas of mutual interest, shared challenges and potential solutions
were raised and explored. The benchmarking exercise has therefore been
characterised and to some extent driven by two important and potentially conflicting
assumptions – on the one hand that the results of institutional benchmarking should
remain confidential to the institution and those with whom it chooses to share, and on
the other that the programme will succeed in revealing the state of e-learning across
the sector. In 2008, HEFCE commissioned a review of its 2005 Strategy for e-Learning,
which reported regret in some quarters that the benchmarking exercise has not been

Page 14 of 50
 able to provide a clear picture of the landscape of e-learning in the sector. However,
the review also acknowledged the value to individual participating institutions of in-
depth, candid, reflective review which relies on confidentiality and trust relationships.18
6.4 There was a very clear conclusion from the pilot that there was no case for imposing
one benchmarking methodology on the sector, and several compelling reasons for
offering diversity. These reasons included the relative immaturity of benchmarking
methods at that time, coupled with the development of an emerging market within
which it was not appropriate for the Academy and JISC to advocate or impose a single
approach. Diversity of methodology was also demanded by the plurality of institutional
missions and the heterogeneous and fiercely independent nature of the sector. It was
noted that institutions might find different methodologies appropriate at different stages
in their development, for different purposes or different contexts.
6.5 There is an emerging interest across the programme as a whole in a subject-based
approach to sector-level benchmarking, with particular reference to the Subject Centres
and some CETLs as both a focus for benchmarking activities and as a forum for
reflecting and acting on some benchmarking outcomes and recommendations. The
technology affordances in different academic disciplines has been a focus of particular
interest for some participants, and may be worthy of further investigation in the sector.
Institutional experience, collaborative relationships, developments in benchmarking
expertise and the methodological refinement that has taken place through this exercise
should provide a robust basis for discipline-based benchmarking, appropriately
configured, resourced and managed.

What is e-Learning?
6.6 Throughout the benchmarking exercise, the extent to which it is useful to separate e-
learning from mainstream learning and teaching in thinking about processes, practices
and strategic planning has been a recurrent theme. e-Learning has always been a
problematic term and institutions in earlier phases went to great lengths to try to define
what was meant by the term in order to scope the boundaries of benchmarking. In
some institutions there has been a move away from the use of terms such as e-
learning and blended learning to more inclusive concepts of ‘academic innovation’. By
Phase 2, most benchmarking institutions eschewed in-depth ontological discussions
about the nature and scope of ‘e-learning’, and focused their attention on the use of
technology to enhance learning and teaching, to support all aspects of the institution’s
business, and to help to meet key challenges.
6.7 Developments in the scope and understanding of what is meant by e-learning which
have taken place in the course of the benchmarking exercise are evidenced in
successive strategic policy documents for UK HE. The interim review of the 2005
HEFCE Strategy for e-Learning recommended adapting the language to reflect the
terms and concepts in current and projected use in the sector to describe and define
the use of technology to support and enhance learning and teaching.19 In March 2008
HEFCW published Enhancing Learning and Teaching through Technology: a Strategy
for Higher Education in Wales20. There is increasing strategic recognition that
technology-based solutions are integral to all aspects of the sector’s business, with
particular reference to the core activities of learning, teaching and assessment.

18
Review of the 2005 HEFCE Strategy for e-Learning, Glenaffric Ltd, May 2008 (to be published
shortly on http://www.hefce.ac.uk/learning/elearning/)
19
See above
20
http://www.hefcw.ac.uk/Publications/circulars_5137.htm

Page 15 of 50
 Strategic Management and Development of e-Learning
6.8 For most institutions, there is consensus on the value of some sort of strategic
document that focuses attention on e-learning. The extent to which it is desirable or
necessary to have a discrete e-learning strategy, distinct from a general institutional
learning and teaching strategies, has been the subject of some debate throughout the
Programme. Some HEIs with substantial engagement in e-learning after a period of
having a separate e-learning strategy have seen the strategic integration of e-learning
with institutional learning and teaching strategies as a way of embedding activity.
Others see a separate e-learning strategy as a way of providing focused support and
widespread academic engagement.
6.9 Benchmarking has raised understanding of the need for overt senior management
support for e-learning in institutions, and for strong central planning facilitating the
strategic alignment of e-learning developments with other relevant institutional policies
and processes. Collaborative arrangements – some facilitated or further development
through benchmarking activities – have emphasised the value of open exchange and
dissemination between institutions, particularly targeted at higher levels of
management and strategy development. The focus on e-learning as a lens on general
learning and teaching practice and policy has had significant change management
implications in a number of institutions.
6.10 The exercise has highlighted the need for a better understanding of the costs of
e-learning, including workload requirements and costing the time and effort required by
both academic and support staff. A large element in the challenge in developing this
understanding is the false premise that e-learning can somehow be isolated from other
aspects of organisational activity for management purposes.

Benchmarking Scholarship
6.11 Integral to the Academy’s overall strategy in developing the benchmarking exercise
were plans for the establishment of a Special Interest Group (SIG) with a scholarly
focus on benchmarking e-learning. In the course of Phase 1 it became apparent that, in
the main, participating institutions were more concerned with addressing the issues
that benchmarking was revealing about e-learning in institutions than in developing a
scholarly understanding of benchmarking methods and processes. This is not to say
there was no interest in benchmarking scholarship, but that the focus of attention and
dedication of available resources to practical institutional realities was of more pressing
concern.
6.12 The benchmarking wiki, developed by Professor Paul Bacsich with a view to supporting
a scholarly community around benchmarking e-learning in UK HE, was used as a
reference source by participating institutions. Several institutions made submissions to
ALT-C 2007 based on their benchmarking experiences, which were conflated into a
symposium which presented a summary of the methodologies used and their
affordances. However, there were few actual contributions to the wiki from participating
institutions, and conference submissions and workshops have focused principally on
the outcomes for institutions and plans for action rather than the methodologies used
for their development.
6.13 That said, the Academy’s policy of ‘constrained diversity’ has undoubtedly served to
facilitate developing understanding of benchmarking methodologies and their
application. This policy has made possible the development of Pick&Mix as a
methodology particularly appropriate to the needs of UK HE, and its subsequent
adaptation to ELDDA, currently being used for benchmarking institutions in Wales in
support of the implementation of the 2008 HEFCW Enhancing Learning and Teaching

Page 16 of 50
 through Technology21 strategy. It supported the implementation of the internationally
recognised eMM methodology in UK HE, with attendant benefits both for participating
institutions and for the New Zealand-based methodology champion in the further
development and wider applicability of the approach. It provided an operational context
for the implementation and further refinement of the ELTI method that was developed
through a previous JISC-funded initiative, including the provision of generic resources
to support the wider implementation of this approach. It provided a forum for the
implementation of the MIT90s framework as an approach to benchmarking that suited
the particular needs of a group of institutions, and presented opportunities for synergies
and collaborations with the then nascent Scottish Benchmarking Group. Some
individual participants have produced scholarly outputs focusing on benchmarking
methodologies and processes22.

Communication and Community

6.14 The evaluation of the Pilot phase noted that the attempt to create a weblog culture of
continuous reporting and reflection by institutional participants has been only partially
successful. Through subsequent phases of the exercise the use of institutional weblogs
for reflecting and reporting on benchmarking activities was patchy. It was the stated
intention of the programme to encourage the participants to engage in a continuous
process of reporting, commenting and reflecting on their benchmarking activity, thereby
rendering the need for final reports unnecessary. Although every participating
institution contributed a blog, linked from the main Academy benchmarking weblog,
postings were in the main characterised more as a minimalist log of events than an
open reflection on progress that would discuss issues and problems as well as
achievements.
6.15 Despite the difficulties experienced in the attempt to establish a genuine community of
practice in which blogging would become the norm, many of the participants expressed
a change of attitude towards blogging during the pilot. An initial resistance to the idea
of blogging was replaced by a quite positive view of its worth, though still tempered by
the insistence that to be truly reflective about the exercise while it was in progress
demanded time commitment and could potentially compromise the internal trust
relationships that were essential to the success of the exercise in participating
institutions. The use of blogs as the main communication medium for the programme
provided a degree of consistency in the interface with the various participating
institutions across the three Phases, despite the use of different blogging engines. The
Academy’s benchmarking weblog presented a unified public face for the exercise and a
window into the multifarious and multi-faceted benchmarking activities of the
institutions. There was also a developmental aspect to the use of blogging, which is
now a much more widespread and widely accepted form of social communication in
2008 than it was at the outset of the benchmarking exercise in 2005. While some of the
institutional representatives remained extremely reluctant bloggers, most derived
benefit both from contributing and accessing blog postings, and some developed a
genuine enthusiasm and appreciation for blogging and other social networking
approaches which has in turn influenced their professional practice.
6.16 The Helga social networking platform was set up in response to concerns from
benchmarking projects that the Academy's blog was 'too public' for private reflections
and team discussions. Institutional benchmarking teams agreed in principle with the
benefits of reflection and with the need for an ongoing log of experiences and findings,

21
See above, paragraph 6.3.2
22
See for example ‘Benchmarking e-learning: trialling the “MIT90s framework’”, Virendra Mistry,
Centre for Excellence in Learning and Teaching, University of Glamorgan, in Benchmarking: An
International Journal, Vol. 15 No. 3, 2008 (www.emeraldinsight.com/1463-5771.htm)

Page 17 of 50
 but wanted a private and confidential space in which to do this. They also agreed that
some kind of public face for their activities was required (the Academy blog). Pathfinder
projects also needed a private space for team communication and resource sharing. At
the outset, EDSuT was also concerned to establish a means of cross-programme
communication and collaboration on a thematic basis, and to provide a forum for
establishing and maintaining cluster relationships.
6.17 There was some concern about potential confusion and repetition among multiple
communication media (the Academy blog, Helga and the institutional benchmarking
websites). Following discussion and with ensuing support from the EDSuT, however,
there seemed to be general clarity about the different purposes each of these served.
The main issue with Helga seems to be that as a social networking platform, the onus
is on the individual to establish their own communities, or ask to join other
communities. Users have to take responsibility for how they interact with the site,
where they go, what resources they want to access. Using Helga also presented an
opportunity actively to engage with a Web 2.0 technology for community building and
engagement, which was a (direct or indirect) focus of interest for many Benchmarking
and Pathfinder projects. One of the key lessons for the Programme support team has
been the need constantly to encourage people to engage with online environments, to
make this engagement task-oriented, and clarify its relevance to the main focus of
participants’ interest and activity.

Recommendations for Future Initiatives

6.18 There is overwhelming evidence from all three phases of the exercise that institutions
have drawn benefit from participating in benchmarking and found the exercise to be
worthwhile and useful in a number of ways. The value of some kind of in-depth periodic
review of the use of technology to enhance learning and teaching has been recognised
both at an operational level and in terms of strategic planning and management in
participating institutions. There was general appreciation of the role of the Academy not
just in terms of the overall management of the exercise, but as an external validator
and driver in developing sector buy-in, establishing impetus and helping to create a
sense of community.
6.19 The support model that was implemented for benchmarking – the provision of
dedicated consultants offering expertise, advice and guidance at a strategic level to
help an institutional team through a structured process of scoping, data gathering,
analysis, reporting and action planning – has proved to be highly successful.
Throughout the programme, the role of external consultants has been consistently
highlighted as a crucial factor in facilitating dialogue at different levels in the institutions
about e-learning and the role of technology in changing practice, management and
organisational culture. Groups of institutions collaborated with one another in a context
of mutual trust and respect, developing synergies, drawing on shared experiences and
collective expertise. Generic support from the EDSuT provided co-ordination, sharing
good practice and emerging themes, issues and challenges across different
methodologies and cohort groups. Consideration should be given to the enhancement
and implementation of similar support models for future development programmes.
6.20 There is no clear evidence from the exercise in favour of any particular benchmarking
methodology, or for a criterion-based or narrative-based approach. There is compelling
evidence that institutions appreciate an element of choice, and that the rationale for
choosing to employ a particular benchmarking methodology depends on a number of
internal and external factors. Each of the approaches has served as a starting point for
a developmental process, and there was a widely-expressed view in all phases of the
exercise that the precise nature of the method is much less important than the general
process involved in starting to ask penetrating questions about e-learning in an
institution. It is important that institutions are able to employ a methodology that fits with

Page 18 of 50
 their own internal culture, the drivers, scope and expectations of benchmarking, and
the resources available. Institutions also valued an element of choice and negotiation in
the formation of methodology-based cohort groups.
6.21 The achievement of some drivers for benchmarking e-learning (for example cost
reduction and retention benefits) can only be determined by explicit forms of
measurement, whereas others (for example the general enhancement of teaching and
learning) may be much more qualitative in outcome and difficult to determine in
quantifiable terms. It follows that it might be regarded as good practice that HEIs
should be clear about the key drivers for introducing e-learning and then adopt
appropriate measures for indicating the extent to which the drivers are being met.
6.22 Senior managers are most likely to take notice of the benchmarking process and
outcomes when these can be specifically related to QAA guidelines and the
enhancement process, or the National Student Survey and other published
performance information. While the availability of funding for specific initiatives helps to
secure institutional involvement in developments in the use of technology, the longer-
term relevance of technology implementation to strategy planning in relation to sector-
level priorities is more significant to institutional managers.
6.23 It was implicit in the strategic context for the exercise that helping institutions to
understand and appreciate their strengths and weaknesses in the development and
provision of technology-enhanced learning and teaching should lead to a more
coherent approach to e-learning development activities in the sector and hence to more
effective use of development funding. The formative evaluation of the Pathfinder
Programme offers early evidence of the role of benchmarking in identifying institutional
development needs, then tailoring resource provision, providing implementation
structures and consolidating communication channels to meet these needs. While it is
too early to confirm the role of benchmarking in making better use of internal and
external development funding, many institutional responses to recent JISC calls for
funding in curriculum design and delivery have drawn evidence from benchmarking in
support of their proposals. The longer term impact of the periodic review of e-learning
(technology-enhanced learning and teaching) provision in institutions on the relevance,
quality and effectiveness of innovation and developments in academic provision and
practice and institutional processes should itself be monitored and reviewed on an
ongoing basis.

7 The Pathfinder Programme

Pathfinder’s Relationship to Benchmarking
7.1 All the Pathfinder projects express a positive benefit from having had the opportunity to
carry out an internal benchmarking exercise before embarking on Pathfinder. There is
a widespread agreement that the Pathfinder projects have in general been successfully
tailored to each institution’s priorities and key requirements. This programme,
therefore, represents a successful model for a national enhancement activity – where
institutions are given expert help in understanding their own situation, in devising
activity to address their needs, and in implementing the desired changes. The
Pathfinder projects are best characterised as enhancement projects, where there are
many different starting points, and where enhancement can involve making a very
modest advance from a low base. Nevertheless, the inclusion of an institution in a
programme that was advancing the state of e-learning across the sector was, in many
cases, the incentive for senior management to become involved in the project.
7.2 Across the whole programme it is possible to draw conclusions about the outcomes of
the institutional benchmarking from the nature of the development work that the
Pathfinder projects have undertaken. The overall picture is one of internal

Page 19 of 50
 consolidation, widening involvement, and raising awareness, rather than innovation.
This suggests that institutional benchmarking involves an inevitable bias towards
levelling-up, dealing with perceived weaknesses rather than building further on possible
strengths. In articulating a rationale for their Pathfinder approach most of the projects
have referred explicitly to capacity building in areas of weakness revealed by
benchmarking, rather than in areas of strength.
7.3 A subset of the projects have explicitly put in place in the institution a permanent
benchmarking activity – to be carried out as a regular (yearly, for example) activity.
This joins up the current enhancement initiative with internal quality procedures.
Indeed, this idea should emerge from the programme as an important strategic
recommendation for HEFCE. In Scotland, the enhancement themes are achieving an
integration of quality procedures and development activities in learning and teaching.
Of particular note at this juncture is one of the five Pathfinder Network projects23 led by
the Institute of Education which has focused on the quality issues raised by e-learning
and has attempted to initiative a wider debate, to which the QAA has contributed.
7.4 In most Pathfinder projects senior managers who had played a role in, or had taken a
close interest in, benchmarking, stayed involved with the project, and have committed
to some kind of follow-up activity. In some institutions, however, the Pathfinder projects
have received a lower level of senior management involvement than in the
benchmarking phase. In these institutions the setting up of a ‘project’ – particularly
when backed by external funding – led to the responsibility for its outcomes being
delegated downwards, and the activity then began to fade out of executive
consciousness. In these cases the role of the Critical Friend has become central, in
attempting to refocus senior manager attention on the strategic importance of
addressing the issues highlighted by benchmarking.
7.5 The Pathfinder reports provide evidence of strategic issues revealed by benchmarking
being taken forward in Pathfinder in a systematic way across institutions. Even in
projects where there is no reference to a direct link it is clear that benchmarking started
a process that has been developed further in Pathfinder. Most institutions rolled their
blogs forward from benchmarking to Pathfinder. There is an opportunity now to conduct
a more systematic study of the influence benchmarking had on the subsequent
development activity, and the extent to which the Pathfinder projects were influenced
by the programme’s emphasis on embedding. The relationship between the two
activities is an important one for policy, and while we can currently assert that a
majority of projects have explicitly reported that Pathfinder was based on the
benchmarking outcomes, it would be valuable to be able to be more analytical about
the relationship. The issue is certainly not completely straightforward since in some
instances it was noted that Pathfinder encountered institutional challenges to the
development of e-learning that were not exposed by benchmarking.

8 The Design and Operation of the Programme

The Design of Pathfinder
8.1 The focus for Pathfinder was stated clearly in the ITT for the Pathfinder pilots:
The focus of the Pathfinder Programme is on the design, planning, implementation,
and evaluation of transformation processes and activities which are intended to lead,
ultimately, to the full and effective embedding of e-learning into the learning and
teaching processes of the entire institution, i.e. the aim is long term change and not
just short-term innovation.

23
http://elearning.heacademy.ac.uk/weblogs/pathfinder/?p=179

Page 20 of 50
8.2 The Pathfinder programme had three phases: a pilot phase that ran from October 2006
to September 2007 with nine institutions and which led into the main phase (called
phase 1) in which a further 28 institutions carried out projects from May 2007 to the end
of June 2008. Five of the original pilot projects were awarded additional funding to
develop ‘network’ projects which ran from January to the end of July 200824. Entry to
phase 1 of the Pathfinder Programme was competitive. Twenty-eight institutions who
had undertaken a Phase 1 benchmarking exercise were selected for entry. Grants
ranging from £23,000 to £170,000 were awarded. The selection of network projects
was also competitive.

Steering Groups
8.3 Each project was expected to set up a Steering Group, with senior manager
involvement. The main function of some Steering Groups was to review and approve
documentation, but others had a more reflective, visioning and developmental role. The
projects were asked to include the Critical Friend as a member of the Steering group
and a majority of projects did so. Critical Friends reported a very variable set of
approaches here, ranging from a formal standing item on a CETL Steering Committee
or even on an institutional Learning and Teaching Committee, to a completely informal
meeting in a PVC office. Most had a formal Steering Group and separate meetings of
the project team, but Critical Friends reported varying levels of institutional confidence
and readiness for new developments and approaches, and communicating
developmental activity in an open forum.
8.4 The extent to which project reports and Steering Group minutes fed into University
committees was a valid indicator of institutional interest and support. The Critical
Friends worked hard in some institutions to counteract a tendency to view the project
as peripheral to mainstream institutional activity. This was most noticeable where the
operation of the project had been delegated to support staff level.

The Programme Events

8.5 At both stages of the Pathfinder programme there was a start-up event, a progress-
reporting event roughly half-way through the year, and a closing event. Each project
was encouraged to send at least three participants. The pilots continued to attend
during phase 1, increasingly playing a programme-wide mentoring role. During Phase 1
the events were primarily organised around opportunities for the clusters to reinforce
their joint activity. There was a good professional level of support from the Academy in
the organising and running of the events, and the Academy’s programme leader, Derek
Morrison, used each occasion to re-emphasise the overall aims of the programme.
Representatives of external stakeholders (eg, QAA, DIUS) attended the close-of-
programme event on 26 June 2008. The Pathfinder events were designed bearing in
mind the feedback from programme events during the benchmarking pilot, that they
were too ‘presentation-heavy’.
8.6 One innovation that was tried with mixed success was the ‘carousel’ approach, where
participants moved quite rapidly around various ‘presenters’ who engaged in informal
discussion. This proved very effective when used to orient institutions considering
whether or not to join the benchmarking exercise. At the Phase 1 start-up event a
‘reverse carousel’ was tried, where the cluster groups stayed in one room and were
visited in turn by the newly introduced Critical Friends and other potential support
agencies, such as JISC InfoNet, JISC Legal, and an Academy staff development
expert. Feedback from this experiment was less positive (the ‘experts’ were not
regarded as having been appropriately briefed, and the cluster teams found

24
http://elearning.heacademy.ac.uk/weblogs/pathfinder/?p=159

Page 21 of 50
 themselves giving repeated mini-presentations) and this approach was not attempted
again.

The Pathfinder Weblog

8.7 A weblog site had been developed for the benchmarking pilot, and an attempt had
been made to create a culture of blogging across the programme. This had met with
only limited success during the benchmarking pilot, but it was accepted by the
Pathfinder pilots that they would continue with this approach and each set up a blog
that was linked to the Academy site. This was continued for Phase 1 projects. The
Pathfinder site essentially became a programme news/announcements blog, and a
repository of programme documents and links to relevant resources. This fulfilled a
valuable function and achieved some consistency of web interface across the whole
programme. The Academy’s weblog became one of the mechanisms of public
dissemination and also acted as a tool for day-to-day project communication. There
was some evidence of cluster members (see 4.5) reading and commenting on each
others’ blogs within a cluster, but not more widely. While Phase 1 projects in general
continued a low level of engagement with blogging, there were a few exceptions, such
as De Montfort, whose blog site was a model of how real value can be added to a
project through reflection and continuous sharing through blogging.

Clustering the Projects

8.8 A conclusion from the pilot phase of Pathfinder was the need to encourage more of a
‘sharing’ culture in Phase 1, a response to the interest expressed by some of the
projects for the opportunity to exchange views with other Pathfinder projects. However,
this seemed to be restricted to those who had shared the benchmarking experience
using the OBHE method, and it is worth noting that almost no-one took the opportunity
to use the blogs to comment on, or ask questions about, each others’ projects. The
CAMEL25 (Collaborative Approaches to the Management of e-Learning) project, at that
point recently published, offered a suitable model. CAMEL was funded by the HEFCE
Leadership, Governance and Management programme. It set out to explore how
institutions who were making effective use of e-learning and who were collaborating in
regional lifelong learning partnerships might be able to learn from each other in a
Community of Practice based around study visits to each of the partner institutions.
Consequently, it was decided that Phase 1 of Pathfinder would be structured into
seven clusters of four institutions each. The clusters were agreed by the projects at the
Phase 1 start-up event in York in May 2007, where Seb Schmoller of ALT, who had
been the originator of the CAMEL idea, introduced the origins and assumptions behind
the approach. Additional funding to each project was allocated at this point to cover the
costs of hosting a cluster meeting at each institution.

8.9 It should be noted that these CAMEL collaborations were essentially informal meetings
held under the Chatham House Rule, rather than in any sense constituting a
consortium. One Critical Friend noted that development initiatives in UK and other
countries have often tended to assume that a consortium approach is the best way to
secure transformational change, but this is rarely the case. The more partners there
are the less evident is the footprint in the individual institutions. In CAMEL, however,
the impact should be evident from mutual encouragement, a sense of confidence, and
enhanced learning of the project members themselves.

25
http://www.jiscinfonet.ac.uk/publications/info/camel-publication

Page 22 of 50
 Configuring the Clusters
8.10 The seven clusters and their Critical Friends were quite carefully configured, taking
account of thematic focus, existing affinities, institutional types, geographical location
and specific requests from projects. Some, but not always all, of these factors were
able to be addressed in each cluster. However, the rationale for the groupings was not
explained as clearly as it could have been and there was initially a view expressed by a
number of projects that their cluster was not particularly well matched. However, these
misgivings quickly faded as the clusters took ownership of their group activities.
Meetings were often focused on a particular project theme, with one project taking the
lead. The February 2008 Programme meeting appears to have been instrumental in
reassuring any remaining institutions with residual concerns, and also for confirming
the value of collaboration to those cluster that were already working well together.

Operating the Clusters

8.11 Almost every project reported some benefit from working in some kind of partnership
relation with the other institutions in their cluster. Each cluster developed its own style
of working. Some clusters ran cluster meetings ‘impeccably’ along recognised CAMEL
lines. Some project representatives visited one another on an individual basis outside
formal cluster meetings to progress areas of common interest. Some clusters
developed peer research networks, and are producing joint conference submissions
and publications. These groups include both traditionally research-led institutions and
others with a much less developed tradition of scholarship. One cluster arranged that
each project Steering Group meeting was attended by a representative from another
project in the cluster. Another has established peer evaluation pairings. Clusters
developed various communication methods. A few used Helga, the Elgg social
networking site maintained by Glenaffric (part of EDSuT), although most developed
alternative methods, including their own communication platform, eg using wikis.
Project teams in development programmes are in a learning situation. Like many
learners they develop their own communication methods and platforms in preference to
those provided for them. This is as true of Helga and the Academy’s weblog as it is of
course blogs and e-portfolio systems in institutions.

Sustaining the Clusters beyond the Programme

8.12 Some clusters very quickly overcame their initial misgivings to develop a model for
collaboration which has in turn generated a clear will and intention to continue working
together. Several clusters have firm intentions to continue their collaboration.
Relationships have developed at different levels across institutions (learning
technologists, academics, quality managers, staff development managers). The extent
to which these intentions are met will depend on maintaining a clear purpose for the
collaboration

9. Reporting
Projects were asked to produce:

Final Project Report

Each project was required to produce a final report for the Academy as a ‘sign-off’ on
the grant-funded phase of the project work. Although submitted at the end of the
Academy-funded phase of project activities, this report was intended to reflect the
declared aim of Pathfinder to instigate or continue some form of transformation in the
institutions, and reinforce the point that projects have a timeline that extends beyond
the Academy funded period. In that respect it was both a ‘completion’ report on the

Page 23 of 50
 funded phase of the project and a ‘progress’ report on the ongoing transformational
activity in the project institutions.
The ‘Pathfinder Journey’
A complementary, reflective, outward facing report for the sector was requested. This
was intended to relate the journey (tell the story) of the project in an engaging,
interesting and thought-provoking way.
Briefing Papers
Each project was asked to produce at least one briefing paper on a theme or area of
specific project focus. This was aimed at describing some lessons or recommendations
for a defined audience (eg senior managers, academic teaching staff, IT services, staff
developers). Templates for these reports were produced by the EDSuT.

10. Critical Friends

Selecting the Critical Friends
10.1 The selection of Critical Friends was a key success factor for the programme. Their
seniority and recognised position in the sector has enhanced perceptions of the role
and its acceptance in projects and institutions. Newly recruited to the programme for
phase 1 were Professor Betty Collis, Professor Grainne Conole, and Andrew Comrie.
Consultants from the benchmarking phase, Professor Paul Bacsich, Cliff Wragg and
Professor Peter Chatterton were invited to continue with the programme in the new role
of Critical Friend. Professor Terry Mayes (team leader of the EDSuT) also became a
Critical Friend (this was not entirely desirable from the point of view of clarity of role,
but the cluster for whom he acted as Critical Friend seemed readily to accept that he
was acting in different roles across the programme).The idea of clustering into CAMEL
groups clearly suggested that one Critical Friend should be allocated to each cluster.

The Practice of being a Critical Friend

10.2 Development of a trust relationship was crucial to the success of the role, both at the
level of personal confidante and at a strategic level in institutions. Critical Friends had
to work hard to make contact with key people in the organisation to raise the profile of
the Pathfinder Programme and the institutional project. The sustainability agenda and
the need for senior management buy-in were emphasised from the outset. The Critical
Friends had different areas of expertise in a range of relevant areas, and each brought
a slightly different perspective to the role. Some particularly encouraged academic
development and research; others placed more emphasis on strategic change
management and communication. An important element of the role in some cases was
supporting individuals and project activities in working with organisational structures
and political contexts. There is no set formula for the Critical Friend role, which can
evolve and develop depending on the nature and culture of the institution, the interests
and expertise and working context of project team members, and Critical Friends’
specialisms.
10.3 Some Critical Friends were more directive than others (depending on context and
institutional need). The role was vital to success of some projects requiring high levels
of external direction – they would have floundered and not delivered anything near
what they have achieved without this support. Critical Friend activity involved directing
and focusing them, challenging what they were doing, helping them to develop a new,
more focused plan of action. All Critical Friends had some level of engagement with
project Steering Groups. Some were heavily involved, for example instigating a series
of themed Steering Group meetings to help develop understanding of the scope of
Pathfinder activity. There were several occasions where the Critical Friend was able to
raise the profile of the project internally by explaining how importantly it was viewed

Page 24 of 50
 externally. This dynamic seems important – using the strong sensitivity of an institution
to its relative position in the quasi-market-driven environment of UK HE to drive change
through the need to stay competitive.

Evaluating the Critical Friend Role

10.4 By the end of Phase 1 it was possible to assert that the concept of the Critical Friend
for supporting institutional development at both a strategic and practical, operational
level is proven and effective. The Programme has refined and developing the Critical
Friend/mentor approach to a transferable and scalable model for programme and
institutional support. Critical Friends facilitated the development of research capacity
across a range of institutions and this work is helping to enhance scholarship about
learning and teaching. Critical Friends helped to keep the profile of each project higher
than it might otherwise have been within the institution, and acted as a resource, and a
source of encouragement, for the project in a range of areas of expertise. However, as
with any relationship, success cannot be guaranteed. Effectiveness in the role is very
dependent on individual circumstances, and on the sensitivity of the Critical Friend to
the needs of the project team, the people within it, their influence within the institution,
and the investment of time that everyone can give. The external support provided by
Critical Friends needs to be centrally co-ordinated and supported in itself. Collaborative
methods of working among Critical Friends, sharing experiences, challenges and good
practice could have been achieved to the benefit of the programme through the
allocation of a higher resource to the Critical Friend role. Some Critical Friends
expressed the view that the resource that was allocated was not commensurate with
the levels of support that they had provided, but this was compensated by the extent to
which Critical Friends found their engagement with the Programme to be stimulating
and personally rewarding.

11. The Pathfinder Project Themes

11.1 As the following section reveals, the Pathfinder projects26 have covered a wide
spectrum of approaches to building capacity in e-learning, and embedding best
practice. The projects have ranged from, at one end of the spectrum, a few highly
specific attempts to develop particular approaches to e-learning itself, or a particular
approach to institutional procedures, to the majority who have taken a range of
approaches to the development and empowering of staff in e-pedagogy, or who have
constructed new pathways for communication and influence, or who have targeted
course redesign as a key opportunity. At the other end of the Pathfinder dimension lie
a number of attempts to understand more deeply the nature of the changing student
experience of e-learning, and to begin to grapple with some of the challenges such
understanding brings to institutional policy.

11.2 The following are some of the broad themes that emerge across the Pathfinder
spectrum.
Embedding into course redesign
Taking curriculum or course-redesign as a key opportunity for the e-learning specialists
to enter into dialogue with subject-based staff. To construct staff development
opportunities that are timely (responding to an acknowledged need) and which
introduce subject teaching teams to new opportunities for innovative pedagogies.

26
: Links to all reports and project blogs are available at
http://elearning.heacademy.ac.uk/weblogs/pathfinder/

Page 25 of 50
 Embedding into decision-making structures
Setting up a permanent infrastructure for pedagogy-influencing relationships to be built
and developed. This usually involves representing e-learning relationships within a
committee structure, and thus, ultimately sustaining them within quality procedures.
Embedding e-learning into wider policy
Using the opportunity afforded by institutional policy not aimed specifically at e-
learning, e.g. partnership building with FE, new lecturer training etc.
Focus on the student experience
Within the pilots there was comparatively little focus directly on the student experience
or learning outcomes. In Phase 1, however, starting a year later, there was a clear
strand of activity that was starting to investigate the potential for Web 2.0 approaches
(particularly wikis, social networking and podcasting), and another that was attempting
to understand the way in which students are using these tools.
Inter-institution collaboration
The five Network projects represent the programme’s clearest examples of capacity
building. Each has succeeded in recruiting wide participation across the sector in taking
forward the topics of:
• quality and e-learning (PREEL 2 led by the Institute of Education)
• intensive course redesign events (CABLE Transfer led by the University of
Hertfordshire, and Carpe Diem led by the University of Leicester)
• podcasting (PPP, led by the University of Chester)
• researching the student experience (ELESIG led by Oxford Brookes University)

Summary of themes and dimensions across Pathfinder

11.3 Appendix B provides a summary of each individual Pathfinder project. The following
table of themes (areas of activity or main focus) and dimensions (perspectives on the
activity) represents a way of capturing the main categories of outputs from the
programme.

Themes  Dimensions 
Quality management, audit,
Sustainability and embedding
enhancement
Strategy and policy
Staff empowerment
Communication, collaboration and sharing
Organisational change
Capacity enhancement
Web 2.0
Building on benchmarking
Mobile technologies, podcasting
Outputs, outcomes and benefits realisation
Course redesign
Learner engagement 

12. Key Messages from the Programme

The review concludes with the following high-level points about the overall programme.
Building relationships: transforming people rather than technology innovation
12.1 Most of the Pathfinder projects have taken very seriously the main aim of the
programme – to embed existing good practice as widely as possible in the mainstream
provision of the institution. In all institutions there are pockets of teaching that are
positioned at the leading edge of innovation. Yet, the emphasis of the programme has
tended to encourage others in the institution to embrace such methods rather than
building further on the innovation. At a meta-level a majority of the projects can be
characterised as aimed at building relationships, rather than at technology
development. Within institutions we can view the Pathfinders as situated within a long

Page 26 of 50
 process of awareness raising and persuasion. In that sense several of the projects are
patiently building relationships between, on the one hand, learning technologists and
educational developers in support departments, and on the other, academics who are
designing and teaching modules. (There is also a strand of activity discernable in
several projects which is, in a social-constructivist sense, building relationships among
learners). The main theme of capacity building has been implemented as help for
people, not so much with developing strategies but with their implementation,
reconfiguring processes, rethinking traditional learning and teaching approaches,
redesigning higher level processes for learning and teaching, enabling new models to
emerge and be successfully developed.
From supply to demand
12.2 It is arguable that e-learning development is currently right on the cusp of a
fundamental shift. This sounds a surprising statement given the ‘consolidation’ theme
described above. Nevertheless, in discussion across the programme, we see an
emerging awareness that the VLE-driven approaches of recent years are increasingly
out of alignment with the changing expectations of students revealed in recent JISC
studies, and in some of the survey work carried out in the programme itself. Student
expectations will increasingly set the agenda for institutions. The VLE era has not in
general led to a culture shift in teaching methods but academics are increasingly
squeezed between the demands for a more student-centred form of teaching on the
one hand and pressures from the social networking generation of learners on the other.
One possible approach is to support moves from a supply-side view of e-learning to a
demand-side one, but whether many institutions are yet thinking in these terms, with
some huge implications for the curriculum, is doubtful. There is very little evidence yet
from the programme that institutions are moving strategically in this direction, though
some of the participants are beginning to discuss these issues more widely within their
institutions.
Building research capacity
12.3 The Institute of Education’s (IoE) project focused on a theme that reflects the special
nature of that institution, but which also represented a thread of capacity building
across the programme. This theme was the underpinning of learning and teaching by
the institution’s own research. Even for the IoE’s own internationally respected e-
pedagogy research there had previously been little impact on its own learning and
teaching. Across all the pilots, though, is recognition of the importance of positioning
Pathfinder work within a research culture. This is clearly relevant in research-intensive
universities but it also emerges as an important issue for all HEIs in the programme,
which has played a part in raising the esteem of learning and teaching researchers in
many institutions.
Sustaining the progress
12.4 The short-term funding in Pathfinder is not well matched to the need for sustainable
outcomes at an institutional level. However, it is a tribute to the success of the two-stage
model of benchmarking and Pathfinder that there are tangible signs of sustainable
change in many of the institutions, though it is too soon to make claims about the extent
of this. Some of the projects have been able to define the Pathfinder project with a two-
year duration, using matched funding from the institution to fund the second year. A
frequently expressed view is that the sustainability of the Pathfinder developments
depends, in most institutions, on the extent to which benchmarking was conducted at an
institutional level, rather than being owned by a support group responsible for e-learning.

Page 27 of 50
 Ownership of change: the need to engage the subject communities
12.5 A key strategic question that emerged is where to place the ownership and
responsibility for development projects. Central support units have limited impact and
have to expend a lot of effort to get buy-in from the academic faculties, which is where
the key changes can occur. Support units can and do play important roles, but they
often sit in an uneasy position between the policy-makers and the faculties. It is
important to have these special units and the connection to policy, but faculties pay
more attention to people who speak their language. Some element of direct funding to
faculties (e.g. the University of Hertfordshire’s Change Academy approach) may have
optimum impact. However, this programme has engaged those parts of the Academy
that have primarily an institutional focus, through the generic learning and teaching
community, and of course through the e-learning specialists. Future development
programmes of this kind could usefully involve the Subject Centres, with the aim of
tapping into discipline-based development communities.
Joining up the issues: challenge for the agencies as well as for the institutions
12.6 The Pathfinder Programme has served to highlight some of the benefits for the sector
of collaboration between JISC and the HE Academy. The Programme has also brought
clearly into focus the close relationship between e-learning developments in practice,
and those procedures that directly relate to programme and module design, where the
QAA have a sector-wide responsibility. The PREEL 2 network project, led by the
Institute of Education, explored this relationship between e-learning and quality as its
main focus, and succeeded in attracting the interest and involvement of the QAA. One
achievement of the programme is certainly the highlighting of the need to bring the
QAA into a more central role in enhancement (as in Scotland, where the SFC works
closely with QAA Scotland on the enhancement themes). It has also been suggested
that bringing the Leadership Foundation into the fold would present a coherent set of
sector supports for transformation. Critical Friends have noted that at senior level in
institutions there is sometimes compliance with the rhetoric of e-learning and
technology-enhanced learning and teaching, with no clear evidence that this is truly
understood. The experiences of Pathfinder have indicated that in many cases, PVCs
are not close enough to the ground to make effective decisions about practice. There is
a real need for leadership programmes at PVC and VC level, and for figures in national
agencies to help them to visualise e-learning possibilities and scenario plan for the
future.

Page 28 of 50
Appendix A: Benchmarking Institutions by Phase and Methodology

Institution Methodology
Pilot: Jan - Jun 2006
University of Manchester eMM
University of Bristol ELTI
University of Hertfordshire ELTI
University of Wales Institute, Cardiff ELTI
University of Strathclyde MIT90s
Coventry University (with Warwickshire College) OBHE/ACU
Institute of Education OBHE/ACU
Oxford Brookes University OBHE/ACU
University of Warwick OBHE/ACU
Staffordshire University Pick&Mix
University of Chester Pick&Mix
University of Leicester Pick&Mix
Phase 1: Oct 2006 - Mar 2007
Bradford College ELTI
Nottingham Trent University ELTI
University of East London ELTI
University of Sunderland ELTI
West Nottinghamshire College ELTI
Manchester Metropolitan University ELTI
Thames Valley University MIT90s
University of Bradford MIT90s
University of Brighton MIT90s
University of Glamorgan MIT90s
Bournemouth University OBHE/ACU
Brunel University OBHE/ACU
Buckinghamshire Chilterns University College OBHE/ACU
College of St Mark & St John OBHE/ACU
De Montfort University OBHE/ACU
Edge Hill University OBHE/ACU
Keele University OBHE/ACU
London South Bank University OBHE/ACU
Trinity College, Carmarthen OBHE/ACU
University of Birmingham OBHE/ACU
University of Cambridge OBHE/ACU
University of Central Lancashire OBHE/ACU
University of Exeter OBHE/ACU
University of Greenwich OBHE/ACU
University of Hull OBHE/ACU
University of Lincoln OBHE/ACU
University of Middlesex OBHE/ACU
University of Northampton OBHE/ACU
University of Reading OBHE/ACU

Page 29 of 50
Appendix A: Benchmarking Institutions by Phase and Methodology

University of Westminster OBHE/ACU

University of Wolverhampton OBHE/ACU
Canterbury Christ Church University Pick&Mix
Cumbria Institute of the Arts Pick&Mix
Kingston University Pick&Mix
Ravensbourne College Pick&Mix
University of Derby Pick&Mix
University of Gloucestershire Pick&Mix
University of Teesside Pick&Mix
Phase 2: May - Dec 2007
The Open University eMM
University of Leeds eMM
University of Sheffield eMM
University of Southampton eMM
University of Wales, Aberystwyth eMM
University of Wales, Bangor eMM
University of London External eMM
Glasgow Caledonian University OBHE/ACU
Goldsmiths College OBHE/ACU
Leeds Metropolitan University OBHE/ACU
Newman College OBHE/ACU
Queen's University Belfast OBHE/ACU
Sheffield Hallam University OBHE/ACU
St Martin's College OBHE/ACU
University of Bath OBHE/ACU
University of Portsmouth OBHE/ACU
University of Wales, Lampeter OBHE/ACU
Anglia Ruskin University Pick&Mix
St Mary's College Pick&Mix
University College Falmouth Pick&Mix
University College for the Creative Arts Pick&Mix
University of Bolton Pick&Mix
University of Chichester Pick&Mix
University of Northumbria at Newcastle Pick&Mix
University of Salford Pick&Mix
University of Wales, Newport Pick&Mix
University of Worcester Pick&Mix

Page 30 of 50
Appendix B: Pathfinder Projects by Institution

PATHFINDER PILOT PROJECTS

CRITICAL FRIEND: Terry Mayes

UNIVERSITY OF BRISTOL

KEY FEATURES Benchmarking recognised need for central e-learning support to

make an impact on real teaching practice in the strong
autonomous research-intensive Faculties. Created ‘hub and
spokes’ structure through appointment of a ‘Pathfinder’ in each
Faculty to facilitate e-pedagogy, and the creation of a cross-
institution e-learning network.
INDICATORS Hub and spokes structure in place.

BRIEFING PAPER Embedding e-learning in the disciplinary context at research-led

universities.
UNIVERSITY OF CHESTER

KEY FEATURES Benchmarking confirmed value of in-house VLE (IBIS). Range of

development activities at both structural (E-Learning Central
Support Unit created, e-learning network set up) and programme
e-pedagogy level (e-learning module in staff PGCert, podcasting
for student feedback). Awarded Network project: Podcasting for
Pedagogical Purposes SIG.
INDICATORS New support unit in place. Evaluation measures taken of impact.
Survey agreed to allow comparison with benchmarking data.
BRIEFING PAPERS 1: Influence of institutional context on e-learning policy
development.

2: Chester’s E-Portfolio for Staff: Enhancing support for CPD in

partner FE Colleges.

3: Podcasting student assessment feedback.

COVENTRY UNIVERSITY/WARWICKSHIRE COLLEGE

KEY FEATURES Joint project aimed at understanding the role e-learning can play
in FE-HE transition, and in underpinning collaboration more
generally across the sectors. Separate benchmarking revealed
challenges of inter-operability and practices. Project has
succeeded in defining a way forward around repository
development – progress also in joint approach to student
‘transition web’ around e-portfolios, involving templates.
INDICATORS Collaboration in Curve repository development (JISC).
Introduction of e-portfolios into Warwickshire College

Indicators around transition defined but not yet measured.

BRIEFING PAPERS Good practice guide to FE/HE Collaboration.

UNIVERSITY OF HERTFORDSHIRE

Page 31 of 50
Appendix B: Pathfinder Projects by Institution

KEY FEATURES Local version (CABLE) of the Change Academy model,

managed through the Blended Learning Unit (a CETL), involving
direct support for innovative blended design in six Schools with
an intensive four day residential event. CABLE Toolkit
developed.
INDICATORS Awarded Network project (CABLE Transfer) to roll out the model
more widely into the sector. CABLE process taken to
Sunderland, Worcester, St Mary’s Twickenham and Reading.
External evaluation.

New partnerships between BLU and Academic Schools in place

in School plans. CABLE 2 underway with 5 new teams.

Senior management commitment to embed the CABLE process

as permanent procedure.
BRIEFING PAPERS 1. Briefing Paper for Academic Staff Involved in Curriculum
Design: Involving Students
2. Briefing Paper for Heads of Academic
Departments/Schools: Blended Learning Enhancement
3. Briefing Paper for HEA Subject Centre: Health and Sciences
Practice

INSTITUTE OF EDUCATION

KEY FEATURES ‘Salutary experience’ of benchmarking led to project to create a

clear connection between own research on pedagogy, and own
teaching. Addressed through a review of relevant research and
staff development events aimed at course redesign. Awarded
Network Project to generate sector-wide SIG on relation
between e-learning and quality.
INDICATORS E-learning now central in the IoE’s new learning and teaching
strategy.

QAA involved in Network Project.

BRIEFING PAPERS 1: Course Redesign

2: Linking Research and Practice

UNIVERSITY OF LEICESTER

KEY FEATURES Benchmarking identified weakness in e-pedagogy. ‘ADELIE’

project organised around two-day intensive re-design workshops
(‘Carpe Diem’) with subject teams (16 course teams from 11
depts.) E-moderator courses mounted. Network project awarded
(’CHEETAH’) to extend the approach to other institutions (
University of Bath, University College Falmouth, Leeds
Metropolitan University, Newman University College, University
of Worcester, Oxford Brookes University).
INDICATORS CHEETAH symposium indicated emergence of wider community
around Carpe Diem. Collaboration with Oxford Brookes, and

Page 32 of 50
Appendix B: Pathfinder Projects by Institution

informally with other Pathfinders, to share and improve

methodology.

Write-up from University of Worcester on their Carpe Diem

experience. Indicator of change at
http://pebblepad.worc.ac.uk/webfolio.aspx?webfolioid=9824
Evaluation data on impact due.
BRIEFING PAPERS 1: E-tivity Designs for Learning

2: Changes in Teaching Practices

UNIVERSITY OF MANCHESTER

KEY FEATURES e-Change project focused on development of eMM framework

for driving change in newly merged complex institution. Six use
cases across subject areas, involving ‘change team’ and
process models. Reference model established. Extension
project awarded to test and disseminate proof-of-concept
Decision Support Tool aimed at supporting e-learning planners
and managers in HE. Requirements captured from interviews
with senior management.
INDICATORS Convergence of the knowledgebase with the JISC Innovations
Knowledgebase, resulting in common user interface.

ALT-C 2008 workshop

BRIEFING PAPERS 1: Implementing an e-repository.

2: Using eMM to support staff development.

3: Developing a Knowledge Base for Change Management.

OXFORD BROOKES UNIVERSITY

KEY FEATURES Focused on evaluation and evidence-informed practice, though

components of the project contributed to the design of a
repository architecture, and the exploration of social networking
applications for reflective learning. Large scale institutional
survey of student uses of technology led to the award of the
ELESIG Network Project (with University of Bradford and
University of Greenwich) to create a community around
researching the student experience of e-learning.
INDICATORS ELESIG evaluation shows high interest level across the sector
and commitment by members to sustain the SIG.

Collaboration with CABLE at the University of Hertfordshire, and

Carpe Diem at the University of Leicester, to produce evidence-
based common approach to intensive course redesign.
BRIEFING PAPERS 1: Technology, Applications and Organisation (TAO) for
Learning.

Page 33 of 50
Appendix B: Pathfinder Projects by Institution

2: Course Redesign Intensives.

3: Researching Student E-Learning Experience: Methodological

Issues and Challenges.
UNIVERSITY OF WARWICK

KEY FEATURES Benchmarking highlighted, along with many strengths, some

weakness in coordinating the different support services involved
in e-learning, and dissemination of good practice between
academic areas. Project addressed this by attempting a multi-
service approach, re-focused on academic departments. Range
of development projects coordinated across Faculties. WELA
(Warwick e-learning award) offered free to other institutions, and
e-learning Showcase days and Faculty e-learning lunches
organised.
INDICATORS Project coordinated from Teaching Quality office. Participated in
International Benchmarking exercise with the Observatory on
Borderless Higher Education (OBHE).

Embedding E-Learning (EEL) Challenge Fund supported.

‘Teaching Grid’ – part of high profile Library Remodelling, giving

physical space for support and advice.
BRIEFING PAPERS 1: E-learning Showcase Day.

2: Teaching Quality and Pathfinder.

Page 34 of 50
Appendix B: Pathfinder Projects by Institution

PATHFINDER PHASE ONE PROJECTS

CLUSTER A (Critical Friend: Paul Bacsich)

UNIVERSITY OF BRADFORD

KEY FEATURES Benchmarking fed into development of coherent e-strategy.

Unique student profile led to e-learning focus on the ‘extended
student’ – pre-entry and for first year. E-learning integrated into
coordinated support –materials development, survey of student
needs, support for e-assessment.
INDICATORS Member of ELESIG consortium.

Institutional Exemplar for JISC thin client (e-assessment).

BRIEFING PAPERS 1: Developing the Model of the Extended Student: How using an
institutional skills and personal reflective activity can support the
process of transition and help students create a meaningful
Personal Development Plan.

2: Summative Online Assessment: Lessons Learned and

Recommendations .

UNIVERSITY OF BRIGHTON

Benchmarking revealed leading-edge integrated technical

KEY FEATURES
environment (studentcentral; Community@Brighton) not well
used pedagogically. Staff development focus requiring
integration of technical and pedagogical support. Intensive
workshops for course teams. Particular emphasis on social
networking.
INDICATORS University new Learning and Teaching strategy specifically
commits to sustain the approach.
BRIEFING PAPERS None submitted.

UNIVERSITY OF GREENWICH

KEY FEATURES Student experience of e-learning laboratory (SEEL). Project has

collected data from a range of methods investigating the student
experience of e-learning and has disseminated early findings
widely through a range of workshops and case studies. Impact
on learning and teaching policy.
INDICATORS Partner in ELESIG. Host of ELESIG symposium, as part of

Page 35 of 50
Appendix B: Pathfinder Projects by Institution

annual e-learning conference.

Three year commitment to continue the project.

BRIEFING PAPERS No briefing papers as such but an extensive ‘Journey’ Report
addressing key issues.

THAMES VALLEY UNIVERSITY

KEY FEATURES Benchmarking revealed initially low level of e-learning use.

[R]elearn Project designed to kick-start major initiative in
embedding sustainable staff development and technical support
in all academic areas. E-Learning support posts (ELTs) created.
Drop-in surgery approach devised. Impact difficult to assess as
confounded with major restructuring.
INDICATORS University policy paper emerged: Mainstreaming Blended
Learning.

ELT appointments continued through internal funding.

BRIEFING PAPERS None submitted.

Page 36 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER B (Critical Friend: Cliff Wragg)

BOURNEMOUTH UNIVERSITY

The eRes project focuses specifically on engaging learners to

KEY FEATURES
use quality e-resources in e-learning activities using Web 2.0
technologies within the VLE. Learning technologists, librarians
and staff developers provided joint support for academics in 12
case studies.
eRes SIG created.
INDICATORS
Project has already generated 10 publications and 13
conference presentations. 
1: e-Res: Innovative Learning with e-Resources
BRIEFING PAPERS
2: e-Res: Reading Strategies in the Digital Environment
3: e-Res: Support Needs
4: Role of the Learning Technologist

COLLEGE OF ST MARK & ST JOHN

KEY FEATURES Benchmarking highlighted need for specific e-learning goals in

the Learning and Teaching Strategy. Survey of staff and student
needs. New e-learning support service created.

Project focuses on exploiting the agreed ‘universal student

access to a laptop’.
INDICATORS New institutional e-learning strategy produced, approved by
Learning and Teaching Committee. Contains metrics for
measuring progress.

High level of interest in the ‘Guild HE’ sector at events.

BRIEFING PAPERS 1: Development of RLOs. 2: Staff & student support.

UNIVERSITY OF GLOUCESTERSHIRE

Modestly-funded, narrowly-focused project on exploring the

KEY FEATURES
potential for reflection, learning and assessment of digital
storytelling. Copyright, assessment and ethical issues
highlighted. Workshops delivered.
INDICATORS Held 2-day conference on digital storytelling .

BRIEFING PAPERS
1: Emerging Pedagogies 2: The Learner Voice

Page 37 of 50
Appendix B: Pathfinder Projects by Institution

UNIVERSITY OF KEELE

KEY FEATURES Benchmarking had revealed unease concerning the status and
use of digital resources. Project explored issues around the
development of a digital repository linked with an existing VLE.
‘Proof of concept’ study around digital resources, metadata and
embedding the resultant new practices across the university.
Encouraging subject teaching staff in the purposeful use and re-
use of the resources.
INDICATORS By end of project 70 course modules were accessing 350
scanned resources.

Exemplars of good practice for staff use.

University has committed to continued funding to sustain the

work started in the project.
BRIEFING PAPERS Briefing Notes for the IntraLibrary Repository from Intrallect: A
Technical Perspective.

Page 38 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER C (Critical Friend: Grainne Conole)

LONDON SOUTH BANK UNIVERSITY

KEY FEATURES Detailed development and implementation of ICT Literacy

benchmarks. The Compass project focussed on the
identification and development of academic staff skills with the
benchmarks, mapped to staff roles. Staff ITC survey conducted
and ICT course designed. 3 Carpe Diem workshops delivered.
INDICATORS London South Bank University (LSBU) and University of
Northampton to facilitate Carpe Diem workshops in each others’
institutions.

Partner in JISC Curriculum Design Project, generated by Cluster

C and the OU.

EdMedia symposium and ALT-C workshop by Cluster C.

BRIEFING PAPERS Seize the team: engaging academics in the process of
developing material for online delivery using the Carpe Diem
methodology.

UNIVERSITY OF READING

KEY FEATURES The ‘DIRECT’ project developed an approach to transform its

institutional quality management processes. A framework for
institutional change developed. 4 Depts across 3 Schools piloted
new framework, involving tools, templates, staff development
workshops. Partner in QA/QE SIG Network Project.
INDICATORS New framework incorporated into University Periodic Review QA
procedure.

2 Pathfinder Posts extended for 15 months with University

funding.

Partner in JISC Curriculum Design Project, generated by Cluster

C and the OU.

Hosted QA/QE SIG event, July 2008.

CABLE Transfer event planned for September 2008.

EdMedia symposium and ALT-C workshop by Cluster C.

BRIEFING PAPERS 1: Raising awareness of the costs of e-Learning.

2: Getting the most out of adopting a cluster approach:

synergies and collaboration.

3: A framework to enable quality management and

enhancement.

Page 39 of 50
Appendix B: Pathfinder Projects by Institution

BRUNEL UNIVERSITY

KEY FEATURES Project gathered information on emergent e-pedagogical

practices within Schools. An ‘Appreciative Inquiry’ (AI) was
conducted within 4 Schools. The generated local evidence-base
is used to inform a change management programme with
Schools.
INDICATORS Remaining Schools committed to completing the process.

Partner in JISC Curriculum Design Project, generated by Cluster

C and the OU.

EdMedia symposium and ALT-C workshop by Cluster C.

BRIEFING PAPERS 1: The implementation of Appreciative Inquiry (AI) to gain ‘buy-
in’ from academic staff for improved e-learning practices.

2: Using a Personal Response System (PRS) for Teaching &

Learning in Higher Education.

3: Educational Considerations for Blended Learning.

UNIVERSITY OF CAMBRIDGE

KEY FEATURES The Learning Landscape Project (LLP) has gathered data from
staff, students alumni and providers of learning support across
the collegiate university, analysing their perspectives on a range
of key issues related to teaching and learning. This has led to
the identification of ‘clusters’ of practice and patterns of
innovation that will inform future collaboration and development.
Developed an innovative online evidence base to inform the
University’s Learning and Teaching Strategy. 
INDICATORS Informing a major TQEF funded initiative on supporting the
development of transferable skills across Undergraduate
courses.

Partner in JISC Curriculum Design Project, generated by

Cluster C and the OU.

EdMedia symposium and ALT-C workshop by Cluster C.

BRIEFING PAPERS 1: Research Methods: Case Records.

2: Research Methods: Movers and shakers focus group.

3: Research Methods: Coffee meeting interviews.

4: Research Methods: Project steering group.

Page 40 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER D (Critical Friend: Terry Mayes)

EDGE HILL UNIVERSITY

KEY FEATURES The project focus on e-learning in partnership delivery

arrangements and off-campus locations emerged from e-
Benchmarking. Range of interventions, informed by the
approach of the SOLSTICE CETL around ‘new academic
teams’. Developed guidance materials and workshops for
professional development in off campus e-learning development.
Also metrics for costing e-learning. Online modules for student
induction to e-learning. Audit tool for off-campus.
INDICATORS Learning Services audit procedures now include off-campus
provision.

E-learning strategy manager has brief to work across FE

partners.
BRIEFING PAPERS Raising awareness of the costs of e-Learning.

DE MONTFORT UNIVERSITY (DMU)

KEY FEATURES
Overarching aim for the DMU Pathfinder was to develop a
read/write culture across the university. Project focused upon:
engaging university leaders and managers, and PGCertHE
practitioners, with Web 2.0 tools; enhancing professional
development opportunities for support and academic-related
staff; podcasting; evaluating read/write tools to support retention
and progression. Project led by example in the use of read/write
technologies. 
INDICATORS
Dept of Academic Quality and the e-Learning Coordinators have
created a network of 52 local champions. DMU SIGs created on
podcasting, games for learning, support staff, e-assessment.

Project nominated by DMU for the national ICT Initiative of the

Year Award (THE).

Partnership arrangement with DMU Students’ Union Executive

for continuing Pathfinder developments around student use of
technologies.
BRIEFING PAPERS 1: Podcast.

2: The impact of read/write web technologies and approaches

on Professional Development: a briefing for Professional
Developers.

3: The impact of read/write web technologies and approaches

on academic-related staff: a briefing for academic-related
managers.

Page 41 of 50
Appendix B: Pathfinder Projects by Institution

MANCHESTER METROPOLITAN UNIVERSITY

Benchmarking revealed lack of Intra-institutional collaboration in

KEY FEATURES
e-learning development. Development of institution-wide
strategies for Faculties and Departments to increase
collaborative e-learning design, development and use/re-use.
Evaluating infrastructure to support Podcasting and the
dissemination of “good practice” via a repository of 12 case
studies. Cross-institutional CoPs set up. Independent formative
evaluation team fed recommendations for change to the team
and to senior management.
INDICATORS Critical Friend and the Pathfinder Team invited to meet
University Directorate in July 2008. VC gave undertaking to
continue to support what the project had initiated.
BRIEFING PAPERS 1: Projects and Institutions: working together for a change.

2: Developing communities of academic practice.

UNIVERSITY OF CENTRAL LANCASHIRE

KEY FEATURES Benchmarking revealed need for blending of VLE use with more
open, student orientated platforms. Project explored the use of
institutional-based online social networking to support student
engagement. Also focused on online orientation and support
materials for students about e-learning tools and resources. New
options trialled for staff development for e-learning, to explore
the nature of disciplinary differences in approaches to e-
learning. Case studies and exemplars of e-learning practice
within disciplines (20 studies produced). Investigated viability of
online peer network for off-campus liaison.
INDICATORS Project will continue for further year with internal funding.

BRIEFING PAPERS {pending}

Page 42 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER E (Critical Friend: Peter Chatterton)

UNIVERSITY OF DERBY

KEY FEATURES The project has establishment the Flexible Learning Network to
coordinate an enhancement programme across a number of
strands of activity in staff development and support, technical
issues, quality assurance, course design and pedagogic
research. The FLN facilitates cross institution coordination and
the e-learning Teaching Fellow scheme expanded through the
project. Work underway with the QE Dept to embed TEL in the
review of QA procedures. Staff development needs analysis tool
delivered to Faculties. Project participated in Network Project
PREEL 2.
INDICATORS FLN produced Communication Strategy and Plan for TEL.

BRIEFING PAPERS Developing and implementing a communication strategy for

Technology Enhanced Learning, a vehicle to promote academic
engagement.

UNIVERSITY OF TEESSIDE

KEY FEATURES Project focused on E-Quality based on Benchmarking outcomes.

The main institutional benefits were the development of explicit
quality processes and procedures for e-learning and the
development of an institution-wide approach to supporting and
enhancing the student experience. These included minimum
standards, diagnostic tools and sharing of good practice,
including staff development activities. Analysis of students’
experiences and needs informed changes to practice including
enhanced online pre-induction support, increased student
involvement in e-learning design, and the development and
trialling of new learning technologies. Outputs include a
threshold standard for VLE use, a range of support materials for
staff including guidelines and diagnostic/audit tools. Partner in
Preel2 Network project. Hosted one of the Quality A/E in E-
learning workshops.
INDICATORS A new peer review scheme for teaching and learning has been
developed including guidelines and procedures for peer review
of both face-to-face and online teaching activities. This will be
implemented for all academic staff from September 2008.

Learning and Teaching team in CLQE are developing

programme design workshops based on recommendations
produced by the Pathfinder project. These will be rolled out to
Schools from the start of the new academic year for targeted
programmes due for revalidation or review.
BRIEFING PAPERS Students’ E-learning Experiences and Expectations.

Page 43 of 50
Appendix B: Pathfinder Projects by Institution

UNIVERSITY OF EAST LONDON (UEL)

KEY FEATURES The UEL DEEP Pathfinder project arose from Benchmarking
where the need for a comprehensive e-learning staff
development portfolio was identified. This has been designed
and implemented. It includes a detailed staff development needs
analysis tool, a suite of activities and exercises for change
management among staff and a portfolio of case studies of good
e-learning practice within the university. The project has also
facilitated the development of internal and external
communication strategies to support e-learning.

Have run CABLE workshops through collaboration with the

Blended Learning Unit CETL (BLU) at the University of
Hertfordshire. 
INDICATORS E-learning benchmarking exercise to be run as a sustainable
activity.
BRIEFING PAPERS Routes to Success: Finding the UEL Path through Staff
Development.

UNIVERSITY OF SUNDERLAND

KEY FEATURES A major outcome from benchmarking exercise was identifying

that some academic and learning support staff had a skills gap
in being able to use e-learning appropriately. The pathfinder
project used the existing HE Academy Professional Standards
Framework (PSF), areas of activity and professional values
to design, plan, implement and evaluate criteria for the effective
use of e-learning in a formal programme of study for new staff
and in staff development for existing staff: output is a new PSF
for e-learning and a transferable staff development plan.
INDICATORS University-level new Technologies for Learning Group set up.

University task group created to look at legal issues around

social networking tools.
BRIEFING PAPERS 1: Professional Standards Framework for Teaching and
Supporting e-Learning in Higher Education
Example for Standard Descriptor 3

2: Professional Standards Framework for Teaching and

Supporting e-Learning in Higher Education
Example for Standard Descriptor 1/2

Page 44 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER F (Critical Friend: Betty Collis)

KINGSTON UNIVERSITY

KEY FEATURES Specific aim of Kingston’s project is to conduct research on the

in-classroom use of mobile technologies to support diagnostic
and formative assessment. The mobile technologies used were
Electronic Voting Systems, Tablet PC’s, Interactive tablets, i-
pods and Mobile phones. Thirteen academic staff members
from 7 different faculties used mobile technologies in the
classroom for formative assessment purposes and to provide
rapid feedback to students on their knowledge and
understanding. Eleven successfully implemented mobile
technologies in their classroom and plan to keep using the
technologies in the next academic year. The use of a staff
mentor approach to introducing new approaches to learning and
teaching has been successful. Mentors have been available to
support staff prior and even during lectures. Their support has
been very appreciated by the participants. Preliminary analysis
of the student evaluation data indicates a very positive
response from students.  
INDICATORS Seven conference presentations on the research already given,
or accepted.
BRIEFING PAPERS 1: Using mobile technologies to enhance student engagement
and collaboration in the classroom.

2: Using a Staff Mentoring Scheme to Embed New Learning

Technologies.

3: Strategies for the effective and appropriate implementation of

electronic voting systems in the classroom.

UNIVERSITY OF EXETER

KEY FEATURES Exeter has an extensive history of using video and conferencing
technologies (VC) for teaching and learning. With the launch of
the Peninsula Medical School in 2002, and the University’s
Cornwall campus over 100 miles away in 2004, Exeter’s portfolio
of programmes expanded rapidly, with VC being used to build
cross-campus capacity in several disciplines. The Pathfinder
project has gathered data on which to base a review of VC
practices and procedures and has developed a staff
development workshop for teachers new to VC, grounded in
research and evaluation. By evaluating current practice and
assessing the impact of VC on students and staff, Pathfinder
has informed inter-campus strategy and curriculum
development, and the long-term development of VC
infrastructure within the University.

Page 45 of 50
Appendix B: Pathfinder Projects by Institution

INDICATORS Over the past two years (2006-2008), around 800 VC teaching
sessions have taken place between Exeter and Cornwall
campuses - a saving equivalent to 180,000 miles’ travel and 113
weeks’ academic staff time, plus reducing the University’s
carbon footprint by approximately 60 tonnes of CO2.
BRIEFING PAPERS 1: Rationale, research design and Recommendations.

2: Using video and conferencing technologies for large-group

cross-campus teaching.

3: Medical student’s perceptions of telematic lectures.

4: Using video and conferencing technologies for small-group

and innovative teaching.

5: ‘Can you see me? Can you hear me?’ – Staff development
for teaching cross-campus through video conferencing.

UNIVERSITY OF WOLVERHAMPTON

KEY FEATURES
Project aimed to embed the use of ePDP activities into the Level
1 curriculum of two modules from each of the 10 Schools. The
chosen methodology was to use ePDP peer mentors to support
Level 1 tutors in their planning, design, use and assessment of
integrated ePDP tasks. Staff development in support of the
mentor role and for tutors implementing ePDP was brokered
through a series of three away-day retreats following a
developmental role of mentoring. Nine staff carried out their role
as an ePDP mentor and nineteen staff integrated a variety of
ePDP tasks aimed at different aspects of PDP into their
curriculum. 1810 Level 1 students engaged with these ePDP
tasks in a range of subjects. Modules ranged from groups of 15
to the largest module with 350 registered students. The project
was important to the institution to move past ‘champions’ and
early adopters of the ePortfolio tool to build personal capability in
staff and institutional capacity to increase the quality and amount
of PDP activities in the level 1 curriculum.  
INDICATORS
The use of ePDP has been incorporated as a distinct strand of
the University’s new Blended Learning Strategy.Successful bid
to University Executive to approve that ePDP mentors can be
funded at a 0.2 level in each School.
BRIEFING PAPERS 1: Emerging pedagogies using webcams, podcasting and
YouTube in eportfolio.

2: Student evaluation and the learner voice.

3: A mixed model for staff development – ePortfolio,

developmental mentoring and total immersion retreats.

4: What do we mean by PDP? ePortfolio? and ePDP?

Page 46 of 50
Appendix B: Pathfinder Projects by Institution

CANTERBURY CHRIST CHURCH UNIVERSITY

KEY FEATURES The DEBUT project has piloted and evaluated a digital literacy
staff development programme. Its focus stemmed from the
benchmarking exercise which highlighted that despite the
institutional VLE being used within nearly all academic
programmes, there were only limited examples of e-learning
strategies that fully exploited a range of technologies. DEBUT
explored a situated approach which aimed to develop the overall
digital literacy of the individual, rather than their technical skills
on a particular system. From sixty applications twenty-five
participants were accepted onto the project. A suite of twenty
three digital experiences were captured. The DEBUT tools were
supported by a variety of staff development approaches, giving
the participants a range of digital experiences which had
meaning for them at a time which best suited them. A range of
evaluation methodologies was used to evaluate DEBUT,
including the use of a digital literacy scale to provide a
benchmark against which participants could position
themselves.
INDICATORS Regional JISC Centre to host dissemination conference on
DEBUT.

Researcher funded to continue the project beyond the He

Academy funded period.
BRIEFING PAPERS 1: Mind the Gap – Bridging the digital divide: an evaluation of a
staff development programme focusing on digital literacy.

2: Learning the lesson: a briefing paper for educational

developers on the effectiveness of a range of different staff
development strategies used within a digital literacy
development programme.

3: Tools of the trade: A briefing paper for educational developers

providing responses to a variety of digital tools explored and
evaluated by a group of HE staff on a digital literacy
development programme.

Page 47 of 50
Appendix B: Pathfinder Projects by Institution

CLUSTER G (Critical Friend: Andrew Comrie)

BUCKINGHAMSHIRE NEW UNIVERSITY

KEY FEATURES The project has produced enhancements in both the general
and technical support provided by central services and the
academic and personal support provided by the faculties for
Work-Based Learning students. It makes recommendations for
changes in course design and development, quality assurance
internally and externally, improved marketing, improved IT set-
up, improved IT induction and support, as well as redesign of
enrolment processes.

Outputs include: briefing paper on support for Work Based

Learning; a guide to QEC on appropriate points for course
validation and review and a guide for QAA auditors to support
audit of e-learning provision. The project has provided enhanced
technical and general support for on-line Work Based Learning
(WBL) students, as well as enhanced academic and personal
support for on-line WBL students off campus. The project
outputs have also informed the e-strategy aspect of the new
Teaching, Learning and Assessment Strategy.  
INDICATORS The project outcomes will be embedded in the revised quality
assurance processes for validation and review, and the new
Teaching, Learning and Assessment Strategy.

Guidelines for audit endorsed by the SCONUL QA Working

Group.

An enhanced reputation for supporting non-traditional students

on-line, has produced student applications for next year showing
dramatic growth.

UCISA and SCONUL committees will discuss the implications of

the project during the coming year as the project leader is Chair
of UCISA Teaching, Learning and Information Group.
BRIEFING PAPERS Support for Work-Based Learning.

UNIVERSITY OF NORTHAMPTON

The BITE project has designed and developed 'bite' size chunks
KEY FEATURES
of staff development materials in e-learning. These are
personalised to specific subject areas, and aimed at helping staff
develop their abilities to teach more effectively using technology.
The case studies and resources focus primarily on the potential
benefits of e-learning and the advantages of using new and
emerging technologies within teaching. BITE is embedded as
part of school support modules on the existing PGCTHE.

Page 48 of 50
Appendix B: Pathfinder Projects by Institution

A second e-learning benchmarking snapshot will take place

INDICATORS
Autumn 08 as part of an overall learning and teaching
benchmarking exercise (ELBME 2) and the report from this will
measure the impact of BITE in the specific schools and subject
areas that have been covered as part of this project.
1. The Qualitative Cost of E-Learning
BRIEFING PAPERS
2. Academic engagement with e-learning in the creative arts
(with University of Cumbria)

BRADFORD COLLEGE

KEY FEATURES Using Foundation Degrees, delivered through HE in FE, to

identify opportunities offered by learning technologies to tie
together three primary stakeholders; Employer, HE institute and
learner. A conceptual model of FD development and delivery is
a key output. Project is leading a re-assessment of the highly
instructive approach to pedagogy that is inherent in traditional
employer engagement programmes; perhaps engendering,
through the use of social networking technologies, a more
collaborative learning environment.
INDICATORS The projects’ influence has been reflected in the e-learning
policy of the college. All 58 learning resources staff have been
given new roles within a staff restructure that have a clear focus
on supporting e-learners.  
BRIEFING PAPERS 1: Creating connections and peer learning:
A diverse, flexible and virtual approach to staff development and
collaboration between Academics and Learning Technology
Support Staff.

2: A Conceptual Model for embedding e- learning in Foundation

Degree delivery.

UNIVERSITY OF CUMBRIA

KEY FEATURES The EPLE (Enhancing Practice-Led E-learning for Art, Design
and Media) project sought to investigate the innovative
application of e-learning in creative arts disciplines by providing
a foundation of training and support for academic staff. This has
led to discrete findings from three subject areas, and common
emergent themes. A model approach for engaging academic
staff in the design of e-learning activities for creative arts
disciplines has emerged out of the project. Whilst the front-line
focus of the project was the investigation of innovative use of
Blackboard in Journalism, Life Drawing and Photography
modules, the process of working with staff in order to achieve
this has been resolved as an outcome in itself. Carpe Diem
Leicester workshop to be contracted. 
INDICATORS Plan approved by Faculty of Arts to continue the Pathfinder

Page 49 of 50
Appendix B: Pathfinder Projects by Institution

work.

Collaboration over briefing papers with the HEA Subject Centre

for Art, Design and Media for dissemination to their community.
BRIEFING PAPERS 1: Academic engagement with e-learning in the creative arts.

2: Experiences of e-learning innovation in art, design and media

disciplines.

Page 50 of 50