2020 Book TechnologyDesignAndTheArts-Opp

Springer Series on Cultural Computing
Rae Earnshaw · Susan Liggett ·

Peter Excell · Daniel Thalmann Editors
Technology, Design
and the Arts—
Opportunities and
Challenges
Founding Editor
Ernest Edmonds, De Montfort University, Leicester, UK
Series Editor
Craig Vear, De Montfort University, Leicester, UK
Editorial Board
Paul Brown, University of Sussex, Brighton, UK
Nick Bryan-Kinns, Queen Mary University of London, London, UK
David England, Liverpool John Moores University, Liverpool, UK
Sam Ferguson, University of Technology, Sydney, Australia
Bronaċ Ferran, Birkbeck, University of London, London, UK
Andrew Hugill, University of Leicester, Leicester, UK
Nicholas Lambert, Ravensbourne, London, UK
Jonas Lowgren, Linköping University, Malmo, Sweden
Ellen Yi-Luen Do , University of Colorado Boulder, Boulder, CO, USA
More information about this series at http://www.springer.com/series/10481
Rae Earnshaw Susan Liggett Peter Excell
• • •
Daniel Thalmann
Editors
Technology, Design
and the Arts—Opportunities
and Challenges
Editors
Rae Earnshaw Susan Liggett
Department of Computer Science, Faculty of Faculty of Art, Science and Technology
Engineering and Informatics Wrexham Glyndŵr University
University of Bradford Wrexham, UK
Bradford, UK
St John’s College Daniel Thalmann
Durham University MIRALab Sarl
Geneva, Switzerland
Durham, UK
Faculty of Art, Science and Technology
Wrexham Glyndŵr University
Wrexham, UK
Peter Excell
Emeritus Professors, Executive Office
Wrexham Glyndŵr University
Wrexham, UK
Faculty of Engineering and Informatics
University of Bradford
Bradford, UK
ISSN 2195-9056 ISSN 2195-9064 (electronic)

ISBN 978-3-030-42096-3 ISBN 978-3-030-42097-0 (eBook)
https://doi.org/10.1007/978-3-030-42097-0
© The Editor(s) (if applicable) and The Author(s) 2020. This book is an open access publication.
Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adap-
tation, distribution and reproduction in any medium or format, as long as you give appropriate credit to
the original author(s) and the source, provide a link to the Creative Commons license and indicate if
changes were made.
The images or other third party material in this book are included in the book’s Creative Commons
license, unless indicated otherwise in a credit line to the material. If material is not included in the book’s
Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the
permitted use, you will need to obtain permission directly from the copyright holder.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publi-
cation does not imply, even in the absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with regard
to jurisdictional claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
The editors dedicate this book to their
partners and parents
Foreword by Jill Journeaux
Charcoal, the interaction of fire and carbon, and naturally occurring pigments such
as red clay were the first materials used by homo sapiens to record their presence in
the world. Throughout time, artists have been curious about tools and technologies
that have emerged from adjacent or alternative domains and have been open to
reapplying those tools and technologies into new areas in innovative ways. In recent
times, it is computers and interactive systems that have driven this curiosity, but we
should also remember the great variety of forms of technology that have expanded
and accelerated art practice throughout history. These include photography, the use
of hot metals such as bronze and steel to create form, the unique qualities of
plastics, the simple processes of fire in raku ceramics and in making charcoal, and
the creation of synthetic pigments in 1704. The application of these inventions and
innovations to artmaking has involved highly skilled technologists who have
worked in the spaces between art and science to manifest new possibilities of form
and expression.
In order to make sense of the world and our place within it, we attempt to
construct conceptual models. Artists are often attracted to the apparent certainties of
science but bring to it expressions of doubt and uncertainties about a rationalist
approach. They utilize dialogues between interiority and exteriority in order to
confirm the nature of being in the world as known or understood through inner
emotional and spiritual worlds, and use artmaking as a process that searches for
established and new truths based on knowledge and science but felt and intuited
through visual recognition by others. They place value on the unknown as well as
the known.
In this book, Rae Earnshaw, Susan Liggett, Peter Excell and Daniel Thalmann,
explore the spaces between technology, design and the arts and ask questions of the
challenges and opportunities that are presented by interdisciplinary research across
these fields. The inclusion of a series of case studies allows for consideration of the
value of interdisciplinary research which incorporates practitioners, practice and
practice research and enables explorations of the manner in which technological
innovations shape cultural manifestations and extend the capacity for artists to
articulate experiences of being, thinking and questioning.
vii
viii Foreword by Jill Journeaux
Rae Earnshaw and Susan Liggett first collaborated in 2015 when Liggett curated
the exhibition Carbon Meets Silicon, in Oriel Sycharth. This exhibition included
works by artists who collaborate with scientists, or technologists and was accom-
panied by a symposium as part of the 6th International Conference on Internet
Technologies and Applications. The conference brought together researchers and
developers from academia and industry across engineering, computing and art and
design, and was inspired by Alan Turing’s explorations of artificial intelligence as
opposed to cognitive psychology. Artwork in the exhibition explored a range of
media practices including film and video installation, painting, printmaking, jew-
ellery and sound-based works.
As part of the 7th International Conference on Internet Technologies held in
2017, Earnshaw and Liggett organized the exhibition Carbon Meets Silicon II
which brought together fifteen diverse artists using science and technology to better
understand the world we live in. The thematic exhibition used carbon and silicon as
metaphors for the evolving nature of art practice in a digital era. Carbon was used to
refer the materiality of art objects and silicon to refer to how an artworks’ physical
presence may be questioned by new media and technologies, thus bringing artist
and scientist closer together.
The editors of this book are concerned to reframe perceptions of dualism and
difference between artists and scientists to ensure that they are not set apart but can
instead find ways of working together to solve problems. However, if we consider
the processes and ambitions associated with creativity supposed differences
between artists and scientists may become less distinct. Creativity theorist Mihaly
Csikszentmihalyi reminds us that: creativity does not happen inside people’s heads,
but in the interaction between a person’s thoughts and a sociocultural context. It is
systemic rather than an individual phenomena (1997: 23). In his book, Creativity
Csikszentmihalyi argues that levels of creativity do not just depend upon creative
individuals but are also affected by the willingness of domains and fields to rec-
ognize novel ideas and artefacts (1996: 31).
Writing about successful collaborative partnerships in both science and the arts
Vera John-Steiner emphasizes the value of complementarity stating: Each indi-
vidual realizes only a subset of the human potential that can be achieved at a
particular historical period (2000: 40). She argues that the modes of thought
involved in generating explanatory principles are not necessarily the ones used in
communicating scientific discoveries and refers to Reuben Hersh’s idea of the
‘backstage’ of discovery, which may consist of fragments, the conversational,
informal and intuitive propositions, and tentative notions. As with artists some
scientists only share their complete and finished products, but others reveal their
thought processes through notebooks, drawings, journals and other preparatory
thinking processes and formats [3]. Technologists, artists and scientists visualize the
as yet unmade and the possible as well as the actual and the clearly thought and
understood.
In an interview with Alfred Appel Jr., in 1966, Vladimir Nabokov reacted to
C. P. Snow’s assertion that the gap between the two supposedly separate cultures of
science and the arts was unbridgeable. He argued that science has an artistic and
Foreword by Jill Journeaux ix
creative side and that the arts require scientific truths, saying: I certainly welcome
the free exchange of terminology between any branch of science and any raceme of
art. There is no science without fancy, and no art without facts (1973: 78–79).
Jill Journeaux
Professor of Fine Art
Centre for Arts, Memory and Communities
Coventry University, UK
References
1. Csikszentmihalyi, M.: Creativity: Flow and the Psychology of Discovery and Invention, p. 31.
Harper, New York (1996)
2. Earnshaw, R.A., Liggett, S., Excell, P.S., Thalmann, D. (eds.): Technology, Design and the
Arts—Challenges and Opportunities. Springer, Cham, Switzerland (2020). ISBN
978-3-030-42096-3
3. Nabakov, V.: Strong Opinions, pp. 78–79. McGraw-Hill Book Company, New York (1973)
4. John-Steiner, V.: Creative Collaboration, pp. 40–46. Oxford University Press, New York
(2000)
Foreword by Jon Peddie
Technology, Design and the Arts—Opportunities and Challenges is a book for the
times. Never before has there been such a confluence between technology and art as
today. Amazing new developments in computer graphics, not the least of which is
the democratization of the tools, has empowered and emboldened designers, and
want-to-be artists all over the world. Art, from fine art to set backdrops, video
games and graphics novels, has permeated our lives, imaginations and culture at all
levels—and we are richer for it all.
It all begins with the software tools, which as the book beautifully illustrates,
have been used from the earliest times to create and modify artistic works. And the
concept is not limited to just graphics art, but various types of tools for creating
image and musical instruments—computer technology has been used to create
digital images and audio.
The authors are to be congratulated for being brave enough to examine and
discuss the symbiotic relationship between artistic works and the cultural context in
which they are produced.
Computers have been used to mimic the way classical artists did things, their
attempts were not always successful. The technology can provide continuity by
taking traditional methods and techniques and making them more efficient and
effective. However, it can also provide divergent results and open up new per-
spectives and paradigms—sometimes intentionally, and sometimes accidentally.
The authors argue that it can produce a greater understanding of artistic processes
and how they are implemented in practice. The reader will be the judge.
The authors believe a tighter coupling of the interplay between the goal of the
creator, the selection and use of appropriate tools, and the materials and repre-
sentations chosen has been created. They cite Gombrich who argued that, There
really is no such thing as Art. There are only Artists. Thus, although creative works
may generate a variety of interpretations in the minds of observers, say the authors,
the focus also remains on the creator, and the vision and insight that causes them to
create. Thus, what has been created provides insight into the cultural context and
social environment at the time of creation. Powerful ideas are very effectively
argued and demonstrated in the book.
xi
xii Foreword by Jon Peddie
The book proposes to define and detail the relationship between the artist and
their works by the types of tools they use, and the environments that have facilitated
and extended these processes. Such tools for graphics say the authors have
encompassed computer technology, computer environments, and interactive devi-
ces for a range of information sources and application domains. They have also
provided new kinds of created works which are able to be viewed, explored, and
interacted with, either as an installation or via a virtual environment such as the
Internet.
I agree, and think this introduces new dimensions of understanding and expe-
rience for both artist and the public’s contact with, and hopefully understanding
of the works that are produced. But it doesn’t come for free. A variety of inter-
disciplinary opportunities and issue are raised, and the authors bravely examined
them in detail. Spanning from the classics such as Leonardo da Vinci to contem-
poraries like David Hockney the opportunities for artistic and creative expression
have transformed the worlds of which they are an integral part.
The contributions and themes in this book are rich in information in various
forms. The interaction of technology and computer generation of information with
artefact creation and dissemination is discussed and the reader is left with a sense of
wonder, questions and hopefully a few answers. The topic is, and may always be, a
work in process. That is reflected in that this book is a continuation of, and
development from, the research and development detailed in the earlier book—
Earnshaw R. A. (2017): Art, Design and Technology: Collaboration and
Implementation.
Dr. Jon Peddie

Jon Peddie Research
Tiburon, CA, USA
Preface
This volume explores emerging aspects of the relationship between artists (and
other creatives) and their created works, and also how a variety of tools and
environments have facilitated and extended these processes. Such tools encompass
computer technology, computer environments, and interactive devices, for a range
of information sources and application domains. They also provide new kinds of
created works which are able to be viewed, explored and interacted with, either as
an installation or via a virtual environment such as the Internet. This introduces new
dimensions of understanding and experience for both artist and the public’s rela-
tionships with the works that are produced. This has raised a variety of interdis-
ciplinary opportunities and issues. From Leonardo da Vinci to David Hockney the
opportunities for artistic and creative expression have transformed the worlds of
information of which they are an integral part.
Tools have been used from the earliest times to create and modify art works and
sculptures. Naturally occurring pigments have been used for cave paintings, but
these were more than pictorial representations: they were inextricably linked to
narrative (of hunting) and this form of linkage is now returning to the forefront, due
to the facilities of digital media. What has been created provides insight into the
cultural context and social environment at the time of creation. There is an interplay
between the goal of the creator, the selection and use of appropriate tools, and the
materials and representations chosen. Gombrich argues that, There really is no such
thing as Art. There are only Artists [1]. Thus, although creative works may generate
a variety of interpretations in the minds of observers, the focus also remains on the
creator, and the vision and insight that causes them to create.
The symbiotic relationship between art works and the cultural context in which
they are produced is examined. Technology can provide continuity by making
traditional methods and techniques more efficient and effective. It can also provide
discontinuity by opening up new perspectives, paradigms and dimensions of
interaction with the viewer. This can produce a greater understanding and expan-
sion of artistic processes and how they are implemented in practice.
xiii
xiv Preface
This book includes the arts in general, so is not limited to the visual arts. The arts
may be defined as the theory and physical expression of creativity found in human
cultures and societies. A key component of this is communication and narrative.
This book brings together a variety of national and international authors who
present current research and development at the interface between technology and
the arts and humanities. The chapters are grouped into themed parts as follows:
Part I: A Panoramic View of the Field
Part II: Facilitating Communication Between the Arts, Technology, and Audiences
Part III: Interactions Between the Arts and Data
Part IV: Audio Visual Installations to Generate Collective Human Responses
Part V: The Convergence of Digital Design, the Arts, Computing, and the
Environment
Part VI: The Use of Virtual Reality and Augmented Reality to Extend Creativity,
Reach, and Engagement in the Arts
Part VII: The Future of Interdisciplinary Research
These parts present an increasing reach across the interface using the following
modalities—communication, interaction, installation, convergence, and the uti-
lization of VR and AR technologies. The book concludes with an assessment of the
extent to which the current opportunities and challenges are being addressed and
realized.
The convergence of IT, telecommunications, and media is bringing about an
explosion of data and also a revolution in the way information is collected, stored
and accessed, plus the major scope of enhancement into time-based media. There
are three principal reasons why this is happening—reducing cost, increasing quality
and increasing bandwidth. This is likely to result in a closer relationship between
technology and the arts and humanities.
Where the references are to online papers and documents, the authors have
endeavoured to provide those that are open source and in the public domain rather
than behind a paywall. The current move to a requirement for open source publi-
cation in Europe will assist this situation in the future (on the basis that the taxpayer
has already contributed to the funding of research and development and therefore
should be entitled to read the publications without further charge). Where publi-
cations are currently behind a paywall, readers can normally read the abstract and
see the list of references before deciding whether to purchase the paper.
There are many references to online sources on the Internet. Readers of the
e-book can access these directly as they are embedded in the text as hot links. Some
URLs of web pages change over time due to site names being changed by their
owners, or the position of the web site in the site hierarchy being altered. Where the
link does not access the required page, the correct page can often be located by
putting the URL into Google. If this doesn’t work due to Google’s cached copy
of the original website having been over-written, then the title of the reference can
be typed into Google.
Preface xv
This volume is published as a Springer Open Access book in order to make the
e-book freely available to everyone for study and further research, particularly
students who may be least able to afford the normal cost of a book. Although the
e-book contains colour figures, these are rendered in monochrome in the printed
version, as this is current Springer policy. Therefore, because many of the colour
figures have been produced by artists, these are best viewed in the e-book. There is
a nominal charge for the printed book to cover the cost of printing.
It is hoped that this book makes a useful contribution to an important area of
significant ongoing research, development and application.
Leeds, UK The Editors

Wrexham, UK
Bradford, UK
Lausanne, Switzerland
December 2019
Acknowledgements Thanks and appreciation are due to all those who read draft versions of the
chapters and provided comments to improve technical content and readability. However,
responsibility for the final text rests with the authors.
Thanks are expressed to Prof. Jill Journeaux and Dr. Jon Peddie for providing the Forewords to
the book.
Thanks and appreciation are also due to Springer for assistance and support with the editing
and production of this book.
Reference
1. Gombrich, E.H.: The Story of Art, p. 15. Phaidon Press Ltd., London (1995)
Contents
Part I A Panoramic View of the Field

1 Introduction and Background to Technology and the Arts . . . . . . . 3
Rae Earnshaw
2 Positioning the Arts in the Research Process: Perspectives
from Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Susan Liggett
Part II Facilitating Communication Between the Arts, Technology,

and Audiences
3 Framing the Conversation: The Role of the Exhibition
in Overcoming Interdisciplinary Communication Challenges . . . . . 25
Susan Liggett and Mike Corcoran
4 Modern Communication Technologies and the Marxist
Understanding of Scientific Cognition . . . . . . . . . . . . . . . . . . . . . . . 45
Naira Danielyan
5 Use of Digital Holography to Re-Encode and Image Chinese
Movable Type Printing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Shuo Wang, Ardeshir Osanlou, and Peter Excell
Part III Interactions Between the Arts and Data

6 The FOREVER-DO Game: A Big Data Fishing Expedition . . . . . . 83
Jill Townsley and Carlo Ferigato
7 Searching for New Aesthetics: Unfolding the Artistic Potential
of Images Made by the Scanning Electron Microscopy . . . . . . . . . . 103
Anastasia Tyurina
xvii
xviii Contents
8 Interspecific Interactions: Interaction Modes Between Sound

and Movement in Collaborative Performance . . . . . . . . . . . . . . . . . 121
Manoli Moriaty
9 Between Presence and Program: The Photographic Error
as Counter Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Tracy Piper-Wright
10 Signs of Surveillance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Daniel Buzzo
Part IV Audio Visual Installations to Generate Collective Human

Responses
11 Coral Voices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Marlena Novak and Jay Alan Yim
12 Cyberdreams: Visualizing Music in Extended Reality . . . . . . . . . . 209
Jonathan Weinel
13 Augmenting Virtual Spaces: Affective Feedback
in Computer Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
Stuart Cunningham, John Henry, and Jonathan Weinel
Part V The Convergence of Digital Design, the Arts, Computing,

and the Environment
14 Chandini (A Bride for the Moon) . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Rachel Davies and Daniel Saul
15 Digital Moving Image Installations and Renewable Energy:
1994–2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Chris Meigh-Andrews
16 From Digital Nature Hybrids to Digital Naturalists: Reviving
Nature Connections Through Arts, Technology and Outdoor
Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
L. Edwards, A. Darby, and C. Dean
Part VI The Use of Virtual Reality and Augmented Reality

to Extend Creativity, Reach, and Engagement
in the Arts
17 Virtual Reality Holography—A New Art Form . . . . . . . . . . . . . . . 317
I. Pioaru
18 In Darwin’s Garden: An Evolutionary Exploration of Augmented
Reality in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
Alan Summers
Contents xix
19 Creation of Interactive Virtual Reality Scenarios as a Training

and Education Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Rinat R. Nasyrov and Peter S. Excell
Part VII The Future of Interdisciplinary Research

20 Interdisciplinary Research and Development—Opportunities
and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Rae Earnshaw
Editors and Contributors
About the Editors
Prof. Rae Earnshaw is Professor of Electronic

Imaging at the University of Bradford, UK since
1995 (now Emeritus), Honorary Visiting Professor in
Creative Industries at Glyndwr University and Visiting
Fellow at St John’s College, University of Durham. He
gained his Ph.D. at the University of Leeds and is a
chartered engineer and chartered information technol-
ogy professional. He was Dean of the School of
Informatics (1999–2007) and Pro-Vice-Chancellor
(Strategic Systems Development) (2004–2009). He
has been a Visiting Professor at Illinois Institute of
Technology, George Washington University, USA and
Northwestern Polytechnical University, China. He is a
member of ACM, IEEE, CGS and a Fellow of the
British Computer Society and the Institute of Physics.
He has authored and edited 42 books on computer
graphics, visualization, multimedia, design, and virtual
reality, and published over 200 papers in these areas.
Book publishers include: Academic Press, Cambridge
University Press, Addison Wesley, Springer (UK),
Springer (USA), John Wiley & Sons Inc and IEEE
Computer Society Press. Although playing a significant
role in academic leadership and management over the
past 20 years, he has maintained his research and
publication record. He is a member of the Centre for
xxi
xxii Editors and Contributors
Visual Computing at the University of Bradford which

performs world-class research and development in the
area of visual image data processing.
https://sites.google.com/site/raearnshaw/home.
https://www.bradford.ac.uk/ei/media-design-
technology/research/centre-for-visual-computing/.
Dr. Susan Liggett is Reader in Fine Art and Associate

Dean for Research in the Faculty of Art, Science and
Technology at Wrexham Glyndŵr University. She has
a Ph.D. from the University of Wales, a Post Graduate
Diploma in Painting (MA) from the Royal Academy
Schools, London and a BA (Hons) Fine Art from
Nottingham Trent University. Her work as a practicing
artist includes paintings, fine art films and arts in health
collaborative research projects resulting in her artwork
being exhibited in different and varied contexts includ-
ing galleries, public spaces conferences and festivals.
As a curator, she has organized exhibitions, chaired
conferences and published work on the interface of
art/science. She is a member of the Royal Cambrian
Academy and SUITE Studio Group, Salford.
https://www.susanliggett.com/.
Prof. Peter Excell is Emeritus Professor of

Communications and former Deputy Vice-Chancellor
at Glyndwr University. His interests cover computing,
electronics and creative industries, with a strong spirit of
interdisciplinarity that is needed for the digital knowl-
edge economy. He gained his B.Sc. in Engineering
Science at the University of Reading and Ph.D. in
Electronic Engineering at the University of Bradford.
His principal research has been in the area of future
mobile communications technologies and services and
this has been carried out in conjunction with colleagues
from wider discipline areas, analysing human commu-
nications in a holistic way and developing new ways of
using mobile multimedia devices. He has published over
500 papers. He is a Fellow of the British Computer
Society, the Institution of Engineering & Technology
and of the Higher Education Academy, a Chartered IT
Professional and Chartered Engineer.
Editors and Contributors xxiii
Prof. Daniel Thalmann is a Swiss and Canadian

Computer Scientist. He is currently Honorary Professor
at EPFL and Director of Research development at
MIRALab Sarl. Pioneer in research on Virtual Humans,
his current research interests include social robots, crowd
simulation and Virtual Reality. Daniel Thalmann has
been the Founder of The Virtual Reality Lab (VRlab) at
EPFL, Switzerland, Professor at The University of
Montreal and Visiting Professor/Researcher at CERN,
University of Nebraska, University of Tokyo, and
National University of Singapore. From 2009 to 2017,
he was Visiting Professor at the Nanyang Technological
University, Singapore. Until October 2010, he was the
President of the Swiss Association of Research in
Information Technology and one Director of the
European Research Consortium in Informatics and
Mathematics (ERCIM). He is coeditor-in-chief of the
Journal of Computer Animation and Virtual Worlds, and
member of the editorial board of 12 other journals.
Daniel Thalmann was a member of numerous Program
Committees, Program Chair and CoChair of several
conferences including IEEE VR, ACM VRST and
ACM VRCAI. Daniel Thalmann has published more
than 600 papers in Graphics, Animation and Virtual
Reality. He is coeditor of 30 books, and coauthor of
several books including Crowd Simulation (second
edition 2012) and Stepping Into Virtual Reality (2007),
published by Springer. He received his Ph.D. in
Computer Science in 1977 from the University of
Geneva and an Honorary Doctorate (Honoris Causa)
from University Paul-Sabatier in Toulouse, France, in
2003. He also received the Eurographics Distinguished
Career Award in 2010, the 2012 Canadian Human
Computer Communications Society Achievement
Award and the CGI 2015 Career Achievement.
Wikipedia: http://en.wikipedia.org/wiki/Daniel_
Thalmann.
CV: https://www.dropbox.com/s/wfh7fxi1bagv65r/
CV%20%28engl.%29.pdf?dl=0.
xxiv Editors and Contributors
Contributors
Daniel Buzzo University of the West of England, Creative Technology Lab,

Bristol, UK
Chris Meigh-Andrews University of Central Lancashire, Preston, UK
Mike Corcoran Wrexham, Wales
Stuart Cunningham Centre for Advanced Computational Science (CfACS),
Manchester Metropolitan University, Manchester, UK
Naira Danielyan Philosophy, Sociology and Political Science Department,
National Research University of Electronic Technology, Moscow, Russian
Federation
A. Darby Lancaster Institute for Contemporary Arts, Lancaster University,
Lancaster, UK
Rachel Davies Kingston University, Kingston, UK
C. Dean Edge Hill University, Ormskirk, UK
Rae Earnshaw Department of Computer Science, Faculty of Engineering and
Informatics, University of Bradford, Bradford, UK; St John’s College, Durham
University, Durham, UK; Faculty of Art, Science and Technology, Wrexham
Glyndŵr University, Wrexham, UK
L. Edwards School of Computing and Communication, Lancaster University,
Lancaster, UK
Peter Excell Centre for Ultra-Realistic Imaging, Wrexham Glyndwr University,
Wrexham, UK
Peter S. Excell Wrexham Glyndŵr University, Wrexham, UK;
University of Bradford, Bradford, UK
Carlo Ferigato European Commission, Joint Research Centre (JRC), Ispra, Italy
John Henry Centre for Advanced Computational Science (CfACS), Manchester
Metropolitan University, Manchester, UK
Susan Liggett Faculty of Art, Science and Technology, Wrexham Glyndŵr
University, Wrexham (Wales), UK
Manoli Moriaty University of Salford, Salford, UK
Rinat R. Nasyrov Department of Power Electrical Systems, Institute of EPE,
National Research University for Power Engineering “MPEI”, Moscow, Russia
Marlena Novak Department of Film Video, New Media and Animation, The
School of the Art Institute of Chicago (SAIC), Chicago, USA
Editors and Contributors xxv
Ardeshir Osanlou Centre for Ultra-Realistic Imaging, Wrexham Glyndwr

University, Wrexham, UK
I. Pioaru Wrexham Glyndwr University, Wrexham, UK
Tracy Piper-Wright University of Chester, Chester, UK
Daniel Saul Royal College of Art, London, UK
Alan Summers University of Chester, Chester, UK
Jill Townsley University of Huddersfield, Huddersfield, UK
Anastasia Tyurina Queensland University of Technology (Brisbane), Brisbane,
QLD, Australia;
National Research University MIET (Moscow), Moscow, Russia
Shuo Wang Beijing Institute of Graphic Communication, Beijing, China
Jonathan Weinel London South Bank University, London, UK
Jay Alan Yim Composition and Music Technology, Bienen School of Music,
Northwestern University, Evanston, USA
Part I
A Panoramic View of the Field
Chapter 1
Introduction and Background
to Technology and the Arts
Rae Earnshaw
Abstract Interactions between disciplines are reviewed. The long history of the
relationship between the arts and sciences is summarized, and the challenges at the
interface are outlined. The historical developments between technology and the arts
are summarized, including computer arts, computer animation, digital media, and
digital humanities. The current enablers for progressing interdisciplinary collabora-
tions are presented. The possibility for a new Renaissance between technology and
the arts is discussed.
Keywords Forms of knowledge · Scientific revolution · Cultural context ·

Computer arts · Story-telling · Reciprocal relationships · Renaissance teams
1.1 Interactions Between Disciplines
Interactions between disciplines have always had the potential to be exciting and
ground-breaking. It can take the participants on all sides into new areas, often uncov-
ering new understandings and new forms of knowledge. However, cutting across the
boundaries of disciplines can take researchers out of their traditional comfort zones,
and can be uncomfortable and challenging for all parties. It has been assumed in
the past that it is a priority for a particular discipline to protect its legacy, its areas
of interest, and the particular body of knowledge that it has established and would
claim as its heritage. This is the foundation upon which new knowledge is expected
to be built. This been mirrored to some extent by the structures set up in the academy
to study and disseminate knowledge and research in the disciplines. This foundation
built over many years can result in inertia and a wish to maintain the status quo.
R. Earnshaw (B)
Department of Computer Science, Faculty of Engineering and Informatics,
e-mail: r.a.earnshaw@bradford.ac.uk
St John’s College, Durham University, Durham, UK
Faculty of Art, Science and Technology,
Wrexham Glyndŵr University, Wrexham, UK
© The Author(s) 2020 3

R. Earnshaw et al. (eds.), Technology, Design and the Arts—Opportunities and Challenges,
Springer Series on Cultural Computing, https://doi.org/10.1007/978-3-030-42097-0_1
4 R. Earnshaw
However, new disciplines can arise at the boundaries between existing disciplines.
Examples are oceanography, cognitive science, genetic engineering, tribology, and
digital media. Developments and advances such as these have focused attention on
the interface areas between disciplines, and they are increasingly being recognized
as important for support and investment.
1.2 Relationships Between the Arts and the Sciences
There has been a long history of antipathy between the arts and the sciences due
to different modes of discourse, different forms of language, and different ways
of working. This appears to have been caused principally by the early forms of
education from the foundation of the academy until the scientific revolution from the
sixteenth to the eighteenth centuries. Prior to this revolution, many understandings
of the natural world were often misguided and erroneous because they were not
based on a systematic and rigorous methodology. It is understandable therefore that
attention in the academy should concentrate on what was regarded as the development
and disciplining of the mind by means of subjects such as grammar, rhetoric, and
logic. Later on, physics, metaphysics, and moral philosophy based on an Aristotelian
framework were also included. Mind took the pre-eminence over matter. The natural
world was regarded as an environment for the use of tools and the work of tool-smiths,
and therefore not suitable for academic study. It was not until the key discoveries
of the scientific revolution that a systematic enlargement of the curriculum in the
academy took place, to produce the arts and the sciences as we have them today.
1.3 Historical Developments in the Relationship Between

Technology and the Arts
1.3.1 Computer Arts
The Computer Arts Society [1] in the UK was founded in 1968. Its objective is to
promote the creative uses of computers in the arts and culture. It acts as a forum to
bring together those with interests in the cultural impact of information technology,
and the various ways this impact can occur. This can involve those who create cultural
artefacts by information technology or manage collections or those who are seeking
to interpret and understand the cultural implications of the artefacts. An archive of
the collections of the Computer Arts Society is hosted by the Victoria and Albert
Museum in London, UK, and is part of their Computer Art Collections. Mason [2]
and Brown et al. [3] detail the early history of computers and the arts up to 1980.
Franco [4] detailed the work of Edmonds on generative systems art from the
1960s to the present day. This explored the relationship between art and computer
1 Introduction and Background to Technology and the Arts 5
technologists in terms of concepts, tools, and forms of art. Candy et al. [5] reviewed
the history of art and technology collaborations highlighting the contributions of
practitioners and researchers at the interfaces between technology and the arts.
In the UK in the 1980s, Lansdown and Earnshaw brought together the output from
the joint work of the Computer Arts Society and the Displays Group of the British
Computer Society. This work consisted of contributions to visualization, computer
art, design, and animation [6]. It was recognized that the boundaries between these
various disciplines were blurring due to the increasing power and capability of the
computer and the facilities of software packages, with interfaces that were more
accessible and user-friendly for arts users.
Computer Art and Technocultures was a 3-year project supported by the Arts and
Humanities Research Council (AHRC) in the UK to study the history of computer-
generated art. The project was based jointly at Birkbeck College and the Victoria
and Albert (V&A) and was completed in August 2010. A display at the V&A on
Digital Pioneers was exhibited from December 2009 until April 2010. An associated
symposium, Ideas before their Time [7], was held at the British Computer Society,
and a two-day conference, Decoding the Digital [8], at the V&A on 4–5 February
2010. Dodds [9] and Beddard and Dodds [10] provide an account of the V&A’s
collections and their relationship to art history, and their social and technological
context and implications. It also examines the outputs of the AHRC project. These
are also summarized as part of the project [11].
1.3.2 Computer Animation
Making images move has been a preoccupation of computer technologists from

the earliest days of computers [12]. Prior to this, successive frames of an ani-
mated sequence had to be composed on print media and then filmed, which was a
time-consuming process. With the advent of computer displays and software, image
sequences could be created and displayed directly. This facilitated ease of editing
and re-display, which was more time consuming with print-based media.
Today there is a wide variety of desktop software packages that enable home
users to create animations with audio tracks. Vince [13] discusses animation tech-
niques, animation hardware, and animation software such as Softimage, Maya, 3d
Studio Max, and Lightwave. Post-production techniques are presented, and anima-
tion applications are reviewed. Such Computer-Generated Imagery (CGI) has found
widespread use in the applications such as special effects in films, computer games,
advertising, computer-aided design, and simulation. However, computer technology
of itself does not provide an automatic or easy solution. The imagination and creativ-
ity of the person formulating the animation need to be the fundamental driving force.
The hardware and software technology are just tools and need to be used appro-
priately [14]. Animated films such as Pixar’s Toy Story [15], which won awards,
illustrate this point. It is the story which draws in, and captivates, the viewer. There-
fore, in order to be effective artistically, animation sequences are designed and chosen
6 R. Earnshaw
to best represent the story. Story-telling is itself an art and has a long history [16],
whether in oral form or using various forms of media that were available at the time
in earlier centuries.
1.4 Digital Media
Collaboration in the production of digital media may involve a variety of discipline

specialists. This is because of the diversity of its constituent parts such as hard-
ware, software, digital images, digital, audio, sensors, games, interaction devices,
and social media. This in turn may involve different constituencies such as indus-
try, the academy, research and development organizations, and Small and Medium
Enterprises (SMEs) with all their different cultures and working practices [17, 18].
The challenges are therefore significant. However, for those with a good collabo-
ration methodology geared to generating successful outcomes, the rewards can be
substantial.
1.5 Digital Humanities
On the wider front, the humanities disciplines such as history, literature, and phi-
losophy are increasingly using computational tools and facilities to advance their
research. Digitized texts can be searched, word frequencies can be calculated, and
indexes and concordances can be produced [19]. In addition, image processing and
3D recording can be useful for pictorial data and 3D spaces such as museums and
heritage sites [20]. Two broad areas of digital humanities research may be identified:
firstly, the use of digital tools to perform and extend research in the humanities and,
secondly, the use of humanities applications to perform research in computer sci-
ence in the development of new interfaces and more advanced tools. Thus, there is
a mutual reciprocity between technology and the humanities, even if this is not ini-
tially recognized [21]. The cultural context of computing also needs to be taken into
account when evaluating its contributions. Such interdependencies are of increasing
importance when seeking to understand current interdisciplinary activity and define
potential ways forward for the future.
1.6 A New Renaissance?
The Renaissance in the fourteenth century onward marked a transition from the
medieval era to modernity, and it opened up the possibilities of new horizons and new
ways of thinking and working. Similarly, Renaissance Teams [22] in the twenty-first
1 Introduction and Background to Technology and the Arts 7
century offer the opportunity to utilize the expertise now available and work together
toward a common objective which can transcend the traditional boundaries of the
past.
This point is illustrated and exemplified by the contributions to this book.
This book brings together a variety of national and international authors who
present current research and development at the interface between technology and
the arts and humanities. The chapters are grouped into themed sections as follows:
Section 1: A Panoramic View of the Field.
Section 2: Facilitating Communication between the Arts, Technology, and
Audiences.
Section 3: Interactions Between the Arts and Data.
Section 4: Audio Visual Installations to generate Collective Human Responses.
Section 5: The Convergence of Digital Design, the Arts, Computing, and the
Environment.
Section 6: The Use of VR and AR to extend Creativity, Reach, and Engagement
in the Arts.
These sections present an increasing reach across the interface using the following
modalities—communication, interaction, installation, convergence, and the utiliza-
tion of VR and AR technologies. The book concludes with an assessment of the
extent to which the current opportunities and challenges are being addressed and
realized.
The convergence of IT, telecommunications, and media is bringing about an explo-
sion of data and also a revolution in the way information is collected, stored, and
accessed. There are three principal reasons why this is happening—reducing cost,
increasing quality, and increasing bandwidth. This is likely to result in a closer
relationship between technology and the arts and humanities.
References
1. Computer Arts Society. https://computer-arts-society.com/

2. Mason, C.: A Computer in the Art Room: The Origins of British Computer Arts 1950–1980.
Quiller Press, Shrewsbury, UK (2008). https://www.amazon.co.uk/s?k=Mason%2C+C.+A+
Computer+in+the+Art+Room%3A+The+Origins+of+British+Computer+Arts+1950-1980&
i=stripbooks&ref=nb_sb_noss
3. Brown, P., Gere, C., Lambert, N., Mason, C. (eds.): White Heat Cold Logic: British Com-
puter Art 1960–1980. MIT Press, Cambridge, MA (2009). https://mitpress.mit.edu/books/
white-heat-cold-logic
4. Franco, F.: Generative Systems Art: The Work of Ernest Edmonds. Routledge, Abing-
don, UK (2018). https://www.routledge.com/Generative-Systems-Art-The-Work-of-Ernest-
Edmonds-1st-Edition/Franco/p/book/9781472436009
5. Candy, L., Edmonds, E., Poltronieri, F.A.: Explorations in Art and Technology. Springer,
London (2018)
6. Lansdown, J., Earnshaw, R.A. (eds.): Computers in Art, Design, and Animation. Springer, New
York (1989, 2012). https://link.springer.com/book/10.1007/978-1-4612-4538-4
8 R. Earnshaw
7. Ideas before their Time. Symposium summary. http://www.technocultures.org.uk/

Symposiumoutline.html
8. Decoding the Digital. Symposium outline. http://www.technocultures.org.uk/
decodingthedigital.pdf
9. Dodds, D.: Computer Art and Technocultures: Evaluating the V&A’s Collections in the Digital
Age. In: Proceedings of the Conference on Electronic Visualisation and the Arts (EVA) (2008).
https://www.bcs.org/upload/pdf/ewic_eva08_paper10.pdf
10. Beddard, H., Dodds, D.: Digital Pioneers. Roli Books, New Delhi, India (2012)
11. Outcomes of the Computer & Technocultures project. http://www.technocultures.org.uk/
outcomes.html
12. Badler, N.I., Barsky, B.A., Zeltzer, D. (eds.): Making them Move: Mechanics, Control and
Animation of Articulated Figures. Morgan Kaufmann, San Mateo, CA (1991)
13. Vince, J.A.: Essential Computer Animation fast: How To Understand The Techniques And
Potential Of Computer Animation. Springer, London (2013)
14. Catmull, E.: Creativity. Bantam Press, Inc., London (2014)
15. https://en.wikipedia.org/wiki/Toy_Story
16. https://en.wikipedia.org/wiki/Storytelling, https://www.thehistorypress.co.uk/local-history/
storytelling/
17. Earnshaw, R.A.: Research and Development in Art, Design and Creativity, Springer, Lodnon,
Sections 3.1 (pp 32–33), 5.2.5 (pp 74–75) (2016). ISBN 978-3-319-33005-1. http://dx.doi.org/
10.1007/978-3-319-33005-1, http://www.springer.com/gb/book/9783319330044
18. Earnshaw, R.A.: State of the Art in Digital Media and Applications, Chapter 3 Collaboration on
Digital Media. Springer, London (2017). https://doi.org/10.1007/978-3-319-61409-0. https://
www.springer.com/gb/book/9783319614083
19. https://en.wikipedia.org/wiki/Digital_humanities
20. Warwick, C., Terras, M., Nyhan, J. (eds.): Digital Humanities in Practice. Facet Press, London
(2012)
21. Berry, D.M., Fagerjord, A.: Digital Humanities: Knowledge and Critique in a Digital Age.
Polity Press, Cambridge, UK (2017)
22. Cox, D.J: Collaborations in art/science: Renaissance teams. J. Biocommun. 18(2):15–24
(1991). https://www.ncbi.nlm.nih.gov/pubmed/1874707
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter’s Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.
Chapter 2
Positioning the Arts in the Research
Process: Perspectives from Higher
Education
Susan Liggett
Abstract Research in the visual arts has contributed to the creation of environments
that involve cross-disciplinary, multidisciplinary and interdisciplinary or transdisci-
plinary projects in departments within and across universities. An overview is pro-
vided of the historical context of doctoral awards in the arts with a definition of the
terms practice-based, practice-led, and practice as research discussed. It articulates
the challenges when acquiring explicit and exact knowledge alongside more subjec-
tive approaches that utilize tacit knowledge from artistic practice in research projects.
Drawing on examples from art practice and doctoral students work, it analyzes objec-
tive, subjective, empirical, and hermeneutic paradigms, as described by Pierre Bour-
dieu, which can combine empirical approaches and individual understandings to
re-enforce our understandings of the world.
Keywords Practice-based research · Practice-led research · Practice as research

(PaR) creative arts Ph.D. · Collaboration · New technologies · Cross-disciplinary ·
Interdisciplinary
2.1 Introduction
Most researchers are aware of the need to position themselves in relation to other
fields or disciplines in order to reveal the particular characteristics of their research
findings. At the start of the research process, usually in writing the proposal, it can be
challenging for artists when “oneself” often plays a significant role in the object of an
enquiry. Research in the arts can be largely hermeneutic; that is the understanding and
interpreting of it exposes the subjective limits of the artists’ ways of doing and seeing.
It is this epistemological method of socially constructing meaning that gives the arts
authority within a process of enquiry. However, it is all too easy for knowledge
gained tacitly to be “overlooked because it is subsumed into the rational logic of
discursive accounts of artistic production” [1]. The articulation of possibilities of
artists for shaping consciousness and providing cultural capital can be difficult even
S. Liggett (B)
Faculty of Art, Science and Technology, Wrexham Glyndŵr University, Wrexham, UK
e-mail: s.liggett@glyndwr.ac.uk
10 S. Liggett
for the more experienced academic. How can, as Barrett [1] says, “the interplay
of disparate areas of knowledge create new analogies, metaphors and models for
understanding objects of enquiry”?
Cross-disciplinary research involves viewing one discipline from the perspective
of another. Interdisciplinary research involves integrating knowledge and methods
from different disciplines that use a synthesis of approaches and multidisciplinary
research that involves people from different disciplines working together, each draw-
ing on their disciplinary knowledge. Transdisciplinary is close to multidisciplinary
but it moves beyond a discipline taking more of a holistic approach creating a union
of intellectual knowledge that goes beyond the disciplinary perspectives.
From experience of supervising doctoral students in the creative arts, I have
noticed that an artist’s identity can be lost when research methods are combined
from different disciplines, such as those from the social sciences where is it easier to
articulate the possible new knowledge that may emerge from interviews or surveys,
for example, than those resulting from the art practice. This is particularly evident
when the student begins to scrutinize artwork in unfamiliar ways to seek valida-
tion from research communities in different disciplines. For example, at Wrexham
Glyndŵr University all research students are required to present their work to the
whole research community in an event called “Open House for Researchers”. Art
students need support when combining different methodological paradigms in their
research to bring fresh perspectives and new possibilities to problems. Artists in
academia need to convince practitioners from other fields of enquiry within and
beyond the university that they can reflect new objective realities that may be of
benefit to society.
2.2 Artistic Research and the Academy
It is useful to decode some of the terminology surrounding the arts in academia to

facilitate analysis of what artists do in a meaningful way. The word “practice” is used
in the arts as it is in other professions to recognize knowledge production through
action and doing. We can only fully understand what happens when we make artwork
through the process of doing it, as opposed to purely thinking about it and the way
artists create work goes beyond the physical activity to include influences, ideas, and
critical reflections. The term practice refers to all artists work, but not all art practice
can be termed research in an academic context. Despite the fact that most practice
involves some form of research as part of the process, for example taking photographs
in preparation for a painting, is not considered research in an academic context if
the resulting work is not shared and its contributions to knowledge challenged. The
theory/practice nexus helps define research in the arts by the linking of theory and
practice through a method of intentionally investigating the process and outcomes
informed by practice. The term “praxis” is often used to describe the making of
artwork in a research context. Practice without the theory remains “practice” which
can get very confusing for those involved in the arts, and also for those from the
2 Positioning the Arts in the Research Process: Perspectives … 11
outside looking in. However, the adoption of the term practice is clear and implies
that the production of artworks is part of society and connected to lived experience
rather than existing in an ethereal world empty of everyday concerns and therefore
unimportant and insignificant. Integrating the arts in society forces us to articulate
how the purpose and meaning of art is constructed, challenged and its investments
accounted for.
Bringing the arts into academia gives it status and legitimizes it as a profession. In
the UK the first training in the arts was available in 1768 when The Royal Academy
of Arts opened; art schools were further developed in the nineteenth century and
re-designed in post-war Britain under the influence of the Bauhaus [2]. In 1974
in the UK the National Council for Diplomas in Art and Design was merged into
the Council for National Academic Awards (CNA) allowing Polytechnics with art
departments to issue honors degrees for the first time. Further academization of the
arts happened in 1992 with former polytechnics becoming Universities [3]. This put
the arts on an equal footing with other subject areas historically considered more
academic. With the home of the Art School now residing within Universities, artists
have the opportunity to study for research degrees and there has been an exponential
growth in uptake of doctorates in the arts over the last decade. Between 1986 and
1995, 181 students received research degrees in the arts subjects in the UK [4] and
in 2016/17 alone, 880 students were awarded research degrees in art and design
(HESA) [5].
The Ph.D. is now a standard requirement for teaching in an art department in a
UK University and with this, it brings pressure on academics to compete for research
funding in ways comparable to other academic subjects. With the arts being newcom-
ers to universities system, ontological truth claims between the Realist1 worldview
and the Constructivist2 world view are challenged with artists claiming that often
regulatory frameworks within Universities consist of a strong Realist components
with the use of the terms “question” and “answer” rather than the terms “issue” and
“response”.
If the arts do indeed operate within a new paradigm they should be able to say: our ontological
position is this, our epistemological position is this, our methodological position is this and
all of these are coherent and this is why we warrant special different conditions to the ones
that have formally been recognized [6].
Art departments have had to fight hard to ensure that systems were set up to support
their students in practice-based research. Between 2000 and 2006 debate surrounding
the doctorate in the arts was highlighted at the University of Hertfordshire’s annual
conferences on the foundations of practice-based research which published Working
Papers on Art and Design [7]. Particular concern regarding academic standards being
imposed on art practices and artistic research were further addressed internationally
[8] with European League for the Institute of the Arts (ELIA), with the creation of an
1 Realist view here refers to the idea that reality exists independently of observers and their thoughts,
feelings, intuitions, and opinions.

2 Constructivism her is recognition that reality human construct that interacts with our experience
in the real world.
12 S. Liggett
advocacy network to address issues facing the arts in higher education. Its aim is to
collaborate to create networks that strengthen, support, promote, and encourage arts
institutions across Europe. A specific working group to enhance doctoral research in
the arts known as the SHARE3 network holds annual events to raise the platform for
research in the arts.
2.3 Knowledge Production

and Practice-Based/Practice-Led Research
Practice-based, practice-led, practice as research (PaR), artistic research, and arts-

based research are all widespread terms in creative arts research which are not inter-
changeable and exhibit qualities that need to be understood regarding the nature of
the art practice and the resulting production of knowledge. Different perspectives
on these can be explored through the writings of Candy and Desmond [9, 10], Bar-
ratt and Bolt [11], Grey and Malins [12], Macleod and Holdridge [13], Briggs and
Karlsson [14], Nelson [15].
In summary, “Practice-based Research is an original investigation undertaken
in order to gain new knowledge partly by means of practice and the outcomes of
that practice”. By contrast “Practice-led Research is concerned with the nature
of practice and leads to new knowledge that has operational significance for that
practice” [16]. Practice as research (PaR) distinguishes artist-scholars in a university
context from professional practice. Nelson [15] describes PaR as a research project in
which practice is a key method of enquiry where the practice is submitted as evidence
of a research enquiry. An important difference between the personal or professional
artist and artwork in the doctoral research is the form the knowledge produced takes.
For example, “understandings about audience experience, taxonomies, models of
collaboration and the artwork themselves” [17]. This is not to be confused with
arts-based research which is another term used principally in the fields of education
where it is used to understand education through arts-based concepts, techniques,
and practice [18]. A practice-based Ph.D. includes creative works that arise from the
research process as part of the submission.
Despite the literature available on different perspectives on practice-based
research, there is still not an “an integrated discourse on the place of practice-based
research in Ph.D. programmes” [17]. The definition of research within the univer-
sity is clear and Wrexham Glyndwr University had articulated it as a “process of
investigation leading to new insights collectively shared” [19]. Two expectations of
new knowledge resulting from by research are firstly, that it has to be open to being
3 SHARE stands for Step-change for Higher Arts Research and Education and was an international
networking project, comprising 39 partners working together on enhancing the “3rd cycle” of arts
research and education. It created a Europe-wide exchange framework for the widely different
experiences, practices and ideas that make up the lively domain of artistic and cultural research.
[https://www.elia-artschools.org/activities/artistic-research/share].
challenged and, secondly, that it has to be verified. Creative arts research outcomes
and methodologies are sometimes difficult to understand and quantify in terms of
traditional scholarship, which can lead to a devaluing of studio-based enquiry [20].
It is only when artist researchers, with their practice, firmly root themselves in
their discipline, and position themselves in relation to other disciplines, that they
can then truly reveal the characteristics particular to their research. Students enrolled
on Ph.D. programmes are frequently expected to present their research to the wider
community of students from other disciplines. This can have a “de-centering” effect
on the newly enrolled student. On writing the research proposal it can, initially, be
confusing if “oneself” plays a significant role in the object of an enquiry. This has
also presented challenges to supervisors at Wrexham Glyndwr University guiding
students in cross-disciplinary teams.
2.4 Cross-Disciplinary, Interdisciplinary,

and Multidisciplinary Collaborative Research
in Doctoral Studies
Bourdieu notes that on-going privileges of positivistic and instrumentalist approaches

to research persist [20, p. 4]. The researchers in the case studies described have
experienced a de-stabilizing effect on their confidence when writing proposals. A
lack of experience sometimes leads to an over reliance on scientific or social science
paradigms to justify “fuzzy” concepts.
The following examples demonstrate a research project conducted by Liggett and
two that she has supervized to illustrate how these challenges have been overcome
by researchers having a clear understanding of themselves and their motivations for
the research. They all adopt a mixed-method approach and involve interdisciplinary
or cross-disciplinary collaborations through the research design of the supervisory
teams comprising academics from different subject areas. Supervisors recognized
the importance of openness and a willingness to seek the advice of other academics
to ensure the support was provided for cross-disciplinary dialogues in the following
projects.
Firstly, Heald and Liggett undertook a multidisciplinary research project in col-
laboration with a Consultant psychiatrist and Professor of social psychiatry both
from Bangor University titled In-between-ness: using art to capture a sense of self
during anti-depressant treatment [21]. The rationale for the research was to try and
understand how reality is constructed and whether this process can be manipulated
with medication and also, to determine the implications for the authenticity of “self”
for someone undergoing treatment for depression. The artists worked one-to-one
with four participants over 6 weeks. Service users were issued with a video camera
and asked to film on a weekly basis with visual prompts introduced as a stimu-
lus. The process allowed participants to explore how views of themselves changed
as they recovered from depression. Interviews and psychometric tests were carried
14 S. Liggett
out before, during and after their treatment, in collaboration with the trial psychia-
trist. The findings revealed that the interview transcripts recorded positive benefits
to participants in terms of increased responsiveness to sensations, their surrounding
environment, the quality of their feelings and their sense of “self” when engaged in
this creative process. The psychometric tests, by contrast, evidenced no change in
data.
The artists’ practice acted as a personal rationale in the approach to the research
and initial research questions stemmed from the issues arising from their work.
These questions were recognized by the medical professionals and validated by the
psychiatrists in the collaboration. This external rationale then drove the art practice
as the artists made new artworks in response to the situations and experiences they
encountered on their research journey. Figure 2.1 shows Liggett and Heald’s film
work made in response to feeling “in-between” working as artists at a psychiatric
unit. The artwork cannot be detached from the research project, and the artwork
produced by participants was seen as integral to the project and exhibited alongside
the artists’ work in the final exhibition concluding the project.
Secondly, Braisby’s Ph.D. project, which was practice-led and interdisciplinary
[22], identified how galvanic processes can enhance the work of the artist printmak-
ers and act to reverse the current decline in the teaching of intaglio etching. His
work involved making artworks to gain a deeper understanding of the chemical and
electrochemical processes involved. Scientific advice was sought from academics
in the chemistry department and workshops were designed for artists to learn about
Fig. 2.1 Paper Interior, film duration 9 20 , Heald and Liggett (2013). Photo credit Karen Heald.
Copyright © K. Heald, 2019
the processes. The learning generated from the workshops was analyzed using Soft
Systems Methodology (SSM) developed by Checkland [23]. It involved drawing a
rich picture; a process that uses images and words drawn out on a large sheet of paper
to describe the problem situation. Exposing the richness of the data helped in the
framing and re-framing of the situation and generated new ideas and insights into
the problem situation.
This rich picture process identified two aspects to the research question. The
first relates to the research into the galvanic techniques and the second to their
application as an artistic medium. The first is a well-defined problem that can be
addressed through a scientific problem-solving experimental approach under the
rubric of the “hard systems” [23]. The second part of the question needed to engage
with messy, real-world issues. This world was engaged through: workshops (electro-
etching), presentations (using the rich picture), collaborative work (other artists), and
interviews.
The research strategy developed was based on a hybrid model of both the practice-
based and practice-led methods [16]. The quantitative (practice-led) research col-
lected data by experimentation and developed a new process to produce the artwork.
The qualitative data (practice based) developed new understandings of the process.
Figure 2.2 shows experiments in enameling or plating etching plates with a differ-
ent metal in its impact on the practice of etching was evident in the creation of the
Fig. 2.2 Don Braisby, Experimenting with Enamelling (2019) 23 × 29 cm Copyright © D. Braisby
2019, reproduced by permission
16 S. Liggett
artwork produced. The research required the researcher to move out of his artistic
environment and engage with chemists, scientists, and physicists.
Finally, Wyatt’s Ph.D. cross-disciplinary project [24] investigated how people
living with dementia engage with, and experience, painting whilst working alongside
an artist-researcher. Arts-based inquiry4 was adopted as a method creating a body
of paintings to provide insights into different experiences and forms of engagement
when painting. Figure 2.3 is an example of a painting Wyatt made in response to
the memory of a beach in Wales. These were used to inform a qualitative study that
used social science methods of interviews observations and video recordings. The
use of both arts and social science research methods presented a challenge initially
but the research demonstrated how an artists’ perspectives can contribute to the field
of health by promoting new approaches to research.
In all three projects, the subjective strength of their work has grown out of the
positioning of the researchers in their studies by a self-scrutinizing process that is,
perhaps, unfamiliar in other disciplines. This process is necessary to prevent the artist
becoming a “pseudo-scientist”, having a “split personality” or being over narcissistic.
The in-between-ness project involved a collaborative multi-method pluralistic
approach to the research questions and monthly dialogues within the research team.
Initially, the challenge for the artists was to emphasize their individual practice in
the research design. Ostensibly their role could be seen to be purely one of facilitator
Fig. 2.3 Megan Wyatt, Whistling Sands (my Favourite Beach), 2017 Copyright © M. Wyatt 2019,
reproduced by permission
4 Arts-based inquiry uses artistic expression as data for inquiry.

for the participants. In reality, it was much more than this. The collaborative process
offered the artists a new method of challenging ideas that grew out of their practice.
It wasn’t until each researcher’s motivations and position within the research process
was established that a meaningful collaboration existed where all academics equally
owned the research.
In the “In-between-ness” project with mental health participants and Wyatt’s
project involving people living with dementia the relationship between the researcher
and the participants is an intrinsic aspect of the research. When viewing and making
artwork, it is not merely the deciphering of reality which occurs, because new under-
standings emerge, and connections and explanations are exposed to both viewer and
artist. Viewing and creating artwork can transform understanding to inform and gen-
erate new and valid knowledge [25]. Practice-based research is used in these projects
as it allows for new outcomes which do not oppose a verbal explanation [26].
In Braisby’s research involving galvanic etching, the known data from multiple
points of view is gathered together and made visible. This data enabled the researcher
to identify hidden links and make new connections and insights with their personal
practice to facilitate the refinement of the research. This process exposed the richness
of the data that emerged from the research and helped in re-framing the situation.
This firmly positioned the researcher in this research field, and this practice generated
new ideas and insights into the problem of why electro-etching has never become
mainstream in institutes of art.
2.5 Art, Collaboration, and Technology
The use of art as tool for social change is gaining traction with advocates such as
Alistair Hudson, the director of the Whitworth and Manchester Art Gallery since
2017, which promotes projects that have a real impact on people’s lives. People need
to value art so that they no longer say that they need a hospital rather than a gallery.
The mission is to get people to understand that the two are not mutually exclusive
and that people need both hospitals and art galleries [27].
According to the World Economic Forum, creativity is a key skill needed for
future employment [28]. The Fourth Industrial Revolution [29] brings advances in
technology that are challenging traditional jobs in the shift from the human to the
digital. Robots and artificial intelligence (AI) are more efficient in certain areas than
humans in the workplace. In the future, 47% of work may be lost to automation pro-
cesses [30]. Mobile Internet and cloud technologies are impacting society. However,
robots do not yet possess innate creativity (though software may soon be able to
emulate some of its characteristics) so business leaders, educators, and governments
still recognize the importance of human creativity.
Artists are increasingly collaborating with scientists and technologists in a multi-
tude of ways that extend the boundaries of knowledge. For example, The Foundation
for Art and Creative Technology (FACT) a leading UK visual arts organization has
developed a number of projects promoting creative media and creative technology.
18 S. Liggett
In 2019, it worked with the NHS, BBC, charities, and businesses in a project called
“The Future World of Work” to provide an arts-led critique of innovation and tech-
nology, reimagining work in relation to gender, the gig economy and careers advice
for uncertain times [30]. It worked with a number of artists who undertook workplace
artist residencies who make artwork in response to their experiences culminating in
an exhibition at FACT’s gallery [31].
Artists can provide useful critiques of society and can raise awareness of societal
issues in various ways. The German artist and filmmaker Hito Steyerl has an interest
in how the media, technology, and the global circulation of images impact society.
We live in a world that is already deeply edited and PhotoShopped. It is cut and pasted, and
the people who know these practices because they work with them everyday – let’s call them
artists – they understand the importance of these practices, not only understanding the world
but also making it as it is [32].
Steyerl’s video installation Factory of the Sun examines the desires and threats
of image circulation and the possibilities of shared resistance when surveillance is
everywhere in our increasingly virtual world. Factory of the Sun tells the surreal story
of workers whose forced moves in a motion capture studio are turned into artificial
sunshine [33, 34].
2.6 Conclusions
The artists’ subjective and hermeneutic approaches offer new perspectives to research
but there are challenges in articulating the unique qualities their work offers. Cross-
disciplinary supervisory teams at Wrexham Glyndwr University have enabled critical
dialogues for the students and supervisors to set out clearly and test the concise
methodological approaches to their work. The unpredictability of artistic research
has brought challenges, particularly when adopting a less prescriptive approach to
outputs in the research proposal. The researchers here often found it easier to talk
about the results of their interviews, or scientific experiments, than articulating the
impact of their practice within their projects when applying a multi-method approach
to research questions. In the In-between-ness project, for example, the therapeutic
effect of the project was not the primary measuring stick for assessing the success
of their work.
According to Polanyi (1962), aesthetic insights “motivates the early stages of
much scientific research” [35]. Mixed disciplinary supervisory teams at Wrexham
Glyndwr University have forced us to challenge perceptions of ourselves as artists
and consequently enriched our projects. We often cannot predict what others will
learn from our work, but it is the process of learning and identifying one’s individual
position in the research process that will allow for new avenues of knowledge to
emerge. The arts can be a powerful tool for social change, for wellbeing and for
connecting people. As Candy states in the summary of her new book, there is a
considerable amount we can learn from paying close attention to the world and it
takes a creative mind to do this with insight [36].
References
1. Barrett, E., Bolt, B.: Practice as Research, Approaches to Creative Arts Enquiry, pp. 4–6.
Bloomsbury Publishing Plc, London (2007)
2. Elkins, J.: Why Art Cannot Be Taught, pp. 37–39. University of Illinois Press, USA (2001)
3. Davis, M.C.: The Council for National Academic Awards 1964–74: A Study of A Validating
Agency. Loughborough University, UK (1979). Available online: https://en.m.wikipedia.org/
wiki/Council_for_National_Academic_Awards. Accessed April 2019
4. Elkins, J., Motram, J.: cited in Artists with PhD’s, p. 13. New Academia Publishing,
Washington, DC, USA (2009)
5. HESA. Available online: https://www.hesa.ac.uk/news/11-01-2018/sfr247-higher-education-
student-statistics/qualifications
6. Biggs, M., Buchler, D., Nelson, S.: Social Dynamics of Knowledge Communication Within
Higher Education: The Case of An Art-Science Collaboration (2011). Available online: https://
uhra.herts.ac.uk/bitstream/handle/2299/7582/905511.pdf?sequence=1. Accessed March 2019
7. Biggs, M., et al.: Editorial: the foundations of practice-based research, vol. 1. Working Papers
in Art & Design (2000). Available online: https://uhra.herts.ac.uk/bitstream/handle/2299/7325/
101063.pdf?sequence=1. Accessed April 2019
8. Wilson, M., Van Ruiten, S.: ELIA SHARE (2013). Available at: https://www.elia-artschools.
org/activities/artistic-research/share. Accessed May 2019
9. Candy, L.: Research and creative practice. In: Candy, L., Edmonds, E.A. (eds.) Interacting: Art
Research and the Creative Practitioner, Chap. 2, pp. 24–59. Libri Publishing Ltd, Farringdon,
UK (2011)
10. Candy, L., Edmonds, E.: Practice Based Research in the Creative Arts: Foundations and Futures
from the Front Line. Leonardo 51(1), 63–69 (2016)
11. Barrett, E., Bolt, B.: Practice as Research, Approaches to Creative Arts Enquiry. Bloomsbury
Publishing Plc, London, UK (2007)
12. Grey, C., Malins, J.: Visualising Research: A Guide to the Research Process in Art and Design.
Ashgate Publishing Ltd, Farnham, UK (2004)
13. Macleod, K., Holdridge, L.: Thinking Through Art: Reflections on Art as Research. Routledge,
Abingdon, UK (2006)
14. Biggs, M., Karlsson, H. (eds.): The Routledge Companion to Research in the Arts. Routledge,
Abingdon, UK (2011)
15. Nelson, R.: Practice as Research in the Arts. Palgrave MacMillan, London, UK (2013)
16. Candy, L.: Practice Based Research: A guide (2006). Available: http://www.
creativityandcognition.com/resources/PBR%20Guide-1.1-2006.pdf. Accessed April 2019
17. Candy, L., Edmonds, E.: Practice Based Research in the Creative Arts: Foundations and Futures
from the Front Line. Leonardo, vol. 51, Issue 1, February, pp 63–69 (2016)
18. Smithbell, P.: Arts-Based Research in Education: A Review. The Qualitative Report, 15(6),
1597–1601 (2010). Available online: http://nsuworks.nova.edu/tqr/vol15/iss6/14. Accessed
May 2019
19. Wrexham Glyndŵr University Vision & Strategy to 2025: Inspiring, Transforming, Enabling
and Sustaining. Definition of Research taken from Vision and Strategy to 2025 (2018). Available
online: https://www.glyndwr.ac.uk/en/media/156309%20Strategy%20English.pdf. Accessed
April 2019
20. Barrett, E., Bolt, B.: Practice as Research, Approaches to Creative Arts Enquiry, pp. 1–13.
Bloomsbury Publishing Plc, London, UK (2007)
20 S. Liggett
21. Liggett, S., Heald, K.: In-between-ness: Using art to capture changes to the self during antide-
pressant treatment (2013). Available online: http://in-between-ness.co.uk/about/synopsis-of-
research-project/. Accessed May 2019
22. Braisby, D.: How Can the Development and Use of Galvanic Etching Techniques Enhance the
Work of the Artist Printmaker? Ph.D. Thesis, University of Wales (2017)
23. Checkland, P.: Systems Thinking, Systems Practice. Wiley, London (1981)
24. Wyatt, M.: A cross-disciplinary study into how people living with dementia engage with and
experience painting alongside an artist researcher. Ph.D. Thesis, University of Wales (2019)
25. Sullivan, G.: Arts Practice as Research: Inquiry in the Visual Arts. Sage Publications, Newbury
Park, CA, USA (2005)
26. Niedderer, K., Roworth-Stokes, S.: The Role and Use of Creative Practice in Research and its
Contributions to Knowledge (2007). Available online: http://niedderer.org/IASDR07SRS.pdf.
Accessed May 2019
27. The Guardian 21 Oct (2019). Available online: https://www.theguardian.com/culture-
professionals-network/culture-professionals-blog/2014/oct/21/alistair-hudson-mima-
modern-art). Accessed May 2019
28. World Economic forum. Available online: [https://www.weforum.org/agenda/2018/12/why-
the-fourth-industrial-revolution-needs-more-arts-graduates/. Accessed May 2019
29. Schwab, K.: The Fourth Industrial Revolution (2015). Available online: https://www.
foreignaffairs.com/articles/2015-12-12/fourth-industrial-revolution. Accessed May 2019
30. Benedikt Frey, C., Osborne, M.: The Future of Employment, Published by the Oxford
Martin Programme on Technology and Employment (2013). Available at https://www.
oxfordmartin.ox.ac.uk/downloads/academic/future-of-employment.pdf. (Benedikt Frey, C &
Osborne). Accessed May 2019
31. FACT Future World of Work Project (2019). Available online: https://www.fact.co.uk/future-
world-of-work. Accessed May 2019
32. Steyerl, H.: 12th ELIA Biennial Conference ART, SCIENCE AND SOCIETY: Art Questions,
Art Knows, Art Matters Hosted by the University of Applied Arts Vienna at the Museums
Quartier Wien, 8–10 November 20 (2012)
33. Steyerl, H.: Factory of the Sun, February 21–September 12, 2016 courtesy of The Museum
of Contemporary Art, Los Angeles. Available online: https://www.moca.org/exhibition/hito-
steyerl-factory-of-the-sun
34. Tattersall, L.: Curator MOCA LA. Available online: https://www.moca.org/exhibition/hito-
steyerl-factory-of-the-sun. Accessed May 2019
35. Polanyi, M.: Personal knowledge: towards a post critical philosophy. Routledge, London (1962)
cited in Rust, C. Unstated contributions: how artistic inquiry can inform inter-disciplinary
research. Int. J. Des. 1(3), (2007). Online at: http://www.ijdesign.org/index.php/IJDesign/
article/view/201/80
36. Candy, L.: The Creative Reflective Practitioner: Research Through Making and Practice. Taylor
Francis, Abingdon, UK (2019)
Part II
Facilitating Communication Between
the Arts, Technology, and Audiences
Chapter 3
Framing the Conversation: The Role
of the Exhibition in Overcoming
Interdisciplinary Communication
Challenges
Susan Liggett and Mike Corcoran
Abstract The role of the exhibition in overcoming interdisciplinary communication

challenges is explored. The meaning of interdisciplinary communication is estab-
lished, alongside the challenges it brings, where their route causes lie and the extent
to which the exhibition is equipped to address them. It concludes that exhibitions of
a particular nature are uniquely equipped to facilitate better quality interdisciplinary
communication in certain contexts, but that further work is needed to develop mature
curatorial models to facilitate this. A structure for such models is proposed, along-
side the steps that now need to be followed if exhibitions are to fulfil this untapped
potential at the interface of art and technology.
Keywords Curatorial model · Communication · Exhibition · Interdisciplinary ·

Art · Technology
3.1 Introduction
Interdisciplinary solutions are becoming the lingua franca in the discussion of the
global challenges we face in the twenty-first century. Whether it’s the education of
our young people, the care of our elderly, the fight against disease or the protection
of our environment, the answers will not be found in any one place. Rather, progress
relies on the coming together of diverse expertise and perspectives, and the shared
commitment of a vast number of individuals, unified by a common goal.
At the meeting point of art and technology, this is felt more sharply than perhaps
anywhere else. The ‘fourth industrial revolution’ has blurred the boundaries between
man and machine, with the designer’s role fundamental in managing the interface
between the two. Furthermore, artists have ever-increasing responsibility for bring-
ing the challenges of our times to the masses in ways which are tactile, accessible
S. Liggett (B) · M. Corcoran

Wrexham, Wales
e-mail: s.liggett@glyndwr.ac.uk
M. Corcoran
e-mail: mike@macorcoran.com

26 S. Liggett and M. Corcoran
and engaging to all: driving political and social agendas. Conversely, technology is
opening up endless new ways to create, display and interact with art, spawning new
branches of artistic practice, and changing our perceptions of what art is and what it
can achieve.
Communication is central to the success of any group, working towards any objec-
tive. When groups are interdisciplinary, subject matter is technical, stakeholders are
global and success is critical, this presents unique challenges. However, in spite of
this, the nuance of communication in such contexts is still poorly understood. Exhi-
bitions are a unique communication tool. Their potential as a means to support better
quality communication amongst interdisciplinary teams is apparent; yet this potential
is under-investigated. The processes by which it could best be realized are ill-defined,
and the models and tools required to facilitate the execution of such processes are
absent.
Here, a forensic examination of communication amongst interdisciplinary art-
technology teams is undertaken to establish the root causes behind the challenges
which arise and the extent to which exhibitions are equipped to provide solutions to
them. Assumptions are tested against the experiences of the principal actors in such
communications and the way forward is posited, with proposals set out for the further
work required if exhibitions are to fulfil their potential in improving interdisciplinary
communication: for more people, more of the time.
3.2 Defining Interdisciplinary Communication
3.2.1 Defining Interdisciplinary
Before we identify interdisciplinary communication challenges, we must first

establish what is meant by interdisciplinary and communication.
The terms interdisciplinary, multidisciplinary and transdisciplinary and applied
inconsistently, and used interchangeably by many [1]. However, the National
Academies provide the following working definition of interdisciplinary research:
“a mode of research by teams or individuals that integrates information, data, tech-
niques, tools, perspectives, concepts, and/or theories from two or more disciplines
or bodies of specialized knowledge to advance fundamental understanding or to
solve problems whose solutions are beyond the scope of a single discipline or area
of research practice” [2].
It is in this sense that interdisciplinary will be referred to herein: referring to the
coming together of teams or individuals from two or more bodies of specialized
knowledge in research, or otherwise.
3 Framing the Conversation: The Role of the Exhibition … 27
3.2.2 Defining Communication
In order to precisely identify communication challenges, it is helpful to adopt a sim-

ple conceptual model of human communication which clearly delineates each of
its component parts. The Shannon–Weaver Model [3] was developed by Bell Lab-
oratories engineers Elwood Shannon and Warren Weaver in the 1940’s. It is widely
regarded as the forerunner of modern communication studies and provides such com-
ponents. Shannon and Weaver extrapolated a general theory of communication from
the process by which radio technologies function and identified five components of
all communications, as represented in Fig. 3.1.
An Information Source produces the message which is to be communicated.
A Transmitter encodes that message into appropriate signals for transmission. A
Channel provides the medium through which signals are transmitted (and as signals
propagate through this Channel, they are vulnerable to ‘Noise’ which can block,
interrupt or distort signals). A Receiver decodes the signal to reconstruct the original
message, and a Destination is an individual, group or entity for whom the message
is intended.
Consider saying hello to a friend as you walk down the street. You, or perhaps
we should say, the appropriate regions of your brain, are the Information Source,
taking the decision to transmit the message, ‘Hello!’. Your lungs, vocal folds, tongue
and mouth are the Transmitter, working together to produce the appropriate sound
waves and air is the Channel which carries these sound waves between you and
your friend. Passing buses, sudden gusts of wind and much else can provide noise to
distort the message at this stage, before your friend’s ear receives the sound waves
transmitted, and their brain converts and translates them to reveal their intended
meaning. The message has now reached its intended destination and your friend
Fig. 3.1 Shannon and weaver’s five component parts from which all communications are
comprised. Copyright © M. Corcoran 2019
responds appropriately, perhaps with a wave in your direction. Through this new
transmission (and feedback), the iterative process begins again.
3.2.3 Defining the Principal Actors in Interdisciplinary

Communication
Alongside a conceptual model of communication, it is also crucial that we have

an understanding of who the principal actors (in Shannon and Weaver’s language,
the Informational Sources and Destinations) in interdisciplinary communications
typically are. We propose the following groups, as shown in Table 3.1.
With working definitions of interdisciplinary and communication established,
alongside an understanding of who such interdisciplinary communication occurs
between, we can now consider the challenges of interdisciplinary communication
and their direct relationship to activities involving artists and technologists.
3.3 Interdisciplinary Communication Challenges
Shannon and Weaver proposed that, considering the five parts from which all com-
munication is comprised, three fundamental types of communication challenge can
arise:
Table 3.1 The principal actors interdisciplinary communication

Principal actor Definition
Specialists The individuals and teams who carry out the work. They possess knowledge of
one or more technical discipline and are unified by a common set of objectives
Benefactors Those who sponsor the work. This includes philanthropic funders, public
bodies and private investors
Beneficiaries Those who reap the rewards of the work, including governments, private
companies and proportions of the general public
Publics A meta-group that include all those within a given population who are
stakeholders in an interdisciplinary activity, either as specialists (contributing
their insights with respect to a particular problem), Benefactors (through the
public funding of interdisciplinary work through taxation or others) and
Beneficiaries (in virtue of an activity’s impacts). Often Publics will hold
multiple roles simultaneously
Translators Facilitating communication between two or more of the aforementioned
groups in any combination. For example, they may assist specialists in
engaging with Publics, Benefactors in evaluating Beneficiaries, or Publics in
lobbying Benefactors
1. Technical Challenges: relating to how accurately messages can be issued.

2. Semantic Challenges: relating to how precisely meanings can be conveyed.
3. Effectiveness Challenges: relating to how successful received messages are in
bringing about their desired effects.
3.3.1 Communicating Accurately
Technical challenges arise due to the many different means artists and technologists
have to adopt to get their messages across.
The typical interdisciplinary communication involves multiple information
sources, and multiple Destinations. Diverse information is shared in all directions:
both within specialist interdisciplinary teams, and between those teams and their
Benefactors, Beneficiaries, Publics and Translators. Every group (and every actor
within every group) has their own preference with respect to transmitting and receiv-
ing information (some like to write or talk, others like to draw or experience and
so on), and information tends to go through many iterations of transmission and
reception on its journey between actors, highly subject to noise and interference.
This is especially true when artists and technologists work together. Consider
what it takes for the latest research in machine learning to reach the desk of the fine
art professor or for the biomedical scientist to encounter theories in the teaching
of painting. Consequently, before the insights of one actor enter the consciousness
of another, they may be transmitted through many cycles of performance, discus-
sion, publication or otherwise, with each increasing the probability of degradation,
mistranslation and dilution of the overall message [4]. Even before interdisciplinary
teams have been constructed, the basis for their very construction can run the risk of
being underpinned by inaccurate information.
3.3.2 Communicating Precisely
Semantic challenges arise due to the difficulty of explaining things in a simple way
which artists, technologists and everyone they engage with can understand.
There is no universally understood language amongst the principle actors in inter-
disciplinary activities. As Derrick et al. observe, “Scientific disciplines have different
cultures, languages, and standards” [5] and the same can be said of non-scientific dis-
ciplines. The vocabulary and reference points of specialists are highly technical, and
a deep understanding of issues is often contingent upon a contextual understanding
of the wider subject, not always available to every specialist within an interdisci-
plinary team. This is particularly true when artists and technologists come together.
Their respective disciplines having long been siloed into what CP Snow famously
characterized as ‘The Two Cultures’ [6].
This problem is only compounded as specialists (of any discipline) look to com-
municate to their Benefactors, Beneficiaries and Publics. Translators can assist, but
ultimately, communication is often heavily reliant on metaphor and simplification,
inevitably impacting on the precision with which information can be shared, and
preventing much information from being shared at all.
3.3.3 Communicating Impactfully
Effectiveness challenges arise due to the many different things artists and technolo-
gists have to simultaneously achieve when they communicate.
Often, a single communication will have multiple Destinations and multiple
diverse objectives. As communication takes place amongst specialists within an
interdisciplinary team, the outcomes and impacts of such work are simultaneously
subject to the scrutiny of Benefactors, Beneficiaries and Publics, each with diverse
needs and expectations. To achieve such objectives can be ‘complex, nuanced and
highly resource intensive’ [4]. When it is simply not possible to tailor the perfect
transmission for each target Destination, a ‘one size fits all’ approach can be con-
sidered as an alternative, but in striving to please all, there is an inherent risk of not
pleasing any.
Taking into consideration the main components of communication, the fundamen-
tal types of communication challenge posited by Shannon and Weaver, and reflecting
on each in the context of the principal actors in interdisciplinary communication,
especially artists and technologists, we can conclude that such communication is a
challenge because:
1. Technical challenges lead to the accuracy of information shared being impeded
by the many different ways and means required to communicate.
2. Semantic challenges lead to the precision of information shared being impeded
by the lack of common vocabulary and reference points amongst the principle
actors who are communicating.
3. Effectiveness challenges led to the impact of communication being impeded by
the multiple objectives it is required to serve.
In transitioning from an understanding of the challenges which lie at the heart
of interdisciplinary communication to an understanding of the ways in which these
challenges can be overcome and the role of the exhibition in facilitating this, it is
essential to gain a deeper appreciation of the validity of our conclusions above. This
includes the extent of their generalisability, and the contexts in which they most
strongly apply. This starts through dialogue with those with lived experience of
interdisciplinary collaboration from a variety of different perspectives.
3.4 Interviews with Interdisciplinary Practitioners
The following perspectives are derived from interviews conducted with various
experts and specialists in the domain of this research. That is, those who are primarily
responsible for the management and administration of interdisciplinary projects and
activities, and the communication that comes with it. Their thoughts and experiences
are summarized below.
3.4.1 The Specialist Technologist’s Perspective
The specialist technologist’s perspective was offered by a software engineer working

predominantly in the rendering of digital holograms.
This involves communicating primarily with:
• Artists (Benefactor/Beneficiaries): including both independent artists and com-
mercial customers looking for holographic solutions within their work and
• Optical engineers (Specialists): in order to understand and provide feedback on
the strengths and restrictions of working in the medium.
Communication with fellow technical specialists (i.e., software and optical engi-
neers) tends to be the most straightforward. This is in part thanks to the common
vocabulary and reference points these specialists share, and the readily quantifiable
nature of the information they typically share (for example, the technical parameters
of the hologram printing equipment). For communication of this nature, communi-
cation tools such as email are very effective, allowing for precise information to be
shared back and forth asynchronously.
Conversely, communication between artists and engineers is more challenging,
because there is no common vocabulary and the information being shared is typically
subjective and qualitative. Engineers and artists are often exploring the ‘art of the
possible’, and the broad objective which the creative work is attempting to realize.
In these circumstances, live and iterative communication, supported by visual aids,
can prove effective.
With regards to exhibitions, they can play a valuable role in setting the context for
discussions between any actors, alongside demonstrating the nature and potential of
the technology, especially when bringing such state-of-the-art technology to general
public audiences.
3.4.2 The Specialist Artist’s Perspective
Jessica Lloyd Jones, an artist who exhibits internationally combining elements of art,
science and technology provided the specialist artist’s perspective. Lloyd Jones’ work
Fig. 3.2 Jessica Lloyd Jones ‘Forest Fuel’ 2018, permanent projection onto a biomass plant,
Gothenburg, Sweden. (Copyright © J. Lloyd Jones 2018) included by permission
explores concepts of energy, matter and natural phenomena through the interaction
of materials and light through installations, projection and sculpture created from a
variety of materials and processes. An example of this work is shown in Fig. 3.2,
where a digital image is projected on to a biomass plant in Sweden.
Producing works of this nature requires effective liaison with a wide group of
people. These include project managers, funders and commissioners (Benefactors),
technologists, engineers, architects, fabricators and design teams (Specialists) and
curators (Translators).
Internal communication between these people is, in part, influenced by the time
sequence in which the various actors signed-up to the project. For example, com-
missioners and project managers will often be in place before the artist becomes
involved, whereas others (for example, fabricators) can only be selected by the artist
once a project has commenced. The structure, roles and responsibilities of a team will
often be assigned in line with the common goals identified between the artists, tech-
nologists/scientists and the other stakeholders involved, with ultimate management
responsibility depending on who instigated the project. Maquettes and visualizations
provide useful aids here, illustrating ideas and concepts to stakeholders and potential
stakeholders.
External communication to an artwork’s intended audiences must simultaneously
speak to many diverse individuals. Those with an active interest in the arts are the
easiest to reach as they tend to be more open-minded and curious. This is especially
true when the information to be communicated is broad-based and thematic, for
example, communicating ideas about energy, beauty, innovation or creativity. When
audiences have little knowledge, understanding or exposure of the arts, this can
directly impact upon enthusiasm, open-mindedness and curiosity, and subsequently
on the success of the communication which follows.
No one tool is the secret to successful communication, with the most effective
means of communication depending on the context, though when engaging in a
crowded market-place, novelty can be key. Audiences are more likely to be engaged
if the artwork presents something they haven’t seen before.
3.4.3 The Benefactor’s Perspective
The benefactor’s perspective was offered by a senior manager in a UK-based public

services innovation foundation, often delivering projects bringing together artists,
technologists and public bodies.
Audiences in communication in this environment are diverse. This is not only
a diversity of organisations, but a diversity of individuals within each organisa-
tion, which communication must be sensitive to. However, across all audiences, the
communication of ‘facts and figures’ tends to be easier than the communication of
‘concepts and ideas,’ thanks to:
• The clear and unambiguous nature of the information: usually concerning who,
what, when, where and how;
• The clearly delineated audiences for the communication: with eligibility criteria
defining the types of organisation for whom the information is relevant;
• The simple and well-established channels through which communication is
shared: for example, application forms to access funding and;
• The simple feedback loops associated with these channels: for example, via
workshops, emails and phone calls.
Conversely, communicating concepts and ideas, especially the learning arising
from projects, is generally more challenging. Points can be nuanced and subtle; the
learning potentially applicable to any audience and the channels through which such
learning is shared are less well-established resulting in feedback loops which can be
vague or non-existent.
Additional challenges can arise due to the differing organisational cultures in place
between partners in a given project (especially those bringing academic, public and
private sector partners together). Each may have different reasons for being involved
in the project, and these need to be taken into account. At the same time, this diversity
can be a strength of the project because it can lead to innovation and new ways of
viewing the project and its outcomes, which can add value to the original concept.
Exhibition is a tool used regularly to communicate concepts and ideas, present the
outcomes achieved and learning gained through collaborative projects and to foster
multiple conversations simultaneously between many partners. It is often easier to

show someone something than tell them about it. The innovation foundation often
use ‘pitching exhibitions’ to invited audiences of key stakeholders at targeted events,
which they find to be a highly effective method.
3.4.4 The Translator’s Perspective
The translator’s perspective was offered by a public engagement manager at a lead-

ing biomedical research facility. A permanent gallery at the facility, along with an
accompanying public engagement outreach programme, aims to generate excitement
and interest in discovery of science, human health and wellbeing: through artistic
means and otherwise.
Exhibitions have to cater to many audiences, marrying rich content for high
science capital peers (including those researchers based at the facility) simulta-
neously with that for a low science capital lay-public. Often, they also need to
cater to visitors from different countries (with different cultural backgrounds), politi-
cians, distinguished guests, as well as the accompanying press: simultaneously and
seamlessly.
To bring these exhibitions together requires close communication with and
between:
• Public engagement and education teams (Translators)
• Press and marketing teams (Translators)
• The organisation’s scientists (Specialists): including those directly involved in
each exhibition, and the wider community of those working in the organisation.
• Exhibition funders (Benefactors).
• Contractors (Specialists) of diverse backgrounds and specialisms.
• Focus groups (Publics and Beneficiaries) contributing to each exhibition’s
relevance and accessibility.
• Peers (Specialists and Translators): working in museums, galleries, and other
science-communication environments, sharing best practice, knowledge and
experience.
In practice, the planning of each exhibition can require communication with a
diverse group of 30–40 people.
Internal communications are managed with a high degree of structure. An Internal
Advisory Panel of senior managers and key stakeholders communicate the exhibi-
tion’s broad themes and messages. It also translates these into the wider strategic
direction and brand identity that the organisation is seeking to achieve. This helps to
build trust—so that the senior managers and key stakeholders feel confident in the
processes being followed and the expected outcomes. An exhibition steering com-
mittee comprised of colleagues with a variety of roles across the organisation can
provide feedback on various aspects of the exhibition plans. A project group is nor-
mally required to discuss the practicalities of bringing each exhibition together and
meeting the needs of all stakeholders within the constraints of the space available. It
is also important to maintain communication with any external teams of contractors
and keep track of timelines, deliverables and issues as they arise. Communication
within these structures is aided by video conferencing, formal presentations, docu-
ment sharing and Gannt charts, but the most effective method is often to communicate
face-to-face (or via video conference for more remote collaborators).
Communication is most effective when it is between individuals who all work
in the same context, with the same strategy, and the same audience. The greatest
challenge is when the time is limited, audiences are disparate, and a contextual
understanding is absent, communication is far more complex.
With regard to the content of the exhibitions, to achieve their intended impact of
making a tangible impact to the feelings and perceptions of their audiences, consid-
eration needs to be given to the diverse ways in which they may be interpreted by
different kinds of audiences, particularly where advanced or specialist subjects are
involved. There is also a distinction between the explicit information the exhibition
communicates directly and the implicit communication of its values, character and
context. The latter can be assisted, amongst other things, by the atmosphere cre-
ated by the exhibition and the personnel who bring it to life. Both are needed if the
exhibition is to be successful.
3.4.5 The Combined Perspective
In many cases, one individual may take on multiple communication roles in their
work.
Such a combined perspective was offered by Jamie Harris, founder of Elder3D, a
company specialising in making artworks affordable to the masses through utilization
of cutting-edge 3D design and manufacturing technologies (Fig. 3.3). Harris is a
Fig. 3.3 A metallic 3D Print (Left) and its supporting digital 3D Model (right). (Copyright © J.
Harris 2019) included by permission
Specialist in both artistic and technological practices, a Benefactor commissioning

work from his international network of associate artists and technologists and a
Translator, ensuring the wants and needs of his customers are always catered to.
In this context, understanding and trust are the basis for successful communication.
Artists and 3D print specialists have to understand that the quality of the final product
is dependent on the quality of information provided initially. The print specialists
have to understand every detail in the brief and customers have to understand what
is and what is not possible within the limitations of the technology.
Communication is easiest when the information shared is quantitative and unam-
biguous (for example, specifying the material and dimensions for a 3D print run)
and when it is between actors who speak a common language, sharing common
skills and experiences. Conversely, when the information to be shared is nuanced
and qualitative (for example, when a family wishes to convey the ‘spirit’ of a person
in a portrait) and when commonality of language is missing, communication is a
challenge. Commonality lies in the shared experience of actors, but also on the num-
ber of individuals comprising each ‘Information Source’. When a customer artist or
printer is in practice, and when decisions are made by committee, information can
quickly become confused, unclear or contradictory.
Trust is a prerequisite to understanding allowing honest and open communication
to take place customers must trust that their vision can translate into a high-quality
product, and artists must trust that a printer can bring their designs to life, print
specialists must trust that the designs will be suitable for 3D printing in the spec-
ified materials and so on. Therefore, establishing a strong rapport and a personal
relationship with everyone is essential to the business.
With regards to exhibition, showing the actual work is often the best way to
communicate because people can touch and handle the work. This creates excitement,
promotes questions and produces a better understanding.
3.4.6 An Interdisciplinary Communication Tool Wish List
In Sect. 3.3 we proposed that the challenge of interdisciplinary communication was

rooted in the many different ways and means required to communicate (impair-
ing accuracy), the lack of common vocabulary and reference points amongst the
principal actors communication is between (impairing precision), and the multiple,
simultaneous objectives such communication is often required to serve (impairing
impact).
The views of those with direct experience of working in these environments cer-
tainly do not contradict these conclusions. Rather, we observe many instances of
these challenges in action. We also gain a deeper insight into how these challenges
arise, the other challenges they are closely related to and how such challenges might
be addressed.
Table 3.2 An interdisciplinary communication tool ‘wish list’

An interdisciplinary communication ‘wish list’
More accurate communication (to overcome technical challenges)
1 Reduce the total number of channels needed to communicate messages
2 Reduce the quantity and improve the quality of Transmitters/Receivers
3 Reduce the risk of noise within multiple communication-feedback cycles
More precise communication (to overcome semantic challenges)
4 Bring clarity to open ended, subjective and qualitative messages
5 Bring shared language, reference points and context to messages between diverse actors
6 Bring clarity to single messages compiled by multiple actors
More impactful communication (to overcome effectiveness challenges)
7 Provide simultaneous communication of multiple (implicit and explicit) messages
8 Build trust and rapport amongst the principle actors in communication
Taking all of this into consideration, we propose a simple ‘wish list’ (see Table 3.2)
of outcomes for any tool which could make interdisciplinary communication
involving artists and technologists easier to facilitate.
With a wish list for an interdisciplinary communication tool established, we can
assess the extent to which the exhibition has the potential to be such a tool. This can
be done by considering what exhibitions are and how well aligned (and potentially
impactful) they are with respect to each of our desired outcomes.
3.5 The Exhibition as an Interdisciplinary Communication

Tool
3.5.1 Defining Exhibition
Exhibition is a broad and all-encompassing term. It may refer to any “event at which
objects are shown to the public, a situation in which someone shows a particular
skill or quality to the public, or the act of showing these things” [7], and it is this
general sense that exhibitions are discussed here. This definition excludes private and
informal day-to-day interactions. Furthermore, when we discuss exhibitions herein,
we refer primarily refer to them in their most common temporary and ephemeral
form, as opposed to permanent installations and collections.
Our wish list (Table 2.2) is composed in the language of the Shannon–Weaver
Model and so before we can assess exhibitions against each of its desired outcomes,
we must first ensure we can describe exhibitions in a consistent way. For virtually
any exhibition, in any form:
• The ideas of the exhibiting artist and/or the curator serve as the Information
Source.
• The artworks, in their particular arrangement, are the Transmitters of these ideas.
• The exhibition environment is the Channel through which these ideas pass.
• The senses of the exhibition’s audiences are the Receivers of these ideas, and
• A given subset of the exhibition’s total audience is the intended Destination for
these ideas.
Furthermore, as done so throughout Sect. 3.4, the principal stakeholders in exhibi-
tions can be simply mapped to the principal actors in interdisciplinary communication
which we have defined as follows:
• Specialists: the exhibitors and/or the exhibition’s curator.
• Benefactors: the sponsors, institutions or others who fund the exhibition.
• Beneficiaries: the audiences who are the intended ‘Destination’ for the exhibition’s
messages.
• Publics: formed from some combination of the above.
• Translators: curators and exhibition assistants.
With these definitions in place, our desired outcomes can now be assessed in turn.
3.5.2 Assessing Impact
Exhibitions can address technical challenges, leading to improved accuracy in com-

munication, by reducing the number of Transmitters, Receivers and Channels, needed
to communicate information. Each Information Source has one Transmitter, an
exhibit, fewer transmitters implies fewer receivers to get a message to its intended
destination, and the exhibition provides the sole Channel for communication. Fur-
thermore, its structured nature allows for multiple stable feedback cycles, provid-
ing a constant background, providing a general context against which all further
communications can be located.
Exhibitions can address semantic challenges, leading to improved precision in
communication, by front-loading the communication process with the information
which does not require the use of technical and discipline-specific vocabulary and
reference points to understand: that is, the general qualitative messages that lie behind
the technical detail. The exhibits themselves become the common, visual and tactile
transmitters used to communicate these messages, to actors who do not require
any technical knowledge to receive them. By precisely communicating the general
messages behind work at the earliest opportunity, subsequent discussion of that work
(through symposia, informal meetings, exhibition guides or otherwise) and all that
follows thereafter, becomes much easier, whether between one induvial or many.
Exhibitions can address effectiveness challenges, leading to improved impact in
communication, by simultaneously communicating to multiple and diverse actors.
For some (for example, Publics), a general sense of ‘what the work is about’ is
the desired outcome. For others (for example Benefactors), this general sense is a
precursor to a desired outcome of a more detailed understanding why this work is
needed, at this time. For Specialists, this detailed understanding is also a precursor
to understanding who they should work with, how they should work with them, and
where they need to widen their technical understanding (or recruit the support of
translators) to facilitate this. Each of these can use the same core set of transmitters,
the exhibits, and the sole channel of the exhibition. The exhibition can be curated so
as to allow multiple and mutually compatible routes through it. Some may view the
exhibition online; others may view it in person. Some may use an exhibition guide;
others take a guided tour. Some may participate in symposia; others use it as a venue
for meetings. Some may spend only minutes in the space; others hours or days. All
of this can happen at the same time. There is no direct analogy for this flexibility with
a research paper, a seminar or many of the other traditional tools of interdisciplinary
working.
Not only that, but by putting an initial emphasis on common thoughts and feelings,
in an environment which lends itself to being inclusive, social and highly creative,
the exhibition plays a role in building the relationships, trust and rapport that are
prerequisite to good communication, in a space which can entice and motivate new
audiences to join in the fun.
3.6 A New Curatorial Model
The potential of the exhibition to provide solutions to interdisciplinary communica-

tion challenges, as outlined in Sect. 3.5, is clearly encouraging, but our discussion
so far is insufficient in and of itself. To support meaningful action and generate real
outcomes, this potential must be pinned down and converted into the clear guidance
required by exhibition curators and those responsible for making interdisciplinary
communications work. Our observations need consolidating into a model, stipulat-
ing what sort of exhibitions, will deliver what sort of outcomes, for what sort of
audiences, and how.
To date, there is no off-the-shelf conceptual model describing the functions of
the exhibition as an interdisciplinary communication tool, and perhaps this should
not be surprising. Exhibition and curatorial studies, and the discourse and theoretical
models which underpin them, do not share a common genus with communication
theory. They have evolved divergently, fuelled by different motivations, setting out
to solve different problems and thinking about those problems in different ways.
The Show-Talk-Do Model [4] (Fig. 3.4) describes the process by which high-level
analysis of opportunities for research collaboration, alongside simultaneous non-
expert engagement, can be achieved in environments which are rich with technical
information and highly interdisciplinary. It contains three simple steps, which can
be summarized as follows:
1. Show It: Focuses on initial organic reactions. A public exhibition presents the
organic products of the ongoing research of participating researchers, from across
a variety of disciplines and interdisciplinary areas. Exhibits may include (but
Fig. 3.4 The Show-Talk-Do Model for multi-audience engagement (© M. Corcoran and Liggett.
S. 2018)
are not limited to) artworks, performances, documentation of practical research

activities, visualization of ‘live’ research data and interactive demonstration of a
process or procedure under development.
2. Talk About It: Focuses on developing contextual understanding. A symposium
(or similar) takes place, where exhibiting researchers discuss with one another
the context in which their exhibit (and by proxy, their research) is situated. This
discussion should be structured (in accordance with pre-defined rules) and be
located within the exhibition itself.
3. Do It: Focus on technical understanding and action. Research papers (or similar)
are made available to researchers, and where synergy has been identified through
Steps 1 and 2, the scope for research collaboration can be discussed.
The exhibition itself lives within Step 1. Open to all (with Specialists alongside
Benefactors, Beneficiaries and Publics in their various capacities), its primary role
is to build common understanding on the level on thoughts and feelings, free from
any pre-convinced ideas and prejudices:
The exhibition should be curated as to allow each work to be considered in isolation, as well as
all considered in combination … Exhibit labels (if used) should not refer to exhibitors’ subject
specific disciplines, their academic status or seniority, and should provide no information
about the research presented. The exhibition should be made open and accessible to all the
defined project’s stakeholders, including researchers, funders, policy makers, and the general
public. Visitors (including the exhibiting researchers themselves) should be encouraged to

explore the exhibition with an open mind and reflect upon their spontaneous thoughts and
feelings regarding the individual works and the combination [4].
When principal actors in communication have connected on an emotional level,

they are then best disposed to discuss the wider context in which research is being
undertaken (Step 2), and only after that, they are best disposed to engage in a labour
intensive and technical discussion regarding the scope of future collaboration (Step
3). These things can take place within the same exhibition environment, and over the
same time period, but it is only when they are in this sequence, that the process will
achieve its desired results. Also, different principal actors will be required to progress
through different numbers of steps. For Beneficiaries, Step 1 may be enough, being
inspired, challenged, provoked or otherwise by the works presented to them. For
many Specialists, Step 2 will be enough, understanding their peers’ motivations and
objectives, and only for those Specialists committed to working closely together (in
virtue of what they have learned through Steps 1 and 2), are the challenges of getting
to know the technical details of one another’s practice (Step 3) required.
Show-Talk-Do was developed with a focus on interdisciplinary research. How-
ever, this structure is applicable to communication in interdisciplinary contexts more
broadly. It provides a simple conceptual model concerned with communication,
considering the exhibition as a communication tool alongside all others, and pro-
vides a helpful foundation for assessing the potential impact of the exhibition to
bring about particular outcomes in particular circumstances, well aligned with the
interdisciplinary communication challenges that have been identified.
3.7 Conclusions
This work has sought to establish what role, if any, the exhibition had in overcoming
interdisciplinary communication challenges.
The adoption of a conceptual model of communication identified that the chal-
lenges that arise can be grouped in terms of accuracy, precision and impact. Through
discussion with those with lived experience of these challenges, eight outcomes
were identified which the ideal interdisciplinary communication tool would be able
to tackle. The general ways in which the exhibition is well placed to meet these out-
comes were identified. The adoption of the Show-Talk-Do Model enabled discussion
on where these impacts might be most greatly felt, and how.
Therefore, the exhibition, by virtue of its open, creative and engaging nature, its
simple means of communicating messages and its situation at the nexus of many
audiences, does have an important role to play in overcoming interdisciplinary
communication challenges. It is likely to be especially significant when:
• The actors engaged in communication are large in number and diverse in nature.
• The messages being communicated are broad, qualitative and contingent on an
‘emotional/experiential’ understanding.
• The messages to be communicated are numerous and communicated to many

actors simultaneously, each with their own objectives.
• Open and transparent communication amongst and between all actors is desirable.
• The communication process itself is intended to attract and grow its own audience.
The exhibition does not replace other tools. On the contrary, It is most likely
to have a major impact when delivered in conjunction with other tools, not only
achieving its own ends, but serving as a means to foster higher quality discussions,
and ongoing research and partnerships.
However, if the exhibition is to consistently achieve these impacts and optimize
its role as an interdisciplinary communication tool, there is still much work to do.
We suggest the following three next steps are now required.
(i) Better conceptual models
The more mature the conceptual models, and the more articulate they can become
at describing how exhibitions bring about their effects, the more useful they will
become to the curators and other stakeholders in those exhibitions. Show-Talk-Do
is a helpful start point, not an end. A sign of progress will be when it is rigorously
scrutinized, refined and ultimately replaced, with better, more powerful models.
(ii) Greater exhibition infrastructure
It must become easier for those who want to use exhibition as a communication tool,
to do so. Often, such tools are used not because they are the best option, but because
they are the available option. Art and educational institutions, curators and research
funders all have a responsibility to make exhibition an option: offering the expertise,
guidance, facilities, resources and finance that are required to do exhibition well, and
reap the just rewards.
(iii) Leadership
For exhibitions to be considered an essential communication tool, occupying the
same status as a research paper or seminar in professional communication, it will
require a shift in mind-set amongst the interdisciplinary community, and leaders
from within that community to initiate change. Early adopters will face the biggest
challenge, but each exhibition delivered will make the next one easier to deliver, and
the impacts should follow closely behind.
If these challenges are met, there is no reason why the exhibition should not be
the mainstay of 21st communication practice in the age of the interdisciplinarian.
Acknowledgements The authors wish to thank all those who shared their time and expertise to
contribute to this chapter, most notably those we interviewed: Tal Stokes, Jessica Lloyd Jones, Rob
Ashelford, Bryony Benge Abbott, and Jamie Harris.
References
1. Derrick, E., Falk-Krzesinski, H., Roberts, M.: Facilitating Interdisciplinary Research and
Education: A Practical Guide. American Association for the Advancement of Science
and the Colorado Initiative in Molecular Biology of the University of Colorado (2012)
[online] Available at: http://researchdevelopment.byu.edu/wp-content/uploads/2012/12/Full_
Report_Creating_Interdisciplinary_Science.pdf. Accessed 23 June 2019
2. National Academies.: Committee on Facilitating Interdisciplinary Research, Committee on Sci-
ence, Engineering, and Public Policy. Facilitating Interdisciplinary Research. National Academy
Press. p. 2 (2004)
3. Shannon, C.A.: Mathematical theory of communication. Bell Syst. Tech. J. 27 (July and October),
379–423, pp. 623 656, (1948). Available at: http://math.harvard.edu/~ctm/home/text/others/
shannon/entropy/entropy.pdf. Accessed 5 March 2018
4. Liggett, S., Corcoran, M.: Interdisciplinary research unmasked: a new curatorial model for
multi-audience engagement. Stud. Vis. Arts Commun.: Int. J. 5(1) (2018) online ISSN 2393-1221
5. Derrick, E., Falk-Krzesinski, H., Roberts, M.: Facilitating interdisciplinary research and
education: a practical guide. In: American Association for the Advancement of Science
and the Colorado Initiative in Molecular Biology of the University of Colorado (2012).
[online] Available at: http://researchdevelopment.byu.edu/wp-content/uploads/2012/12/Full_
Report_Creating_Interdisciplinary_Science.pdf. Accessed 23 June 2019
6. Snow, C.P.: The Two Cultures, p. 3. Cambridge University Press, London (1959, 2001)
7. Cambridge English Dictionary.: Exhibition (2019). [Online] Available at: https://dictionary.
cambridge.org/dictionary/english/exhibition. Accessed 05 July 2019
Chapter 4
Modern Communication Technologies
and the Marxist Understanding
of Scientific Cognition
Naira Danielyan
Abstract The bicentenary of the birth of Karl Marx occurred recently (2018). It is
thus apposite and insightful to reconsider his theories that relate to modern develop-
ments in the field of communications technology. Marxism proposed the view that
the basis of the human-specific attitude to the world is focused on practical activity
directed to the transformation of natural and social objective reality. Scientific results
focus on natural and social objects and processes and this blends into science-based
technologies, interacting and being changed by humans who realize some objec-
tive or social activity. Communication practice is connected with definite forms of
worldview and a definite level of cognition, which reflects practical achievements
and promotes their further reproduction and perfection. According to Marx, primacy
of practice means that the efficacy of its real influence on nature, society, and per-
sonality is a characteristic of the development of a subject’s cognitive activity. It is
concluded that Marxism’s achievement is the development of the scheme: object—
objective—practical activity—subject as the foundation of scientific cognition and
the criterion for the correspondence between knowledge and reality. Nowadays this
scheme is taken as primary while constructing and understanding communication
technologies, considered as a special form of practice: this implies the dialectic unity
of material artifacts on the one hand and human skills and experience on the other
hand.
Keywords Co-construction · Cognition · Communication technology · Marx ·

Marxism · Object · Practice · Subject · Technical activity · Trans-humanism
4.1 Introduction
Humans interact with the environment in varying degrees, ranging from passivity to
aggressive pursuit of possibilities to transform it actively. Active transformation can
be seen as being driven by the process of personal development as a social being
N. Danielyan (B)
Philosophy, Sociology and Political Science Department, National Research University of
Electronic Technology, Moscow, Russian Federation
e-mail: vend22@yandex.ru

46 N. Danielyan
in the system of material and spiritual culture. The bicentenary of the birth of Karl
Marx took place in 2018, and his ideas and theories still have plausibility, credibility,
and relevance as tools to analyze and extrapolate from the situation today. Thus, it
is relevant to consider their implications and application to the most transformative
technologies of the present day.
Marx considered man’s practical activity as the background of his specific attitude
to the world (both natural and social). He wrote in “Economic and Philosophic
Manuscripts of 1844” that man is not a product of circumstances and upbringing
and we should not consider our mind as a passive sensation recipient. According to
him, nature is a “man’s inorganic body with which he must remain in continuous
intercourse if he is not to die” [1, p. 87]. While transforming the world, man creates
both a new reality and new conditions of his existence that are not given to him in
a ready-made form by nature. He also develops and perfects himself, specifically in
his creative abilities. Hence, this results in the actual transformation of the physical
world, which can be considered as the foundation of other manifestations of original
human activity. Marx stresses that man is an “objective being” and “acts objectively”
[1, pp. 92–94]. Further, he defines the way of including man in the natural world
and society surrounding him through the active transformation of real objects and
phenomena in the process of his praxis. This framework gives some initial guidelines
to represent Marx’s interpretation of man in all the variety of his relations with the
world as a whole.
According to modern epistemological tendencies, the fundamental characteristic
of practice as a specific output of man’s existence is its openness to objective reality,
which always exceeds his abilities to master it. In contrast, there are still limitless
human capabilities to develop new ways and means in order to interact with reality.
As a result, the basis of the human attitude to the world is the creation of such a
system that allows man to construct definite artificial renewable tools and means to
influence his reality. He passes these from generation to generation; however, while
creating, reproducing, and perfecting this “transformed nature” humanity transforms
itself due to the formation and improvement of its appropriate skills and actions.
Marx asserted that the realization of any forms of practical and transformative
activity occurs only under conditions of social cooperation [1, p. 95]. It follows
that people’s attitudes, as subjects of practical activity, to the objective reality that
they transform always assume real human relationships as a concomitant. Thus,
practice as a philosophical category is formed, reproduced, and developed in the unity
of “subject–object” and “subject–subject” relations: it is a manifestation of public
activity. A human’s self-motivated intellectual evolution in practice is connected
with the development of their communication skills and communication culture as
necessary preconditions of mastering the external world.
According to the theory of communication, this approach coincides with the infor-
mation interpretation of the communication process, as can be found in the works
of Shannon [2]. Its main characteristic is specific attention to the linear character
of communication, i.e., it is an irreversible process. Following the communication
paradigm, practices influencing the mass audience intentionally, e.g., mass media,
government information, political structures or marketing, are dominant. They have
4 Modern Communication Technologies … 47
various forms of information-based influencing and techniques for manipulation of

human consciousness in their foundations.
Hence, communication practice, as with any others, will be always connected with
definite forms of perception of the world and a definite level of cognition, focusing on
practical achievements and promotion of their further reproduction and perfection.
According to Marx, primacy of practice means that the efficacy of its influence on
nature, society, and personality is a characteristic of the development of the subject’s
cognitive activity [1].
Thus, practice as a specific form of human activity in the world possesses a
complex system arrangement. It includes the following:
(1) real transformation of the external environment by artificial tools and instru-
ments (subject–object relations);
(2) communication during this transformation (subject–subject relations);
(3) a complex of norms and values (value and goal-oriented structures) that
facilitates the targeted practical activity.
4.2 Modern Epistemology About Subject–Object Relations
Everything that really exists and is external to the subject, can be regarded as an
object of cognition. It can be not only an item, but a whole complex or a system,
either natural or ideal.
Any physical thing existing in space and time, any objective and real situation can be con-
sidered as an object of cognition. It can be the subject’s body, his/her consciousness or
personality in general, another person, their consciousness, any cultural items (including
books) and their meanings [3, p. 157].
The spectrum of the objects to be studied has significantly widened with time,
due to an evolving system of ideals and norms of the cognition process, and this
has allowed the discovery of ways to research and master complex, self-regulating
systems.
Special attention is currently being paid to discourse as a communication, i.e., ver-
bal communication practice. Alexy [4] separates the following discourse rules, which
he identifies as the most important points to understand the rational communication
process:
• Any subject who knows a language and is capable of functioning can participate
in discourse;
• Any subject can question any assertion;
• Any subject can introduce any assertion into discourse;
• Any subject can express his/her aims, wishes, and necessities.
Such an approach allows the achievement of a consensus due to the overcoming
of personal subjective opinions by the participants of a discourse in favor of an
48 N. Danielyan
agreement motivated rationally. Both language and practices outside of language

become important. They generate various forms of communication among discourse
participants.
As an example of substituting subject–object relationships by the process of obser-
vation, it is useful to consider the ideas of the theory of autopoiesis by Maturana and
Varela [5]. According to this theory, a human as “a living system” does not just reflect
the world surrounding him/her, but constructs the worldview in correspondence with
his/her cognitive, existential, and social sets [5]. As an autopoietic system, a human
progresses independently while broadening his/her relations with the world. Accord-
ingly, this leads to the appearance of inner and outer relations that then form his/her
personal space.
Autopoietic systems have the following features:
(1) Operational insularity—the self-organization process is possible due to feed-
back based on the system’s inner laws: it is not guided from the outside;
(2) Informational insularity—any outer influence should be transformed into an
inner cerebral activity because such a system can respond only to inner activity;
(3) Physical openness—there is an exchange of substances and energy with the
environment, and its character is defined by the autopoietic system.
A system’s inner state forms the subject’s idea of the environment. However, there
is always an outer observer who could play the role of the subject observed. In fact,
this process of changing observers is limitless as there can be no last observer, i.e.,
an exclusive observation system does not exist. This means that absolute objective
knowledge about the world cannot be achieved. Thus, it is possible to conclude that
the achievement of some balance between reality and the result of a subject’s cogni-
tive activity is impossible in practice: since a subject constructs his/her worldview,
everyone has a personal reality. In this sense, a subject does not feel any bound-
ary between their own experience and their conception of reality. Hence, cognition
equates to adaptation: an object exhibits something that is correlated with the subject’s
cognitive activity.
According to the “observation of the second order” theory of Luhmann, observa-
tion can consist of characteristics that create a complex system [6]. Cells, organisms,
societies, and systems of artificial intellect are able to play the role of an observer. A
second-order observation (or an observation of an observation) is different because
of the circumstances of its observation, i.e., depending on whether an observer itself
has performed it or someone or something has been observing him/her. According to
the theory, an observer cannot see itself, i.e., it is not able to recognize itself during
the observation. At the same time, however, another observer can see it. In this case,
the object has the ability to see the observer and to see what is being observed. Luh-
mann defines this process as a second-order observation (auto-reflection concept)
and believes that it is these conditions that serve as the foundation of epistemology.
The concept of observation allows us to avoid the traditional epistemological
terminology that can be found in Aristotle’s works, in particular, the subject–object
scheme. For example, it is possible to consider the relationships between a system
and its environment while observing the system. “Countless discrete systems operate
inside a man as a condition of his life. They define the operations they implement by
means of their own structures, despite being dependent on each other” [6, p. 11].
Thus, this explanation promotes the notion that the world of observation is created
in the same way as any other system activities.
Accordingly, the answers to the ontological questions: “what is it?” and “what is
its state?” depend on a definite observer, who is restricted by the world of his/her
observation. We can say that the observer is similar to some empirical entity that
stipulates the selection of him/her as an object by other observers. It is obvious that
the subject and the environment in which it acts are interconnected constructively
and therefore experience mutual formation in the process of the subject’s activity.
It is possible to conclude that nowadays cognition is considered as an active pro-
cess of a subject’s activity. Firstly, cognition is the process of constructing the reality
perceived by a subject through his/her cognitive operations, thus explaining why the
knowledge that a subject receives, is a construct of the reality rather than of its reflec-
tion. According to Apel [7], the subject of perception is now becoming involved in
cognitive practice rather actively, as a result of interaction between perception means
and objects on the basis of “significant thought”. This category requires a symbolic
interaction, the nature of which is revealed most strongly in language [7]. There is
such a statement in the works of Habermas, where he considers “communicative
action”: according to him, this helps to overcome a number of antinomies, which
have not been overcome by previous concepts of rationality [8].
Thus, the cognition process organizes a subject’s inner world model, but it does
not solve the problem of how to describe the objective ontological reality. A subject
is considered as a part of the world that s/he is going to observe or as a part of some
system that is being observed by other observers. Thus, subject and object determine
each other reciprocally.
According to the interpretative constructivism theory of Lenk [9, 10], the inter-
pretations connecting cognition and action run through all approaches to the analysis
of the world and the subject of cognition: “Cognition is an action, but action is an
interpretation… Hence, cognition is an interpretation. It means the process of cog-
nition uses the results of the interpretation process or, to be exact, manifests itself in
them” [9, p. 18]. Interpretation is not realized only by means of language as it is a
more complex process:
We think we might somehow grasp things immediately by language without taking into
consideration that these matters are in reality much more complex. Language developed
mainly as an instrument for preparing actions which we use in a similar vein like other
means and instruments [10, p. 3].
According to these ideas, any view of reality depends on theoretical concepts,

specifically the linguistic models that dominate in the society concerned. They
include fundamental ideas, axioms, formal and linguistic instruments, and so on.
The concepts of how we ‘grasp’ these ‘realities’ of different sorts and levels will be one
main focus. …methodologically speaking, we need a systematic approach to the processes
of how the phenomena of knowledge and (re)cognition, perception, action and meaning can
be analyzed from a systematic point of view [10, pp. 4–5].
50 N. Danielyan
Hence, interpretation is an activity to create the so-called “mind constructs” which

involve both ordinary and scientifically conceptualized structures. This approach
allows the making of a model that contains only the aspects relevant to a subject’s
purposes and its actions. The subject does not care about the model cognized: he/she
thinks only of compensating the deviations while trying to achieve the appropriate
purpose.
As a result, any perception of reality will depend on theoretical concepts that
prevail in the surrounding society, including fundamental ideas, axioms, formal and
linguistic tools, etc. Hence, the preconditions to the formulation of modern ideas
concerning the possibility of knowing the world are as follows:
1. What a person takes to be the experience of the world does not dictate the concepts
which can explain the world.
2. The concepts which can explain the world are products of historically situated
interchanges among people.
3. The degree to which a given form of understanding prevails or is sustained across
time is dependent on the vicissitudes of social processes (e.g., communication,
negotiation, conflict, rhetoric).
According to this new approach, forces and relations of practice co-construct
each other. This term, “co-construction”, belongs to recent constructivist technology
studies, which have rediscovered Marx’s idea of social and technical interdependence
[11]. Contemporary technology studies offer concepts useful for analyzing these
developments and also for identifying what they hide. In this domain, technology is
conceived not as a pure product of inventive genius or an application of science, but as
a “construction” of social actors. This leaves room for social choice between different
designs that have overlapping functions, but better serve one or another social interest.
This means that context is not merely external to technology, but actually penetrates
its rationality, conveying social requirements into the human practical activity.
According to Sukhodub [12]:
Rationality in modern conditions serves as a method to develop critical thinking that is
able to estimate crisis phenomena and to conceptualize the ways of overcoming them in
social, economic, political, cultural and other spheres. It helps to make a conversion from
authoritarian ideas that function formally in the society to the dialogic mind that is perceptive
to ‘Others’ experience [12, p. 531].
Thus, it is possible to conclude that modern thinking culture is formed as a tran-

sition from the postulating mind, which is based on some sustainable content of
ideologies prevailing in the society, to the communicating mind, which is founded
on the multiplicity of viewpoints and openness of the rational cognition system.
4.3 Practice as a Unification of Subjective and Objective

Reality
Practice is the foundation of all forms of human public activity. Its openness to
the external world and its possibility to master new layers of objective reality in
the process of practical transformative actions suppose an opportunity for regular
development of a subject’s practical skills, e.g., his/her creative attitude to reality, or
participation in various forms of social life. It leads to regular revision and perfection
of the fundamental programs, which form the basis of practice. As a result, all the
history of mankind, including material and spiritual culture, can be considered as
a process of human creative approaches to nature in the form of new programs of
practical activities.
According to Hegel, subjective and objective reality contradict each other at the
very beginning of the cognition process, as they constitute opposites. The life process
starts with the individual necessity “to remove” objective reality that is opposed to
it and, thus, “objectify” itself. This can be achieved through a subject’s “violence”
to an object [13, p. 824]. It is possible to suppose that practice can be taken as a
kind of objectification of a subject. Marx writes: “as objective activity everywhere in
society becomes one of man’s essential efforts, all objects become his objectification,
realization of his individuality, his objects” [1, p. 121]. Thus, practice is the unity
of objective and subjective aspects of individual activity. It can be represented as a
complex net of various acts of object transformations with the “objectification” of
a human’s essence. Objects from one stage of this activity turn out to be the initial
ones for another stage or another activity program.
The structural characteristics of an elementary act of practice can be recognized if
the labor process is taken as a pattern. Marx in his work “The Poverty of Philosophy”
[14], discussing the cost of labor, reveals its following elements: man with his aims,
knowledge, and skills; operations of his rational activity; and objects included in
definite interactions in the course of these operations. Paying special attention to the
labor object, he considers, that “labour is qualitatively defined by object and object
in its turn is defined by specific labour qualities” [14, p. 32].
This pattern can be applied to the structure of practical activity and introduced
as the unity of subjective (man with his abilities, aims, and rational actions) and
objective (production means, initial materials, products of labor from initial mate-
rials) aspects of the cognition process. There are various forms of human practical
activity: according to Marx, the initial form of practice that lies in the foundation of
other kinds and forms of human life activity, in general, is material production as a
way of getting material welfare, which, in its turn, becomes the main driving force
of social development.
52 N. Danielyan
4.4 Technical Activity as a Special Form of Practice
Forming and developing public relationships is also a necessary form of practical

transformative activity: it does not influence the environment surrounding a human,
but it involves people themselves and their relationships. However, this form of prac-
tice is initially connected with the material production practice. According to Marx,
“the way of producing material life defines its social, political and spiritual processes
in general” [15, p. 7]. Hence, this view proposes that there is a unified practical activ-
ity that includes two aspects: man’s attitude to nature and his attitude to himself. It is
worth noticing that during public interactions these aspects differentiate further and
further since, due to the universal character of public activity, the subject can be both
pieces of nature undergoing transformation and people whose “properties” change
because of their inclusion in various social subsystems interacting in society as an
integrated organism. Studying the first possibility, we can conclude that humankind
deals with the human influence on nature, but the second postulated subject concerns
the “subjective” character of practice as aimed at the transformation of social objects.
From this point of view, humans can be considered as both the subject of practical
activity and as its object.
The early stages of social development witnessed the unification of subjective and
objective aspects of practical activity. As the labor process became more complex,
the operations that had been performed by man started to be materialized. They were
carried out as a sequence of influences of one tool on another and after that on the
object being transformed. Science as a special form of practice, having more narrow
social significance than production or social instances, gained an increasing influence
in society. It then turned into a direct production power and a means for control of
social processes.
Scientific cognition by itself possesses such qualities as a reflection of existing
objects and these can be received or reproduced by different forms of practical
activity. It also has projective and constructive functions, i.e., it provides knowledge
about the objects being mastered during production and during social activity in the
future. To check the validity of the knowledge requires a scientific experiment as a
special form of practice: as a rule, this is typically based on production opportunities
and human social experience within the current stage of development of society, but
it overcomes the contemporary level of entities and foresees future technologies and
ways of control of social life activity.
4.5 Social and Cultural Perspectives of Communication

Technologies
In view of the accelerating pace of technological advancement, it is appropriate

to consider ideas about the likely evolutionary paths of technologies that strongly
influence society (especially communications technologies) before returning to
consideration of the correlations with Marx’s concepts.
At first, it is necessary to understand what the term “trans-science” means. It is

often used by participants of international philosophical forums and conferences.
According to Weinberg, the trans-science period started with the first A-bomb tests
in the middle of the twentieth century. It stipulated the shift from researching nuclear
power by some scientists to team projects:
Many of the issues which arise in the course of the interaction between science or technology
and society, for example, the deleterious side effects of technology, or the attempts to deal
with social problems through the procedures of science – hang on the answers to questions
which can be asked of science and yet which cannot be answered by science. I propose the
term trans-scientific for these questions… [16, p. 209].
Biological effects of weak radioactive influence on the environment, consequences

of applying nanotechnologies to prolong the life expectancy, emergence of vir-
tual reality, creation of artificial intelligence, changes of human ethics under new
technological impacts are among such questions.
The main trans-scientific peculiarity is to unite problems from various fields of
human knowledge, i.e., it is based on a trans-discipline approach. An outstanding
current example is emerging and spreading of the Internet technologies. In compari-
son with rather rational technological methods, used in the past, modern technologies
are able to provide for numerous negative effects due to the availability of mecha-
nisms possessing an opportunity to manipulate the human mentality. Moreover, there
has been no comprehensive reflection of their properties so far. So, they might be
accepted as the starting point of philosophical deliberations due to the fact that the
traditional approach to understanding the technology cannot reflect all the range of
arising challenges. As a result, the main question is how all the spectrum of tech-
nologies and the social life are changed with penetration of the Internet technologies
into absolutely all aspects of human existence.
Scientific knowledge, as viewed from the position of trans-science, is understood
as suggesting probable hypotheses, which pass through bifurcation points where the
choice of trajectory for further development occurs. This subject of research particu-
larly addresses complex, dynamic systems that include technical, managerial, social,
and other levels. Any cognition process turns into a social act, since communica-
tion among representatives of different knowledge fields stimulates the emergence
of special norms and standards that are not connected with a specific author, but
acknowledged as valid by all the scientific community involved in the process. As
a result, they become a characteristic of a specific public style of thinking, with a
consequential transformation of the communication language, which then acquires
a universal character and influences linguistics and linguistic methods. These norms
and standards permit cognizance of the complete range of science functions:
• To analyze languages of different knowledge fields.
• To discover “scientific discourse as a net of communications with their mutual
intentionality and inter-reflection” [17, p. 496].
• To study natural sciences in the context of communicative relations: it is plausible
that this view is also correct for social and humanitarian sciences.
54 N. Danielyan
However, these models have a number of drawbacks. Their acceleration, the exten-
sion of their mobility and flexibility and their reduction of local attachment stimulate
the arrival of location-independent micro-societies and new social institutions of
global communications that can exchange information globally, e.g., in the form of
text messages or, more powerfully, as social media. According to Nasarchuk, “man
becomes a message generator” [18, p. 69], hence, a message performs a role as the
fundamental item of the society, not a human. The whole set of messages, depend-
ing on their intention and content, form the lifestyle of a person, a micro-society
or a social institution. It does not matter who is a message carrier, because it is
the content that gains a dominant significance. This means that its content and its
author’s (supposed) competence are more important than its sender. Yet, a message
and its perception are not always correlated with each other. The example of Internet
technologies indicates a conclusion that a sender often knows nothing about his/her
message recipient. In turn, the recipient is often not sure that the message has been
sent to him/her personally. The recipient thus frequently cannot deduce what should
be perceived as the message information as contextualized by some intention, i.e., the
objectification of the discussion acquires some fuzziness. Antonovsky and Emelin
suggest that “communication on the society level is still possible, but can’t form
self-adjusting stable successions of messages” [19, p. 106], since the knowledge
presented loses its discrete boundaries and metamorphoses regularly.
According to the above conceptions, it is possible to highlight the following
social and cultural perspectives of trans-humanism predictions under the influence
of communication technologies:
• Forming a new lifestyle.
• Presenting a possibility of the phenomenon of “secularized eternity” in the public
consciousness as a result of blurring the borders of reality.
• Changing man’s life goals due to his rapidly evolving status as creator of natural
and social worlds.
Let us consider the final thesis in detail. An active role of cognition is the most
important aspect of the application of these concepts to trans-humanism methodology
[20]. It supposes the dominance of activity of the mind in all levels of perception,
while nonstructured and nonclassified sensor data are absent.
According to the ideas of trans-science [21], modern technologies will pro-
vide new opportunities to create artificial intelligence and, as a result, to realize
new social forms and psychosocial processes. The website of the Russian Trans-
Humanism Society [22] cites some of its “radical possibilities” that could follow
from the abovementioned trans-science issues and ways of their realization, such as
the following:
• Super intellectual machines.
• Extension of life expectancy significantly.
• Transfer of human consciousness into virtual reality (or brain downloading) and
similar.
Trans-humanism manifests itself in three aspects: practical activity; technologi-

cal achievements and social transformations. Trans-humanism can be considered as
a synthesis of discrete doctrines being applied in technological practice, especially
in the field of communication technologies. In this conception, a human being is
substituted by a virtual existence, i.e., there is the potential elimination of anthro-
pocentrism principles, which were laid in the foundations of traditional humanism
and traditional science.
Thus, the exploration of trans-humanism issues needs to be coordinated with the
“stable development” conception [23], which can be understood as the necessity of
the immediate protection of the environment, on the assumption of further conver-
gence of natural and humanitarian sciences aimed at getting a more comprehensive
understanding of a viable forward path. This approach raises a question concerning
the boundaries of man’s constructive activity, its involvement, and correspondence to
the real world. A human being is becoming more and more “technological” with the
evolution of communication technologies; however, s/he doesn’t stop being sentient:
the human, its body and consciousness turn into an integral part of complex eco-,
sociocultural, and socio-technical systems [24].
In the context of the developing ecological and humanitarian crisis, the prob-
lem of the human future can be considered as a task of preventing the degradation
of human and natural systems, achieving co-evolution of science and society, and
forming a civilization based on stable evolution by means of searching out sensible
answers on topical matters: this correlates with a comprehensive worldview outwith
trans-humanism. Communication technologies have a great potential to dominate
these issues and define future perspectives of not only science, but the whole of the
development of civilization. They can be both production tools and the final product
of human fabrication activity. That is why they are becoming more and more impor-
tant components of the production process, defining its character and content (e.g.,
“Industry 4.0” [25]). Communication technologies require a range of specific skills
from their participants and as a result, imply the dialectic unity of material artifacts
on the one hand and human skills and experience on the other hand.
4.6 Extrapolation into the Future
The development and introduction of new technologies have led to the emergence
of a new sociocultural reality that raises new ethical issues. They are closely con-
nected with the realization of various projects, e.g., complete description of think-
ing processes and perception of reality by the human brain; the slowing down of
aging processes; the opportunity of human organism rejuvenation; the develop-
ment of brain/brain or brain/computer interfaces; the creation of robots and other
devices possessing at least partial individuality; and similar speculations. Ethical
problems arise from the realization of the above projects and the ethical principles that
are conventionally followed nowadays will be transformed. Development and
56 N. Danielyan
penetration of these new technologies will provoke a cultural effect that is likely
to see the intensification of some ethical values and the devaluation of others.
For example, the future achievement of neurointerface technology would lead to
the unification of human and machine on a qualitatively new level. It could change
the level of virtualization of the human mind and social relations. Penetration of vir-
tual technologies into human sensibility will create the situation of “hybrid reality”,
which could obliterate distinctions between man’s virtual personality and his physi-
cal localization in a body. However, the current virtual world of social networks, and
especially social media, leads to egocentrism and the human’s preoccupation with
itself and its thoughts: the result of this can be the loss of relationships between the
human and reality. For this reason, discussion of change of the spatial conception
concerning the physical boundaries of interpersonal communication and identifi-
cation is very desirable. This change will involve a reconsideration of the human
presence in the communication environment: whether it should be treated in both
real and virtual forms simultaneously. Such an approach indicates a potential trend
toward a completely new phenomenon of human existence: the boundary between
real and virtual exists rather clearly nowadays.
Thus, sociocultural perspectives of the current development of communication
technologies include the following:
• The evolving appearance of a new lifestyle.
• Introducing a phenomenon of “secularized eternity” in public consciousness;
• Changing the meaning of human life in a substantial way as humans will be able
to feel themselves to be creators of their world.
Marx died in 1883. The telephone was invented in 1876, although the practical
deployment of the telegraph dates from 1838. Thus Marx was alive at the beginning
of near-instantaneous electrical communications, although the means of propagation
of ideas most familiar to him would have been printed works: books, newspapers, and
broadsheets. Extrapolation of his ideas to the circumstances of the present day must,
therefore, be seen in this light. For him, the industry would have meant the making
of hardware products and the concept of the “knowledge economy” would have been
difficult to embrace: it is significantly different from material production as a way of
getting material welfare (see Sect. 4.3, above). On the other hand, Marx’s assertion
“the way of producing material life defines its social, political and spiritual processes
in general” (Sect. 4.4, above) represents a more information-oriented view and his
view that the realization of any forms of practical and transformative activity occurs
only under conditions of social cooperation (Sect. 4.1, above) correlates closely with
modern concepts of social media and co-production of media artifacts.
4.7 Conclusions
The philosophical exploration of social and cultural results of current technological

development is becoming more and more topical. Today there is a real necessity to
bring out the distinctive features of these technologies and to analyze their impact
on social reality. It is also very urgent to consider a new approach to humanism,

as still understood traditionally, to clarify transformations of social values and the
meaning of human life in the perspective of these developments, to study new cultural
stereotypes emerging nowadays under the influence of communication technologies
and emerging so-called “virtual reality”, social media, and artificial intelligence.
Latour demonstrated the relation between technology and society [26] and the
theme of democratization of technology has recently begun to receive the attention it
deserves. As Latour has argued, the exclusion of technology from the social scientific
concept of society is untenable. But once technology enters the picture, the issue
of rationality appears in a new guise. It was Weber who introduced the concept
of rationalization to explain many of the processes Marx had earlier identified as
central to modernity [27]. Whereas in Marx’s works such processes were examined as
potentially opposing (capitalist or socialist), Weber argued that they were the same for
all modern societies. When there is a strong focus on technologies as there is today, a
basis for questioning Weber’s simplification emerges again. It is clear that the modern
world does not provide a unified view on the question of whether mankind is able
to get a single homogeneous outcome to both technological and social development.
The traditional practice activities as highlighted by Marx are blurring more and more
as there is no sharp division between objective reality and the subjects of practical
activity. The scheme “object– objective –practical activity–subject” as the foundation
of scientific cognition and the criterion for the correspondence between knowledge
and reality introduced by Marx has undergone a number of transformations in modern
epistemology. Communication technologies with their new sociocultural reality erase
differences between material and nonmaterial, nature and culture, theoretical and
practical, etc. [26]. We do not need them anymore as the transition to “hybrid reality”
and trans-humanism due to the equal interaction of living objects, social structures,
theories or scientific-research programs, etc., generates a network consisting of a
number of active elements. Its main characteristic is their collective actions. The
coordination of their functioning in the network occurs due to the connections among
elements, which makes all components interact.
It is possible to conclude that there is a special meaning in these circumstances
for such a form of practice as technical activity in the field of communication, and
the analysis of Marx continues to give useful insights. This can be directed to the
creation of not only material values and artificial environments, but also thinking,
cultural, and cognitive processes.
References
1. Marx, K.: Economic and Philosophic Manuscripts of 1844. Works, Marx, K., Engels, F., vol.
42, pp. 43–174. Publishing House of Political Literature, Moscow (1974) (in Russian). English
edition: Prometheus Books, New York (1988)
2. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423,
623–656, July, October (1948)
58 N. Danielyan
3. Lektorsky, V.A.: Classical and Non-classical Epistemology, 256 pp. Editorial URSS, Moscow
(2001) (in Russian)
4. Alexy, R.: Legal documentation as rational discourse. Russ. Yearb. Law Theory 1, 446–456
(2008) (in Russian)
5. Maturana, H., Varela, F.: Der Baum der Erkenntnis: Die biologischen Würzeln des men-
schlichen Erkennens, 217 pp. Scherz, Bern, München (1987). English edition: The Tree of
Knowledge: The Biological Roots of Human Understanding. Shambhala Publications, Boulder,
Colorado (1992)
6. Luhmann, N.: Erkenntnis als Konstruktion, 74 pp. Benteli, Bern (1988)
7. Apel, K.-O.: Types of rationality today: the continuum of reason between science and ethics.
Rat. Today 307–340 (1979) (Ottawa)
8. Habermas, J.: Aspects of rationality of action. Rat. Today 185–204 (1979) (Ottawa)
9. Lenk, H.: Handlung als Interpretationskonstrukt: Entwurf einer konstituenten und beschrei-
bungstheoretischen Handlungsphilosophie: Handlungstheorien interdisziplinar, pp. 18–317.
Karl Alber Verlag, München (1978)
10. Lenk, H.: Grasping Reality: An Interpretation-Realistic Epistemology, 282 pp. World Scientific
Publishing Company, Singapore (2003)
11. Feenberg, A.: Marxism and the critique of social rationality: from surplus value to the politics
of technology. Camb. J. Econ. 34(1), 37–49 (2010)
12. Sukhodub, T.D.: Rationality in philosophical tradition and modern culture: the problem of
non-incumbency. In: Kiyashchenko, L.P., Stepin, V.S. (eds.) Post-Non-Classics: Philosophy,
Science, Culture, pp. 519–538. Publishing House “Mir”, St. Petersburg (2009) (in Russian)
13. Hegel: Science of Logic, 912 pp. Publishing House ACT, Moscow (2018) (in Russian). English
edition: Prometheus Books, New York (1991)
14. Marx, K.: The Poverty of Philosophy, 160 pp. Publishing House “”, Moscow (2017) (in
Russian). English edition: International Publishers Co., New York (1999)
15. Marx, K.: A Contribution to the Critique of Political Economy. Works, Marx, K., Engels, F.,
vol. 13, pp. 1–167. Publishing House of Political Literature, Moscow (in Russian) (1959).
English edition: Lawrence and Wishart, London (1969)
16. Weinberg, A.: Science and trans-science. Minerva 10, 209–222 (1972)
17. Ogurtsov, A.P.: Philosophy of Science: XXth Century: Concepts and Problems, vol. 1, 744 pp.
Publishing House “Mir”, St. Petersburg (in Russian) (2011)
18. Nasarchuk, A.V.: Net society and its philosophical understanding. Russ. Stud. Philos. 7, 61–75
(2008) (in Russian)
19. Antonovskiy, A.Y., Emelin V.A.: Informational society and the technology of adaptation to its
structure. Epistem. Philos. Sci. XXXI(1), 90–107 (2012) (in Russian)
20. More, M. Transhumanism: Toward a Futurist Philosophy. Originally published in Extropy
#6 (1990, revised 1996). https://www.scribd.com/doc/257580713/Transhumanism-Toward-a-
Futurist-Philosophy
21. Danielyan, N.V.: Internet technologies and trans-science: orientation for future. In:
RPTSS2017—International Conference on Research Paradigms Transformation in Social
Sciences, pp. 246–253. Future Academy, UK (2018). https://doi.org/10.15405/epsbs.2018.
02.28/
22. Russian Transhumanist Movement. http://www.transhumanism-russia.ru/
23. Los, V.A., Ursul, A.D.: Stable Development, 254 pp. Publishing House “Agar”, Moscow (2000)
(in Russian)
24. Danielyan, N.V.: Scientific Rationality. Publishing House MGOU, Moscow (2010)
25. Lasi, H., Fettke, P., Kemper, H.G., Feld, T., Hoffmann, M.: Industry 4.0. bus. Inf. Syst. Eng. 6,
239–242 (2014). https://doi.org/10.1007/s12599-014-0334-4
26. Latour, B.: On recalling ANT. In: Law, J., Hassard, J. (eds.) Actor Network Theory and After,
pp. 15–26. Blackwell, Oxford (1999)
27. Weber, M.: Gesammelte Aufsätze zur Wissenschaftslehre. Mohr, Tübingen (1968); Wirtschaft
und Gesellschaft, Bd. 1. Mohr, Tübingen (1956)
Chapter 5
Use of Digital Holography to Re-Encode
and Image Chinese Movable Type
Printing
Shuo Wang, Ardeshir Osanlou, and Peter Excell
Abstract In the long history of the development of civilization, Chinese movable

type printing has played a pivotal role for disseminating information and it can be
seen as the ancestor of the modern information age. From traditional engraving to
type, from clay to lead type, from laser printing to 3D holographic printing, the tech-
niques of encoding have been gradually developed and recorded, and these printed
texts convey the progress of mankind. A brief introduction to encoding in the con-
text of Chinese cultural communication is presented and, specifically, the role of
encoding will be explored. In addition, digital holographic printing is introduced as
a new approach and new medium to explore the enlightenment conferred by Chinese
movable type printing on today’s society and especially the information revolution.
Further, the future of encoding through the advanced technology of digital holo-
graphic printing is discussed. Two holographic artworks are presented to illustrate
the principles expounded and the response to their display in national museums
discussed.
Keywords Chinese movable type · Digital holography · Holographic printer ·

Printing technologies · Textual encoding
5.1 Chinese Movable Type Printing
5.1.1 Introduction to Chinese Movable Type Printing
The invention of Chinese movable type is a remarkable milestone in the history of

printing, as it ushered in a new printing era, well before the well-known developments
in Europe. Chinese movable type printing has been used for nearly one thousand years
and greatly promoted the development and exchange of world culture; in addition,
S. Wang
Beijing Institute of Graphic Communication, Beijing 102600, China
A. Osanlou · P. Excell (B)
Centre for Ultra-Realistic Imaging, Wrexham Glyndŵr University, Wrexham, UK
e-mail: p.excell@glyndwr.ac.uk

62 S. Wang et al.
it contributed to the historical progression of world civilization [1]. Movable type

involves making individual movable characters in advance, which can then be chosen
according to the manuscript, arranged in line on plates and finally printed on paper. All
of the movable characters may be used repeatedly, as each of them can be separated
and rearranged.
According to historical records from the ancient book MengXiBiTan [1] (see
Appendix), Chinese movable type originated in the Qingli period (1041–1048 CE),
during the Song Dynasty, with clay movable type characters invented by Bi Sheng
(Fig. 5.1) [2]. It should be noted that this was around 400 years before the introduction
of movable type in Europe by Gutenberg [3]. Later, with the progress of technology,
movable type characters made of wood, tin, lead, copper, and further derivatives
appeared [4]. Figure 5.2 shows the Uighur wood movable type examples which were
found in cave sediments of the northern Dunhuang area of China in 1908: these were
the earliest surviving examples of wood movable type in the world [5].
The invention of Chinese movable type printing enabled the simplification of
the process of printing plate creation, reducing labor requirements, and improving
production efficiency. However, Chinese movable type printing did not immediately
replace engraving-type printing until the Qing Dynasty, when the government began
to promote text-based cultural transmission. After this, the number of movable type
books greatly increased. Printing techniques also varied, as the Qing government
directly organized human labor and material resources to support the development.
Fig. 5.1 Replica of clay movable type characters based on Beisheng’s technology, in the Chinese
Printing Museum
5 Use of Digital Holography to Re-Encode and Image Chinese … 63
Fig. 5.2 Uighur wood movable type examples, as found in 1908
For instance, during the Yongzheng period in the Qing Dynasty, copper movable type
was used to complete as many as 160 million words on the integration of ancient
and modern books [6] (Fig. 5.3). Further, in the Qianlong period, wooden movable
type was used to print several thousand volumes of the Hall of Martial Valor book
[6] (Fig. 5.4). Government participation was central in promoting the application of
movable type and cultural communication with other countries.
Today, movable type, embracing many of the ideas of the ancients, still has great
value. Its most valuable point is its encapsulation of creative ideas, that is, the transfer
of whole-plate engraving printing into individual units, whether phrases or characters.
In today’s language, this may be called a code, symbol, or material carrier. This idea,
in computer typesetting, is widely accepted as a general principle, which drives
forward modern printing techniques [5]. Moreover, movable type printing is still
popular among Chinese people.
Fig. 5.3 The reference book—Qingding Ancient and Modern Integration

64 S. Wang et al.
Fig. 5.4 The book—Hall of Martial Valor (Wuyingdian), one of the largest quantity of books
produced by wooden movable type printing in China
5.1.2 The Influence of Chinese Movable Type Technology

in the Wider World
Movable type was introduced into Korea not long after Bi Sheng invented it during the
Song Dynasty. The original reason for using such an innovative printing technique in
Korea was that Buddhist scriptures and Confucian classics imported from China were
in short supply. In the beginning, Korea used ceramic movable type characters, but
they were made in different sizes and this led to unsatisfactory printing quality until
the Goryeo Dynasty, when copper movable type was gradually developed, which not
only enhanced cultural identity between the two countries, but also promoted a new
development of Asian culture [7].
After the invention of Chinese movable type printing, Japan adopted the tech-
nique, mainly using wooden movable type printing characters to produce Buddhist
scriptures for use among monks. A large number of Japanese books were printed,
including the Japanese scripts modeled on Chinese cursive script [8]. Movable type
printing was a major turning point during the process of social development as it
improved the efficiency of mass book production and greatly promoted the dissem-
ination of scientific, technical, and cultural knowledge. The spread of knowledge
expanded from the aristocratic classes to the general population [8].
Vietnam commenced the use of movable type around 1841–1847 when the Viet-
namese Dynasty bought a pair of Chinese wooden movable type sets and printed
a large number of national legal books with it. This can be seen as the direct
manifestation of Chinese type printing in Vietnam [7].
Besides Korea, Japan, and Vietnam, Chinese movable type printing also had a great
influence on nations in other countries. In particular, Christian books in the Philip-
pines were originally made by the ethnic Chinese people in local printing presses;
however, after 1608, the Filipinos set up their own printing houses. In addition, Iran
was a transit station on trade routes, facilitating the introduction of China’s printing
technology into Europe and hence making great contributions to the westward spread
of the technology [6].
Europeans’ first contact with Chinese printing began in the Yuan Dynasty (1271–
1368 CE), with the paper money brought by European businessmen and travelers
returning from China [7]. Around 1450, A German inventor, Johannes Gutenberg,
derived inspiration for a form of movable type printing from the technology of wine
pressing machines [3]. Although this was more than four hundred years after Bi
Sheng’s invention, the European movable type printing techniques introduced sig-
nificant progress in the use of fatty ink, movable type made of lead, etc. Its mechanical
printing technology laid the foundation for modern printing. These machines greatly
increased the speed of production and promoted the formation and dissemination of
new ideas: this became the driving force of modern civilization, and advanced the
modernization of European society.
5.2 Encoding in the Context of Chinese Cultural

Communication
5.2.1 Re-Definition of Encoding Concepts
Encoding in semiotics can be interpreted as the process of creating a message trans-

mission from an addresser to an addressee [9]; in psychology, encoding can be
interpreted as the process of entering the memory system, storing, and subsequently
retrieving [10]; in computers, encoding can be interpreted as the process of efficiently
transferring or storing a sequence of characters (letters, numbers, punctuation, and
symbols) into a professional format. In the book “Encoding and Decoding in the
Context of Television Discourse,” by contemporary cultural researcher Stuart Hall,
encoding is interpreted as meaning production of a media message and this plays
an important part in cultural communication, and similar processes [11]. Based on
these concepts, encoding is a specific process and has different definitions in different
discipline areas.
The present work attempted to identify the encoding of movable type printing
in the context of Chinese cultural communication, hence encoding is defined for
the present purpose as a transferring process from one role to another during the
historical development of China’s movable type printing. In a physical view, movable
type printing breaks the earlier whole-plate engraving printing into individual units
and this may happen during the processes of deconstructing phrases or words. In the
psychological view, movable type printing evolves from transmitter to protector of
information (in cultural context), and from protector to inheritor.
66 S. Wang et al.
5.2.2 The Role of Enlightenment
The appearance of words can be regarded as the gestation phase of encoding. The
invention of words was a great advancement for human civilization. In primitive ages
in China, humans tied a number of different knots or different shapes of knots on
ropes to express specific meanings: the so-called knot note (Fig. 5.5). Afterward,
ancient humans portrayed certain things on stones and stone walls to convey certain
meanings. This kind of pictogram directly gave birth to the origin of human writing:
the archeological discoveries of ancient Egyptian hieroglyphic characters, Sumerian
cuneiform characters, Chinese characters, and similar are all such.
Chinese characters have a long history, but the exact time of their emergence is
still a historical mystery [5]. Judging from the archeological findings, the Oracle
script system was fully developed in the Shang Dynasty, around 3500 years ago.
Chinese characters are ideograms whose form and meaning are closely connected.
Early characters were of pictographic form: with such a glyph, people could easily
know what it stood for.
The usage of Chinese characters is closely linked with their transmission medium,
but there were some limitations for expedient information transmission [12], for
instance, scattered oracle bones were composed of animal bones with engraved sym-
bols (Fig. 5.6); bronze inscriptions were heavy and difficult to move (Fig. 5.7); the
words on bamboo slips were hard to modify (Fig. 5.8); silk as a writing substrate was
expensive and susceptible to be eaten by insects (Fig. 5.9). The invention of paper was
convenient for text recording, but it was not able to change the transmission mode
of manual copying and thus still constrained the efficiency of transmission. Until
the invention of whole-plate engraving printing, information could not be spread
widely. Compared with manual copying, engraving printing had great advantages in
saving time and improving efficiency, but it also had shortcomings: it was very costly
as each version of the text required creation of a new set of plates. However, with
the invention of movable type printing, whole-plate engraving printing was divided
Fig. 5.5 An example display of a knot note

Fig. 5.6 Oracle bone with

inscription in the China
Printing Museum (late 2nd
millennium BCE)
Fig. 5.7 Bronze tripod bowl

cast for Duke Mao during the
King Xuan period of the
western Zhou Dynasty (828
BC–782 BCE): the vessel
has 497 characters in 32
lines. (China Printing
Museum display)
68 S. Wang et al.
Fig. 5.8 Words on bamboo slips in China Printing Museum (140–110 BCE)
Fig. 5.9 Lao Zi silk manuscripts in the China Printing Museum, dated around 2000 years ago
into separate re-usable units, overcoming many of the above problems and starting
widespread information transmission.
5.2.3 The Role of Encoding
With the development of information dissemination, the role of encoding evolved

from transmitter to protector then to inheritor, or in a global view it can be considered
as a navigator, directed toward civilization.
Whether as whole-plate engraving or movable type printing, the most obvious
role at the beginning is as the information transmitter. In this process, information is
transferred from one form or format to another. It is also a medium for a communicator
to translate its message or meaning into languages, images, symbols, and so on. In
view of relationship between language and literacy, printing provides a platform for
converting spoken language into written language, which promotes the development
of language, especially the development of written language in the Chinese literary
and grammar frameworks. Considering the relationship between written language
and communication efficiency, movable type printing provides an acceleration effect,
and a new language experience can be built on its platform. This process of evolution
with experience is also a role of the transmitter. In other words, the transmitter
influences the efficiency improvement between language and literacy.
If the transmitter only transmits information and lacks cultural connotation, it
cannot assist in advancing human civilization. Therefore, during the encoding pro-
cess, the dominant role is held by the transmitter while the recessive role is that
of the culture protector. In ancient times, the advancement derived from printing
could be identified with history, religion, politics, military matters, and many other
issues where printing had important historical and cultural values. The role of pro-
tector is crucial in the transfer process, for instance there are still preserved the Vajra
cchedikā Prajñā Pāramitā (in English: The Diamond Sutra) (Fig. 5.10) based on
engraving printing. This was made in the Tang Dynasty (868 CE), while the ency-
clopedia MengXiBiTan [1] (see Appendix), based on movable type printing, was
written in the Northern Song Dynasty (written c. 1086–1093 CE) [6]. There can be
little doubt that the ancient imperial system also contributed in establishing special
rules and orders for specific cultural preservation.
Protection aims at re-dissemination and then assured inheritance. As the founda-
tion of modern printing, movable type printing is the basis of pre-digital era printing
technology (digital printing can be argued to be closely connected—see Sect. 3.1).
Fig. 5.10 Vajra Cchedikā Prajñā Pāramitā (The Diamond Sutra), printed in the Tang dynasty: this
is a well-preserved piece with the name of the carving master and the year of carving (868 CE), it
was discovered in the cave of Dunhuang in 1900. The picture on display is of a replica
70 S. Wang et al.
Fig. 5.11 Brush pressing by a local craftsman onto wooden movable printing type in the Rui’an
area of Zhejiang Province
Here, another role of encoding is linked to deep ethnic cultural conventions. For
instance, wood type printing was derived from the custom of renewing genealogy
records, especially in the southern part of Zhejiang Province, China: the clan concept
of root-seeking remains quite strong there. Every clan’s family tree is renewed every
12 years or so. Figure 5.11 shows the use of wooden movable type printing in the
Rui’an area of Zhejiang Province. In the wider view of communication theory, the
expansion of social and cultural communication has promoted the popularization of
education and the exchange of knowledge. Additionally, mass production of printed
books increases the chances of book retention, reduces the possibility of handwritten
copies becoming extinct due to limited collections, and has a huge positive influence
on cultural heritage.
Looking at the role of movable type printing in the history of civilization, it has
always guided humans toward civilized societies. As a driver of social progress,
movable type printing has effects on almost every aspect of modern civilization. Due
to its roles as transmitter, protector, and inheritor, new ideas are quickly disseminated
and popularized, and speculative thoughts are promoted.
5.3 Digital Holographic Printing
5.3.1 Modern Digital Printing
The rise of computer-mediated printing has caused a radical change in printing tech-
nologies, although this did not happen immediately. For many years the standard
computer printers were the line printer and the electric typewriter, both of which
used metallic characters of a similar form to movable type, although not assembled
in the same way as in a printing press: the daisy wheel printer can be said to be
similar. Toward the end of the twentieth century, however, raster-based printers (e.g.,
laser and inkjet) became common and now totally dominate the market. These do
not use hardware type characters in the same way as the earlier technologies, but the
philosophical difference is not so great, since they store a vast range of characters
in software, made available to the user as “fonts”, exactly mimicking the jargon of
the traditional movable type printing technician. The fundamental point here is that
human readers have become completely attuned to the use of discrete characters, as
established by movable type, and the raster-based systems have to display text in the
same way for it to be meaningful.
A more radical difference with this technology is the ease of inclusion of
illustrations, as these can be printed with exactly the same raster technology.
5.3.2 Digital Holographic Printing
Holograms can represent objects and scenes in three dimensions without the need for
special viewing glasses. Computer-generated or computer-mediated holograms are
intrinsically digital and hence also have a raster structure, but in three dimensions,
hence the two-dimensional raster pixels are replaced by three-dimensional voxels
in the software. However, the hologram is a two-dimensional object which contains
the three-dimensional information. This means that a different implementation of
raster pixels is required. Thus digital holographic printing is composed of a matrix
of holographic pixels, also termed holopixels or hogels. Holopixels are created by the
interference of three-color lasers in a holographic medium, such as a photopolymer
or silver halide film.
A digital printed hologram can visually represent an object in a realistic way and it
also can be a creative medium to re-interpret the object in a narrative way. In addition,
digital holograms can be duplicated repeatedly through a digital holographic printer.
A digital holographic printer includes several components, such as the data source,
tri-color lasers, a spatial light modulator (SLM), beam-steering optics, and a computer
to show the images on the SLM and to control the whole system. The data sources
should include a selection of resources, such as general artwork and illustrations,
photographs taken by people, or digital imagery produced by software such as Maya,
Sketch-up, 3D Studio Max, point clouds, and laser scans. The laser-writing scheme
utilizes an object and reference beam pair. The object beam is modulated through
the SLM using two-dimensional (2D) information synthesized from a scene.
Digital holographic printing as an artistic medium has been practically developed
and applied in cultural presentations. For example, Fig. 5.12 shows a holographic
artwork created by author Shuo Wang: in this artwork, the digital printed hologram,
as an innovative medium, not only enables the viewer to recover an archeological
excavation space as it appeared in 1980, and to have a realistic spatial visualization
of it, but also the losses of archeological detail owing to the difficulties of conditions
72 S. Wang et al.
Fig. 5.12 Digital holographic artwork representing a reconstruction of an archeological scene, by

Shuo Wang
at that time could be bridged, bringing a profound influence upon archeological,

historical, geological, and artistic representation of the priceless cultural relics that
were recovered [12].
5.3.3 The Evolution of Chinese Movable Type Printing

to Digital Holographic Printing
Understanding the past is an essential component for projecting developments in the

future. Chinese movable type printing has developed in the course of its historical
changes. The process has been complex and diverse with both positive and nega-
tive aspects, which are closely related to the social system, philosophies, aesthetics,
and humanity of Chinese society in the context of historical development at the time
[13]. With the initiation of the modern information age, the rise of digital holographic
printing also needs to be explored in the context of today. Since the development
of digital holographic printing and movable type printing occurred in very different
historical periods, it is hard to compare them. However, if they are regarded as trans-
mission media and linked in their aspect as information carriers with their respective
medium characteristics, then they appear to have some similarities in particular peri-
ods in their history. Thus, some comparisons and suggestions of similarities may be
attempted:
On the positive side:
1. They both are carriers of information, that is, media.
2. In different historical periods, both movable type printing and digital holographic
printing have provided a new information dissemination platform for societies
and have promoted language development globally. Compared to movable type
printing, digital holography carries more information or data and has the potential
for the inclusion of highly realistic graphics.
3. Both movable type printing and digital holographic printing have opened up a new
mode of information transmission in their respective times. With the development
of movable type printing, efficient replication using various materials has been
created; digital holographic printing will also replicate with more materials in
the future, not be limited to glass and photopolymer, but the digital generation of
characters gives infinite scope to replicate not only the entire Chinese alphabet,
but characters from other languages, mathematical symbols and new, fictitious,
creations.
On the negative side, neither of them has had a great speed of popularization.
Chinese movable type printing was invented in the Song Dynasty but it was not widely
used to replace engraving printing, which required intensive labor [4]. Currently,
digital holographic printing is not widely used either. The main reasons are as follows:
1. From the perspective of aesthetics, Chinese scholars advocated the beauty of
carving, so, movable type printing was not rapidly developed after engraving
printing [14]. Similarly, the digital printed hologram is not at present as realistic
as the traditional analogue hologram, and it cannot provide the same beauty and
verisimilitude. This deficiency is expected to disappear as holopixel resolution
increases.
2. From the perspective of the development period, after the invention of engraving
printing, people constantly improved its technology. In the Song Dynasty, it
was the golden era of engraving printing when Bi Sheng invented movable type
printing [1]. Digital holographic printing also seems to have its rival technologies,
such as head-mounted virtual reality displays, but the usage scenarios for these
are very different.
3. From the perspective of economics, due to the features of Chinese characters,
it is difficult for ordinary people to invest in the necessary labor and material
resources (excepting the very wealthy) as it is very costly for one-time use.
Before the emergence of movable type printing, there were numerous engraving
workers. Because of that, the cost of engraving printing was lowered, but the
cost of movable type printing was effectively increased. At present, it costs a
great deal in time and money to produce a digital printed hologram, and digital
holographic printers are at present very rare.
4. From the perspective of media-use habits, after movable type printing was
invented, handwritten and engraved books were still generally used at that time.
Now digital color printers and 3D display monitors are also competitors for
digital holographic printing.
74 S. Wang et al.
5.4 The Exploration of Encoding in the Future
The encoding of digital holographic printing can be seen as paralleled by movable

type printing in some cases. In view of the medium features and advanced appli-
cations, digital holographic printing will also experience the roles of transmitter,
protector, and inheritor. However, due to the rapid growth of information and the
diversification of information media, digital holographic encoding ideally needs to
be combined with artificial intelligence, so as to optimize it. For example, throughout
the whole process of digital holographic modeling, this process can be performed
with a large number of databases based on people’s creative ideas, functioning as
a replacement of manual modeling, improving the models, material, lighting, ren-
dering and other steps, and designing the frame narrative through smart thinking.
During the printing process, smart devices can hasten transmission speed, shorten
printing time, and achieve efficient replication.
5.5 Artistic Works
5.5.1 Three Manifestations of a Chinese Movable Type Text
Based on the research on encoding of movable type printing, as well as the relation-
ship between movable type printing and holographic printing from the perspective of
media analysis, an artwork entitled “CUBES” was created to further explore the dif-
ferent roles in the future. The name “CUBES” involves two levels of significance: the
straightforward meaning is descriptive of the artwork represented by different pieces
in the form of small cubes; secondly, CUBES is an acronym, meaning “CUlture
Broadly Encoding Spreading”. The work was reproduced in the form of collocated
wooden movable type, 3D printed movable type and holographic movable type, and
the record of movable type printing in MengXiBiTan [1] was taken as the con-
tent (see Appendix). This follows the philosophy of the four “RE” creation methods
(Re-definition, Re-creation, Re-construction, and Re-presentation) devized by author
Shuo Wang [12], as well as holographic montage ideas. Each individual text com-
ponent has been created in the 3D software: initially, the length, width, and height
of the components have been calculated based on the wooden texts; furthermore, the
polygonal modeling method follows the style of Chinese wooden engraving and the
corresponding writing style, Fig. 5.13 shows holographic modeling of the content of
MengXiBiTan and Fig. 5.14 shows the holographic rendering result, which simulates
the effect of Chinese wooden engraving.
Fig. 5.13 The “CUBES” artwork: holographic modeling in Maya
Fig. 5.14 The “CUBES” artwork: rendering effect in Maya
5.5.2 Four Manifestations of Chinese Movable Type

in a Hybrid Timeline
This artwork (Figs. 5.15 and 5.16) explores the different roles in the context of
encoding. In addition, digital holographic printing was used as a new approach and
76 S. Wang et al.
Fig. 5.15 Artwork: Hybrid Plate as a timeline
Fig. 5.16 Interpretation of this artwork

Fig. 5.17 The psychological and philosophical view of encoding
new medium to explore the enlightenment conferred by Chinese movable type print-
ing on today’s information revolution. This artwork consisted of a wooden plate and
266 individual blocks, including wooden movable type blocks, analogue holographic
blocks, digital holographic blocks, and black blocks, these four parts (Fig. 5.2) briefly
present a timeline of the information dissemination from ancient times to the future.
In creating this work, the author Wang assumed two roles/identities in the
creation process: information receiver and information presenter. Furthermore, he
attempted to explore the relationships among transmitter, receiver, presenter, protec-
tor, and inheritor to address the psychological and philosophical views of encoding
(Fig. 5.17).
Considered as a receiver, the information received was drawn from the content
of the “Ancient Chinese Wikipedia”—MengXiBiTan [1], In the process of receiving
the information, the whole content has been summarized into 70 characters, and the
these characters have been engraved into wooden blocks, based on the traditional
method of Chinese wooden engraving: this aims to present the ancient method of
information transmission for audiences.
Considered as a presenter, two approaches have been explored, based on the
medium characteristics of analogue holography and digital printing holography. Ini-
tially, the 70 characters were produced using a Holo-Camera (Fig. 5.18): these holo-
graphic blocks can be seen as one of the modes of protection for wooden blocks (via
substitution, since the 70 wooden blocks are considered as heritage objects); addi-
tionally, they can encapsulate the moment of creation of the wooden blocks, which
provides a spatial view for audiences.
As a further insight, in the process of the digital holographic modeling, other
spatial aspects of the characters can be discerned, as well as the opportunity to re-
interpret the character in a narrative way. In this artwork, when each character was
78 S. Wang et al.
Fig. 5.18 Using a Holo-Camera to produce the analogue hologram
in the processing of being created, the structure revealed a layer of new meaning and
spirit.
In such an era, facilitating the production of the hybrid printing plates, each
individual block carries different dimensions of time and space, from the ancient
wooden type block to the latest digital holographic blocks. Thus technology, medium,
and culture have been developing together: they act the different roles in the process
of information dissemination. Hence the development of printing technology and
culture has always been developing in a synergistic way.
5.6 Conclusions
The historical development of Chinese movable type printing has been reviewed
and its influence on the World’s culture emphasized. The encoding of movable type
printing has been explored and this expanded to include the evolution of Chinese
movable type printing to digital holographic printing, including reinterpretations
of the concept of encoding. Art works have been presented that demonstrate the
coexistence of ancient movable type printing and modern digital holographic printing
and the insights that can be drawn from this. The power of the holographic image has
been demonstrated since, when the artwork was illuminated by the appropriate light
source, virtual (holographic) character cubes, and real character cubes were combined
together, thus presenting a new form of encoded space within the culture of human
communication. In addition, the artwork reflects the passage of thousands of years of
historical space and time, linked through the continuity of Chinese characters. The
re-encoding of Chinese movable type printing reveals that this printing technology
continues to develop from the past to the future—in digital holographic forms—with
the potential to inform new interpretations of human communication.
Appendix: Records of Movable Type Printing

in MengXiBiTan
MengXiBiTan is a book written by Shen Kuo, a scientist in the Northern Song

Dynasty. It was written around 1086–1093 (CE) and documented Shen Kuo’s views
and opinions in his life. It is regarded as an Encyclopedia of ancient China by western
scholars, with many foreign language versions published around the world.
In the Tang Dynasty, people had not yet adopted engraving printing to print books
on a large scale. In the Five Dynasties (907–960 CE), namely, the Later Liang Dynasty
(907–923), the Later Tang Dynasty (923–936), the Later Jin Dynasty (936–946), the
Later Han Dynasty (947–950), and the Later Zhou Dynasty (951–960), Feng Dao
began to print the Five Classics with the engraving printing technique. After that,
various classics and books were printed. During the Qingli period, a civilian named
Bi Sheng created movable type printing. His method was to use clay to engrave
characters and the thickness of each word was like the edge of copper coins. Each
character was a matrix, hardened by firing. Firstly, he set up a piece of iron plate,
covering it with turpentine and wax mixed with burnt paper ash. When printing, he
put an iron frame on the iron plate, then arranged the word (i.e., character) matrix
appropriately and closely. He filled the iron frame as one printing plate, then held it
near the fire to bake; when the rosin and other bonding materials began to melt, he
pressed the surface with a flat plate. In this way, the pattern on the plate was flat and
smooth. If people wanted to print only two or three books, this method would not be
convenient; but this method would be particularly efficient for tens or even hundreds
of books. Usually two iron plates were produced while printing. One plate was used
for printing and the other one for arranging the matrix. As soon as one plate finished
printing, the other one would be ready to use. The process was quite fast with two
plates. Each word had several matrix elements. For example, the Chinese words “
之” and “也” usually had more than twenty matrix elements in case of repetitive
characters in one plate. When the words were not used, they would be marked by
pronunciations with paper strips. Each rhyme part was marked with a label, then
stored in a wooden lattice. If an unfamiliar character was needed but unprepared,
people engraved it and then grilled it with a grass fire, hence it could be made very
quickly. The reason why people did not use wood to make movable type matrices
was that the texture of the wood varied between sparse and dense. Once stained with
water, it would expand, and easily got stuck to the frame, making it hard to remove.
Thus, clay was a better choice. After the matrix was used, the coating material could
be melted by fire. Wiped by hand, the matrix would fall off and not be contaminated
80 S. Wang et al.
by coating material at all. Bi Sheng’s matrices were understandably still treasured

by his cousins and nephews after his death.
References
1. Shi, Jinbo.: The history of movable type (in Chinese). J. Guyuan Teachers College (Soc. Sci.),
39–40 (2001)
2. Hao, Liu.: Movable Type: The Rebirth of Wood Movable Type—The development and Inno-
vation of Wood Movable Type Under the Modern Aesthetic Demand. Design (J. China Ind.
Des. Assoc. Beijing). ISSN 1003-0069. (2016)
3. McLuhan, M.: The Gutenberg Galaxy: The Making of Typographic Man (1st ed.). University
of Toronto Press (1962). ISBN 978-0-8020-6041-9
4. Zheng, Yefu.: The origin of movable type printing (in Chinese). Social Sciences of Beijing.
ISSN 1002-2054. (2015)
5. Shi, Jinbo.: The history of movable type (in Chinese). J. Guyuan Teachers College (Soc. Sci.),
41 (2001)
6. Shi, Jinbo.: The earliest extant printing products in the world (in Chinese). J. Beijing Lib.
(1997)
7. Wang, Xiaoxue.: Review and Reflection on the research of Gutenberg’s metal movable type
printing (in Chinese). Research on the History of Publishing in China, pp. 59–67 (2016)
8. Xu, Yinong.: The Significance of East Asia Typography in World History (in Chinese). Nanjing:
New Century Library (2016)
9. Encoding (semiotics). https://en.wikipedia.org/wiki/Encoding_(semiotics). Last accessed 14
March 2018
10. Encoding. https://www.alleydog.com/glossary/definition.php?term=Encoding. Last accessed
14 March 2018
11. Encoding and Decoding in the Television Discourse: Centre for Cultural Studies. University
of Birmingham, UK (1973)
12. Wang, S., Osanlou, A., Excell, P.S.: Case Study: Digital Holography as a Creative Medium
to Display and Reinterpret Museum Artifacts, Applied to Chinese Porcelain Masterpieces,
Chapter 6. In: Earnshaw, R.A. (eds.) Research and Development in the Academy, Creative
Industries and Applications, pp. 65–88. Springer, Cham, Switzerland (2017), ISBN: 978-3-
319-54080-1
13. Li, Bin.: Looking for the living fossils of movable type printing. Human. Photogr. 5–10 (2016)
14. Zhu, Qin.: Discussion On the Reason of the Slow Development of Ancient China Printing
(in Chinese). China Publishing Update (China News Publishing & Media Group, Beijing),
pp. 40–42 (2002)
Part III
Interactions Between the Arts and Data
Chapter 6
The FOREVER-DO Game: A Big Data
Fishing Expedition
Jill Townsley and Carlo Ferigato
Abstract The FOREVER-DO Game is a participatory visual artwork, that takes

the form of a relational game and installation. It is one outcome from an art science
collaboration supported by the European Commission’s Joint Research Centre in
Italy. The artwork was developed under the theme of ‘Big Data and Identity’ it
considers the causal chain that our individual and social actions may have as they
flow across the interface of the human and the digital network. The work focuses on
data flow that has some correspondence with coordination and communication, as
identified by Petri Nets theory. Carl Adam Petri introduced the Nets in the 1970s,
they are one of the techniques used today for the analysis of Process Data. When
data is ordered in time, the causal chains of data are collected in ‘processes’. One
simple representation of a process net is the ‘bucket chain’ used to represent two
fundamental aspects of data flow: selection and transfer. By using the bucket chain
as a platform this project is focussed on a discipline of data where some ‘causal
flow’, possibly represented by nets, has a role in the data analysis game. The resulting
artwork encourages the viewer/participant to flow through an installation following
a set of rules on a box, their actions coordinate according to the incidence of chance,
selection, transfer and the similarity or difference of everyday objects.
Keywords Petri Nets · Coordinated behaviours · Data · Flow · Communication

discipline · System · Time · Space · Repetition · Difference · Unfolding ·
Folding · Selection and transfer · Participation
J. Townsley (B)
University of Huddersfield, Huddersfield, UK
e-mail: jill@jilltownsley.com
C. Ferigato
European Commission, Joint Research Centre (JRC), Ispra, Italy
e-mail: carlo.ferigato@ec.europa.eu

84 J. Townsley and C. Ferigato
6.1 Introduction
Artistic activity is a game, whose forms, patterns and functions develop and evolve according
to periods and social contexts; it is not an immutable essence [1].
The FOREVER-DO Game is a socially relational public artwork or happening, in

which people flow around a space, moving between a network of coloured doormats,
according to a set of instructions printed on a box. This seemingly absurd game
activates interactions between players, who meet randomly in space and time. The
coordinated nature of the rules-based system dictates the path that the players travel,
it is in itself a socially related construct. A construct that rationalises data flow, mak-
ing visible the hidden nature of all data flowing within our big data systems. The
work presents us with a symbiotic relationship between scientific theory and art prac-
tice, while also raising questions about socially mediated, rules-based interactions,
especially those that cross the physical and digital interface.
The game is positioned within today’s digitally networked age, it mirrors the
theoretical construct of a ‘Bucket Chain’ network, a simple form of Petri Net system,
proposed by the computer scientist Carl Adam Petri in the 1970s. In theoretical
Computer Science, nets are instruments used for the analysis and design of systems,
distributed in time and space. The strength of these nets is their explicit representation
of fundamental situations of coordination and concurrency among system agents.
Agents can be anything: objects, computers and/or human beings.
The first manifestation of this work was played/exhibited during the Milan Digital
Week in March 2019, in the Palazzo dei Giureconsulti, near the Duomo, in collabo-
ration with the MC3 research group on concurrent systems at the University of Milan
Bicocca. The game was repeated in October 2019 for the Resonance III festival, at the
European Commission’s Joint Research Centre (JRC) in Italy, this time with addi-
tional digital visualisations developed through the application of Radio Frequency
Identification tracking (RFID). Trackers were placed inside each of the 2,000 boxes
used to play the game, the feedback from the trackers was detected by antennas with
the data then displayed digitally alongside the installation (Fig. 6.1). It is the first
presentation of the FOREVER-DO Game in Milan that we discuss here.
The cross-disciplinary collaboration between artist Jill Townsley and computer
scientist Carlo Ferigato was important to the way that the artwork has developed. The
work was commissioned by SciArt, an art science organisation embedded within the
JRC. The JRC is the European Commission’s science and knowledge service which
employs scientists to carry out research in order to provide independent scientific
advice and support to EU policy. The SciArt mission:
brings together scientists, artists and policy makers to discuss matters of concern from various
points of view, not only to the JRC and the European Commission but also more widely to
society [2].
Additionally, SciArt set the agenda of ‘big data and identity’ as the theme for the
collaboration, stating:
6 The FOREVER-DO Game: A Big Data Fishing Expedition 85
Fig. 6.1 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019
Big Data have erupted in our daily lives and we want to join the discussion on how to turn
Big Data into culture, not leaving it only in the hands of technicians, trolls and corporations.
We want to become hackers of our own future. We think it is time we make them ours, these
Big Data, this digital transformation, in a vast collective effort open to all [2].
To do this SciArt have proposed a new concept ‘DATAMI’:

We see the datami as a virtual tatami [a Japanese mat], made of the data that we cherish, our
data and those of our families and friends, our discoveries and curiosities, our roaming and
conversations, all of these making up our new identities [3].
This is the theme for the Resonances III Festival and Exhibition held at the JRC
in October 2019, and it is within this overarching context that the FOREVER-DO
project was formed (Fig. 6.2).
In order to help convey the significance of the FOREVER-DO Game we look to the
work of art critic and curator Nicolas Bourriaud and his concept of relational aesthet-
ics. This supports the identification of some formal qualities in the artwork especially
the interpersonal cooperation that is present within the FOREVER-DO Game and
the significance of this model of interaction as a creative tool, to critically reflect on
Fig. 6.2 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019
society. We also look to the media theorist, critic and philosopher Boris Groys and
his writing on ‘flow’. The flow of people and objects within the game is important
and Groys helps identify how flow may define moments of temporary happening.
Happenings that can have some governance over social interaction, presenting as ‘an
event not as a thing’ [4]. Groys’s theory helps us draw some conclusions about the
way that the game mediates the viewer/participant in order to highlight the interface
of the digital and physical realm, especially in relation to big data operations and to
net theory.
Finally, to help us consider how the artworks analogue process interacts with the
human digital interface we will be looking not only to Petri Nets theory, (as the
inspiration for the way that the game has been developed), but also reading Carl
Adam Petri’s writing as a critical tool. Looking specifically at his Communication
Disciplines delivered as a set of twelve conceptual tools. He proposed them in a
lecture in 1976 to support reasoning in communication systems design, and sug-
gested that they could help build the ideal computer system [5]. A system that could
enhance human cooperation and overcome some of the many problems he foresaw
for a computerised world going forward. Petri’s principal aim was to help realise
a computer system whose applications could support humanity to develop its best
possible future. It is this call from the 1970s that dovetails so significantly with the
concept of DATAMI proposed by the SciArt team and motivates the creation of the
FOREVER-DO project.
6.2 A Fishing Expedition—the Method
Meeting within the context of the JRC, and in response to the SciArt theme of
Big Data and identity, we (artist Jill Townsley and computer scientist/researcher
Carlo Ferigato) realised that ‘process’ held an important position within both of our
working practices. Ferigato’s research around the process design of communication
systems, specifically systems identified by Petri Nets theory, and Townsley’s artwork
emphasising the process of the art object, definitive beyond subject and object. The
process presented us with common ground to articulate our practices in language that
cut across our respective fields, producing a rich dialogue and exchange of creative
thinking. It identified important key words beyond the process itself, such as system,
time, space, flow, coordinated behaviour, repetition, difference, unfolding, folding,
selection and transfer.
Through analysing our processes, nets became a key link between the art and
science used in our collaboration. Put poetically, we became fishermen casting our
nets, to fish across disciplines into the sea of big data sets in order to extract coher-
ent patterns and procedures. We found patterns pertinent to Petri Nets theory, that
could be visualised as art. From this, we have developed two symbiotic artworks the
FOREVER-DO Game and the FOREVER-DO Infestation.
While we consider only the FOREVER-DO Game here, it is important to know
that the placement of boxes within the game became data that has driven another
artwork, the FOREVER-DO Infestation. The data, gathered from one piece of work
flows on to inform the realisation of another. In this way, authorship is shared with
every person who has played the game by placing a box on a pile.
While the FOREVER-DO Project as a whole (the Game and Infestation) explores
the idea of ‘fishing’ into data sets, the nets we use are multifariously allegorical.
Beyond being a fisherman’s tool, the word ‘net’ has reference to our digital networks,
the Internet. Nets are also the title given to the formal computer science tools ‘Petri
Nets’ (as outlined in the introduction). One of Petri’s examples, the ‘Bucket Chain’
is the main source of inspiration for the FOREVER-DO Project. It is outlined in
some detail in the theoretical context below but is an explanation of the mechanics
of the FOREVER-DO Game.
6.3 The FOREVER-DO Game—the First ‘Catch’
This artwork propels its human participants on a physical journey that mirrors a
‘causal flow’ of data (Fig. 6.3). The basic idea of causal flow is to understand how
factors influence one another. The game seeks to envision the invisible influence
between human action and data. As the game is played, a sculptural installation con-
sisting of towers of boxes develops in the gallery space. These box-towers simulate
a data pile, presenting a physical object that records the coordinated meetings of
each individual who (following a set of instructions on the box) encounters another
Fig. 6.3 The FOREVER-DO Game (2019 October JRC exhibition) Copyright © J. Townsley 2019,
included by permission
player at a coloured mat and finds (through random chance) that their boxes contain
identical objects.
The mechanics of the game are as follows. In a large room there are placed a set
of objects:
• Sixteen coloured doormats, placed randomly around the room, specifically,
– 4 Yellow,
– 4 Red,
– 4 Blue and
– 4 Black doormats.
• 2,000 boxes (made to hold a ream of A4 paper).
– 1,000 brown boxes—randomly dispersed through the installation,
– 1,000 white boxes—randomly dispersed through the installation.
• Each box has instructions (the rules of the game) printed onto the top.
– There are 6 sets of rules offering slight variations.
– The 6 sets of rules are randomly distributed (shuffled) throughout the 2000
boxes.
• Each box contains (randomly) one item of mass-produced wooden cutlery:
– a knife,
– fork or,
– spoon.
With this absurd combination of objects, the game is ready to be played.

Visitors to the exhibition are invited to pick up a box and follow the instructions
printed on the top (Fig. 6.4). The instructions send people on a journey travelling
Fig. 6.4 Boxes and

Instructions Copyright © J.
Townsley 2019
through the space from coloured mat to coloured mat. There are 6 sets of instructions,
all presenting a different order of coloured mats to visit:
(1) go to the closest BLUE mat;
(2) go to either a RED or to a BLACK mat;
(3) go to another mat that is the same colour as your present one;
(4) go to any YELLOW mat;
(5) repeat forever from rule one above.
Players physically ‘flow’ through the space of the gallery, moving according to
the rules on their box. The brown or white boxes contain an everyday object: a knife,
a fork or a spoon. When individuals meet at a mat, they open the box to compare the
contents. Depending on the ‘local’ circumstance around the similarity or difference
of those contents, players encounter one of two consequences:
1. They continue to flow around the game:
• this happens when their compared objects are different (non-repeated)—then
the objects are exchanged and the flow continues according to the rules.
2. They stop and exit the game:
• if the objects are the same (repeated) then players are asked to leave their box
and object behind, placed on a tower at the point of coordination.
Fig. 6.5 Players of the game Copyright © J. Townsley 2019, photograph Pierre-Stuart Rostain,
included by permission
In this way a ‘causal flow’ happens, when non-repeated coordination occurs,

objects are exchanged and continue their journey, independent of player or box—
flowing on, to continue the FOREVER-DO. Alternatively, when repeated coordina-
tion occurs (people meeting with the same objects), the flow is halted and boxes build
up around the mats. Over time the number of boxes in the towers gather randomly
around the mats, infesting the space and forming a sculptural installation (Fig. 6.5).
The installation makes visible a poetic data trail of coordinated human interac-
tions, recording the incidence of two people meeting at a mat, to find that through
random chance, their boxes contain identical objects. The coloured doormats act like
nodes in a giant network. Receiving the flow of information as people and boxes
move from mat to mat, enabling coordination and a place for human interaction
(Fig. 6.6).
As the box-towers grow, data may be extracted from the game in many different
ways. The piles of white and brown boxes could be interpreted as binary code.
The number of boxes clustered around a coloured doormat also give that mat an
integer value. From this data, new digital, conceptual and physical constructs may
be developed. Reorganised data is used to form a new sculpture: the FOREVER-DO
Infestation. Alternatively, a new event or happening could be derived as presented
in the FOREVER-DO Game 2. In other words, a new fishing expedition can begin
(Figs. 6.7 and 6.8).
More generally, the gathering, reorganisation and extraction of information, as
presented in the FOREVER-DO Game, is indicative of the way data flows today. Data
(collected in a myriad of ways) flows forever through our digital systems. Systems
and data unimaginably vast in form, yet invisible in time and space. This data can be
organised and extracted in whatever way society chooses: it can be reorganised and
Fig. 6.6 Players of the game and the box-towers Copyright © J. Townsley 2019 included by
permission
extracted to help us do great things for humanity, such as understand diseases; or it

can be used less responsibly, to smartly target us to purchase more mass-produced
goods.
6.4 Petri Nets, Communication and the Bucket Chain
The structure of the FOREVER-DO Game takes its principles from Petri Nets theory,
in experiential and analogue terms it makes visible a form of communication that is
happening across the interface of our physical and digital worlds. The installation
presents a data trail made from the flow of people around a room, flowing until random
instances of coordination happen between players. This has resonance with Petri Nets
theory, in its desire to rationalise information flow as a resource in communication.
The history of Petri Nets can be traced back to the Ph.D. thesis of Carl Adam Petri
in 1962 [6]. His aim was to establish a new theory of communication-based on two
assumptions:
(a) there exists an upper limit on the speed of signals;
(b) there exists an upper limit on the density with which information can be stored.
These assumptions can be considered as natural ones today, but they were not so
natural in the 1960s, when the theory of abstract computing machines (automata) was
Fig. 6.7 Image of towers

around a mat Copyright © J.
Townsley 2019 included by
permission
developing rapidly and, sometimes without regard for its potential limits in actual
hardware realisations.
As a consequence of these natural assumptions, Petri analysed the subject of
‘information flow’ in a new way, based on the relations between the structure and
behaviour of automata and on the role of information as a resource in communication.
In his view, communication becomes an organised activity involving automata and
people. Communication with automata, the title of the thesis, refers precisely to this
organised activity where ‘with’ has the double meaning of ‘between automata and
people’ and ‘using automata as communication medium’. Considering automata as
organised communication mediums, as we do today through the Internet for example,
Petri opened the way to new applications of the discipline ‘pragmatics’, as a branch
of linguistics, in view of a modern theory of communication systems [7].
Petri’s bucket chain is a simple example of net theory and one way of rational-
ising data flow. It presents a scenario of coordinated behaviour between firemen
extinguishing a fire. Figure 6.9 shows figures coordinating a task to carry water from
a tank to a fire, using a chain of buckets.
The sequence in Fig. 6.10 explains how the coordination of behaviour (the figures)
and the flow of data (the water from the tank to fire) operate. The theoretical net
Fig. 6.8 Image of towers around a mat Copyright © J. Townsley 2019 included by permission
Fig. 6.9 The Bucket Chain (drawing from Ferigato’s blackboard, 2019) Copyright © J. Townsley
2019
Fig. 6.10 The Bucket Chain and its unfolding in space and time (Ferigato 2018) Copyright © J.
Townsley 2019
‘unfolds’ through time and space to illustrate how the coordination of the figures
putting out the fire operates. This is Petri’s clever graphical way of explaining simple
net theory through the bucket chain scenario.
As the information, in this case, the water in bucket B, flows smoothly from the
water tank to fire, it is also possible to see how each fireman moves left and right
until they meet to exchange the buckets (Fig. 6.11).
The proposition: ‘A walks in the direction right’ can be represented, following the
formalism of Petri’s Occurrence Nets, by a circle, representing in an abstract space
the persistence of the proposition.
Importantly, the truth (or value) of the proposition ‘A walks in the direction right’
is changed by the occurrence of events. For example, as in Fig. 6.10, two men
walking in opposite directions along the same line will eventually meet at points in
space represented as squares in Fig. 6.11. This ‘meeting event’ changes the respective
directions of the two men, as if they were bouncing balls. So, we can represent a
chain of firemen extinguishing a fire by carrying back and forth buckets of water in
an abstract way, as in Fig. 6.11.
Fig. 6.11 The bucket chain

as a net (Ferigato 2018)
Copyright © J. Townsley
2019
The net presents events, the meetings between two firemen, or, at the edge between
the first fireman on the left and the reservoir of water and the last fireman on the right
and the fire. Note that meetings between firemen do not happen in exactly the same
place, nor are they regulated by any definite intervals of time.
This is the principle illustrated in the game, people flowing from coloured mat to
coloured mat according to a set of rules on a box. They are moving through the space
as if in a deep fog, meeting other players outside of a determinate system of time and
space. The rules written on the box propel the player according to a networked system,
but when another player is encountered, the meeting event happens outside of the rule
of a clock or defined by a specific determination of place. Also, the proposition for
flow is changed with each encounter as objects are exchanged. In our digital world
information also flows forever, it can be extracted at any time or place that interaction
happens. This model also presents the basic principle of artificial intelligence, where
the information has the ability to change according to an occurrence or meeting event.
6.5 Theoretical Context: Design and Analysis

of (Coordinated) Communication Systems
computer technology supplies us not [just] with a medium for artificial intelligence nor with
a machine which may be used solely for computation, but with a medium for communication
and for strictly organised information flow, a medium which may induce major changes in
the modalities of co-operation between human beings [5].
The contemporary designer of communication systems is forced, in his profession,

to consider relations between computers and society that are not yet consolidated
in a unique practice or competence. Along the history of design of communication
systems, plain consideration of measurable components like memory, throughput
and availability, has shown its weakness in respect to less measurable subjects like
delegation of tasks to computers, keeping control on copies of data and identify data
as belonging to a given category.
Petri proposed the first set of twelve communication disciplines that are today
still a valid starting point for designing and analysing communication systems.
He addressed the design of communication systems from a wide perspective, as
coordinated activities involving humans and computers.
The ‘Communication Disciplines’ were presented in September 1976 during a
lecture given at Newcastle University and later published in ‘Computer Systems
Design: Proceedings of the Joint IBM-University of Newcastle upon Tyne Seminar
(B. Shaw editor). He stated:
I tried to classify practical problems from my long list, according to my stated viewpoints
of the role the computer [5].
His categories were:

1. Synchronisation
2. Identification
3. Copying
4. Addressing
5. Naming
6. Cancellation
7. Composition
8. Modelling
9. Authorization
10. Valuation
11. Delegation
12. Reorganisation
His principal aim was to look into the future of computer systems and their
applications, in order to support humanity to develop towards a brighter future. His
conclusion was that computer system design should ideally be developed to ‘organise
information flow’ in order to support ‘cooperation between human beings’. As the
true maverick he was, Petri proposed a list that he hoped computer systems developers
could potentially use to ‘concentrate our attention on’ building a computer system
that would enhance human cooperation and solve some of the problems he foresaw
for a computerised world in the future.
6.6 Fishing the Net of the Communication Disciplines
To identify some of the processes and functions operating within the FOREVER-DO
Game, we will use a selection of the 12 Communication Disciplines (highlighted in
bold) as critical tools. In doing so we are also contextualising the game within a
broader critical framework.
Petri states that the discipline of identification ‘covers the question of pattern
recognition’ [5]. The installation is a result of pattern recognition, specifically, a
record of all the players whose synchronised meetings resulted in the identification
of copied objects. Identifying sets of objects also indicates a destination at a mat. This
is most obvious as the composition of stacked boxes grows around the mats in the
room. Petri proposes that composition ‘is concerned with determining the structure
of documents relative to a material or conceptual carrier’ [5]. In traditional art
terms, there is a strong correlation here between ‘function and form’ in material
composition. The towers are subject to the rules of the game, the rules act as a
‘conceptual carrier’ determining the number of boxes in a pile. This takes the form
of an ever-emerging flow of data, offered through time, as boxes move around the
installation.
Contemporary art escapes the present not by resisting the flow of time but by collaborating
with it. It produces artistic events, performances, temporary exhibitions that demonstrate the
transitory character of the present order of things and the rules that govern contemporary
social behaviour. Imitation of the anticipated future, may manifest itself only as an event not
as a thing [4].
Groys’s observation is a good way of thinking about the game, especially in the
many ways flow can be identified. Flow happens in the reorganisation that ensures
the slow evolution of the installation. Petri talks about reorganisation in relation to
mechanised tasks within a nuclear power plant, he highlights the potentially ‘dis-
astrous’ consequences if the flow of tasks and information are not adequately reor-
ganised to meet changing situations. In the game, reorganisation is the process of
becoming, an ever-changing contingent object. Reorganisation is embedded within
the flow of the game and also enforced by external contingencies.
The idea of artwork adapting to contingent conditions is not new, artwork can exist
for just a moment within a definitive contingent space, temporary artwork such as:
Allan Kaprow’s happenings and environments, or the installations of Doris Salcedo,
Olafur Eliasson and teamLab et al. Martha Buskirk considers ‘The Contingent Object
of Contemporary Art’ in her book by the same name [8]. She refers to the artwork
Gnaw (1992) by Janine Antoni where the artist chews, over many days, two 600-
pound cube’s, one made of chocolate, the other of lard. The action of gnawing results
in a constant reorganisation. Buskirk says:
Thus the experience of the work, including the relationship of its components to one another,
changes depending on when and in what part of the cycle one sees it, and also diverges from
photographic records of its appearance [8].
This is pertinent to the FOREVER-DO, not only from the position of a passive
viewer, but also for the player who is not only witnessing the reorganisation of boxes
in a room but accepting active agency within that reorganisation. This implicates
the player as an author within the ongoing realisation of the game. Petri says of
authorization:
this discipline is concerned not only assigning and schematically representing access rights
but also with scheduling obligations [5].
By thinking about authorization in this way, a parallel can be drawn between

authorization within a system, such as Petri indicates, and authorship commonly
placed within art production, the ‘authorial’ process of an artwork. If, within the
FOREVER-DO Game, authorship is shared across participants, objects, systems
rules and networks, it cannot be solely located. Rather, obligations are shared, there
is no single owner or receiver. Petri died in 2010 yet his inspiration is still implicit in
the game, an authorial obligation through time. Authorial agency is delegated across
all the agents of the game, flowing across the human, object and conceptual realm.
In this way Delegation is forming an ever-changing collaboration between the
mechanics of the game and the society of people who participate in it, Petri identifies
in his networks a ‘delegation of tasks from one agency to another’ [5].
This shared social context is fluid, outside of a singular vision. There is, however,
a fixed position in the naming of the groups of objects: a knife; a fork; a spoon.
‘Naming is understood as denoting objects structures’ [5, p. 177]. Naming—becomes
the subject of coordination, the similarity or difference, triggering the selection or
transfer and the cancellation of the player in the game. As players track objects in
the game they also have an experience of a route travelled and can anticipate a future
route to travel. The pattern of flow through the network is in this way an experiential
one, as infinite possibilities for coordinated behaviour are generated through random
instances.
The synchronisation is ‘partially’ controlled by the system mechanism, which
embraces chance, ensuring that synchronicity happens outside of a deterministic time
or place. In this way, the game is mirroring Petri’s premise that synchronisation is
based on the ‘partial ordering in terms of causality as opposed to an ordering to
time’ [5]. This mirrors Groys’s observation on flow being collaborative of time rather
than fixing it [4].
Value, such as it is in FOREVER-DO, resides in the moment of participation
or observation, not in a fixed object. Petri’s consideration of valuation states: ‘the
instant the information is registered by the observer it already loses its value because
it may not be presented anew.’ [5]. With this definition, valuation offers a very tricky
problem to the traditions of an art market. Traditionally value in art has been placed on
the object but, in the game, creative value is in the performative relational aesthetic of
the work; it is essentially objectless, a process rather than an object. In this way, value
is transient, unable to be ‘presented anew’ a data value, that can only be experienced
within the moment.
In an attempt to contextualise art made in today’s global context, the French
curator Nicholas Bourriaud coined an art movement, the Altermodern. He defined it
as a moment when it is possible to produce something:
from a vision of human history as constituted of multiple temporalities … a positive vision
of chaos and complexity [1].
In the Tate triennial exhibition by the same name, ‘Altermodern’ (2009). The
triennial presented artworks from artists such as Marcus Coats, Mathew Darbysher
and Franz Ackerman whose artworks offered a document, narrative or engagement
with change. Earlier in his book ‘Relational Aesthetics’ (1998), he offers examples
of art practice as a game:
Artistic activity is a game, whose forms, patterns and functions develop and evolve according
to periods and social contexts; it is not an immutable essence [1].
Art practice, based on the relational form, is presented in FOREVER-DO as a

meeting point of exchange, a physical human interaction, within a networked system.
In our digital age, these exchanges are often an opportunity for data mining. Points,
relational in time and space, where data is gathered, producing much of our big data;
data that is held outside of our physical realm.
The role of artwork is no longer to form imaginary and utopian realities, but to actually be
ways of living and models of action within the existing real, whatever the scale is chosen by
the artist [1].
The FOREVER-DO Game also represents a ‘model of action within the existing
real’. Big data, whose functions control us beyond our everyday perception, is here
and is real. And yet, the otherness of the nature of big data, its invisible procedures,
make it hard for us to perceive our own individual relationship with it—our essential
role within the phenomena of big data systems. Without us, our relational interac-
tions, big data would not exist while at the same time human life is becoming more
dependent on the information big data feeds back to us. In reality, perhaps, we are
already cyborgs in a close and symbiotic relationship with big data itself.
The art historian Edward A. Shanken, in his essay Art in the Information Age
(2001), considers work by artists such as Hans Haacke and Joseph Kosuth, he says:
‘meaning and value are not embedded in objects, institutions, or individuals so much as they
are abstracted in the production, manipulation and distribution of signs and information’ [9].
The process of the FOREVER-DO offers a distribution of signs and informa-

tion, through the search for repetition. Repetition here is the key to unlocking the
FOREVER-DO Game. It is the copying of objects (two knives, forks or spoons) that
signifies the end of the player’s participation, while the process of the game contin-
ues. Copying for Petri is a ‘message—occurrence in a definite pragmatic context’ [5].
In the game, this is a process held within an unfolding never-ending chain of events,
Nietzsche like in its eternal return. In ‘Thus Spake Zarathustra’ (1917) Nietzsche
describes two eternal paths ‘This long lane backwards: it continueth for eternity.
And that long lane forward—that is another eternity’ [10]. Most importantly the
paths come together at a gateway: ‘The name of the gateway is inscribed above: This
Moment’ [10].
In the game, the most significant ‘moments’ are the coordinated meetings accu-
mulating to model the towers of boxes. Petri utilises the discipline of modelling
to guard against rigidity of implementation of established systems across different
fields. He states:
our ‘models of thinking’, tend to gain some illegitimate independence once they have proved
successful on a particular field, and are then – per analogy but without care – transferred to
other fields [5].
Perhaps this is a warning against the transformative analysis between Petri’s dis-
ciplines (as pertinent to computer systems) and the FOERVER-DO Game. However,
the very temporality of the FOREVER-DO Game can perhaps embody modelling,
where:
we are able to deal with mathematical models in which notions of a temporal ordering is
replaced by that of an ordering in terms of causality [5].
Whether you are an active participant in the game or a viewer on the periphery
of the game, you are in communication with the model of the game and its objects.
This is not a fixed exchange but a communication flow (a causality), with all agents
causally influencing one another in a relational sense.
Nicolas Bourriard draws on such symbiotic forms when he proposes that through
the act of ‘inclusion’—the art object is causally significant to the participation of the
viewer. Indicating that there is an energy exchange between the art object and the
participant.
The game is a destination in flux, a place directed by the rules, e.g. ‘go to the closest
blue mat’. Petri offers a fluidity of destination with his discipline of addressing: ‘by
this we mean the description of routes or systems of paths through the net of channels
and agencies’ [5]. This flow of address as a destination, is an agency of the ever
becoming. Addressing within the flow of the game drives it on to new beginnings,
this chimes with Groys when he says:
Being immersed in the flow of things, one cannot return to previous moments in time or
experience the events of the past [4].
Addressing also happens in a subtler way too, as social address between players
in the game. The form of address has a systemic origin beyond the game, residing
in the social etiquette of human behaviour. This presents us with a wider network of
behaviours and systems. Petri has proposed that his net systems could have a role
in conflict resolution. He suggests, nets could help reconcile complex human prob-
lems, emotionally driven problems, beyond the mechanical, systematic or digital, by
offering clear alternative channels of communication.
6.7 Conclusion
Art does not predict the future, but rather demonstrate the transitory character of the present
– and thus open the way for the new [4].
By thinking about art as a process or event, as relational, societal and social, a new
place is offered for authorship and realisation. This is not only the post-modern
reform of authorship, deferred or shared, but authorship within the moment of a
contingent ever flowing process—the FOREVER-DO.
This conceptual space made real through event also requires a temporal place. For
the FOREVER-DO Game the place of the Palazzo Giureconsulti at Duomo in central
Milan, and temporarily occupied by Milan digital week, is important. Milan Digital
Week is itself a social construct, a temporal event into which interested parties’
(visitors and participants) occupy space temporarily. Visitors are in this way already
complicit within the flow of events.
The historically embedded definitions of art and art practice, particularly the
traditional consideration of value in relation to object, are being challenged through
utilising digital systems as integral tools. By visualising, in analogue terms, this
relationship between the participant (the player of the game) and the system (the rules
and network of the game), the FOREVER-DO highlights our individual complicity
within the greater whole. Places of meeting within the game indicated by simple
coloured doormats are also nodes within a networked system. When individuals
meet through a flow of random incidents, objects are either exchanged or offloaded
onto the data pile. The piles of boxes record the incidents of two humans meeting and
randomly possessing a repeated object. This process presents a digital and analogue
description of events through time and offers a visualisation of a similar scientific
modelling that the Petri Nets identify.
Petri’s Communication Disciplines, written in the 1970s still ‘concentrate our
attention on’ building a computer system to enhance human cooperation and over-
come some of the many problems brought about by our human digital interface.
Problems for culture arising from irresponsible systems, technicians, trolls and cor-
porations. We are all complicit in society’s broader data communications. As Petri
outlined, we must all consider the relations between computers and society that are
not yet consolidated in a unique practice or competence. The FOREVER-DO opens
a discussion about the interactive trail we leave behind and the systems acting upon
us. By reclaiming our digital identity, being aware of the symbiotic relation of self to
our larger networks, maybe we can hack our future in a positive way, for the benefit
of all.
Acknowledgements The FOREVER-DO project and its resulting artwork (including the
FOREVER-DO Game) is a collaboration between Jill Townsley (University of Huddersfield) and
Carlo Ferigato (Joint Research Centre of the European Commission—JRC-Ispra, Italy). The project
is supported by advice from the MC3 research group at the Milano-Bicocca University and Otolab,
Milano. The project is funded by the EU Commission through its SciArt initiative.
References
1. Bourriaud, N.: Relational Aesthetics. Les Presses Du Reel (1998)

2. Joint Research Centre (JRC). Available: https://resonances.jrc.ec.europa.eu/. Accessed Oct
2019
3. Joint Research Centre (JRC). Available: https://resonances.jrc.ec.europa.eu/datami. Accessed
Oct 2019
4. Groys, B.: In the Flow, Verso Books. London (2018)
5. Petri, C.A: Communication disciplines. In: Shaw, B. (ed.) Computing System Design: Pro-
ceedings of the Joint IBM—University of Newcastle upon Tyne Seminar, September 1976,
pp. 171–183. University of Newcastle upon Tyne (1977)
6. Petri, C.A.: Kommunikation mit Automaten, Dissertation, Schriften des IIM 2, Rheinisch-
Westfälisches Institut für Instrumentelle Mathematik an der Universität Bonn. (Eng. transl.
Communication with Automata, Technical report RADC-TR-65-377, Volume I, Final Report,
Supplement I) (1962)
7. Bernardinello, L., Cardone, F., Pomello, L.: L’eredit’a di Carl Adam Petri: dagli automi alla
comunicazione, Mondo Digitale, febbraio, p. 175 (2016)
8. Buskirk, M.: The Contingent Object of Contemporary Art. The MIT Press, Cambridge, MA,
USA (2003)
9. Shanken, E.A.: Art in the Information Age in LEONARDO, vol. 35, No. 4, pp. 433–438 (2002)
10. Nietzsche, F.W.: Thus Spoke Zarathustra. Modern Library, New York (1917)
Chapter 7
Searching for New Aesthetics: Unfolding
the Artistic Potential of Images Made
by the Scanning Electron Microscopy
Anastasia Tyurina
Abstract The visual arts have become a powerful tool for alternative approaches to
scientific outputs, but it is crucial that both science and art cultures are aware of their
interdisciplinary capabilities and limitations. It is necessary to differentiate between
images captured by devices designed as resources for scientific investigation and
images that exploit the ‘visual elements’ of scientific images. A great example of
a device designed for scientific exploration is the Scanning Electron Microscope
(SEM), which was introduced to scientific research in the mid-1960s. This chapter
outlines the procedures, processes, and methodologies used in developing a body
of studio work that investigates the artistic potential of scientific images made by
the Scanning Electron Microscope (SEM) with a view to its possible social and
cultural impact of this practice. It also outlines the developed theoretical findings and
proposes that certain images made by the SEM can have esthetic value apart from
that of scientific documentation. The use of artistic manipulations in experiments
with the SEM fuses science and technology with art, and the SEM-based images that
result provide a new meaning for scientific photomicrographs.
Keywords Science-art · Visual arts · Photomicrography · Interdisciplinary ·

Microworld · Interactivity · Aesthetics · Self-organizing processes · Coffee ring
effect · Alternate visualizations · Visual coalescing · Creative coding · New
media · Scanning electron microscope · Water
7.1 Artist Statement
My art practice involves interplay between photography and scientific imaging, pho-
tomicrography in particular. Originally being a technical discovery, photography has
been widely used in almost all fields of human activity, acting as a research tool and
as an independent artistic direction. Modern times offer “modern” interpretations of
A. Tyurina (B)
Queensland University of Technology (Brisbane), Brisbane, QLD, Australia
e-mail: anastasia.tyurina@qut.edu.au
National Research University MIET (Moscow), Moscow, Russia

104 A. Tyurina
scientific photographs and attitudes toward them. Scientific photomicrography is a

powerful tool for receiving and storing information and for providing solutions to
a variety of tasks in many areas of science and technology. At the same time, the
artistic application of photomicrography is capable of revealing a set of complex and
interrelated principles that underlay the materiality of the human environment. Pho-
tomicrographs also expand human visual vocabulary, revealing principles of beauty
which are difficult to access with the naked eye.
Water is the main subject of my research project. I believe that an interdisciplinary
approach is the most appropriate one for deepening knowledge about unique prop-
erties of water and building a sustainable practice of water management through
enhanced visualization of water contamination. My focus is on interconnected artis-
tic thinking and by developing alternate forms of visualization, I aim to transcend
disciplines and contribute to the new ways of seeing water.
My visual artworks consist of series of images made by the SEM depicting the
nature of water (water chemistry), which is invisible to the naked eye, through using
the SEM utilizing the phenomenon of drop evaporation. However, it was not the
purpose of this research to claim that the created images of water after evaporation
are scientifically valid forms of documentation; rather, this research takes an esthetic
approach to scientific photomicrography.
7.2 Introduction
This research was undertaken during my Ph.D. study at Queensland College of Art,
Griffith University and involved esthetic approaches to scientific photomicrography.
Scientific photography is commonly perceived as a way of recording scientific data
through techniques such as photomicrography, high-speed photography, time-lapse
photography, X-ray photography, and aerial photography, among others.
Images made by the SEM are not photographs in the traditional sense; they go
beyond what can be captured with light because the process of producing a picture
is camera-less. Imagery produced by the SEM can confuse the viewer because the
microscopic sample seems as if it is observed in the eye aperture when illuminated,
and light seems to come from a particular illuminant. SEM photomicrographs are
constructed out of pixels synchronized with a distribution map of the intensity of the
signal being emitted from the scanned area of the specimen [1].
Interestingly, the last decades’ use of SEMs in creating scientific images formed
a new, well-established visual culture within a variety of scientific disciplines. As
Klaus Hentschel explains, it became “an image centered science in the sense of being
even totally dependent on photographic images as basis of all further processes of
inference” [2, p. 315].
By exploring the interplay between the indexical and iconic modalities in scientific
photomicrographs, I try to imbue them with new meanings. Both art and science
are experimental in nature. There are different ways of artistically representing and
perceiving an object, some of which may be valuable for science. My practice aims
7 Searching for New Aesthetics: Unfolding the Artistic … 105
to draw attention to the qualities of water through enhanced visual details that aid in
the interpretation of and differentiation between water samples.
This project investigates how to reinterpret photomicrographic images made by
the SEM of micro-scale drops of water after evaporation and thus turns scientific
photography into an art form.
In recent decades, there have been increasing concerns over the ecological man-
agement of water. Waterway pollution is recognized as placing urban ecosystems
around the world at risk. Rainfall that washes oils, metals, and nutrients directly
from streets into rivers and seas is hard to treat [3]. This is a challenging problem for
science and technology. The health and well-being of present and future generations
are dependent on how quickly and well it is managed.
Photomicrography has a particular potential to respond to this issue from both a
scientific and a cultural perspective. SEM-made photographs are capable of visually
representing features related to water composition and, in some cases, the contami-
nation of water. At the same time, they can transmigrate from science into the domain
of art and draw attention to water issues.
7.3 Water Represented by the Scanning Electron

Microscope
The variety and frequency of the unusual properties of water are determined by the
physical nature of its atoms and their association in the molecule and the group-
formed molecules [4, 5]. The composition of water, even that which is entirely
free from mineral and organic impurities, is complex and diverse because water is
constantly in contact with many substances.
During my experimentation with the water droplets collected from different
aquifers, I aligned the scientific method of revealing water composition (the so-called
‘Coffee Drop Effect’) with my artistic practice. The drop evaporation phenomenon,
the so-called ‘Coffee Ring Effect’, has been the subject of studies in the last few
decades. It was first mathematically described by Robert D. Deegan in 1997 as a
natural model for studying the dynamics of self-organizing processes and is actively
used in physical experiments [6]. When the liquid is ‘pinned’ to its contact surface,
the liquid evaporating at the exterior edge is replenished by the constant flow of
liquid from the interior to the periphery. This flow carries nearly all dissolved solids
toward the edge [7].
During experiments for my project, the structure of the water impurities visually
transformed, leading to a unique connection between the processes of evaporation
and solidification. These two processes are shown in Fig. 7.1. This natural process
of drying reveals the unique informative capacity of droplets through the shapes,
patterns, details, and characteristics resulting from each water sample.
At the University of Chicago Materials Research Center, scientists are explor-
ing the driving mechanism responsible for this phenomenon that is found in many
106 A. Tyurina
Fig. 7.1 Anastasia Tyurina, Drop Evaporation and Solidification Scheme, 2015. Image copyright
© A. Tyurina 2019
varieties of liquids. This research validated my process of exploring variations in

water composition for my project. On its webpage, the Center states: “We have
determined that ring formation is a ubiquitous and robust phenomenon. It doesn’t
depend on the solute, the solvent, or the substrate so long as the solvent is partially
wetting and volatile, and the contact line is pinned” [8].
An interesting observation made by the researchers is that “By controlling the
speed and spatial variation of the evaporation, this model predicts that we can control
the shape and thickness of the deposit” [7]. This was particularly important for my
practice because it meant that experimenting with various modes of evaporation
could lead to different visual formations of the drops after they dry.
Specifically, for this research ‘science’ can be described as a process through
which new knowledge about the world is built. The scientific process relies on the
testing of ideas through experimentation. Scientific photography, as a tool, aims to
support the scientific process by capturing data about the world for examination.
Reflecting on the scientific findings, the primary purpose of my visual art project
was to depict the inherent features of water that are invisible to the eye through using
the SEM. To do so, I used the process of evaporation as an alternative and unusual
artistic method of visually presenting the composition of water. My approach is
unique in the specific way in which I use water to create images using the SEM.
During experiments for my project, it became evident that the structure of water
impurities is visually transformed after evaporation and reveals a unique connection
between evaporation and solidification. This process of revealing the nature of water
(water chemistry) allowed me to play with the process like an artist [9].
One of the technical requirements when taking a microscopic image with the
SEM is that any object placed in the chamber of the SEM must be completely dry,
because the SEM operates in a vacuum. As explained in Under the Microscope: A
Hidden World Revealed (1987), “Turning to biological specimens for the SEM, it
used to be necessary first to fix and dehydrate them, and then dry them either on air
or by the use of liquid carbon dioxide (‘critical point drying’)” [10, p. 199]. After
the evaporation process, water is no longer a liquid; dry solids or other substances
become watermarks that represent its previous composition.
The main difference between ‘Artistic’ and ‘Scientific’ use of the SEM in this
project is that the artistic approach has a focus on making aesthetic images. Within
this is the substance of the scientific idea about showing water state and how it is
changing. In my work the results of imagery processes become art pieces with both
artistic and scientific applications (Fig. 7.2).
My artistic intervention into a scientific process through experimenting with the
SEM was a way to find out what different things my images could reveal about
water to a viewer. There was an opportunity to explore some concerns related to the
environmental impact of water pollution in an artistic context and to communicate
the significance of water.
By exploring the integration of art and science in this way, my project focuses
on the use of scientific tools to create science-based art. However, it must be recog-
nized that my approach was primarily one of artistic research and experimentation: a
search for new aesthetics, exploring spatial and temporal dimensions, engaging with
materiality, and involvement of modern technologies in the formation of esthetic
experiences for the viewer. In my project, images are highly variable in what they
show and how they show it because of changes in water content.
7.4 Photomicrography as an Art Form
Contemporary practices in photomicrography are making significant contributions

to the dialog around aesthetics and artistic components in science. As far back as
1963, in his article “The Esthetic and Pictorial Applications of Photomicrography”
for The Photographic Journal, Douglas Lawson wrote:
When mentioned within the hearing of some pictorialists they immediately think it is only for
the scientifically minded, or the record worker. Nevertheless, the so-called record photograph
can be made to look quite attractive. Some of you may remember Mr. H. A. Murch, one of
our great pictorial photographers, once saying, ‘I do understand the desire to apply pictorial
ideas in record work, which is a very different objective, and we ought to welcome such an
application when it can be done without losing anything of the essential factual value of the
record’ Photomicrography offers expression not in what the painter has already done but in
what the painter cannot do. Because this application of photography is highly scientific there
is perhaps a tendency to think of it as being without scope for the artistic application. May I
suggest that a work of art in any medium can be the deliberate creation of unity, and nature
through the microscope is one medium provides us with plenty of scope for such unity [11].
To effectively engage with photomicrography as a social phenomenon, it is crucial

for an artist to demonstrate an understanding of its ‘scientific’ protocols of represent-
ing. But will the viewer be familiar with what he or she sees in scientific images? If
108 A. Tyurina
Fig. 7.2 Anastasia Tyurina, Watermarks Series, 2015 (From left to right: Brown Lake, Brisbane
River, Raby Bay, Rainwater, Mount Gravatt Pond, Mimosa Creek, South Bank Pond, Cylinder
Beach). Image copyright © A. Tyurina 2019
not, how will they perceive such images? Do the images need to be explained or are
they capable of conveying messages in a different context?
Artistic photomicrographs of water after evaporation call upon the viewer’s cre-
ative ability to perceive previously unseen water composition after evaporation as
well as to observe natural phenomena over and beyond the directly visible.
My images transform the microworld to a macro-level and evoke an interest in
water chemistry that is shown as being beautiful. This causes a dilemma for viewers,
particularly because the gallery space is different from a laboratory. The captions
for my photographs refer to the sites where the water samples were collected. It
is intriguing that they can resemble aerial photographs of topographical features of

particular water reservoirs (Figs. 7.3 and 7.4).
Today, the practice of visually pleasing photomicrography is more mainstream.
The most popular international photomicrography competition is Nikon’s Small
Fig. 7.3 Anastasia Tyurina, Photo documentation of the artistic installation for H2O + exhibition,
PoP Gallery, Queensland College of Art, Griffith University, 30 March–16 April 2017. Image
copyright © A. Tyurina 2019
Fig. 7.4 Anastasia Tyurina, Photo documentation of the Shifting the Posts exhibition, Webb Gallery,
Queensland College of Art, Griffith University, 27 November–7 December December 2018. Image
110 A. Tyurina
World Photomicrography Competition, which dates back to 1975. Participating

images are judged for “their scientific and artistic merits” [12] and represent a broad
range of visual studies of the microworld. Some of these photomicrographs have alle-
gorical titles, a fact that highlights the relation of artistic intentionality in producing
images and the scientific objectivity underlying the concept of the competition.
Participation in the competition is restricted to images made by optical micro-
scopes of any kind, which are capable of revealing natural colors of objects. As
mentioned above, the SEM apparatus is camera-less; there is no light involved in the
process; it is not an optical instrument. Thus, colors cannot be reproduced. Rather,
artificial colors can only be added with the help of graphic software such as Adobe
Photoshop. While there is a whole range of possible manipulations available in Pho-
toshop, it is worth mentioning the coloring tools. Through different algorithms, it is
possible to apply different colors to different details of the image. There are many
sources both in the technical literature and on the Internet providing scientists and
artist with various tutorials on image coloring. But the author of the image makes the
final decision of which color to apply to a particular point of the presented object and
thus color choice is very individual. Experimenting with such coloring, I have tested
two options: the coloring algorithm available in Photoshop and the software offered
by Recolored, which was designed for the purpose of recoloring black-and-white
photographs (Fig. 7.5).
Fig. 7.5 Anastasia Tyurina, Colour 32, 2017. Image copyright © A. Tyurina 2019
7.5 Photographs or not Photographs?
It is important to examine the process of image formation in relation to the imagery

obtained by the SEM in modern microscopy. It is also crucial to distinguish image
generation by SEM considering that the process is not one that is based in code. The
image generation process can be seen as a work of high-energy electrons that create a
range of different signals when hit the specimen. Converted into pixels signals result
in image formation appeared on the screen.
Technically, the SEM produces electron micrographs because the image is devel-
oped by either secondary electrons (electrons ejected from the material by the incom-
ing electron beam) or backscattered electrons (electrons from the electron beam,
which have ‘bounced’ off the material into the detector). Dee Breger explains:
Scanning electron microscopes don’t merely use electron beams to illuminate objects so
small they can’t be seen by light. Since they are electronic devices, SEMs can manipulate
isolated groups of electrons from the sample-beam interaction to create separate pictures
(such as secondary and backscatter images) that contain different kinds of information about
a single object. This variety can lead to a ‘compound reality’ or, since no version tells the
whole story, a kind of ‘ironic nonreality’ [13, p. 11].
Images produced by the SEM are beyond light; captured by a focused beam of
electrons, they are not photographs. The apparatus tries to recreate a reality that
is not a visual phenomenon, which scientists then try to analyze through its visual
representation: the photomicrograph.
However, the most commonly used SEM electron detector, named Everhart-
Thornley (E-T),1 typically uses a material that produces light when an electron
collides with it—a scintillator.2
Therefore, the SEM does use photons, but they are converted back into elec-
trons, which are accelerated onto the electrodes of the photomultiplier, producing
an increasing stream of electrons until the final collector is reached [14]. The image
formation in scanning electron microscopy consists of the scanning system, the sig-
nal detectors, the amplifiers, and the display [1]. A physical, material connection of
the SEM apparatus with the studied object is gained by the interaction of these four
elements (Fig. 7.6).
1 “The E-T detector operates in the following manner: when an energetic electron (≈10 keV energy)
strikes the scintillator material (S), light is emitted. Light is conducted by total internal reflection
in a light guide (LG) (a solid plastic plastic or glass rod) to the photocatode of a photomultiplier
(PM). At the photocatode, the photons are converted back into electrons, which are accelerated onto
the successive electrodes of the photomultiplier, producing an ever-increasing cascade of electrons
until the final collector is reached” [14].
2 “A scintillator is a material that accepts incident high-energy electromagnetic or charged particle
radiation and in turn uses that energy to fluoresce photons whose peak emission wavelength is longer
than the wavelength of the incoming radiation. In the case of a SEM, the scintillator disc collects the
secondary electrons that are produced as the electron beam scans the surface of the sample. These
electrons are converted into photons which travel through the light pipe to the photomulitiplier tube
(PMT) so that the signal may be amplified to the level required for viewing” [15].
112 A. Tyurina
Fig. 7.6 Anastasia Tyurina, Scanning Electron Microscope Imaging Facility, 2016. Image
During the imaging process, the energy exchange between the electron beam
and the sample results in the reflection of high-energy electrons, the emission of
secondary electrons and the emission of electromagnetic radiation, each of which
can be measured by specialized detectors. The detectors interpret these signals by the
algorithm and the resultant image appears constructed out of pixels. This algorithm
is defined by each quantum of signal information but it is also implemented in the
physical world in apparatus, computer, and screen [1]. In some sense, the SEM
apparatus interprets (transforms) signals and visualizes signals.
7.6 Scientific Images and New Media
The artworks employ two kinds of medium, comprising two different applications of
SEM photomicrography: still photomicrographs and interactive digital installations.
They can be perceived differently in terms of the esthetic response they generate in
the viewer, but both draw attention to the qualities of water through enhanced visual
details that aid in the interpretation of water samples. I use static SEM-generated
photomicrographs of water, then apply a digital code which allows the image to be
altered by the audience’s interaction with its touch-sensitive interface. Thus, anyone
who touches the screen can create, display, and experience a ripple effect, which is
very similar to the effect we can see and observe when we interact with water surface
by touching or disturbing it. Interacting with the scientific photomicrographs in this
way offers a layered meaning and can enhance the audience’s perception of scientific
data, scientific photography, and water.
Exploring the idea of ‘interactivity’ for my project, I looked for a method that
allows viewers to interact with photographs by altering the work’s visual content.
Utilizing the programming language Processing, I developed an algorithm which
allows viewers to physically interact with my photomicrographs so that they become
direct objects of manipulation. Using the random human touch of some areas of the
developed algorithm alters the static image, which in some senses visually represents
non-visual elements: scientific data.
Through my research, I have explored different digital tools and realized that cre-
ative coding suits my goals because this type of computer programming is designed
to create something expressive instead of something functional. Using creative cod-
ing tool such as Processing, it is possible to create art installations, projections, sound
art installations, and much more within the context of the visual arts.
The interplay between virtual ripple effects and the resemblance of physical inter-
action with water can be seen as embodying the “artist–display–user” paradigm seen
with regard to computer art [16, p. 10]. In their article “The Post-Display Condition of
Contemporary Computer Art,” Toby Juliff and Travis Cox attempt to re-conceptualize
the relationship between artistic intentionality, coding, interface, and user input. They
suggest that the relationship between the artist and computer code should be recon-
sidered so that the paradigm becomes “artist–code–display–user.” They state that
“the code takes input from the user and subjects it to internal semiosis between dis-
tinct elements of the code, before being output to the display. This code, having been
written by the artist, or an agent of the artist, contains within its structure an inbuilt
intentionality, a way of approaching input that is integral to conveying the intended
meaning” [16, p. 10]. In line with artists using code to activate new experiences and
with Juliff and Cox, who argue that the code is not only a product of intention and
meaning but also a producer of them, I seek in my practice to explore the role of
code in the production of meaning, in particular for the scientific photomicrograph
of natural phenomena.
In the series of live images for my project, the photomicrographs displayed on
the screen seem to be still when viewed from a distance; yet, as the audience comes
closer and start touching the screen, the ripples appear.
The photomicrograph transforms into a live picture, producing varying forms of
ripple that seem both ordered and random at the same time. Such transformations
occur continuously until the user-viewer stops touching the screen (Figs. 7.7 and
7.8).
114 A. Tyurina
Fig. 7.7 Anastasia Tyurina, Rainwater (detail) 2016, Digital mixed media. Image copyright © A.
Tyurina 2019
Fig. 7.8 Anastasia Tyurina, Rainwater (detail) 2016, Digital mixed media. Image copyright © A.
Tyurina 2019
Interacting with the image in this way transforms the work into something that
transcends disciplines. It offers a layered meaning, providing audiences with the
opportunity of experiencing the fluid and animated qualities of the effects that con-
nects with these qualities in the subject matter: water. The addition of the interactivity
and animation introduced by the ripple effect is the expanded mode of ‘reading’ or
appreciation. As well as offering a visual engagement, the work offers an embodied
Fig. 7.9 Anastasia Tyurina, Colour 32 (detail) 2017, Digital mixed media. Image copyright © A.
Tyurina 2019
engagement: an important connection to the material significance of water in our

lives.
Colour 32, one of the artworks, portrays a drop of Brisbane River water. This
artwork was exhibited at various venues displayed as both interactive work and
video at screens of different scale (Figs. 7.9, 7.10 and 7.11).
Such digital installations allow the transformative power of water to be explored;
they offer new perceptual experiences and, by the artist–code–display–audience inter-
action, are capable of providing a new meaning for scientific images through their
ever-changing visualization.
7.7 Conclusions
Not every image is art, and art is far more than just an image. Originally a technical
discovery, photography has been widely used in almost all fields of human activities,
acting as a research tool and as a form of artistic practice. Scientific tools have
brought new ways of seeing the world. Although usually reserved for scientific use,
such technology is now being used in the creation of art.
The primary purpose of this research was to show that the artistic use of SEM-
made photomicrography can shift the visual outcomes of scientific photomicrography
116 A. Tyurina
Fig. 7.10 Anastasia Tyurina, Colour 32, North Spine Plaza, Media Art Nexus, Nanyang
Technological University (NTU), Singapore, 2017. Image copyright © A. Tyurina 2019
Fig. 7.11 Anastasia Tyurina, Colour 32, public screening, Adelaide Festival Centre, 2018. Image
to function within the context of art. Drawing on literature, visual analysis, and
theoretical findings, it was possible to develop my artistic practice using the SEM to
visualize the microworld of water. As Breger states, the microworld exists beyond
our cognition; its beauty is hidden but it can be explored using appropriate tools,
such as the SEM [13].
Not many people have access to complex research technologies that can widen
their daily experience such as the SEM. Mostly, people passively receive scientists’
dry results and the interpretations they are exposed to. My work in the field of
photomicrography aims to expand human visual vocabulary, revealing principles of
beauty which are typically difficult to otherwise access. Thus, my artistic intention is
to expand the visual representation of the composition of water, its chemical features,
and their patterns after evaporation. My aims are to also communicate issues related
to the ecological management of water and to raise and deepen responsible attitudes
toward consuming and managing water resources.
During this project, I have placed (both virtually and physically) scientific images
into a ‘hostile’ artistic environment. In the gallery space, the significantly greater
scale of photomicrographs, their impeccable details, the artistic materials used, and
the digital, still and interactive modes of presentation uncover the novel artistic
potential of the scientific image. This leads to the image being disconnected from
pure data and objectivity and allowing the viewer to perceive it as art, helping them
to interpret photomicrographs of water in different ways. Most importantly, through
these images, I hope to uncover the horizon of meanings previously unseen.
Today, many institutions and organizations host scientific photography and visu-
alization competitions (and associated exhibitions), which encourage scientists from
different disciplines to contribute. The images are not only judged by profession-
als (scientists and photographers) but often the audience is also invited to evaluate
the competing images. By doing so, all three groups apply their own esthetic judg-
ment criteria. This illustrates how scientific images can be evaluated from different
contexts. Examples of such events include the annual the Visualization Challenge,
National Science Foundation; the Wellcome Image Awards, Wellcome Trust; Nikon
Small World Photo Competition; The Royal Photographic Society International
Images for Science, just to name a few.
This popular trend indicates the growing interest in the visual coalescing of science
and art through photography, particularly through photomicrography. Nevertheless,
the idea that the photographic scientific image can be artistic and that the arts can
serve science and vice versa still generates skepticism, and the debate will surely con-
tinue. Regardless, it is obvious from the discussion that the arts have the potential
to reveal important aspects of the complex world in their own way, which is com-
plementary to what scientists do and may be useful for both disciplines. Scientific
photography aims to record and illustrate data and experiments that differ according
to specific disciplines. Although the main purpose of scientific photography is to
convey accurate information rather than beauty, its ability to record material in addi-
tion to that which is merely informative allows it also to serve expressive, subjective,
and esthetic purposes.
The new definition other than photographs (or photomicrographs) for SEM images
should be sought considering the nature of the process of image generation by the
SEM. This is particularly interesting due to the complex nature of the apparatus.
Different ways in which images made by SEM can be interpreted create layered
meaning in them especially if they are given in the artistic context.
118 A. Tyurina
References
1. Goldstein, J.I., Newbury, D.E., Echlin, P., Joy, D.C., Fiori, C., Lifshin, E.: Electron Microscopy
and X-Ray Microanalysis: A Text for Biologist, Materials Scientist, and Geologists. Springer
Science + Business Media, New York (1981)
2. Hentschel, K.: Visual Cultures in Science and Technology. A Comparative History. Oxford
University Press, New York (2014)
3. De Haan, F., Rogers, B., Frantzeskaki, N., Brown, R.: Transitions through a Lens of Urban
Water. Environ. Innov. Societal Transitions 15, 1–10 (2015)
4. Chaplin, M.: Water Molecule Structure. Water Structure and Science (blog). Last updated
30 October. http://www1.lsbu.ac.uk/water/water_molecule.html (2016). Last accessed 19 Jan
2017
5. Chemical Association: Accessed on 6 June 2019. https://www.britannica.com/science/
chemical-association (2019)
6. Yakhno, T.A. Yakhno, V.G., Sanin, A.G., Sanina, O.A., Pelyushenko, A.S., Egorova, N.A.,
Terentiev, I.G., Smetanina, S.V., Korochkina, O.V., Yashukova, E.V.: The informative–capacity
phenomenon of drying drops. IEEE Eng. Med. Biol. Mag. 24(2), 96–104 (2005). https://doi.
org/10.1109/memb.2005.1411354
7. Deegan, R.D., Bakajin, O., Dupont, T.F., Huber, G., Nagel, S.R., Witten, T.A.: Capillary flow
as the cause of ring stains from dried liquid drops. Nature 389, 827–829 (1997)
8. The Coffee Ring Effect Accessed on: June 6, (2019). Available: https://mrsec.uchicago.edu/
research/highlights/coffee-ring-effect (2019)
9. Tyurina, A.: The unseen water: experimentation with scientific photomicrography and creative
coding. In: Proc Electronic Visualisation and the Arts (EVA 2017). Available http://dx.doi.org/
10.14236/ewic/EVA2017.15
10. Burgess, J., Marten, M., Taylor, R., McNamee, M., Stepney, R.: Under the Microscope. A
Hidden World Revealed. CUP Archive, New York (1987)
11. Lawson, D.F.: The aesthetic and pictorial applications of photomicrography. Photogr. J. Off.
Publ. Roy. Photogr. Soc. G B 103(1), 1–11 (1963)
12. Thompson, N.: Tiny Photographs from the ‘Limbo Between Art and Science’, CNN News,
30 October (2013). Available: https://edition.cnn.com/2013/10/30/world/nikon-small-world-
photo-contest/index.html (2013). Accessed on 6 June 2019
13. Breger, D.: Journeys in Microspace: The Art of the Scanning Electron Microscope. Columbia
14. Electron Detectors. Available: https://www.scienzemfn.unisalento.it/c/document_library/get_
file?uuid=639f02da-34b4-43b5-b595-13978bae6164&groupId=10122 (2015). Accessed on 31
May 2019
15. Scintillators. Available: https://semicro.org/collections/scintillators. Accessed on 6 June 2019
16. Juliff, T., Cox, T.: The post-display condition of contemporary computer art. EMAJ 8 (2015)
Chapter 8
Interspecific Interactions: Interaction
Modes Between Sound and Movement
in Collaborative Performance
Manoli Moriaty
Abstract Collaboration between composers and choreographers is an approach

possessing a long history and expansive list of creative outcomes. In recent years,
such collaborative endeavours have further engaged with knowledge and practices
from scientific disciplines, consequently giving rise to an emergent field of research
focusing on novel modes of interaction between the expressive media of sound
and movement as facilitated by new technologies and methodologies; from ana-
logue proximity devices permitting dancers rudimentary control over recordings in
the 1965 Variations V, to Marco Donnarumma utilizing biophysical sensors and
machine learning algorithms as means of maintaining detailed command of digi-
tal signal processing (DSP) through specific body postures in the 2016 Corpus Nil.
Whilst the contributions by this relatively new transdisciplinary field have produced
practice and research outcomes of unquestionable value, its main focus concen-
trates on the expression of sound through movement, with lesser emphasis placed on
perspectives of practitioners utilizing interactive sound as means of informing the
arrangement of movement. This chapter aims to reflect those perspectives through
the research conducted during my collaborations with dance practitioners, where we
have together examined areas such as each performer’s role while operating inter-
active systems, employing different modes of system operation according to the
desired determinacy of resulting material, and simplification of discipline-specific
language when engaged in polydisciplinary collaboration. The contribution of this
research concerns a novel taxonomy of interaction modes informed by the biological
phenomenon of symbiosis. Defined as close and persistent relationships between
organisms of different species, symbiosis manifests in three core types—mutual-
ism, commensalism, and parasitism—with each type determined according to the
change in fitness outcome experienced by each of the engaged organisms. In the
context of my collaborative practice, the biological notion of fitness outcome is
interpreted as the expressive range allocated to each performer and their respec-
tive media, while simultaneously relating to the level of determinacy from the score
and choreography. The symbiotic modes of interaction are firstly described through
examining three contemporary precedents, each showcasing distinct approaches in
M. Moriaty (B)
University of Salford, Salford, UK
e-mail: e.moraitis1@salford.ac.uk

122 M. Moriaty
collaboration and use of gesture recognition technologies (GRT), followed by a

practical demonstration of each mode by its activation during a practice outcome
resulting from my collaborations with two dancers, including a detailed analysis of
the developed interactive system and its principles of its operation.
Keywords Polydisciplinary · Collaboration · GRT · Interaction · User interface ·

Symbiosis
8.1 Introduction
For the better part of the past decade, a significant portion of my practice and research
has been concerned with polydisciplinary (see note 1) collaboration in performance
practice, with a particular focus on the processes by which practitioners expressing
through sound and movement engage in collaboration. Since its early conception,
my approach in examining such engagements stems from a perspective of biological
associations, namely the phenomenon of symbiosis. My preoccupation with sym-
biosis is fuelled in equal parts by the phenomenon’s scientific significance towards
evolution, and the way it has informed seminal philosophical observations on social
organization among humans [1]. The outcome of this artistic research resulted in
two publications [2, 3] with symbiosis informing a framework through which I went
to analyze the process of and the social dynamics observed within the collabora-
tive engagements conducted as part of my practice, largely concerning live perfor-
mance works involving interaction between music and dance. At the same time,
however, these publications provided scarce details on the technological means used
to facilitate the interaction between our respective expressive media. Considering the
fundamental role of technology in developing our joint practice outcomes, further
reflection on the conceptual debate between symbiosis and performance practice
complemented the existing framework for interaction between disciplines and prac-
titioners resulted in findings that appear salient towards identifying distinct modes
of interaction between the expressive media of sound and movement.
The findings presented in this chapter have been reached through a Practice
Research methodology, and are presented through a process of analytic autoethnog-
raphy, that is the active reflection of one’s experiences in creative work through the
wider context of associated practices. The contribution of these findings on technol-
ogy focus on the application of well-established GRT devices and associated software
suites within a novel set of interaction modes informed by the taxonomy of symbiotic
relationships, with the activation of these modes presented through both precedent
examples, as well as a representative outcome developed through my collaborative
practice. In other words, rather than aiming towards creating new technologies for
creative expression, this research aims to present new ways of appropriating existing
technologies, particularly in manners accessible to practitioners of diverse levels of
technological proficiency.
8 Interspecific Interactions: Interaction Modes … 123
My motivation towards this research approach stems from my previous practice as

a performer of Electronic Dance Music, where the limited possibilities presented by
analogue disc jockey equipment were a driving force to develop new ways for sonic
manipulation. This exploration continued during my first steps within sonic arts,
albeit this time through predominantly digital tools, with the addition of modulation
devices (e.g. Envelope Generators (EG), Low Frequency Oscillators (LFO), Step
Sequencers), as means of predefining and accurately recreating certain sonic gestures
in the performed material. However, the accuracy of digital devices made away with
the organic and ephemeral feel borne of the imperfections of human movements. This
observation coincides with my first forays into collaboration with dancers, as well
as my initial experimentations with GRT in sonic performance through the widely
used Nintendo Wii Remote (or Wiimote). These early experiments were conducted
in collaboration with contemporary dancer Shona Roberts, an already close personal
friend, with whom we began exploring ways for motion sensors to manipulate and
modulate sound (see Fig. 8.1).
At the time, my main focus was to explore the potential of using physical move-
ment as means of affecting, guiding, and enriching sound, with the connection
between the two expressive media being a lesser concern. In essence, my perspective
at the time could be described as sonic-centric, where the performer’s actions were
utilized as merely another source of modulation to augment the previously used
devices. Contrary to reducing my collaborator’s contributions to the programmed
actions of a machine, this approach allowed Roberts unrestricted range of expression
Fig. 8.1 Performance with Shona Roberts at Anatomy, Edinburgh Summerhall, April 2014
(Copyright Richard Dyson, reproduced by permission)
124 M. Moriaty
through her choreography, while I enjoyed similar liberty in creating sonically mean-
ingful interpretations of the dancer’s arm movement through the data captured by
the pair of Wiimotes.
8.2 Interactivity in Performance—Origins and Recent

Developments
On reflection, this mutual independence during our earliest experiments with Roberts
echoed the collaborative relationship between composer John Cage and choreogra-
pher Merce Cunningham, who in developing their joint works would ‘intentionally
segregate the creation of the sound from the creation of the movement until the per-
formance’ [4]. Dubbing this relationship ‘autonomous complementarity’, Andrew
Uroskie explains that this separate development of disciplinary material was an effort
in alleviating the dependency of movement to embodying sound in a manner that
was to an extent literal, and certainly perceivable by the audience:
Music was understood to govern, implicitly or explicitly, the movement of the dancers.
Choreography, within this propulsive conception, was a kind of musical interpretation, judged
on its ability to form a singular synaesthetic coherence in the experience of the audience. [5]
Challenging the traditional notions of audience experience in the work of Cage and
Cunningham is further evident from the presentation format of the Black Mountain
‘Happenings’, a series of events where the combination of music, dance, and visual
arts was such as to allow each audience member to experience a unique perspective
[5, 6]. The two artists’ focus was shifted from the experience of the audience to that
of the performers, and the manner in which the latter embodied the received aural
and ocular stimuli.
Having explored independence between sound and movement throughout their
work in the 1950s, in 1965 Cage and Cunningham developed Variations V, arguably
the first performance to feature technologically facilitated interaction between music
and dance. Working alongside the artists, engineers Billy Klüver and Robert Moog
developed two distinct approaches in motion-activated control of music equipment.
Respectively employing photosensitive cells and proximity antennas [7], the system
was designed as to react to dancers’ movements by operating the transport con-
trols of several tape machines storing Cage’s sound palette. As such, the notion of
sound governing the ‘propulsion’ of movement was all but eliminated in Variations
V. However, as Uroskie points out, the dancers were not in conscious control of
the sonic palette, which would imply that ‘one model of subordination would have
merely been exchanged for another’; instead, movement was utilized to ‘set a cer-
tain train of sonic events in motion’ [5]. In other words, while dancers performed
Cunningham’s precisely rehearsed choreography [7], the sounds triggered from the
interaction between movement and the two motion-detecting systems resulted in an
indeterminate sonic outcome, an approach in music score that is often associated
with the practice of Cage.
With the benefit of more than fifty years of technological developments, a mod-
ern look on the modes of interaction between sound and movement in Variations V
could deduct that the artists were not able to implement a precise control of sound
through movement due to the limitations of the relatively crude analogue system in
interpreting movement into actuating commands. Nowadays, interpreting movement
into digital data has given rise to GRT devices that allow a detailed level of control-
ling software through physical movement. Such systems are implemented in diverse
applications within the fields of computing, manufacturing, as well as arts. On the
latter field, researchers and practitioners exploring interaction between sound and
movement have formed dedicated communities, such as the annual conference New
Instruments for Musical Expression (NIME).
While much of the focus of this research field has been the development of new
technologies and methodologies facilitating the expression of sound through move-
ment, implementing these contributions demands a certain level of technical profi-
ciency and knowledge of the associated language and vocabulary. In my practice,
I was faced with this issue during my early collaborations with Roberts. To further
contextualize the aforementioned independence between our respective material, our
approach resulted from the lack of a shared vocabulary. In other words, while Vari-
ations V presupposed a degree of freedom between sound and movement due to
imprecise technology, that freedom in my work with Roberts was dictated due to
imprecise language. Although my mapping of the Wiimotes’ motion data to sound
processing parameters made sense to practitioners familiar with audio software, com-
municating these mappings verbatim to my collaborator was inefficient in allowing
Roberts to understand how her movement affected sound, an issue further com-
pounded by my own lack of knowledge in the language of choreography. While
we were able to overcome these issues due to our prolonged collaboration and per-
sonal familiarity, I was aware that the privilege of time and patience will not always
be present. This realization prompted me to explore a way of communicating my
desired intentions in shaping sound through movement via the use of GRT. Prior to
this specific research question, my research contributed a framework comprising of
a set of strategies and precepts towards guiding polydisciplinary collaborative pro-
cess [2], with its core concept being the biological phenomenon of symbiosis. From
its serendipitous appropriation as the name of our initial collaborative engagement,
examining the scientific research of the phenomenon in the context of artistic practice
allowed for the development of a system derived from a disciplinary-neutral field,
thus avoiding presupposing greater importance to neither of our disciplines.
8.3 The Symbiotic Phenomenon
The lexicological definition of symbiosis (Oxford English Dictionary) suggests two

elements existing in a sustained harmonious relationship. However, in the con-
text of biological associations, harmonious coexistence is merely one of the many
manifestations of symbiotic relationships. Biologists define symbiosis as the close
126 M. Moriaty
and persistent relationships between organisms of different species [8]. Organisms

engaged in symbiosis are identified as the host and its symbiont, with the engagement
most often initiated by the symbiont becoming attached to the typically larger host,
motivated by the former’s desire to extract benefit from the relationship. In line with
the different levels of extraction of benefit, or fitness outcome [9], symbiosis mani-
fests in a variety of types. The three core types are mutualism, commensalism, and
parasitism, with each type identified according to the effects it has on the engaged
organisms. And since the symbiont is always benefited, it is the effect on the host
which defines the type (see Table 8.1).
This taxonomy of relationships refers to symbioses as they are observed at a
specific moment in time, described as research on ecological scale [10]. However,
examining symbiosis over longer timescales reveals its function as an evolutionary
mechanism, with the organisms’ evolutionary trajectory reacting to the close and
persistent interactions with their partner’s specific traits. For example, while a par-
ticular partnership may have begun as parasitic (which is often the origin of most
interspecific associations [9]), the host will eventually evolve ways to extract benefit
from the relationship, while the symbiont will also manage its exploitation of the
host with the aim of prolonging the relationship. This reactive evolution serves as
proof for an evolutionary trend from parasitism to mutualism, which is nowadays
widely accepted by biologists [10].
Having explained the core taxonomy of symbiotic relationships, the next step
towards drawing parallels between biological and creative associations is to firstly
identify the elements making up each partnership, and secondly establish a rela-
tionship between these elements. The first shared aspect among the two forms of
association concerns the partners’ motivation towards establishing a relationship;
that is to combine their individual traits as means of jointly overcoming limitations,
respectively borne of environmental [8] and disciplinary [11] factors. From this start-
ing point, the remaining elements are placed through a process of conceptual debate,
and finally become organized opposite each other, as summarized in Table 8.2.
This subjective interpretation assumes the symbiotic relationship as the collabo-
rative engagement, with the organisms as the collaborating practitioners. The inter-
specificity of the engaged organisms is reflected through the different disciplines
employed by each practitioner, with each discipline’s specific expressive media
related to the biological traits carried by each species. In interpreting the roles of
host and symbiont, these are allocated respectively on the practitioner instigating
the collaboration and the one who is guided according to the former’s direction. As
will become clearer later on, the used nomenclature applies to engagements which
Table 8.1 Symbiosis

Type of interaction Fitness outcome
typology and fitness outcome
Symbiont Host
Mutualistic Positive Positive
Commensalistic Positive Neutral
Parasitic Positive Negative
Table 8.2 Related elements of symbiotic relationships and polydisciplinary collaborations

Symbiotic relationships Polydisciplinary collaborations
Organisms Symbiont Instigator Practitioners
Host Directee
Interspecificity Biological traits Expressive media Polydisciplinarity
Fitness outcome Expressive range
Fig. 8.2 Effect direction in

the different types of
symbiosis and collaborative
process
feature a higher level of dependency, be that a parasitic relationship in symbiosis or

a directive collaboration [12] (see note 2) in artistic practice, with the prescriptive
meaning of these roles diffused during commensalistic/interactive engagements, and
almost entirely absent in mutualistic relationships and collaborative modes.
The final interpretation concerns the element of fitness outcome into expressive
range. As mentioned, the types of symbiotic relationships are identified according to
the host’s fitness outcome, or the level of benefit that organism experiences as a result
of its engagement with the symbiont. In the context of my interpretation, expressive
range refers to the level of creative input allocated to each practitioner during their
collaboration (see note 3). Figure 8.2 showcases the shared direction of effect between
the two types of association, with the partner’ extraction (or reciprocal exchange) of
fitness outcome in symbioses, and delimitation (or conversation) of expressive range
in collaborations. In the context of polydisciplinary collaborations, this approach
presents an efficient way of understanding and allocating each practitioner’s liberty
in developing their respective disciplinary material during the process of collaborative
engagements (see note 4), with the same principle available towards organizing the
relationship between musician and dancer while operating an interactive system.
8.4 Interactive Taxonomy: Theory and Precedents
As mentioned in the introduction, the typical approach of using GRT in music-

dance interaction concerns the change of sound through movement. This effect is
achieved by mapping movement data to various parameters of DSP devices, with the
128 M. Moriaty
sound consequently affected as a result of the movement data performing alterna-

tions on the parameter’s values. Considering this relationship between the two media,
sound can be understood as the symbiont medium, with movement being the host
governing the development of sound. With this principle in place, and taking into
account a host’s different fitness outcomes during each type of symbiosis, an equal
number of interaction modes can be derived, where the ‘host’ movement can be ‘ben-
efited’, ‘harmed’, or ‘unaffected’ by its ‘symbiont’ medium of sound. In line with
the previous subjective interpretation between the elements making up each associ-
ation, the biological notions describing changes in fitness outcome are related to the
restrictions, or lack of, placed on the expressive range of the associated media and
their respective practitioners during a performance. The most efficient way to estab-
lish these relationships is by first observing the resulting sonic outcomes, followed
by the restrictions placed on the movement, the provision in which the two perform-
ers develop their respective sonic and movement material, and finally the dancer’s
awareness of how their movement affects sound while operating the system.
Looking at the first association, when the collaborative performance requires a
determined sonic outcome (akin to a fixed music score), the dancer must perform
a specific set of movements in order to alter the values of the DSP parameters in
a predefined manner. As a result, this interaction mode imposes restrictions on the
movement’s range of expression in order to accommodate the desired sonic outcomes.
Furthermore, with the mappings between movement data and DSP parameters hav-
ing been created by the music practitioner, she or he needs to communicate to the
dancer the required movements needed to achieve the determined development of
sound over the duration of the performance. Consequently, the dancer is relieved
from having to fully understand the ways her or his movements may affect sound
beyond the predefined movements. As such, this interaction mode assumes move-
ment as a predefined modulator for the sound. In other words, through the previously
discussed subjective interpretation of the biological notions describing fitness out-
come, the ‘host’ movement is ‘harmed’ in order to ‘benefit’ its ‘symbiont’ sound,
thus establishing a parasitic symbiosis between the two expressive media.
On the opposite spectrum of sonic outcome, an indeterminate score entirely alle-
viates any requirement for the dancer to become familiar with the mappings between
movement data and DSP parameters, with movement remaining independent to
sound. However, from the musician’s perspective, the mapping must be designed
as to accommodate the dancer’s full range of movements which she or he may
deploy in any manner and time throughout the duration of the performance. In a way,
the randomized alteration of DSP parameters during this interaction mode can be
related to a generative music system, or to provide a further simplified reflection, to
the modulations derived by an LFO set to random or noise waveform. In the context
of the symbiotic interpretation, the ‘host’ movement is ‘unaffected’ due to enjoying
a full range of expression, while the ‘symbiont’ sound extracts ‘benefit’ in the form
of randomized modulations that can be used to develop and expand its outcome. As
such, this interaction mode forms a commensalistic symbiosis between sound and
movement, which on reflection can be associated with the interaction employed in
Variations V, when dancers had an effect on sound despite being unaware of the
ways their movements specifically controlled Cage’s tape players. However, while
Cage revelled in employed indeterminacy as a compositional approach, Cunningham
directed his dancers through an explicit choreography. Nevertheless, this is but one
manifestation of a commensalistic interaction mode, and as I present later in this
chapter, free improvization presents itself as fruitful provision for the dancer to fol-
low, with the musician tasked with designing a system able to generate meaningful
sonic outcome through random modulation inputs.
With mutualism being the remaining type of symbiotic relationship, such an inter-
pretation into the context of collaborative performance requires for both sound and
movement to mutually extract ‘benefit’ from their interaction, which considering
the earlier connection between fitness outcome and expressive range, suggest a
simultaneously full range of expression for both media. While the provision of free
improvization may at first appear salient to this mode, developing this mode through
practice showcased that an intermediate provision is more appropriate, that of struc-
tured improvization. Examining this provision in the context of music performance,
structured improvization differs from its free counterpart by the approach of creating
real-time compositions by connecting pre-established material over an arrangement
which is not predefined [13]. As such, while the resulting sonic outcome is not
determined, its characteristics can be anticipated. Structured improvization shares
a slightly different meaning in the context of choreography, with dancers adhering
to a predefined temporal arrangement in relation to stage placement and clustering,
while retaining freedom towards their performed movements during each section
of the arrangement [14]. Considering these provisions for music and dance respec-
tively, in the context of GRT-facilitated interaction the dancer is allocated freedom
towards her or his movements, with the caveat that these movements need to result to
anticipated sonic outcomes. As such, the dancer must be well-familiarized with the
system’s mapping, and be aware of the ways each movement may affect sound. In
other words, the mutualistic interaction mode presents a mutual compromise between
the expressive ranges allocated to sound and movement, with both media mutually
extracting ‘benefit’ up to the level at which one of them can be said to experi-
ence ‘harm’, thus resembling the mechanisms by which mutualistic symbioses are
developed over evolutionary scale in the natural world.
The three symbiotic modes of interaction are summarized in Table 8.3, with
each mode identified according to their specific effect awareness, provision,
Table 8.3 Taxonomy of symbiotic interaction modes with associated elements

Interaction modes
Mutualism Commensalism Parasitism
Affect awareness High Low Moderate
Provision Structured improvisation Free improvisation Score/choreography
Operation Exploration Detachment Instruction
Outcome Anticipation Indeterminacy Determinacy
130 M. Moriaty
operation, and outcome borne of the interaction between the two media. At this
stage, it is worth pointing out once again the subjectivity and conceptual nature of
this interpretation, and furthermore the Practice Research methodology employed
towards reaching these findings, with the latter derived through the accumulated
knowledge from numerous years of collaborative practice alongside several prac-
titioners. As presented later in this chapter through the work Symbiont Zero [15],
in addition to employing distinct modes of interaction, multiple modes can also
manifest during a performance, either consecutively during different sections, or
simultaneously while operating different layers of sound, each controlled via a dif-
ferent mode. Prior to describing our collaborative outcome and designed interaction
system, I demonstrate the taxonomy’s theoretical standing by examining three inter-
active works as means of identifying their specific modes of interaction. Rather than
providing a conclusive literature review of the field’s practical outcomes, the pre-
sented precedents were selected as to represent distinct uses of GRT, collaborative
processes, and employed aesthetics.
8.5 Symbiotic Interactions in Precedent Practices
Stratofyzika is a Berlin-based collective, with its founding member comprising musi-

cian Lenka Kocisova, visual artist Alessandra Leone, and dancer Heather Nicole (Hen
Lovely Bird). Their 2016 work THÆTA [16] presents a dialogue between the three
media of sound, movement, and visual. In the context of the symbiotic concept, the
dancer serves as the host modulating sound and visuals, with the interaction facil-
itated through a wearable Inertia Measurement Unit attached on the dancer’s arm,
with the movement data simultaneously modulating the two software suites gener-
ating sound and visuals. According to Leone, rather than having a constant flow of
movement data into the system, this flow is interrupted by altering the correlation
between movement data and resulting value alteration via algorithms [17], as well as
activating the data stream only during predefined moments. With the dancer aware
of the way her movement affect the other two media, it is during those moments of
activated interaction that the dancer is able to freely employ movements that result
anticipated sonic and visual outcome, such as the floor roll (see [16], 1:10–1:20)
resulting in a white spire and a tone of descending pitch.
The second precedent is one of the audio-visual installations developed with
the danceroom Spectroscopy (dS) visualization system. Using an array of depth
perception cameras (Microsoft Kinect) as its GRT devices, the system is able to
extract movement data from multiple participants, with the date used to modulate
the visualizations. While dS was designed primarily as a research tool for molecular
dynamics, allowing researchers to visualize particle movement, as well as interact
with their visual representations [18], it then became the basis for a series of creative
outcomes with the addition of a sound-generating system sharing the data modulating
the visualizations, with the movement data coming from a troupe of dancers. In
addition to the performances, the team behind dS also presented the system in the
configuration of a public multi-participatory installation, where audience members

are invited to interact with the system. The result of this interaction is that sonic and
visual events are manipulated by movement without the controllers having any prior
training with the system [19], thus demonstrating a commensalistic interaction mode.
A point of interest here is that once an audience member began exploring the system,
they would attempt to understand what their effect is on the media, resulting in an
adaption of the interaction mode. While this does not in itself form an undesirable
aspect, I later present a solution devised towards maintaining commensalism during
interaction.
Finally, as an example of practices displaying parasitic interaction between sound
and movement, Marco Donnarumma’s Corpus Nil [20] is a work which appears to
absolutely appropriate the notion. For this work, Donnarumma devised a choreog-
raphy comprising of ‘five key bodily postures’ [21], each designed as to force the
system to generate a specific response from the sound and lights it controls. With
the system operating through machine learning algorithms able to identify specific
positions by their duration, the performer must sustain these positions as to progress
through the arrangement. As such, Donnarumma is restricted by the design of these
key positions, or more accurately, is able to move only within the restrictions posed
by each position. Understanding this approach through the symbiotic framework, the
performer possesses a detailed awareness of the ways their movement affects sound
and lights, and is able to perform the determinate sonic outcome through their move-
ments. This sits in contrast to the approach employed by Stratofyzika and danceroom
Spectroscopy; in the latter, audience members are unaware of the ways the system
interprets their movements as modulations for the other two media, whereas in the
former, the dancer is aware of the effects her movements have on the interacting
media, and is allowed to explore movements along with the effects these have on
sound and visuals. In the case of Corpus Nil, the performer’s movements are intended
to achieve predetermined changes to the other two media.
8.6 Symbiotic Modes of Interaction in Symbiont Zero
As mentioned earlier, the findings presented in this chapter stem from my collabora-
tive practice alongside several dancers. Following the development of the theoretical
frameworks for polydisciplinary collaboration and symbiotic modes of interaction,
my aim was to activate the entire spectrum of interaction modes within a single
creative outcome, with the additional purpose of serving as a demonstration of the
framework. In order to satisfy both purposes, Symbiont Zero has been documented
both as complete performance, as well as a set of demonstration videos, with the latter
displaying a simplified version of each interaction mode, with additional perspectives
providing simultaneous views of the musician’s inputs via external controllers, as
well as the effects both performers conduct on the mapped parameters (see Fig. 8.3
for a representative example).
132 M. Moriaty
Fig. 8.3 Demonstration video with multiple views: dancer (Cunliffe), external controller, and
mapping distribution matrix
The work was initially developed with my long-term collaborator Roberts, and
subsequently performed with contemporary dancers Lucie Lee, Joseph Lau, and
more recently Kelsea-Leigh Cunliffe. The premise concerns a musician-dancer duet
featuring interaction between their respective expressive media through GRT. With a
total duration of approximately fifteen minutes, the piece develops over three move-
ments, or sections, of equal duration, and while each section predominantly focuses
on a single mode of interaction, the performers have gone to implement areas of
adaptation through simultaneous employment of different modes while controlling
multiple layer of sound.
In describing the path of the movement data into the system, the concerned GRT
devices comprise a pair of Wiimotes tethered on the dancer’s forearms as means of
performing alterations to the sound-generating system. This system is hosted within
Ableton Live, receiving a total of eight data streams from the two Wiimotes, follow-
ing their capture and conversion into MIDI CC messages through the application
OSCulator. These continuous messages are then mapped onto two Max for Live
devices, namely, Map8 and Multimap, which serve two purposes; firstly, acting as a
distribution matrix from which to map each data stream onto multiple DSP param-
eters, and secondly, allowing the musician to toggle the activity of each mapping,
as well as alter the range, direction, and curve of each mapping’s effect on their
destination during the performance. The musician’s inputs are achieved through two
hardware MIDI controllers combining continuous and toggle controls. The data path
is graphically represented in Fig. 8.4, (albeit for the data from a single Wiimote
Fig. 8.4 Path of gesture data, from dancer’s movement to DSP mapping
only), with the following subsections describing the work’s three sections in terms
of both system operation and performance planning. Each section is described under
the title of their predominant mode of interaction.
8.7 Mutualism
In line with the aspects of mutualistic interactions, the dancer has been familiarized
with the system, and is able to anticipate the ways her movements may affect sound.
Furthermore, the performers have noted certain combinations of movements and
specific ranges in the distribution matrix which resulted in interesting sonic outcomes,
which the performer has memorized and able to deploy at will.
The main interactive sound during the mutualistic section is generated by a feed-
back delay (Amazing Noises Dedalus Delay), with its parameters mapped to the
Wiimotes’ eight continuous messages, these being X-axis (vertical position), Y-axis
(roll), Z-axis (yaw), and acceleration. In this case, the X-axis of both devices is
assigned to the delay’s two filters, low pass and high pass for left and right arm,
respectively. This particular mapping ensures that the performer maintains control
over the overall signal amplitude, which subsides by simultaneously pointing both
arms upwards due to the removal of high or low frequencies by either filters. The
remaining mappings are arranged to parameters such as pitch shifting, size and
frequency of grain, and addition of parallel distortion and reverberation.
The musician’s role during this section is to adjust the range by which each con-
tinuous message affects the mapped parameters. This is achieved by adjusting the
range parameters on the Map8 device, thus altering the dancer’s level of influence
134 M. Moriaty
on each mapped parameter. The section begins with the filter ranges restricted mid-
dle frequencies, while the dancer concentrates on upper body movements. Halfway
through the section, the musician broadens the range of the filter mapping, causing a
sudden increase in amplitude. This serves as a signal for the dancer to initiate weight
transfer movements, resulting longer movements across the stage.
8.8 Commensalism
As described in previous sections, commensalistic interactions alleviate the need for

the dancer to possess prior knowledge of the ways movement affects sound. However,
as mentioned in the description of dS, participants in control of sound will naturally
begin to discover the connections between their movements and changes in sound,
resulting in an adaptation of mode. Aiming to avoid this adaptation during the com-
mensalistic section of Symbiont Zero, I designed a system able to dynamically alter
the mapping during the performance. This is achieved by mapping a single move-
ment stream of continuous change message to several DSP parameters by inserting
an instance of Multimap between one of Map8’s outputs and the DSP inputs. Fur-
thermore, toggles are assigned to toggle each of the Multimap’s outputs. The result
of this design allows the musician to continuously alter the movement data stream
distribution among the mapped DSP parameters, and thus avoid the dancer to gain a
fixed relationship between her movement and specific sound modulations.
The sound generation is based on a series of granulators (Audiority Grainspace)
processing recordings of synthesized percussive patterns, with the Wiimotes’ CCs
mapped to parameters such as start–end of input sample, pitch and tone variations,
saturation, size of and distance between grains, and level of parallel reverberation.
While the dancer is entirely at liberty to freely improvise her movements, she
is also motivated to ‘forget’ the interaction, and instead treat the resulting sound as
recording, akin to improvising to a fixed (non-interactive) composition. The only
cognisant interaction of the performer occurs after a new layer of sound appears,
made of two percussive samples (respectively kick drum on the left arm and snare
drum on the right arm), each activated by the acceleration streams going over a set
threshold which triggers a MIDI Note on message mapped to an instance of Ableton’s
Drum Rack containing the percussive samples.
Once the performer becomes aware of the new layer, she is able to interact with
the two sounds through a mutualistic mode, while simultaneously maintaining her
commensalistic interaction with the previous sound layer. The section ends with the
initiation of a further sound layer made of arpeggiated synthesiser pattern, signifying
the transition into the final section.
8.9 Parasitism
The parasitic interaction presents the highest level of accuracy from the dancer, while
at the same time imposes the highest level of restriction on their movements. The
Wiimotes’ CCs are mapped to a synthesiser (Ableton Analog) generating a fixed
arpeggio pattern, with the left arm controlling the synthesiser’s low pass filter cutoff
frequency on the X-axis and envelope attack duration on the Y-axis (rotation), while
the right arm controls reverb depth and delay feedback on X- and Y-axis, respectively.
In line with the approach during parasitic interaction modes, the sonic outcome is
determined, in this case being a slow modulation between the wet–dry balance of
the reverb, followed by a filter frequency sweep of equal duration. At the same time,
the determined outcome requires for staccato (sudden jumps in value) modulations
on the two remaining parameters, resulting in altering the note’s envelope shape and
feedback level. Finally, the two percussive samples from the previous sections are
required to be activated at the end of each filter sweep cycle, resulting in the dancer
interacting with this layer through a parasitic mode, as opposed to the previous
mutualistic engagement.
With the determined sonic outcome providing a limited range of movements for
the dancer, the latter is then free to develop their own choreography within this
restricted range. Having performed the piece with four different dancers, a partic-
ularly interesting finding is that while each dancer were able to achieve the deter-
mined sonic outcome, each reached that through distinct sequence of movements.
For example, Roberts maintained her core still while only moving her arms (as to
achieve the modulations on the synthesiser’s parameters), Lee continued to move
across the stage while limiting the movement of her arms in line with the score.
Similarly, while Lee activated the percussive samples by performing waving move-
ments with her arms, Cunliffe opted to perform sudden contractions and extension,
akin to punching motions. This showcases that despite the imposed restrictions dur-
ing parasitic interactions, dancers are able to interpret the score through different
choreographic movements.
In describing the remaining section, once the filter sweep sequence has been
performed twice, the dancer is instructed to resume a mutualistic interaction with the
percussive samples, while at the same time the musician limits the mapping range of
the synthesiser parameters, who is then able to assume control of these parameters
through their external controller. As a result, the musician ‘takes’ away control from
the dancer while she breaks free from the previous restrictions. This continues for
around one minute, before the dancer assumes the previous position, and the musician
returns the control of the synthesiser back to the dancer, who repeats the previous
movements twice. Symbiont Zero concludes with a short section of commensalistic
interaction, where the performer’s movements influence a pair of granulators (New
Sonic Arts Granite) processing two segments of recorded music, namely, the ‘Amen’
and ’Hot Pants’ drum breaks. As with the previous section, the performer is unaware
of the effect their movements exert on the newly introduced sounds, and is instructed
136 M. Moriaty
to perform an improvized sequence, with its intensity influenced by the overall level
of sound made of the combined elements, which begins to fade until silence to the
end of the piece.
8.10 Conclusions
Although the analyzed precedents do not constitute an exhaustive review of the field,
the interactive works portray a selected collection of practices displaying diversity
in GRT (Inertia Measurement Units, depth perception cameras, and biophysical sen-
sors), in the make up of their collective creation (inter/multi/trans-disciplinary respec-
tively), and as explained earlier, in the employed interaction mode between physical
movement and digital audio-visual material. Appropriating the typology of symbi-
otic relationships towards forming a new taxonomy in technologically facilitated
interaction allows for the diversity in which interactive performances manifest to be
identified within just three modes of interaction, each comprising of thee aspects—
provision, operation, and outcome—as well as the level of training or familiarity with
the system required by the dancer. This serves as an efficient approach towards com-
municating the purpose of each mode, particularly assisted by the lack of convoluted
terminology, which can often be counterproductive when used to communicate the
desired outcome to practitioners whose expertise focus on other areas of practice.
This research has thus far focused on dyadic collaboration, with future view of
assessing the symbiotic concept with multiple partners. Furthermore, while the use of
consumer GRT devices was implemented as means of making the research outcomes
more inclusive (due to being both reliable and inexpensive), the limited accuracy and
expansion of these devices have become to emerge in my practice, which serves as
motivation to look into developing bespoke devices, in line with current trends in the
field of interactive arts research.
Most importantly, however, it is my strong belief that the future of interactive
arts will greatly benefit from researchers combining technological innovation with
artistic Practice Research, a symbiotic relationship if you will. In my experience,
although some steps are already taken by few brave individuals, the two fields’ distinct
approaches, cultures, sensitivities, and nomenclature result in hesitation towards
tighter integration. But akin to the parasitic beginnings from which most symbioses
begin their evolutionary journey, persistence and closeness will eventually manifest
in mutual aid among the two domains of creative practice, artistic and scientific. In
the words of Pëtr Kropotkin, ‘In the practice of mutual aid… we thus find the positive
and undoubted origin of our ethical conceptions; we also see the best guarantee of a
still loftier evolution of our race’ [1].
Notes
(1) The term polydisciplinary is used to denote the multiple modes by which disci-
plines interact, as an alternative to the more popular ‘interdisciplinary’, which I
consider to be erroneously habituated for this purpose, as it ignores established
principles of taxonomy. I provide a more thorough discussion on this subject in
Sect. 2.2.2 of [3].
(2) The referenced modes of collaboration are derived from the work of Hayden
and Windsor [12], identified as directive, interactive, and collaborative, each
suggesting a different level of creative liberty between the engaged practitioners.
(3) The social aspects of collaboration, along with the notions of allocating creative
direction duties, privileges, and responsibilities, have been addressed in my
previous paper [2].
(4) For a detailed description of the symbiotic framework on collaborative process,
see chapter three of [3].
References
1. Kropotkin, P.: Mutual Aid: A Factor of Evolution. McClure Phillips & Co., New York (1902)
2. Moriaty, M.: Symbiosis: organisation and mutual exploitation in interdisciplinary collabora-
tion, artistic research will eat itself. In: Proceedings of The 9th SAR International Conference
on Artistic Research, University of Plymouth, UK, 11–13 Apr 2018. Cox, G., Drayson, H.,
Fatehrad, A., Gall, A., Hopes, L., Lewin, A., Prior, A. (eds.): Society for Artistic Research,
pp. 81–96 (2018)
3. Moraitis, E. (Moriaty, M): Symbiotic synergies: adaptive framework for polydisciplinary col-
laboration in performance practice. Ph.D. thesis, School of Arts & Media, University of Salford,
Manchester (2019)
4. Klosty, J.: Merce Cunningham. E.P. Dutton, New York (1975)
5. Uroskie, A.V.: from pictorial collage to intermedia assemblage. Animat. Interdiscip. J. 5–2,
223–241 (2010)
6. Kirby, M.: Happenings: An Illustrated Anthology. R&R Books, Reno (1966)
7. Miller, L.E.: Cage, Cunningham, and Collaborators: The Odyssey of Variations V. The Musical
Quarterly, Oxford University Press, vol. 85–3, pp. 545–567 (2001)
8. Douglas, A.E.: The Symbiotic Habit. Princeton University Press, New Jersey (2010)
9. Paracer, S., Ahmadjian, V.: Symbiosis: An Introduction to Biological Associations. Oxford
10. Sapp, J.: Evolution by Association: A History of Symbiosis. Oxford University Press, New
York (1994)
11. John-Steiner, V.: Creative Collaboration. Oxford University Press, New York (2000)
12. Hayden, S., Windsor, L.: Collaboration and the composer: case studies from the end of the 20th
century. 61–240, 28–39. Cambridge University Press, Tempo (2007)
13. Cage, J., Kirby, M., Schechner, R.: An interview with John Cage. Tulane Drama Rev. 10–2,
50–72 (1965)
14. Pressing, J.: Improvisations, methods and models. In: Sloboda, J. (ed.) Generative Processes
in Music, Clarendon Press, New York, pp. 129–178 (1987)
15. Moriaty, M.: Symbiont zero: multimodal interactive performance, 19 Aug 2019. Accessed 20
Nov 2019 [Online]. https://manolimoriaty.com/symbiont-zero/
138 M. Moriaty
16. Leone, A.: StratoFyzika—THÆTA, 16 Dec 2016. Accessed 20 Nov 2019 [Online]. https://
vimeo.com/195961322
17. Leone, A.: Personal interview, 19 Feb 2018
18. Glowacki, D.R., O’Connor, M., Calabró, G., Price, J., Tew, P., Mitchell, T., Hyde, J., Tew, D.P.,
Coughtrie, D.J., McIntosh-Smith, S.: A GPU-accelerated immersive audio-visual framework
for interaction with molecular dynamics using consumer depth sensors. Faraday Discuss 169,
63–87 (2014)
19. Royal Society of Chemistry: Atoms and molecules are not static by dance room Spectroscopy
(dS), 3 June 2014 [Online]. https://www.youtube.com/watch?v=H_EJquvSBMQ. Accessed 20
Nov 2019
20. Donnarumma, M.: Corpus Nil, 22 Jan 2016 [Online]. Accessed 20 Nov 2019 [Online]. https://
vimeo.com/152710490
21. Donnarumma, M.: Configuring corporeality: performing bodies, vibrations and new musi-
cal instruments, Ph.D. thesis, Department of Computing, Goldsmiths University of London,
London (2016)
Chapter 9
Between Presence and Program: The
Photographic Error as Counter Culture
Tracy Piper-Wright
Abstract Common photographic errors such as over or under exposure, blur, or

inadvertent cropping are increasingly rare as technological developments in digital
photography have sought to eradicate the error from practice and perception. Effi-
ciencies such as camera automation and image preview are often designed to remove
the ‘unreliability’ of the human element in order to produce accurate and consis-
tent images. The error, occurring on the margins of practice and increasingly rare,
provides a counterpoint to this prevailing image culture by revealing the interde-
pendence of photographer and camera through unintended outcomes. This chapter
explores the ideological assumptions entwined in the development of camera tech-
nologies, and how cultures of practice based on a hierarchy of control between
camera and photographer arose. Through examples drawn from the research project
In Pursuit of Error, the chapter demonstrates how the error disrupts this hierarchy by
evidencing the shared subjective agency of camera and photographer. The method-
ological framework of Actor-Network Theory is used to interrogate the relationship
between photographer and camera and to reveal them as equal ‘actants’ in the event of
photographing. The embodied photographer’s attitude of play, experimentation, and
not-knowing is interdependent on the camera as a co-creator of unexpected image
events which disrupt the conventions of photographic representation.
Keywords Photography · Error · Agency · Doubt · Actor-Network ·

Embodiment · Subjectivity · Time · Hierarchy of agency · Digitization ·
Virtuality · Actor-Network Theory · Transformative art · Alternative realities ·
Presence · Cultures of practice · Experimental photography
9.1 Introduction: Histories and Hierarchies
Alone of all the creative arts, photography is the most deeply enmeshed with the
technologies of production. A photograph cannot be made without an instrument
external to the photographer whereas one might argue the technologies of drawing
T. Piper-Wright (B)
University of Chester, Chester, UK
e-mail: t.piperwright@chester.ac.uk
140 T. Piper-Wright
or painting can, on occasion, be replaced by the human hand and the right surface.
The symbiosis between camera and photographer makes for a peculiar push and
pull between the claims of the technical and the subjective, which have been the
hallmark of writing about photography as, variously, an art, a craft and a science
since the inception of the medium. The tension between an apparently objective and
automatic picturing mechanism and the role of the photographer in the creation of
the image dates back to the critics and practitioners of the late nineteenth century,
and centers around a hierarchy of agency—who or what is finally responsible for the
photograph?
The unavoidable connectedness between camera and photographer is not a fault
of the practice, but rather an important distinction which must be recognized in any
discussions about it. In the early twenty-first century the digitization of photography
coincided with the connectivity offered by the Internet, producing the circumstances
for limitless digital images to be created and subsequently shared and viewed in
distributed networks. This rapid development created extra layers of technological
distance between photographer and camera as points of tension emerged between
the physicality of analog film and the ‘virtuality’ of the digital photograph [1, p. 97].
As digital cameras become increasingly abstruse and proliferate the question of the
relationship between it and the photographer becomes one of knowledge as well as
agency. If I cannot know what the camera is actually doing, how much control can I
have? The “black box” [2, p. 27] of the digital camera lies so far outside the realms
of ordinary understanding that photography has almost returned to the realm of the
magical.
What made photography ‘magical’, in its earliest presentations, was its ability to
capture and retain an image of the world with a precision hitherto impossible through
the human hand alone [3]. While early discussions on the invention of photography
refer to the photographic process rather than the camera itself; the technology of the
camera as an anthropomorphized ‘eye’ was first proposed by Da Vinci in relation
to the camera obscura [4] the earliest form of enclosed light projection from which
cameras were developed. Thus, some assumptions about photography were set out at
the early stage of its development: a monocular, perspectival viewpoint which mimics
the focus of attention in human vision and a unique responsibility to reality which
is the product of an “unreasoning machine” that produces “an unerring record” [5,
p. 269].
The camera’s ‘eye’ supersedes the frailties of the human eye through its objective
stance and ability to transmit and reveal the world as it is, without interpretation
or editing. The camera becomes a technology for asserting truth, and the images it
produces are facts that can be used as evidence.
Leading from its perceived status as a truth-telling machine, the quest for sim-
plified and accessible accuracy has been a dominant motivator in the technological
development of the camera. The famous Kodak strap line which accompanied the
launch of its camera in 1888 “You push the button - we do the rest” [6, p. 38] exem-
plified a desire to remove the possibility of error in the creation of a photograph by
eliminating the more complicated or technical aspects both in camera design (the
camera had a fixed focus lens and limited aperture/shutter speed controls) and in
9 Between Presence and Program: The Photographic Error … 141
the loading and development of film (the whole camera was sent back to Kodak for
processing, thence replenished and returned to the owner) [6].
The Kodak camera is evidence of the twin aims of technological progress which
can be found in most subsequent camera development including digital. Firstly, there
is the desire to ameliorate the complexity of the photographic process per se; the
unknowable chemical/algorithmic functions which are performed in the making of
the image. Secondly, the camera sets out to moderate the potential of a faulty, error-
prone human element from entering the photographic equation. The photographer is
reduced to being a button pusher, and no more. The twin aims of camera development:
simplicity and error elimination can be seen in many of the innovations in camera
technology in the twentieth century, such as cassette film and auto-focus, which
continued into the twenty-first century digital revolution.
These reflections on the early history of photography show that the pursuit of
accuracy has been at the forefront of the practice and perception of photography since
the beginning. The linking of photography as a technical yet simplified image making
method that requires the presence of a ‘button pusher’ but which devolves the image
making to the camera, makes for a peculiar awkwardness in the relationship between
human and camera: an oddly mismatched hierarchy in which the photographer is
perceived as both necessary and extraneous to the photographic event.
Despite these advances, errors still crept into the practice of photography. Not
every eventuality could be eliminated, and the material agency of the film and the
chemical process contained many potential pitfalls for the photographer, which were
often only revealed once the prints had returned from the developer. Unsatisfactory
images could be discarded, but were often simply incorporated back into the wallet
of photographs. The gap between the making and revealing of an error effected a
nostalgic distance between event and object in which the significance of the image
could override the obvious flaws in the depiction. In the analog era photographs were
perceived as a finite resource, and an error might appear a charming reminder of a
moment in time.
Digital photography took its cues from film photography in terms of camera
design and it is therefore unsurprising that its subsequent development extended
the simplicity/error-elimination methodology. However, digital photography offered
a level of certainty that far exceeded what had been possible before. Due to the
algorithmic nature of the sensor and digital processor cameras became computers,
and the photographer became less and less the primary decision-maker [7]. Later
developments such as image preview effectively capped the potential for error by
making it possible to delete substandard images at source, narrowing the possible
images that could emerge from any given shoot. In the digital era the opportunities
for ‘casual’ error creation changed; no longer could a substandard image slip through
the net and into the pile of prints. Each image could be vetted so that only the best
were eventually presented in online spaces.
As our consumption of photography as digital images in networked environments
increases the more our perception of photography reflects what we are presented
with—invariably ‘good’ photographs. In this case a ‘good’ photograph does all that
a photograph should: it shows things as they are, to the best of the camera’s ability.
142 T. Piper-Wright
The technology of the camera, now most often embedded in a phone, can control
lighting, saturation, and shadows to produce high quality images that feel ‘just like
being there’. This quest for transparency, for a feeling that you are looking at or
experiencing the ‘real thing’, has been the ultimate goal of consumer digital photog-
raphy, with increased mega-pixels offering more and better rendering of the scene.
The photograph becomes a window through which we observe the phenomenon
depicted. This hyper-verisimilitude removes even the camera as the creator of the
work, ostensibly rendered out of existence by the perfection of its image [7].
Concomitant with these advances is the increase in the quantity of images made
available by digital technology. Not only do the best arrive on the Internet, but pretty
much everything else, so that we wade through a sea of images at every turn. While
photographs enter the network within a particular context they do not remain there
in perpetuity, and can soon break free to circulate and appear in multiple sites and
locations. The plurality and mobility of the digital photograph leads to a dissociation
of the image from its context in time and place and a peculiar sense of authorlessness,
where even the fact that we are looking at an image made by a camera pointed by a
human can escape our perception.
The foregoing commentary sets out the context in which the research project In
Pursuit of Error [8] (hereafter referred to as IPE) emerged and seeks to challenge.
The project collects photographic errors from artists, photographers, and the general
public in order to form an archive of the error in contemporary practice. Contributors
are asked to include a statement that explains how their photograph occurred and
why they feel it is an error. Through analysis of the images and narratives, the project
reveals how artists value the accident as a point of discovery and development and
engage in playing with and against the camera in the creation of ‘deliberate’ errors.
The project aims to question some of the assumptions about representation and
truth which beset photography as a practice, and to challenge the assumption that
technological developments manifest objectively rather than because of cultural and
ideological assumptions about what is correct or good. Thus the project attempts
to destabilize the hierarchy that attaches to the relationship between camera and
photographer (with one or the other ‘in control’ at any given time) in favor of exposing
the horizontality of this relationship [9, p. 283]. The project uses the error to explore
a revitalized concept of photography as an embodied, performative practice, in which
the bodies of photographer and camera play a part.
9.2 Camera Acts and Human Acts
In order to consider how humans and cameras interact in the creation of a photo-
graph, it is necessary to think through the processes that bring a photograph into
being. Despite common assumptions about time in relation to photography—the
split second ‘capture’—making a photograph has a number of stages, from the first
desire to take a photo, to preparing or starting the camera, to focusing or selecting the
composition through viewfinder or on screen and only then the taking of the photo. In
this extended time period both camera and photographer perform a number of acts,
the photographer her gestures, the camera ordered by the principles of its program.
Within these parameters correct images are to be had; however, flouting the rules of
either program or gesture tend to produce unexpected or unintended results.
It is useful to consider the practice of taking a photograph as operating on a
continuum of control between agency and automatism. While one might assume
that ‘agency’ pertains solely to the photographer and ‘automatism’ to the camera,
in practice these roles are interchangeable [10]. Automatic gestures—holding the
camera steady, or straight, or automatic programs such as auto-focus or aperture
control create more or less expected results according to the scene presented. Agency
plays its part in the decisions or mistakes of the photographer but the camera’s
agency can also intrude on the photographer’s intentions, resulting in surprising and
unexpected images.
The photographic errors submitted to IPE cover a range of accidental or unintended
images and ‘deliberate’ errors in which a willful action has been made to achieve an
unpredictable result. In many cases it is near impossible to identify from the resulting
error image which party was the instigator, without the photographer’s identifying
narrative in support (Fig. 9.1). While the visual distinction between the two types
of errors is not necessarily apparent, they create a useful distinction which permits
Fig. 9.1 Image copyright © Gail Griggs and reproduced by permission. Source: In Pursuit of Error
[8]
144 T. Piper-Wright
a focus of attention on either actor in the event of photographing—the camera or

the photographer. The narratives that accompany the contributions to IPE reveal the
push and pull between the photographer’s vision and the camera’s vision, and where
the presence of the camera as agent in the creation of the image is made apparent.
9.3 Agents and Networks
To consider the camera as an agent, or an accomplice, in the act of photographing

is to reconsider the role of the technology and our relationship with it—as user,
as master, or as collaborator. Actor-Network Theory (ANT) creates a framework
through which to reconsider this relationship, based upon acknowledgement of the
“symmetry of humans and non-humans” [10, p. 642] in networks. Networks are
formed of social practices and technical processes as well as human and non-human
‘actants’ and vary in scale. A network could be as small as the photographer and her
camera, or can be broadened out to consider the camera’s developer and manufac-
turer, the photographer’s educators and peers, and the social and cultural context of
photography practice. ANT emerged from the sociology of science and technology
as a means to critique assumptions about the development of technology in soci-
eties, pointing out relationships of power and questioning deterministic models of
scientific development which present such development as rational and autonomous.
ANT’s non-hierarchical emphasis on ‘actants’ erodes distinctions between human
and non-human actors and creates opportunities to consider how technology not only
‘speaks back’ to us through our interactions with it, but also how technology governs
and en-frames our performance with it [11].
In the context of Actor-Network Theory the error sheds new light on the rela-
tionship between camera and photographer. Many of the accidental errors that are
presented to IPE occur due to the mismatch between the photographer’s intentions
and the camera’s program, or more strictly speaking, the settings that have been
selected on the camera. Most modern cameras permit the pre-selection of aperture
and shutter speed to account for the proximity of the subject and light conditions.
Errors occur when the conditions fail to match the camera’s settings, or vice versa
(Fig. 9.2). In this case the situation being photographed becomes another actant in
the network, a set of conditions which intrudes upon and affects the outcome.
In the surprise that is created by these unexpected images the photographer is made
aware of their reduced role in the event of photographing, instead giving over the task
of picturing to the camera. This mismatch between program and situation, ‘translated’
by the camera, presents a wholly new way of perceiving, a form of camera vision
which reminds us of the subjectivity of perception per se. While gross mismatches
between program and conditions create image-worlds far beyond our imagination,
small collisions between settings and situation create spikes of maladjustment in
the image through which the presence of the camera as an observing body can be
detected. The accidental error unequivocally brings the camera’s presence to the fore
(Fig. 9.3). These images do not allow us to look ‘through’ or ‘past’ the photograph
Fig. 9.2 Image copyright ©

Ian Wright and reproduced
by permission. Source: In
Pursuit of Error [8]
as medium, instead, through its faults, the ‘photographicness’ of the image is made
glaringly apparent. The accident is therefore a valuable reminder of the technological
mediation which the camera brings to the act of photography, a subjective vision
which is easily overlooked in conventional photographs.
Photography’s connection to the principles of objectivity and neutrality, which are
socially and culturally grounded, have accustomed us to treat the camera as a perfect
imaging-machine. The accident, revealing the fallibility of this conceit, reminds
us that while automation exists, it is not a panacea which will offer unblemished
facsimiles of the world every time. The accident demonstrates the camera’s vision
as subjectively driven rather than an objective fact and foregrounds the extent and
limitations of the technology that lies behind that vision. These types of errors also
reveal the role of the photographer in choosing the most appropriate functions to
create an image, and thus the interdependence of camera and photographer in the
event of photographing.
Not all accidents are the result of camera malfunction or wrong settings. Acci-
dental images also arise through physical mishap (falling, shaking) or something
inserting itself between the lens and the scene at the moment of capture. These
intrusions into the photographic event evidence the wider field of actants in the Net-
work of photographing. While the camera/photographer nexus would appear to be
146 T. Piper-Wright
Fig. 9.3 Image copyright © Catalina Codreanu and reproduced by permission. Source: In Pursuit
of Error [8]
the sole locus of creation, the error reveals the multiple secondary elements: light,
time, subject, background; the numerous observable and unobservable phenomena
that can impinge on the image. The accidental error reveals the momentariness of
the photograph, its situatedness in time and space and its fixed location and view-
point. Conventional photography practices strive to present a scene which is as far
as possible universal, a witness to an event rather than a participant, working at an
observing distance which lends the scene a sense of objectivity and ‘truth’. This
photographic stance, as common in newspaper reportage as holiday snaps, relies on
a detachment between the photographer/camera and the scene, either through pose or
‘stilling’ of the action (‘smile!’) or through remaining on the periphery. By contrast,
the accidental error provides incontrovertible evidence of the embeddedness of the
camera/photographer within the flow of space and time through which pours all that
is normally left out of the photograph (Fig. 9.4). Through happenstance the error
records these fleeting intrusions and rapid changes that make up our customary lived
experience, fixing for a second the swirl of life that we are part of.
Accidental errors create opportunities to observe the camera at work in the creation
of the image, in many cases an overlooked or taken for granted act. For Borda [12] the
physical materiality of the camera is in danger of being left behind in an increasingly
Fig. 9.4 Image copyright © Brett Chapman and reproduced by permission. Source: In Pursuit of
Error [8]
digitized, automated picturing world which makes the camera “a missing entity that
we no longer perceive.” [12, p. 179]. The physicality of the camera has been greatly
diminished by recent developments such as mirror-less technology which removes
weight and mechanical action and in mobile phones which integrate the camera as a
part of a hand-held computer. If the ‘objectness’ of the camera is somewhat at risk
from these developments, the accidental error reasserts it as a form of what Jane
Bennett would term ‘thing power’: the agency of material objects to act and impact
other materials, thereby producing effects in the world [13]. The thing-power of the
camera is writ large in the accidental error, where the photographer is left wondering
what might have occurred, standing outside the making of the photograph, receiving
the image as a fait accompli. For Bennett, acknowledging ‘thing power’ entails
recognizing what she terms its ‘out-side’, that which we know to exist but can only
indirectly perceive [13]. The camera exercises its ‘thing power’ through the hidden
process of bringing the photograph into being. We know the theory but the actuality
of the process is held in abeyance, because the chemistry or the program is closed
to our perception [2]. This hiatus is where the camera exerts its agency and where
for a few moments the balance of power between photographer and camera is tipped
in the camera’s favor. This gap in our knowledge should remind us that we are not
necessarily masters of our technology, but rather informed collaborators, operating
on a certain amount of knowledge and an element of trust.
148 T. Piper-Wright
9.4 From Tool to Plaything
The hiatus that occurs in the moment of translation from event to image is an aspect
of all photography, correct images, and mistakes alike. This loss of control, how-
ever momentary, seeps into the practice of photography and informs the relationship
between camera and photographer which transpires to be a balance between knowing
and not-knowing, of control and loss of control. Thus, photographs could be orga-
nized on a continuum from those made with complete knowledge and command,
and those produced as a result of a complete loss of control and ignorance. The error
could be said to occupy the middle to latter part of the continuum and thus express
not a simple binary distinction between ‘correct’ or ‘faulty’ images but rather an
elongated space of practice in which minor to major mistakes or maladjustments,
made by either party, impact upon the eventual photograph.
Control could be considered a cornerstone of camera development through techno-
logical enhancements in efficiency and ease of use. The vagaries of the photographic
process are brought under control by innovations such as cassette film, the digital
sensor or ‘Auto’ pre-sets. The control is purportedly for the photographer, who can
now manage the process more readily, but in fact the control, particularly in relation
to programmable functions on modern digital cameras, is situated firmly with the
camera, or perhaps we should say the camera’s manufacturer. Elements of control
can also be exercised through a culture of photographic practice; the way in which
one should hold the camera or operate the shutter was often of particular concern
in manuals for new style cameras where the photographer’s performance with the
new device was as much under scrutiny as the camera. Flusser points out that both
‘apparatuses’ (as he terms cameras) and photographers are subject to the control of
the Program: the systems which define and limit the practice of photography [2].
Flusser is rather negative about the possibility of photographers escaping the limi-
tations of the Program but he does recognize the agency of the individual in testing
or subverting the expectations of picturing. For Flusser, the increasing functional
capabilities of the camera, its automation of the photographic process, have liberated
the photographer from the ‘work’ of making pictures. Now there is nothing else to
do but play [2, p. 29].
Playing with the camera to undermine expected or ‘correct’ image making can be
seen in the ‘deliberate’ errors, or rather, photographic subversions submitted to IPE.
These images are often accompanied by a narrative that suggests the photographer’s
moment of performance with the camera and their action in the moment that has
brought the image into being. Subversions often involve the photographer’s physical
movement, running counter to their expected role of stillness and stability—a human
tripod which facilitates the camera’s task. These movements can be complemented
by a knowing destabilization of the camera functions—extensions of shutter speed
to slow down the camera’s capturing eye, forced de-focusing to render the scene into
amorphous blobs of color (Fig. 9.5).
These actions could be characterized as playing against the camera, forcing
it to see in a very different way through manipulation of the controls. Whether
Fig. 9.5 Image copyright © Melis Cantürk and reproduced by permission. Source: In Pursuit of
Error [8]
through movement or manipulation the subversion foregrounds the photographer as

an embodied presence in the act of making photographs in a similar way that the
accident reveals the camera’s subjective viewpoint. The subversion interposes the
photographer between the camera and the scene as much as if a stray thumb or fore-
finger were to appear in the frame—we are reminded that the camera is not simply
alone and recording by itself as an objective observer. The subversion reminds us that
there is a photographer who is pointing, choosing, moving or otherwise governing
the activity that precedes the image.
These subversions become traces of performance in time, capturing a certain con-
figuration of events that involve camera, photographer, and situation. One may well
ask why photographers feel compelled to intervene in the process of photograph-
ing in this way. Certainly, the desire to do so is evident throughout the history of
photography, notably expressed as a principle of the Provoke photography move-
ment in Japan in the 1950s in their mantra of ‘are bure boke’—“grainy, blurry, out
of focus” [14]. Practices which test the conventions of photography can be seen in
the work of practitioners as diverse as Meatyard [15] and Barth [16] whose out of
focus experiments suggest alternative ways of using the camera as a subjective and
sensory mode of expression rather than an objective recording device. The deliberate
error is therefore more than just play; it is an urgent reassertion of creativity in the
act of photography, one that seems to be increasingly marginalized by technological
advancement which attempts to drive the unreliability of the human presence further
away from the photographic act.
150 T. Piper-Wright
Deliberate errors may also be motivated by a desire to not produce photographs.

One of the main issues in photography theory has been the ‘reality effect’ of the
photograph [17], that is to say that the photograph is not so much a created image
as a recording, a document of what was in front of the lens at any given time. The
reality effect is what lends photographs their ‘transparency’ [18], our tendency to
look through photographs as material media to focus solely on what is depicted within
the frame. The emphasis on transparency is what drives technological development,
a quest to create a photograph which will be as real as being there, a pure and
unadulterated ‘slice of time’. As technological development makes this more possible
it is little wonder that photographers expand their thoughts into new territory and
tend to explore the different forms of picture-making that the camera may afford.
Elements of Francois Laruelle’s thesis [19] can be brought to bear on the deliberate
error in this regard. Laruelle attempts to disentangle photography from its grounding
in doubling or copying by pointing out the philosophical distinction between the
object in the world and its photographed representation [19]. Perceiving the object and
its photographed form as the same is a fallacy that over-prioritizes the world and the
objects in it, without considering the distinct ontology presented by the photographic
image. For Laruelle claiming the non-identity of the photograph releases it from the
dualities of the ‘reality effect’ and permits it to be a related but non-identical depiction
of objects in the world. Set free in this way the photograph is no longer the poor hand-
maiden of reality but another presentation of reality which is distinct and unique to
photography as a material practice [19, p. 55].
The potential of the photograph as a unique form of envisioning is suggested
by Laruelle’s commentary, and speaks of the practical understanding of photogra-
phy as a transformative art. While embedded in an examination of the photograph
as a philosophical problem, Laruelle’s commentary expands photography from a
simplified and primarily technological act toward an understanding of the central
importance of the photographer/artist who, through the “spontaneous philosophy”
of practice can reveal the “very essence of photography” [19, p. 85]. The photog-
raphers exploring the photographic subversion reveal the ambiguity at the heart of
photography by making images that run counter to the principle of representation.
The creation of photographic subversions often centers upon an expansion of
some kind: extended physical movement or elongated shutter times, which result in
strange depictions of seemingly other-worldly phenomena—traces, trails, lines, and
blurs which suggest but do not signify; the original scene lost behind a veil of color
and movement (Fig. 9.6). These images present a divergent concept of photography
which does not adhere to conventional understanding. They are often described as
‘painterly’, a direct response to the evidence of gesture and time that are inscribed
on the image. They are in many respects, a form of ‘non-photography’ divested of
the need to stand in for things in the world, instead presenting another reality which
is lost to our normal human vision. For those who seek a deliberate error there is no
need to make a photograph per se—we have machines that can do that for us. For the
artist the liberation lies in the expanded capacity of the camera as a tool for seeing
the world in radically revised ways.
Fig. 9.6 Image copyright ©

Una Li and reproduced by
permission. Source: In
Pursuit of Error [8]
The satisfaction of the accident, and of being presented with the will of the mate-
rial (or technology), is an intrinsic part of an artist’s developmental method, their
motivation to explore and progress. ‘Not-knowing’ drives practice forward [20].
In photography, the subversion contrives opportunities for not-knowing and to be
surprised by unexpected outcomes. While a certain set of actions are put in motion
it is not the case that the photographer can know how and to what extent they will
affect the resulting image. The photographer may know that certain operations or
actions are likely to produce one type of result, but she cannot predict the image
outcome as when making a photograph in the conventional way. The subversion lies
further toward the knowledge/control end of the continuum discussed at the start of
this section, but it shares similarities with the accident in the gap that opens between
intentions and outcome. This gap is made up of the present tense performance of
photographing and the combination of photographer, camera, light, time, subject,
and situation which coalesce in the resulting image. The error is an improvisation,
always occurring in the immediate, unscripted moment and thus unknowable until
that moment has passed.
152 T. Piper-Wright
For photographers who engage in the deliberate subversion of photographic pro-

tocol the results of their transgression offer a similar level of satisfaction to encoun-
tering an accident, and can perhaps offer new avenues to explore in practice. The
possibility of a different type of photography is offered by the artists who tread this
path.
9.5 Tinkering with Technology
The foregoing has shown how the error steps outside the conventions of photographic
practice and reveals a more complex relationship between camera and photographer
than that between an ‘operator’ and her ‘apparatus’. Photographers often have per-
sonal relationships with their cameras, forming a kind of symbiosis between actants
where one knows what a particular camera will do in a given situation. The pho-
tographer who engages in ‘camera vision’ connects to the photographic seeing that
is a required element of making good pictures and yet as we have seen this can be
disrupted by an unexpected event or knowingly cast aside. These apparent ‘failures’
could be seen as end points, or dead ends from which one must reverse or retreat,
or they could be seen as opportunities presenting new phenomena which stimulate
the imagination. Invariably the contributors to IPE couch their errors in this latter
context, as intriguing surprises which stimulate their practice or their perception
about what a photograph is or could be. The etymological root of ‘error’ lies in the
notion of ‘wandering’, suggesting a deviation which is both intended and imbedded
in not-knowing as a principle [21, p. 21]. To wander is to be open to the possibilities
that arise in that act of wandering, be they positive or negative and might also imply a
return to the conventional route at a later point. This conception of error encapsulates
how photographers use the error in practice to lay claim to previously unexplored
territory.
The territory occupied by the standards and procedures for ‘correct’ photography
is vast and well developed. Since the very early inception of the medium, pho-
tographic practice has been intrinsically tied to rules on the right way to use the
technology in order to produce good photographs. Some guidelines were initially
necessitated by the complex and awkward materials and processes entailed in early
photography, but rapid development of photographic substrates meant that processes
soon became easier, quicker, and cheaper. Rules are not made by technology itself,
but by the culture that surrounds and uses it. In the case of photography, the empha-
sis on progress and refinement exemplified through rapid technological development
worked alongside a perceived need to regulate and define the limits of this emergent
art form. Like all technologies, photography is deeply embedded in the social and
political structures that helped formulate it as a popular and commercial entity and
which continue to inform its development. In this network of influences, cultures of
practice emerge.
The most pertinent distinction in the practice of photography is between the
informed and the casual photographer, a distinction evident since the early days
of the medium’s history. The informed amateur photographer was an early adopter
of new technology, in some cases contributing to the development of the medium
through their experiments. Informed photographers created Photographic Societies
where they could discuss and disseminate treatises on the art and science of photog-
raphy, organizing its philosophical principles, its areas of concern, and its goals. The
casual photographer emerged as the medium became simplified and democratized
by the new easier to use substrates and cameras, culminating in the Kodak, which
offered them a means to enter the realms of photography without the necessary skills
and knowledge to create their pictures other than a willingness to point and shoot.
The first fully-fledged ‘movement’ in photography, Pictorialism, emerged as a reac-
tion to these new ‘snapshooters’ and to the automation and loss of human skill that
the new developments suggested for the medium [22].
Thus the technological development of photography, driven by democratization
and commerce, contributed to a fracturing of perceived elements of skill and knowl-
edge that were in the guardianship of a cognoscenti. Kodak’s invention forced the
Pictorialists towards increasingly ‘unphotographic’ works—those that capitalized on
the artist/photographer’s hand to destabilize the verisimilitude of the image, turning
it instead into proto-painting [22]. The casual photographers, freed from concern
about the quality of their images by the fast and easy methods available to them, took
more chances with their photos; and errors, mistakes, and mishaps became part of
the photographic landscape [23]. As Hand points out, this entry of the banal mistake
into the culture of photography practice challenged “the dominant discursive and
material legitimacy of photography, especially in terms of skill and expertise.” [24,
p. 104].
Arguably, this distinction between informed/casual photographers is still recog-
nizable today. The camera clubs of Britain are filled with those who pride themselves
on their technical skills and expertise while casual photographers take selfies on their
mobile phones. Photography’s intrinsic link with technology makes this distinction,
and the value placed on technical know-how, continually pertinent today.
The distinction between an informed and a casual photographer therefore lies in
the amount of control they are able or willing to exert over the technology of pho-
tographing, which includes the camera, recording medium, and image production.
The informed photographer, through her expertise, exercises control over the tech-
nology, mastering its principles to produce ‘correct’ photographs. The casual pho-
tographer is a receiver of the technology and perhaps somewhat subservient to it in
that its expertise appears to outstrip his own. With less emphasis on control needed
or desired by the casual photographer, technological development for this market
expands with handy, time-saving innovations which bring the ‘ideal’ photograph
within easy reach.
Thus, two paths of activity are suggested by cultures of practice in photography in
which the level of control over the events of photographing favor one or other actant.
As has already been demonstrated, the error appears to undermine this dichotomy and
suggests a third way of working with photographic technology which is akin to Sherry
Turkle’s concept of ‘soft mastery’ [25, p. 56]. This concept emerged from a study
carried out by Turkle and Papert on student programming styles [26] which revealed
154 T. Piper-Wright
differences between those who took a logical, top-down approach to a programming

problem, and those who preferred to work in an iterative, exploratory way with the
computer. The distinction between ‘planners’ and ‘bricoleurs’ was expressed in a
willingness, or otherwise, to engage with the computer as more than just a ‘tool’ but
as a collaborator in the process of developing the program, exploring the nuances
of its unique language and exploiting novel solutions to problems. Observation of
different programming styles revealed that two approaches were at work; on the
one hand there was a level of engagement with the process of programming as a
problem which unfolded in time, and on the other, an approach which sought to pre-
visualize all the aspects of the program before commencing work. This distinction in
approach revealed a difference in the quality of the relationship between programmer
and computer, which could be characterized for the bricoleur as a dialog and for the
planner as a ‘monologue’ [26, p. 136]. The notion of conversation entails responding
to the other party and perhaps adapting the flow of the conversation in accordance
with those responses, while a monologue implies a silent audience. When errors
occurred in the programming process, planners and bricoleurs differed markedly
with how they dealt with this, either as failure or as opportunity:
For planners, mistakes are missteps; for bricoleurs they are the essence of a navigation by
mid-course correction. For planners a program is an instrument for premeditated control;
bricoleurs have goals, but set out realize them in the spirit of a collaborative venture with
the machine [26, p. 136].
Turkle’s concept of ‘soft mastery’ expresses a position of informed knowledge

about the goal or the object to be realized, but an epistemologically different way of
making the journey. Soft mastery relies on a form of negotiation with the materiality
of the object in question, be it a computer or a camera, a programming language
or abstract problem. Turkle adapts the concept of the bricoleur from Claude Levi-
Strauss’s descriptions of knowledge creation in primitive societies, what he terms
a “science of the concrete” [26, p. 135]. To be concrete is to be present, tangible,
handleable, not abstract and conceptual. This notion of the presence of objects as sets
of things that can act and be acted upon is pertinent to a different way of understanding
the camera, not as a tool to be mastered conceptually but as a thing to be worked
with physically by testing it out and seeing what it will do.
This approach is readily witnessed in photographers who experiment with the cam-
era and suggests a level of playfulness and willingness to ‘see what happens’ when
using the technology. This kind of ‘tinkering’ with the parameters of photographing
can be seen in many of the contributions to IPE. It’s important to note that Turkle’s
concept of ‘soft mastery’ does not set out to simply reverse the dichotomy between
logic and practice, control and experimentation, but to expose the hierarchy that
would assume that logic and control were the only or best ways of engaging with
technology. The type of knowledge created through negotiation, through practice,
reveals alternative epistemologies that run counter to the status quo [26].
A good example of a bricoleur approach to photography can be found in the work
of Julia Margaret Cameron. Cameron was an early pioneer of photography as an
art form, creating striking portraits of friends and family in scenarios influenced by
literature and painting. A particular feature of Cameron’s photography is a deliberate

softness of focus. In her extensive writings she notes how this particular feature of
her work arose in practice, and led her to adopt it as a signature style:
I believe…that my first successes in my out-of-focus pictures were a fluke. That is to say,
that when focusing and coming to something which, to my eye, was very beautiful, I stopped
there instead of screwing on the lens to the more definite focus which all other photographers
insist upon [27, p. 77].
It is pertinent that while having complete command of the camera as a piece of

technology, Cameron instead chose to pursue a deliberate ‘mistake’ or subversion
of the accepted protocols for photographing in order to compose an image which
was closer to her intentions. Cameron’s informed but embodied interaction with the
camera was deemed ‘sloppy’ and incompetent by those who had set themselves
as gatekeepers for the craft, a regulatory status quo which set the rules by which
photographs were to be made [28]. The fact that Cameron refers to them as her ‘out
of focus pictures’ is perhaps a significant internalization of this criticism in a nascent
world of cultures of practice which were being formed and entrenched.
What Cameron’s early flouting of the ‘rules’ of photography suggests is a connec-
tion between photographer and camera in which the photographer becomes aware
of the frailties and limitations of the technology, and the creative potential that these
peripheries afford. This awareness stems from a willingness to get inside the camera
in some way, to envisage its ‘camera seeing’ as a unique form of agency which has
a contribution to make to the final image. In their study Turkle and Papert observe
this type of investment into inanimate objects as a form of “reasoning from within”
[26, p. 144] which is embodied rather than abstract. Reasoning from within requires
a proximal relationship with the objects that we use, rather than a distance which
views them as purely functional tools. To acknowledge the camera’s presence in
the making of a photograph is to acknowledge it as a collaborator and changes our
relationship with it.
Entry to this new way of thinking about technology might be as simple as the
first time we encounter something that has gone wrong. Bill Brown observes that we
rarely notice the objects that surround us when they are working correctly. In this state
they become invisible; we almost see past them, or through them. However, when
something goes awry we are forced to reckon with the object as a different order
of ‘thing’, an agent that has impacted on us in an unforeseen way. The breakdown
or interruption in the flow of expected behavior changes our relationship with the
object and redefines it [29].
Error, mistakes, tinkering, and subversion are therefore vital and important aspects
of photography as a creative practice. They create gaps in the armor of accepted
practice, chinks of light where other possibilities for photography can emerge and
be explored. The epistemological pluralism espoused by Turkle and Papert [26]
exposes a new way of relating to photographic technology; as an embodied, proxi-
mal relationship between two agents, two actors who negotiate in the creation of a
photograph.
156 T. Piper-Wright
9.6 Conclusion: Being in Two Minds
The foregoing has suggested some alternative ways to consider the relationship
between camera and photographer, and how cultures of practice emerge which rein-
force oppositional hierarchies between users and their tools. The power of the acci-
dent or the subversion to undermine some of the dominant metaphors that adhere to
photography—facsimile, accuracy, perfection—exposes a counter culture of photo-
graphic practice which has been explored and exploited by practitioners since the
early days of the medium.
To adopt the embodied and proximal methodology of the bricoleur is to be open
to a sense of discovery and unfolding in the creative process, a form of confident
‘not-knowing’ from which to begin.
Vilem Flusser observes that “The act of photography is that of ‘phenomenolog-
ical doubt’.” [2 p. 38]. For Flusser doubt in photography is expressed through the
potential multiple viewpoints enabled by the camera and the aspects of the photo-
graphic process which are invisible to us. Doubt is woven into the elements of the
photographic event in which multiple actants converge at a moment in time and in
the hiatus of control and decision-making that is characterized by the accident or the
subversion.
The etymology of doubt in ‘doubling’ in the sense of having to choose between two
things [30] offers another way of interpreting the not-knowing that we experience in
the event of photographing. To doubt is to be in two minds or to assume two positions.
The error exposes this doubling as it occurs in practice, as a dance between two
epistemological positions: to know and not know, to be in control and to relinquish
control, and as a dance between two actors: camera and photographer. To be in
two minds is to enter a position which permits a relational flow between machine
vision and the embodied photographer, between presence and program, in which the
merging of both exceeds the power of either.
References
1. Deleuze, G.: Bergsonism. Translated by H. Tomlinson and B. Habberjam. Zone, New York
(1988)
2. Flusser, V.: Towards a Philosophy of Photography. Translated by A. Mathews. Reaktion,
London (2000)
3. Parker Willis, N., Porter, T.O. (eds.): The Pencil of Nature. A New Discovery. The Corsair. 1
No. 5 (Saturday April 13): 70–72. Reprinted in Photographic Theory: An Historical Anthology.
In: Hershberger, A. (ed.) Wiley-Blackwell, Oxford, pp. 44–47 (2014)
4. Da Vinci, L. C.: The Function of the Eye. As Explained by the Camera Obscura in Photographic
Theory: An Historical Anthology. In: Hershberger, A. (ed.) Wiley-Blackwell, Oxford, pp. 17–
18 (2014)
5. Petchesky, R.P.: Fetal Images: The power of visual culture in the politics of reproduction. Fem.
Stud. 13(2), 263–292 (Summer, 1987)
6. Pritchard, M.: A History of Photography in 50 Cameras. Bloomsbury, London (2016)
7. Bate, D.: The digital condition of photography: cameras, computers and display. In: Lister,
M. (ed.) The Photographic Image in Digital Culture, 2nd edn. Routledge, London, pp. 77–94
(2013)
8. Piper-Wright, T.: In Pursuit of Error [Online]. www.inpursuitoferror.co.uk
9. Frow, J.: A pebble, a camera, a man who turns into a telegraph pole. Crit. Inq. 28(1), 270–285
(Autumn 2001)
10. Feenberg, A.: A critical theory of technology. In: Felt, U., Fouché, Clark, R., Miller, A.,
Smith-Doerr, L. (eds) The Handbook of Science and Technology Studies, 4th edn. MIT Press,
Cambridge Mass, Cambridge, MA, pp. 635–663 (2017)
11. Latour, B.: On actor-network theory: a few clarifications. Soziale Welt Jahrg H 4, 369–381
(1996)
12. Borda, S.G.: Locating the camera in a digital time. Photographies 11(2–3), 177–192 (2018)
13. Bennett, J.: Vibrant Matter, A Political Ecology of Things. Duke University Press, Durham,
NC (2010)
14. Chong, D.: Tokyo 1955–1970 A New Avant Garde. The Museum of Modern Art, New York
(2012)
15. Meatyard, R.E.: Learning to see ‘No-Focus’ (2011) [Online]. https://www.americansuburbx.
com/2013/01/ralph-eugene-meatyard-reframing-photography-theory-and-practice-excerpt-
2011.html
16. Barth, U.: Artist’s website [Online]. http://utabarth.net/work/untitled-1998-99/
17. Barthes, R.: In: Barthes Reader, A., Sontag, S. (ed.) The Photographic Message [1961]. Hill
and Wang, New York, pp. 194–210 (1982)
18. Walton, K.: Transparent pictures: on the nature of photographic realism. Crit. Inq. 11(2), 246–
277 (1984)
19. Laruelle, F.: The Concept of Non-Photography. Translated by R. Mackay. Sequence Press, New
York (2011)
20. Barthelme, D.: In: Herzinger, K. (ed) Not Knowing in Not Knowing: The Essays and Interviews
of Donald Barthelme. Random House, New York (1997)
21. Nunes, M. (ed): Error: Glitch, Noise and Jam in New Media Cultures. Continuum Books,
London (2011)
22. Warner Marien, M.: Photography: A Cultural History, 4th edn. Laurence King, London (2014)
23. Hirsch, R.: Seizing the Light: A History of Photography. McGraw-Hill, Toronto (2000)
24. Hand, M.: Ubiquitous Photography. Polity Press, Cambridge (2012)
25. Turkle, S.: Life on the Screen: Identity in the Age of the Internet. Simon & Schuster, New York
(1995)
26. Turkle, S., Papert, S.: Epistemological pluralism: styles and voices within the computer culture.
Signs 16(1), 128–157 (1990)
27. Cameron, J.M.: Annals of My Glass House. First Published in Photo Beacon (Chicago) 2
(1890), pp. 157–160. In: Hershberger, A. (ed.) Photographic Theory: An Historical Anthology.
Wiley-Blackwell, Oxford, pp. 76–79 (Reprinted) (2014)
28. Weaver, M.: Julia Margaret Cameron 1815–1879. Little Brown Company, Boston, MA (1984)
29. Brown, B.: Thing theory. Crit. Inq. 28(1), 1–22 (Autumn, 2001)
30. Klein, E.: A Comprehensive Etymological Dictionary of the English Language. Elsevier,
London (1969)
158 T. Piper-Wright
Chapter 10
Signs of Surveillance
Daniel Buzzo
Abstract This chapter discusses the challenges of working with complex digital
information, visual images, meta-data, and temporal and geo-spatial information, as
raw materials for artistic expression. In the chapter the reader will have an insight into
the thought process and working practice, artistic and technical, of forming informa-
tion into a recognizable body of work and processing both the content and the form of
the work into an emotional, reflective experience. It not only discusses the challenges
of working with external, commercial software frameworks and web services and
the challenge of dealing with the brittleness of digital things but also the ‘semantic
affordance’ offered by computer systems in working at a conceptual level with art
materials. The ‘Signs of Surveillance’ project originated as a photographic observa-
tion and collation activity in early 2015 and has grown to a complex digital web and
installation live visualization project. Since that time thousands of photographs of
signs of surveillance and the warning signs, indicating an area or activity is being
monitored by video-camera, have been captured in more than 15 countries, including
Belgium, Canada, China, Denmark, France, Germany, Greece, Italy, Japan, Korea,
Luxembourg, Netherlands, Portugal, Spain, Sweden and the UK. This chapter dis-
cusses the development of one aspect of this multi-part work, dealing directly with
managing and manipulating a large body of digital data and working with complex
visualization systems and online and offline digital distribution technology.
Keywords Digital art · Media art · Installation · Semantic affordance · Digital

brittleness · Surveillance society · Derive · Geo-location · Web technologies ·
Visualization
10.1 Introduction
The pervasive nature of surveillance, as well as the normalization of the signs of it—
both physical and metaphorical, is arresting when it is laid out how clearly intertwined
this silent over-watch of endless surveillance cameras has become with our everyday
D. Buzzo (B)
University of the West of England, Creative Technology Lab, Bristol, UK
e-mail: daniel.buzzo@uwe.ac.uk
160 D. Buzzo
lives. Across Europe and the wider globe, surveillance of the body public, of civic
space, of every interaction in everyday society appears at some kind of saturation
point.1 In 2015, the then UK Information Commissioner, Tony Porter, himself a
former counter-terrorism officer said:
The lack of public awareness about the nature of surveillance troubles me
and as Matthew Thomas wrote in the Guardian newspaper, quoting Porter:

[it] risks changing the “psyche of the community” by reducing individuals to trackable
numbers in a database
This is despite strong evidence that the publicly stated goals of civic safety and
crime prevention are not being fulfilled [25]. The 2005 UK Government Home Office
Research, Development and Statistics Directorate concludes in its official report,
‘Assessing the Impact of CCTV’ [14] that:
the CCTV schemes that have been assessed had little overall effect on crime levels […]
CCTV is an ineffective tool if the aim is to reduce overall crime rates and make people feel
safer. The CCTV systems installed in 14 areas mostly failed to reduce crime (with a single
exception), mostly failed to allay public fear of crime (with three exceptions) and the vast
majority of specific aims set for the various CCTV schemes were not achieved.
In 2004, Richard Thomas the then UK Government Information Commissioner

warned [2]:
My anxiety is that we don’t sleepwalk into a surveillance society where much more infor-
mation is collected about people, accessible to far more people shared across many more
boundaries, than British society would feel comfortable with
Making a comparison to the collection of vast quantities of information on indi-

vidual of Franco’s fascist state and the eastern European communist regimes, he
added in response to the question of whether there was a risk of Britain following
this route,
I think there is a danger. I don’t think people have woken up to what lies behind this. It
enables the government … to build up quite a comprehensive picture about many of your
activities.
It is against this confusing and disturbing backdrop that this project asks a question
of the surveillance colonization of the physical environment around us by building,
visualizing and exploring an international database of signs of surveillance. These
seemingly ubiquitous information/warning displays come in all shapes, sizes and
designs, while often appearing helpful or informative is actually (in the UK at least)
part of the legal obligation, largely unknown by the public and ignored by operators,
1 In its 2013 report, The Picture Is Not Clear: How Many CCTV Surveillance Cameras Are There
in the UK? [3] the British Security Industry Association (BSIA) suggested that camera numbers in
the private sector could outnumber those used by public bodies by as much as 70 to 1. The BSIA
survey covered the whole of the UK, not just London, and its maximum estimate suggested there
was a CCTV camera for every 11 people in the country though it said the most likely figure was
closer to one for every 14 people.
10 Signs of Surveillance 161
on any operator or installer of a CCTV surveillance camera. The Information Com-

missioners Office (ICO), the independent UK authority, explains in the guidance
‘CCTV filming carried out by others: What can I expect?’ [23]
The CCTV operator must let people know they are using CCTV. Signs are the most usual
way of doing this. The signs must be clearly visible and readable, and should include the
details of the organization operating the system if not obvious. (emphasis added)
CCTV should only be used in exceptional circumstances in areas where you normally expect
privacy—such as in changing rooms or toilets, and should only be used to deal with very
serious concerns. The operator should make extra effort to ensure that you are aware that
cameras are in use.
As can be seen from the corpus of collected images in this project, this point
that CCTV warning signs ‘should include the details of the organization operating
the system’ is one that is notable when operator information is included rather than
when it is absent, such is the apparently casual attitude to the use of CCTV and any
adherence to privacy or regulatory policy.
This growth of surveillance and reactions to it, rather than being a recent phe-
nomenon, has been going on for nearly three decades. Several artists and projects
have worked in this area; notable amongst them are recent projects such as ‘Art
and Surveillance’ lead by Susan Cahill in Canada that collates artistic projects on
surveillance [8] and older ones such as http://www.spotthecam.nl created in early
2000s by Maurice Wessling and Bits of Freedom2 in conjunction with the Waag
Society in Amsterdam, Netherlands3 and the ‘Surveillance Camera Theatre’4 from
the Surveillance Camera Players [24], an informal activist theatre troupe formed
in 1996 in New York City between Bill Brown and Michael Carter. Carter, in his
1995 manifesto ‘The Guerrilla Programming of Video Surveillance Equipment’ [9],
proclaims:
It is important to remind oneself of the relationship between the eye of the media and that
of the corporate police state – for they are both the guardian of the commodity, however
nebulous and ephemeral that commodity may become. As a tactic designed to point out
the paradox of a system that turns the lens on a public that has been taught to place more
importance on images recorded by cameras than images seen by their own eyes, we propose
Guerilla Programming of Video Surveillance Equipment (emphasis added).
As a photographic observational project, the existing corpus of images of ‘signs

of surveillance’ comprises over 2000 digital photographs taken across the globe
between 2015 and 2019. Each image contains date, time and geo-location meta-data
allowing geographic mapping and search and retrieval in a variety of ways. The
whole project has several elements:
• A compendium of the designs, locations and forms of the myriad of sign-types in
the collection (see Fig. 10.1)
• A traditional digital photographic print exhibition
2 https://www.bitsoffreedom.nl.
3 https://waag.org.
4 http://www.notbored.org/scp-how-to.html.
162 D. Buzzo
Fig. 10.1 Example of the

variations and number of
different sign shapes, designs
and forms from numerous
countries
• An interactive 3D geo-map-based visualization

• An online web-based searchable catalog that asks for public contribution to extend
and complete the European map of Signs of Surveillance
• Training a machine learning system on the corpus of signs to enable automatic
sign recognition.
This chapter deals with the development of element number three, the develop-
ment of an interactive, dynamic 3D geo-map-based visualization of the photographic
collection.
10.2 Process of Gathering Raw Images and Meta-Data
The process of gathering raw images for the collection of photographs of signs of
surveillance began as a non-verbal act of noticing, in noticing the apparent over-
abundance of these signs in public spaces. After noticing came the wordless desire
to evidence that noticing, to collate and compare, to contrast and reflect. to investigate
if the feelings of noticing were borne out by actual evidence—in this instance the
evidence being the collation of photographs recording the signs. This process of
observation and recording as basic research, and raw material, is common to many
artistic practices, and is a recurring theme I use to undertake non-verbal investigation
of items of importance I see in the world around me. In some ways I use the camera
as one would use a microphone to take field recordings for later study in the studio.
In this work the gathering of images made the proliferation of these signs in public
spaces abundantly clear (see Fig. 10.1); partly as the act of looking for them became
habit, partly as my subconscious eye became trained to see these small icons, images
and visual devices and the kind of places they would appear high up on walls of
large buildings, low down on shop doorways, in the entrances and lobbies of offices
and houses. The act of looking also revealed the signs in unexpected places, among
piles of pastries in display cabinets, buried in hedges and flower beds in residential
gardens, on the sides of cars, buses, trams and trains, and on vending machines and
garden sheds.
10.2.1 Building Semantic Affordance with Artistic Raw

Materials
In the process of gathering the photographs, extending the corpus of images from
country to country as I traveled while engaged in work on other projects, a feel for
the body of work began to emerge. The feeling of emergent, artistic, comprehension
is the one that I recognize when undertaking works of almost any sort, from extended
drawing studies of scenes (see Fig. 10.2) or objects to wrestling with the philosophical
164 D. Buzzo
Fig. 10.2 Example of extensive use of drawing for exploration and visualization, here showing
early explorations of using image meta-data and tagging to map-tile backgrounds
aspects of how best to structure a system when incorporating dense computer code
(such as the Volca project [6, 7] or the Time Machine [4, 5]).
This understanding is the developing of a cognitive, sensory and implicitly non-
verbal relationship with the materials and ideas or objects under scrutiny. It is a
feeling of what I call ‘semantic affordance’, in that one feels an innate ability to
cognitively manipulate the essential conceptual elements embodied by whatever is
the subject of scrutiny. This affordance gives a direct fluidity and competence that
allows further examination and contemplation of the materials at hand at a concep-
tual and ontological level. This affordance or agency to manipulate the conceptual
elements embodied by the material under study could be seen as the central activity
in my artistic practice. It is through this level of deep non-verbal understanding of
the manifold semantic concepts seen held within objects and materials that inform
the aesthetic choices that are made when re-formulating materials in the prepara-
tion and creation of artistic outputs. As the artistic outputs are imagined, sketched,
produced as maquettes and prototypes, a re-connection to the original non-verbal
enquiry is sought. The process is a search to close the loop from initial observation,
non-verbal comprehension, manipulation and synthesis of conceptual structures to a
final embodiment of concepts as an intervention into the physical world. The answer-
ing of the initial question and the final scratching is a mental itch. In this sense the
works are predominantly conceptually driven and operate at a physical level in the
conceptual rather than technical or decorative world. As Shanken [21] raises in ‘Art
in the Information Age: Technology and Conceptual Art’, although the divisions
between conceptual art and technological art have generally been seen as in con-
flict, he explores the view that while the history of technological art has been seen
as separated from that of conceptual art both were rooted in experimentation and
re-interpretation and proposes a re-thinking of their status as separate and divorced
from each other. He quotes Sol Lewitt’s 1967 essay ‘Paragraphs on Conceptual Art’
[16] where Lewitt describes conceptual art as a quasi-mechanical process:
In conceptual art the idea of concept is the most important aspect of the work . . . [t]he idea
becomes a machine that makes the art.
It is this very idea that drives my quest for these conceptual affordances in the work,
which allows mental structures to be created from which the art creates itself. The
arena of the technical in part is a desire to work unbounded by both convention and
the patina of the physical that seem part and parcel of other mediums of expression.
The ‘active’ nature of technology allows for embedding, encoding and subsequently
animating these ‘conceptual machines to make art’ into the systems underlying the
physical surface of the work.
10.2.2 Gathering Images and the Dérive
In this project the initial activity was a process of subconscious harvesting, con-
ducted as an ongoing photographic dérive, a drift, though the world we inhabit with
a watch for these signs of surveillance that had now become artistic items of interest.
Guy Debord, French philosopher and author of ‘The Society of the Spectacle’ often
associated with Situationist International,5 describes the dérive as:
a mode of experimental behavior linked to the conditions of urban society: a technique of
rapid passage through varied ambiances [13]
Over the period beginning with the first photograph of the series, taking in Lisbon
in June 2015, and lasting through 15 countries and more than four years over 2,000
images of signs of surveillance were recorded in dérive fashion.
10.3 Managing and Manipulating Large Digital Image

Corpuses as Artistic Raw Material
Dealing with materials gathered using digital tools as the subject of artistic study
has a number of opportunities and also challenges. Some of these are common to
the practice of much artistic research and some unique to digital materials. Digital
5 https://www.tate.org.uk/art/art-terms/s/situationist-international.
166 D. Buzzo
materials, while fluid, flexible, malleable, and almost instantaneously duplicable and
transferable, are also at their core, inherently brittle. Their flexible, immaterial, nature
and their ability to be versioned, trans-coded, summarized and edited means that
any notion of an original is fleeting and slippery. Attempting to trace the origin, and
original of a digital document, is like grasping at a ribbon blowing in the wind. Partly
this is the nature of sampling the world with digital, that is, numerical processes. It
is important to remember that the sampling process is fundamentally divorced from
techniques of the analog recording realm.6 The connection with the place and instant
of creation is fleeting, and beyond the memory of the place and time that a button
was pressed or a menu item clicked the digital object has no patina or evidence of
its place or moment of creation.
10.3.1 Meta-Data and Artistic Data Management
All digital artifacts have meta-data, that is, there is data about the data. There is
information about the information contained inside any digital document, be it sound
file, image, text, video or software code. What is unusual is that in the digital realm
this meta-data is built of the same substance as the mate´riale it describes. In the
instance of this project each image was recorded with meta-data describing the image.
There are basics facts associated with any digital files, for example, creation date,
modification date, byte size of the file, name, filetype and the current location of a
document in a specific computer system. There also, commonly, exists within image
files additional data using the EXIF (EXchangeable Image File format) standard.7
The EXIF data standard was originally introduced by The Japan Electronic Industries
Development Association (JEIDA) in 1998 and is widely adopted as a standard to
store extended information about digital photographs. This meta-data commonly
includes information about the make and model of the apparatus or camera used
for the original recording and extends to information about lenses, flash, aperture
and shutter speed of the recorded image. The meta-data also includes contextual or
circumstantial data including the date/time of the image capture and increasingly
common with images taken with smart phones and the latest generation of hand-held
cameras, the geographic meta-data, expressed as latitude, longitude and altitude. This
EXIF data and specifically the associated geo-location data became of particular
interest in the course of this project and is processed and manipulated heavily in the
final artworks produced as part of the work (see Figs. 10.3 and 10.4).
6 Sampling puts a mesh to the world and encodes numerically, explicitly sampling reality at positions
on a pre-defined grid—the only parameters of this crude and bombastic approximation are the
frequency and resolution of the grid and the depth of the data captured or sampled at each point.
All information, texture and nuance between the sampling points of the grid are simply ignored or
disposed of.
7 https://www.exif.org.
Fig. 10.3 Early map visualization using C++ and openFrameworks with spline paths generated
following chronological order or recording of individual images. Here showing walks through
Kyoto, Japan over a three day period in 2018
Fig. 10.4 Early map visualization using C++ and openFrameworks with spline paths generated
following chronological order or recording of individual images. This version utilizes only geo-
location and temporal data from each image
168 D. Buzzo
10.4 Developing Artworks Combining Hand-Coded

and Industrial Digital Tools
Developing artworks utilizing technology, whether that be industrially mass-

produced physical materials such as paint, canvas, glass-fiber, electronics or soft-
ware involves a level of compromise and negotiation with the materials an artist
chooses to engage with. Being restricted to colors manufactured by a supplier, being
constrained by the retail costs of a device or being challenged by the features avail-
able in a software API are all within the same canon of challenges faced by artists
working in many contemporary contexts. As an artist regularly working in depth
with technology, and software specifically, this challenge of adapting, working with,
fighting with and customizing industrial products is a regular occurrence. Part of the
aesthetic of much technology orientated art is this engagement with, and subversion
of, industrial products, situating them in new contexts and utilizing them in new and
novel ways. Implicitly tied to this aspirational goal of both mastery and subversion
comes a challenge within the practice of making art with technology, one that is
endemic to all art but is amplified exponentially when engaging with software. This
challenge is the negotiation, both in production and distribution of works, with the
inherent brittleness of digital things.
10.4.1 On the Brittleness of Digital Things
As all physical things decay and change under what appears to be the effect of time,8
so do digital ones, and this decay is seemingly at a rate inversely proportional to
the power of digital material. Just as a digital image can be spread around the world
and seen and manipulated by a billion people simultaneously it can also collapse,
cease and disappear with equal speed. A server outage, a power failure, a change
in the software application programming interface (API)9 or even the simple act of
8 One could begin a lengthy discussion in the nature of the perception of time, correlation and
causality. Suffice to say that in most practical and functional aspects it would appear that as Stappers
and Giarcardi point out in their work on designing for temporal context [22]:
The most important thing designers should know is that there is no such single thing as time.
As Carl Sagan says “[Time] is profoundly resistant to simple definition.” Discussing the nature
of time, Sagan, in an interview during the making of NOVAs TV program ‘Time Travel’, goes on
to note [20].
Ever since St. Augustine, people have wrestled with this, and there are all sorts of things it
isn’t. It isn’t a flow of something, because what does it flow past? We use time to measure
flow. How could we use time to measure time?.
9 Incomputer programming, an application programming interface (API) is a set of subroutine

definitions, communication protocols, and tools for building software. In general terms, it is a set
applying a seemingly unrelated operating system update can bring digital artworks,
not to their knees, but to a state of ceasing to be. Though digital files and lines of
software code may still be intact, software art, like video art before it, is performative
and temporal; it exists at the moment of viewing; it needs to be run, to execute, to
function and to fulfill its promise as art. This base need of instantaneous creation, to
generate itself at the moment of viewing by an observer, is the challenge, excitement
and weakness at the heart of digital artworks. Bit Rot, [10, 15] or occasionally Link
Rot,10 is described by Coupland in his 2016 book of the same name [12] as:
the way digital files of any sort spontaneously (and quickly) decompose
The most important thing designers should know is that there is no such single
thing as time.
As Carl Sagan says “[Time] is profoundly resistant to simple definition.” Dis-
cussing the nature of time, Sagan, in an interview during the making of NOVAs TV
program ‘Time Travel’, goes on to note [20]:
‘Ever since St. Augustine, people have wrestled with this, and there are all sorts of things it
isn’t. It isn’t a flow of something, because what does it flow past? We use time to measure
flow. How could we use time to measure time?’
and is ‘a slang term’ for hypertext links that are broken. Link rot is created when
a web page is moved, taken down or re-organized. Clicking on a rotten link usually
results in a 404 error, which includes a message that the page cannot be found. This
‘digital decay’ affects all computational systems and is in equal parts the ventral
thrill and motivation and also an ever-present challenge in digital art making. As the
internet pioneer Vinton Cerf observes [10]:
There is something ultimately satisfying about keeping information in digital form… But
this blissful outlook may not comport with the reality of digital information preservation and
interpretation.
In the context of this artwork I am writing about—sat in seat 6c on a flight to

Lisbon, tapping on a tablet computer (and now weeks later re-editing on a subway
train under Hong Kong harbor)—hoping that what I write here will synchronize
flawlessly with the master version for this document so I can continue on my laptop
at home, or the desktop computer in my office or the myriad of other computers I
have in my two studios—the challenge of working with industrial tools in the making
of signs of surveillance has been great. Both in the practicality of the making of the
of clearly defined methods of communication among various components. A good API makes it
easier to develop a computer program by providing all the building blocks, which are then put
together by the programmer. An API may be for a web-based system, operating system, database
system, computer hardware or software library. An API specification can take many forms, but
often includes specifications for routines, data structures, object classes, variables or remote calls.
POSIX, Windows API and ASPI are examples of different forms of APIs. Documentation for
the API is usually provided to facilitate usage and implementation. (https://en.wikipedia.org/wiki/
Applicationprogramminginterface).
10 https://www.techopedia.com/definition/20414/link-rot.
170 D. Buzzo
work, designing, programming and forming it, and also in the conceptualization of it,
the philosophical and aesthetic decisions underpin it structurally. This arena is the ill-
defined and rarely talked about area where semantics and ontology meet engineering,
where philosophy becomes embodied in systems and structures and ways of doing
things. Just as William Gibson says [1] with incisive clarity:
We can’t see our culture very well, because we see with it
Perhaps this sentiment is distilled to its essence in the Japanese anecdote that ‘to
a hammer, everything looks like a nail’.
10.4.2 Aesthetics, Frameworks, APIs and Ontologies
When looking through the gathered body of work, mentally sitting among the thou-
sands of photographs from around the world, and reflecting on the journey of col-
lection and understanding, a non-verbal set of connections start to be made, there
begins to form an ontological framework of what is and what is not, what does
go and what does not when considering, comparing, organizing and arranging the
material. Alongside the challenges in managing digital collections, of cataloguing,
versioning, trans-coding, naming, renaming and duplicating and archiving, there
is the selection of applicable tools to manipulate these non-physical collections of
things-that-may-be.
In the development of the work the public exposure of the meta-data inside each
image became increasingly important in relating an individual item in the collection
to the whole. Beginning with a process of drawing, commonly with ink on paper,
ideas of collating, relating and visualizing this relationship, between individual ele-
ment and the whole, took shape. Sketching ideas in code, using the C++ toolkit,
openFrameworks,11 a sense of the dynamic in the material came to the fore. Initially
the project was an observation and dérive exercise and at a certain stage it trans-
formed into a discussion with the gathered material. As is of often the case in life
drawing, one starts with mark making in reference to the subject being observed but
at some point crosses into mark making in reference to the body of marks one has
made on the paper or canvas. Adding to or subtracting from a growing object that
begins to exist independently of that which is originally observed. This is the point
where a piece of work develops its own life or emergent identity, one that is divorced
from being a facsimile of the original scene.
After working with initial crude mappings of the geo-located images into 3D
space using geometric primitives and texture mapping of a variety of representa-
tions of the earth’s surface a move to a more complex visualization was needed. To
access high-resolution maps and images of the surface of the whole earth necessi-
tated engaging with external data sources, initially continuing working writing code
11 ‘OpenFrameworks is an open source C++ toolkit designed to assist the creative process by
providing a simple and intuitive framework for experimentation’ (https://openframeworks.cc/

about/).
in C++, the open-source online OpenStreetMap (OSM).12 Geographical informa-

tion system (GIS) database was accessed through its web-based API.13 While this
brought huge flexibility and access to a wealth of visual and meta-data about the
earth it added a subtle but pivotal disjuncture in the work. Even though the code was
still handwritten C++, compiled to run locally as an executable binary application on
a specific machine, the work now had a brittle chain of failure built into it. The work
now became reliant on an internet connection, reliant upon a connection of sufficient
bandwidth to the outside world and at the other end of a chain of routers and con-
nections. It was reliant upon the servers at OSM being available, and that the access
restrictions and protocols of their API, and the data in their underlying database
were available and had not been altered, patched, upgraded or had their license of
use changed. What also began to happen is that the underlying ontology, the causal
philosophy of the interrelation between elements, images, ideas and concepts that
was embodied in the architecture of the software I was writing changed. The atti-
tude and values in the project, the structure, classifications and ontology that was
emerging, was codified in its most concrete form in the classes, methods, functions,
loops, arrays, iterators and variables that scaffolded and guided the work to become
at the moment of performance. By engaging with this external API, and the underly-
ing philosophy, attitude, ontology and expectation encoded within it, it there began
and evolving and protracted negotiation on how to navigate the external conceptual
space without becoming overwhelmed hegemonistically14 and becoming merely an
extension of it. Illustrating what it could do, rather than a master of it demonstrating
what I could do with it, sculpting it to my will and the will of the evolving work. This
conundrum is somewhere between the Sapir-Worf hypothesis [17] on how language
shapes thought and conceptions of one’s reality and the ominous words of Friedrich
Nietzsche when he said [18]:
Whoever fights monsters should see to it that in the process he does not become a monster.
And if you gaze long enough into an abyss, the abyss will gaze back into you.
This negotiation for control became a battle, metaphorically, for the ground under
the project, and this external empire of protocols and ontologies and the emerging
12 http://www.openstreetmap.org.
13 Utilizing the openFrameworks add-on ofxMaps (https://github.com/bakercp/ofxMaps). ofxMaps
creates an interface to OSM in the ‘SlippyMap’ style (https://wiki.openstreetmap.org/wiki/

SlippyMap). It is written and maintained by Christopher Baker.
14 The Encyclopedia Britannica describes Hegemony as the dominance of one group over another,
often supported by legitimating norms and ideas. The term hegemony is today often used as short-
hand to describe the relatively dominant position of a particular set of ideas and their associated
tendency to become commonsensical and intuitive, thereby inhibiting the dissemination or even
the articulation of alternative ideas. The associated term hegemon is used to identify the actor,
group, class or state that exercises hegemonic power or that is responsible for the dissemination of
hegemonic ideas. (https://www.britannica.com/topic/hegemony).
172 D. Buzzo
artwork—using C++ but communicating into a web-based world, using http trans-
port15 layers—began to shape, not only what was being said but a shift linguistically
into the world of web technologies. That is not to say that the possibility to stay
in the dialect I was writing in was not available, it was entirely, but the challenge
would be to re-write for myself the additional tools, utilities, libraries and processes
I was beginning to rely on to achieve the realization of my increasingly complex and
specific pen and ink drawings of the project. Just as a decision to make your own
paint becomes a significant part of the work so does the decision to make one’s own
digital tools.
As Richardson discussed in his 1998 article ‘New Media, New Craft’ [19]
reflecting on what the status of new digital tools may be in an art context:
Using programming to create a piece of art or design requires an abstraction of thinking,
translating the idea of the final visual form into a structure required to create the work. A
leap of understanding is required to translate a creative idea into a piece of code. Creative
ideas and solutions are thus abstracted into structures and objects that are natural to the
computer material. Such an understanding of material subsequently creates work that echoes
the invisible structure of the code
Richardson goes on to add:

The use of programming, as a way of manipulating and understanding the new material,
also represents the means by which artists and designers are able to get closest to the virtual
material. When they use programming to create a visual work, the underlying process and
structure become of fundamental importance. The invisible structure of the work becomes as
significant as the final outcome. The work experienced by the viewer is a visual translation of
the underlying framework and mental engagement of the artist or designer with the material.
This challenge, between using available tools, and traveling further, faster and
making everything from scratch for oneself, perhaps risking re-inventing the wheel
having taken years to learn how to best make a wheel, is common to many artistic
endeavors but never more so than in self-coding digital artworks. Richardson contin-
ues pointing out the difference that hand-crafted digital tools make in the conceptual
understanding of any artwork:
In direct contrast to the post-modern point-and-click, cut-and-paste approach to creating
a piece of creative work, the use of programming requires that a framework is firmly
established, around which the work is built.
Having made strategic choices and seen where the project wanted to go, and
how it could be formed in a way that allowed it to talk for itself, investigations
of middleware frameworks and software libraries began in earnest. Moving parts
of the codebase into java script and using the browser as interpreter and platform
wheel, rather than the byte code produced by CLANG or the GCC C++ compiler
15 http, or hypertext transport protocol, is a computer protocol that is used to allow clients (commonly
web browsers running on users computers to request and subsequently receive text, image, audio
and video data from web servers).
allowed easy integration with online utilities I was already using, such as map-tiles16
of satellite and geo-political imagery and reverse geo-coding.17 This easy ability to
communicate with remote functions, features and data comes with an additional cost
beyond the loss of independence, self-reliance and overall stability (see Fig. 10.5).
Fig. 10.5 Browser based version written in JavaScript using leaflet.js showing early tests with
meta-data extraction and visualization
16 Map-tiles are a way of breaking large GIS-orientated datasets such as maps into smaller fragments
that can be called and used as needed, remedying the need to load large datasets as monolithic pieces.
A calculation is made of the geo-location that is of interest, the scale that the map is to be viewed
and the size of the viewing port (or in our instance, web browser window); a series of request are
made; the map-tile server and the corresponding map fragments are returned as individual files and
then tessellated in the viewer.
17 Reverse geo-coding is the ability to take and specific geo-location, commonly expressed in latitude
and longitude, and return a descriptive physical world location. For example, the a reverse GeoCode
request for the location 52.356991666666666, 1.2931251573910854 is returned as Oostpoort 12,
Waldenlaan, Winkelcentrum Oostpoort, Amsterdam, North Holland, Netherlands, 1093NH, Nether-
lands by the OpenStreetMap database functions that allowed the work to turn the GPS latitude and
longitude data inside individual images into named localities with street town and country data.
174 D. Buzzo
In this instance it carried a commercial one. Although I was using a free developer
account for one of the major mapping utility providers, the monthly usage limit was
quickly reached for data requests. Due to license limitations on usage of the data that
was provided, it was not legally possible to cache data between sessions of usage, nor
store it in any long term or permanent way. This necessitated performing a reverse
geo-code lookup for the locations of images each time the project was viewed or
run, necessitating over 2000 calls to the remote API each time. Although the cost of
an individual reverse geo-code request was only fractions of a cent per request, this
added up to a cost of USD2 every time the work was viewed. The map-tile imagery
requests came in addition to this cost. Looking at open-source alternatives became
complex, while the service cost was free and the license terms far less restrictive in the
data servers were not configured for heavy use, resulting in the servers banning the
IP address of the requesting machine after the first few hundred geo-code requests. A
strategy was developed to pre-request and cache the geo-code data and serve it from
a self-hosted server by writing a script that requested the relevant information from
the open-source server at a deliberately slow, throttled rate and writing the returned
reverse lookup data to a local file in geoJSON format.18
This conformed with both the terms of use of the open-source server and also its
license on use and retention of data. The second challenge was on how to deal with
the mapping data being requested from the commercial provider. Open-source map
data providers, understandably, were unable to supply maps data in the quantities
the project needed and also separated the idea of individual users of the service from
the creation of applications that would use the service. Unfortunately, there appear
very few instances of a middle ground for artistic or non-profit uses of open-source
services putting most artworks or projects that use third-party services or data firmly
into this arena of being classed as commercial users and being charged accordingly.
One of the options available within the license terms for the core corpus of map
image data—the smaller version being in the region of 65gb of image tiles—was to
clone the data needed and set up and operate a web-based map data server specifically
for the project. This added a layer of technical complexity and a certain amount
of unnecessary redundancy but afforded a break from the reliance on commercial
providers and also added the possibility of generating a machine-specific version
that could run as a single instance of the project with all necessary data held locally.
This would remove the requirement for an internet connection for a greater level of
autonomy. This involves setting part of the host computer to operate as a web server
18 GeoJSON is an implementation of JSON (JavaScript Object Notation) that encodes geographic
information in text form as nested objects, typically containing a latitude and longitude geom-
etry reference optionally followed by additional data. In our instance the data encoded includes
details of the EXIF camera data relating to images and their recording location. An example of
typical GeoJSON entry will look similar to the following: “geometry”: “type”: “Point”, “coordi-
nates”: [ 4.927913888888889, 52.356991666666666, 1.2931251573910854 ], “type”: “Feature”,
“properties”: “url”: “sign6.jpg”, “date”: “2018:04:30 10:07:09”, “orientation”: 6, “make”: “Apple”,
“model”: “iPhone SE”, “size”: 23397, “type”: “image/jpeg”, “locality”: “Oostpoort 12, Waldenlaan,
Winkelcentrum Oostpoort, Amsterdam, North Holland, Netherlands, 1093NH, Netherlands”).
and routing web calls from the work or application back to the locally running web
server, sometimes called LocalHost.
With these two elements in hand, batch generated reverse geo-code data, locally
held, and provisioning the projects own map data server, the forming, tuning and
final choreography of this section the piece proceeded.
10.5 The Challenge of Live Vs Video of Activity

or Interactivity of Exposing the Internal or Relying
on the Surface
In developing live digital works there is a challenge of exposing and communicating

where the work is. In the case of paint and photography and other flat visual arts,
the work is commonly readily identified in the surface. When working with live
digital processes this identification, location and exposition of the work—in this case
meaning the artistic effort, intention or focus of attention—becomes obfuscated and
requires specialist technical knowledge to gain insight into often hidden black-box
processes. Some artists working with digital/technological processes take the route
of exposing the mechanism of systems, disassembling computers into their working
component and exhibiting them exploded on gallery floors, deliberately exposing
the ‘technology’ to indicate complexity and ‘liveness’. Another common route to
provide this exposition to an audience is to make systems interactive or reactive
to some aspect of the audience, making clear that something live and dynamic is
occurring and perhaps leading the audience into consideration that which is beneath
the surface, the ‘deeper mechanic’ inside a work or piece. With work that takes
neither of these routes, work that is active but not interactive, that does not make
its components part of the artwork display, the challenge is to help the audience
understand that what they see is live and note rendered video recordings or pre-made
files being re-played. And that each moment seen is created and becoming at that
moment. It is being performed live, never to be repeated in that fact same way again.
10.5.1 Exposing the Internal or Relying on the Surface
This challenge of exposing the internal mechanism of projects and works and the
delicate balance of the work end up being about the mechanism and the tools rather
than the original intent is a complex one. In this work the multi-part nature of the out-
puts from the project allows a range of expression and nuance which in another work
would not necessarily be available. With the installation version of Signs of Surveil-
lance the clean collection nature of the photographic source material and the exposure
and manipulation of the image meta-data is clue enough to the live/process nature
176 D. Buzzo
behind the work. Alternative versions, such as the artists’ book and the controllable
web-based version address other elements of the project sufficiently.
10.5.2 The Final Work for Gallery Installation
The final instantiation of the work discussed in this article is designed for viewing
in a gallery or installation context (see Fig. 10.6), and has been tuned, sculpted and
shaped accordingly. In an installation context the audience is mixed, with a range of
interests, awareness and contextual insight.
The audience, the viewer also has a range of available temporal modes within
which to receive the work and the underlying themes and idea. Part of the work is
paced and presented for a transitory audience with short attention and an availability
of meaning in the surface of the work that is ‘glanceable’. That will survive artistic
scrutiny of a glance of a few seconds. At another level the work has a deeper pace and
rhythm that is designed to survive deeper scrutiny and draw the viewer into a deeper
dialog with and understanding of the work, allowing nuance and interpretation at
the pace and depth the viewer is comfortable with. The work for gallery installa-
tion is active but unlike its open web-based counterpart (available publicly at http://
signsofsurveillance.com) not explicitly interactive (see Fig. 10.7). To this end, it has
a specific set of behaviors that it exhibits to engage the viewer and draw them through
the corpus of work. When working through the meta-data accompanying the body of
images, a certain irony was revealed, in that, if one followed the time-stamps of the
collected images one could re-create, with a varying range of accuracy, the route that
I had taken through neighborhoods, cities and countries around the world over the
Fig. 10.6 Mockup of gallery installation of early globe version

Fig. 10.7 Final version showing custom map styling, with clustering, image meta-data and dynamic
transitions along the ‘walk-line’ between images over the 5 year period of collection. This location,
Ijmeer, Amsterdam, Netherlands
preceding five years. The time-stamp data combined with the geo-location of each
image showed the points on the globe that I had stood at the press of every shutter
and the capture of each photograph. Beginning to investigate these invisible lines,
simple routines were written to create a single path through each geo-point. Using
both Catmull-Rom19 and Bézier20 curves different paths were visualized showing
the path between each captured image.
10.6 Conclusion
In this chapter I have attempted to expose and discuss some of the approaches and
practices that I have evolved and adopted over the last 30+ years of working with
digital technologies. My original thoughts were that working in conjunction with
complex, highly sophisticated tools and systems which transformed the relationship
from one of artist and tool into much more of an intimate discussion, a collaboration
19 A Catmull–Rom spline, named for Edwin Catmull and Raphael Rom, is a special case of a cardinal
spline. Catmull–Rom splines are frequently used to get smooth transitions between key points and
are often used in animation and similar computer graphic works.
20 Named after Pierre Bézier, a Bézier curve is a parametric curve used in computer graphics. Several
points using Bézier curves can be combined to form a Bézier spline.

178 D. Buzzo
where one asked questions of the systems one engages in, navigating and negotiat-
ing the parameters and avenues in and through the conceptual structures, functions
and processes inside the machine. From this comes the innate understanding of the
systems and materials one is working with, what I have called semantic affordances.
Sadly, this conceptual flexibility and strength that digital materials bring also brings
multiple points of failure, usually with no graceful decline. A point of failure at any
juncture in a digital artwork is usually a complete failure. This is what I mean when
I talk of the brittleness of digital things. For this work, which extensively involved
with the collection of a large body of individual items working in the digital gives
great flexibility to collate, manipulate, transform and express. It also gives myriad
opportunities for instantiation of a work in the physical world, and huge availabil-
ity of choices of how to make an intervention into the mind of the viewer. At this
instance, the challenge came in making gallery-orientated versions and web-based
while using web technologies and remote data in both. The issue of reliance on, and
of cost, when using external systems, such as map-tile data and geo-location services
was a particular challenge but let to new and interesting areas setting up self-hosted
and local versions of some of these external commercial data sources. This does,
however, have its own drawbacks, and the writing discusses in some detail the philo-
sophical, cultural and ultimately ontological struggle when engaging with external
systems, APIs and data. Also that, they have their own political approach, their own
ontologies and that can be a hegemonist force with the potential to overwhelm the
internal ontology that an individual work of art generates for itself. This question of
the internal complexity and dialog that goes on inside active digital works is the last
point in this writing, investigating some of the challenges that are balanced when
dealing with the decision of how much to expose the internal structures of a work
to bring viewers deeper inside, drawing them below the surface of a work while
not making the work purely about the internal structure. Thus, working to retain the
original focus of the work but helping give insight into the live process at work at
the moment of performance, particularly with works with any generative nature to
them.
10.6.1 Final Reflection
The final version of this work, or at least the current version, is part of a body of pieces
that are differing expressions of a formalized process of collecting and collating. This
collecting began as a dérive of observation and photographic collecting around the
world. Begun initially as a completely non-verbal process the project developed its
own vocabulary and logic over the years of photographing, to the point where the
material began to guide the form and the final work. Only then began the process
of exploration in the studio, of applying technological processes to illustrate and
instantiate the mood developing inside of the work. This process is common to much
of my work and my day-to-day artistic activity: one that is an ongoing dialog I
have with the world, of observing, looking, making, and underneath it all ultimately
seeking sense-making.
The work, while not forcing a specific political point, is intimately political. It is
intended as an exposure of pervasive surveillance of our physical world, to a scale
that it continues to surprise me, even as I become inured to it. This journey we have
chosen to take with surveillance technology, and the ability we have built to track,
record and intrude into every aspect of the body public, and our individual bodies,
private is operating at a scale that many would find unimaginable should they be able
to see it clearly.
As Noam Chomsky, never one to use an alarming phrase when calm reasoned
words will do, says [11]:
there are two things to bear in mind, one is that the phenomenon (of surveillance) should not
be at all surprising, the second is that the scale of it is surprising.
10.7 Epitaph for a Digital Artwork
I sit and write, longhand on paper with a pen, the epitaph at the closing of this chapter.
The document and journey written covers six countries, countless hours and edits,
full of insight and reflection. The final edits are yet to be done but what I can write
now is the epitaph. Reading back through this writing, I find I have made a fuller,
more complete explanation of my internal process, discussion and artistic activity
than ever before in one place. At times on technical, philosophical and political, I
trust the reader will forgive me streams of consciousness and occasional melodrama
of the miniscule internal theatre that is the struggle of working with digital tools. This
writing, alongside ongoing development, coding and versioning of the software, has
given me space to reflect on process and practice with a fresh perspective. Thinking
of the work, in progress for nearly half a decade, makes me reflect that most readers
of this will never see it. That by the time this volume is typeset, proofed, approved,
printed and distributed, the work may, in practical terms, cease to exist. Web domains
I pay for will expire. Servers will be upgraded, patched, obsoleted and retired and
I will go on to new projects, works, technology and software. In very real terms
this chapter of ink-on-paper (or the undoubtedly shorter-lived pixels-on-screen) will
outlast the work by a magnitude of scale. Sitting in Hong Kong in late June 2019,
the mood is somber but resolved and the world seems fragile and tenuous. This work
on Signs of Surveillance is a small postcard in a discussion of technological change,
reflecting on the fundamental importance of the ephemerality of art, of the importance
of its functionless-ness. On the innate lack of practicality that those versions and
instantiations of it that require upkeep, maintenance and care to continue to exist are
happily always consigned to memory. This is the epitaph to a work of art that will
soon cease to be. That will fail by nature of its powerful but short-lived reach and
the fickle brittleness of the substance of which it is formed and the un-real substrate
on which it is built.
180 D. Buzzo
As Picasso, Paul Vale´ry or Da Vinci may have said21

Art is never finished, it is only ever abandoned.
and the abandonment of digital works into a continually changing virtual world
affords them a very short lifespan when left to fend for themselves.
References
1. Beeker Northam.: William Gibson, Interview (2015). https://medium.com/handbrain/william-

gibson-57e75f87e525
2. Booth, J.: UK ‘sleepwalking into Stasi state’ (2004). https://www.theguardian.com/uk/2004/
aug/16/britishidentity.freedomofinformation
3. British Security Industry Association.: The Picture is Not Clear: How many CCTV surveillance
cameras in the UK. Technical Report 195, British Security Industry Association, Worcester,
UK (2013). https://www.bsia.co.uk/publications/publications-search-results/195-the-picture-
is-not-clear-how-many-cctv-surveillance-cameras-in-the-uk.aspx
4. Buzzo, D.: What do we know of time when all we can know for real is now? In: Reyes-Garcia,
P., Chatel-Innocenti, E., Zreik, K. (eds.) Archiving and Questioning Immateriality: Proceedings
of the 5th Computer Art Congress, pp. 438–443. Paris, France (2016)
5. Buzzo, D.: The time machine. In: Sena Caires, C., Cardosa, J., Lipovka, A., Bidarra, J.,
Pimentel, L. (eds.) Proceedings of the 8th International Conference on Digital Arts—ARTECH
2017, pp. 199–201. ACM Press, New York, New York, USA (2017). https://doi.org/10.
1145/3106548.3106617.url, https://dl.acm.org/citation.cfm?id=3106548picked=proxCFID=
822909194CFTOKEN=38540610, http://dl.acm.org/citation.cfm?doid=3106548.3106617
6. Buzzo, D.: The volca project: a multi-sensory camera for augmented reality. In: Conference
on Human Factors in Computing Systems—Proceedings, CHI’18 Extended Abstracts. ACM,
Montreal, Canada (2018). https://doi.org/10.1145/3170427.3177767. URL http://dx.doi.org/
10.1145/3170427.3177767
7. Buzzo, D.: The Volca project: a sensory experiment in collaborative visualisation. In: Tran-
simage 2018. The fifth International Conference on Transdisciplinary Imaging at the Intersec-
tions of Art, Science and Culture. Edinburgh Futures Institute. The University of Edinburgh,
Edinburgh, UK (2018). transimage2018.net/
8. Cahill, S.: The art and surveillance project. http://www.artandsurveillance.com/about/
9. Carter, M.: Guerilla programming of video surveillance equipment (1995). http://www.
notbored.org/gpvse.html
10. Cerf, V.G.: Avoiding “bit rot”: Long-term preservation of digital information. Proc. IEEE
(2011). https://doi.org/10.1109/jproc.2011.2124190
11. Chomsky, N.: “What is Anarchism?” Speech at MIT Wong Auditorium. 18 Nov 2013 (2013).
https://www.youtube.com/watch?v=oB9rpSAp2U
12. Coupland, D.: Bit rot: Short Stories + Essays. William Heinemann, London (2016)
13. Debord, G.: Theory of the Dérive. Les Le‘vres Nues #9. Les Le‘vres Nues 9 (1956). http://en.
wikipedia.org/wiki/Drive
14. Gill, M., Spriggs, A., Little, R., Spriggs, A., Allen, J., Argomaniz, J., Waples, S.: Assessing the
impact of CCTV : the Hawkeye case study. Technical report: The home office. UK Government,
London, UK (2005). https://webarchive.nationalarchives.gov.uk/20110218141419/, http://rds.
homeoffice.gov.uk/rds/pdfs05/rdsolr1205.pdf
15. Hayes, B.: Bit Rot. Am. Sci. (1998). https://doi.org/10.1511/1998.5.410
21 Thereis debate about the correct attribution to the quote, including paraphrasing W. H. Auden in
Collected Poems 1965, “Poetry is never finished; it is only abandoned.”.
16. Lewitt, S.: Paragraphs on conceptual art. In: Stiles, S. (ed.) Theories and Documents of
Contemporary Art. University of California Press, Berkley (1996)
17. Lucy, J.A.: Sapir-whorf hypothesis. Int. Encycl. Soc. Behav. Sci. 20, 903–906 (2015). https://
doi.org/10.1016/B978-0-08-097086-8.52017-0
18. Nietzsche, F.: Nietzsche: beyond good and evil: prelude to a philosophy of the future (2002)
19. Richardson, A.: New Media, new craft? electronic art and animation calalogue, pp. 157–159
(1998)
20. Sagan, C.: Carl Sagan ponders time travel: interview with Carl Sagan (1999). http://www.pbs.
org/wgbh/nova/physics/Sagan-Time-Travel.html
21. Shanken, E.a.: Art in the information age: technology and conceptual art. Leonardo 35(4),
433–438 (2002). https://doi.org/10.1162/002409402760181259
22. Stappers, P.J., Giaccardi, E.: Time in the expanding user context of designing. In: Proceedings
of the SIGCHI conference on Human factors in computing. ACM—Association for Computing
Machinery (2013)
23. The Information Commissioners Office.: CCTV filming carried out by others. https://ico.org.
uk/your-data-matters/cctv/
24. The Surveillance Camera Players.: Surveillance camera players:10-Year Report (2006). http://
www.notbored.org/10-year-report.html
25. Weaver, M.: UK public must wake up to risks of CCTV, says surveillance commis-
sioner. The Guardian (2015). http://www.theguardian.com/world/2015/jan/06/tony-porter-
surveillance-commissioner-risk-cctv-public-transparent
Part IV
Audio Visual Installations to Generate
Collective Human Responses
Chapter 11
Coral Voices
Marlena Novak and Jay Alan Yim
Abstract Marlena Novak (visual artist) and Jay Alan Yim (composer/sound artist),
under the collaborative name localStyle, have been making work that addresses
environmental and socio-political concerns through the use of a wide range of media
since 2000, particularly utilizing technological tools in their creative practice. Deep
interest in understanding the contemporary conditions of non-humans and humans
is informed by their engagement with publications and texts, attending, participat-
ing in, and hosting talks, panels, and screenings, and meeting with specialists and
scientists regarding the topics of their research. Choral is a twelve-minute audiovi-
sual installation commissioned by 150 Media Stream for a sculpturally unique LED
panel display. This chapter describes the genesis of the project, from its inception
as an artistic response to news reports of widespread coral bleaching events compli-
cated by climate change, through a concerted phase to properly ground the project
in scientific research, followed by an overview of the methodologies and resources
used to realize the project, and including a discussion of the technical and conceptual
challenges posed by working on a very large scale display medium in a public venue.
Keywords Mutualistic · Symbiotic · Coral reef ecosystem · Climate change ·

Global warming · Ocean acidification · Anthropocene crisis · Endangered
species · Bleaching · Non-humans · Habitat loss · Environmental distress ·
Cultural processing · Shedd Aquarium · 3D animation · Reef-building coral ·
Scleractinian · localStyle · Electronic choir · Forty voice motet · Thomas Tallis ·
Polyphonic · Recomposition · Ambient · EDM · Sound design · Tetrapod
M. Novak
Department of Film Video, New Media and Animation, The School of the Art Institute of Chicago
(SAIC), Chicago, USA
e-mail: mnovak1@saic.edu
J. A. Yim (B)
Composition and Music Technology, Bienen School of Music, Northwestern University,
Evanston, USA
e-mail: jaymar@northwestern.edu

186 M. Novak and J. A. Yim
11.1 Introduction: Corals Are in Crisis
Many coral species cannot adapt quickly enough to warming ocean temperatures and
increasing acidification, threatening reefs worldwide and creating the potential for a
catastrophic loss. The habitat created by coral reefs makes them fundamental to the
sustainable diversity of ocean fauna, and to all living things—including humans—
that rely upon this part of the earth’s ecology.
Our audiovisual installation regards coral reefs as the ‘voice’ of the Anthropocene,
hence the title Choral. Reefs play a foundational role in providing habitat for a
quarter of all marine species and these ecosystems are in crisis. Although they face
challenges from multiple directions, we are approaching our piece from an optimistic
perspective. Techniques for fostering recovery and regeneration are already under
development: a dedicated part of the scientific community is determined to find ways
to make these methods scalable, and this is cause for hope.
Through our creative work, we endeavor to join this collective effort by con-
tributing to a larger-scale human response. Although scientific research informs our
fundamental approach to the visual appearance of various coral species, the way
corals behave in the work reflects our shared imagination through speculative under-
water world-building—rather than emulating a literal documentary. We were invited
in Spring 2017 by the curator for 150 Media Stream [1], Yuge Zhou [2], to make a
moving image artwork for 150’s unique wave-shaped 89 vertical panel LED-screen
lobby display. This custom-designed technology allows for working at a large scale—
the Media Stream display measures 6.5 by 47 m—while the nature of this venue in
the heart of downtown Chicago allows for a larger audience to encounter and con-
sider this topic. Our response, Choral, is a computer-generated 3D animation with
electronic sound that addresses our inquiry focused on the ecosystem of coral reefs
(Fig. 11.1).
11.2 Initial Research Phase
Having attended Ars Electronica for five years, three Documentas, several Venice
Biennales, as well as having participated in the STRP Festival, the National Art
Museum of China’s TransLife Triennial, the Taipei Digital Art Festival, and encoun-
tering a wide range of art installations incorporating sound, moving image resources,
or combinations of the two, in addition to maintaining a collaborative praxis as artists,
both of us are professors who have taught numerous classes since 2003 that involve
new media and installations. We take our responsibilities to stay well informed seri-
ously as it is integral to our pedagogical activities. The following essay may best
be understood as intentionally circumscribed in its scope, since a more expansive
survey of digital installations as a genre would exceed our remit.
As artists we have been fortunate to have a range of public platforms available
to reach a diverse audience. The concern for what we, as a species, have done to
11 Coral Voices 187
Fig. 11.1 Choral at 150 N Riverside Plaza, Chicago Illinois, USA. Photo Michael Salisbury [3],
image copyright ©Michael Salisbury and reproduced by permission
the planet is reflected in our work as individual pieces are used as opportunities to
probe deeper into these topics while assessing the effectiveness of different modes of
expression on audience engagement. Given our history of collaborating on projects—
pr!ck (2006–08) [4], scale (2009–10) [5], Bird (2011–14) [6] —that involved aspects
of what two physicist colleagues have termed “cultural processing” [7], we felt it
necessary to delve into more informed research beyond our initial encounters with
journalistic reports dating from 2014 regarding the catastrophic events that coral reefs
were suffering around the planet. Determined to gain a better grasp of the situation,
we first undertook a program of self-education through a variety of documentary
films and videos, paralleled by perusing a selection of more recent scientific papers.
In the course of learning more about the multiple causes and disastrous conse-
quences of coral bleaching and death, we encountered the research program of two
Chicago-based scientists, Luisa Marcelino (Research Assistant Professor of Civil
and Environmental Engineering, Northwestern University) [8] and Timothy Swain
(Postdoctoral Fellow, Northwestern University and the Field Museum) [9], whose
published articles were a source of inspiration to us, and who agreed to meet us in
person. Prior to our March 2018 meeting, we viewed a WTTW television segment
where both were interviewed and we were struck by Dr. Swain’s observation, “The
coral reef ecosystem may be the first ecosystem lost to climate change” [10]. Though
the topic was initially grim, our discussion with Marcelino and Swain was infor-
mative and productive. An afternoon conversation with these scientists absolutely
validated our concerns regarding the critical condition of the global ecosystem of the
reef and our commitment to developing the project for 150 Media Stream. We learned
that the Marcelino Research Group’s work at Northwestern points to coral polyps as
architects, wherein the fractal properties of their skeletons redistribute light as a pho-
tosynthetic resource within the colony. Further study of this phenomenon suggests a
relationship between faster-growing species and increased vulnerability to bleaching.
At the end of our meeting, Prof. Marcelino offered to curate a number of critical
papers for us to study and Dr. Swain invited us to join him at an upcoming visit to
Chicago’s Shedd Aquarium to observe some of the coral works that he was engaged
with in their laboratory. Having a chance to see these animals close-up and in-person
brought a more immediate level of appreciation of their charismatic qualities. It was
moving to see how tiny these creatures were while playing such a large role in marine
ecosystem sustainability and by extension human economic dependency on that very
ecosystem. As Rebecca Albright says:
Although reefs cover just 0.1 percent of the ocean floor, they support nearly 25 percent of all
marine species, including fisheries that feed millions of people worldwide. They also provide
natural breakwaters that protect coastal communities by reducing wave energy by up to 97
percent and wave height by up to 84 percent. And they generate vast tourism revenue. If we
lose reefs, we jeopardize the livelihoods of 500 million people and more than $30 billion
annually in goods and services [11].
Many people benefit unknowingly from the coral reef habitat; numerous articles
refer to the reefs as the “rainforests of the sea” [12] or “the medicine cabinets of the
21st century” [13], from whence a number of anticancer, antiviral, and pain manage-
ment drugs are being explored, as well as bone graft material. During our behind-
the-scenes encounter at the Shedd Aquarium, one of the staff members remarked
that although they had valuable and instructive coral displays, they felt they weren’t
reaching enough of a diverse public to get their message across, and Swain replied:
“That’s what Marlena and Jay do, as artists”. Consequently, we realized Choral
could play a role beyond expressing our personal commitment to Anthropocene
issues; it might also have the potential to raise public awareness and connect that
awareness to the marine biology community that is actively working on solutions to
the problems. Helping this community become more visible might ultimately trans-
late into the public policy decisions we collectively need to be enacted for corals
to be saved. One of the papers Marcelino referred us to was “Coral Reefs in the
Anthropocene” [14]. Among the noteworthy points found:
“[…] research has revealed provocative complexities in the expanding knowledge base about
corals and their ecosystems, thus stimulating our imaginations regarding alternate ways to
envision the future of reefs” and “Increasingly, coral reef scientists and managers encounter
previously unseen configurations of species” [14]
According to Terry Hughes, the reality is that:

reef ecosystems are more dynamic and patchier, as well as increasingly different to any-
thing that people have encountered before. Embracing this paradigm shift will necessitate a
transformation in the governance and management of these high-diversity ecosystems [14].
There won’t be any single answer to these problems; solutions to fostering sustain-
ability for corals in the future could build upon a wide range of methods developed
11 Coral Voices 189
by Marcelino’s group as well as many other scientists working in the field. Some of
these include the creation of a meta-level tool to correlate bleaching data from all
previous coral studies into a coherent worldwide index [15] and identifying species
better adapted to survive. We learned from Albright’s article that:
Mary Hagedorn of the Smithsonian Conservation Biology Institute has established the first
genome repository for endangered coral species. […] Her team has developed a freezing
system for sperm that can be applied to a wide range of coral species. To date, the team has
successfully banked 16 species from around the world (2 percent of the earth’s estimated
800 species) [11].
Our optimism increased when we learned about the work of the late Dr. Ruth Gates
at the Hawaii Institute of Marine Biology and her collaboration with Madeleine van
Oppen of the Australian Institute of Marine Science to develop next-generation super-
corals. Their labs are working on selective breeding and epigenetic tuning to tolerate
warmer sea temperatures, and selective breeding of the symbiotic zooxanthellae and
the bacteria that comprise the microbiome of coral polyps to breed more robust
colonies. SECORE International is working on both increasing genetic diversity and
fostering the odds in favor of coral larvae surviving past their initial vulnerability to
being eaten by other marine creatures; when they are big enough to be outplanted,
these new generations of young corals can help repopulate damaged reefs. One of
the best designs for an economically scalable substrate for baby corals to grow upon
is a grapefruit-sized concrete tetrapod, textured so that the larvae can readily attach
themselves and start their reef-building. These tetrapods are featured in one sequence
of Choral as a symbolic representation of the collective efforts of humans to channel
their ingenuity toward finding a solution to the bleaching and reef degradation crisis.
(One of our contacts at the Shedd Aquarium, Mark Schick is a collaborator with
SECORE in the tetrapod project.) Other, more speculative approaches under devel-
opment include deploying groups of underwater robots that would re-cement broken
coral fragments to the deep-sea cold-water reefs they came from, aiding re-growth.
Having first-hand discussions with scientists who are dedicated to identifying and
addressing the complex issues that promote coral bleaching was encouraging. These
experiences bolstered our decision to create an audiovisual artwork that engendered
hope and fostered public engagement rather than despair and cynicism. A meeting
with Hobson [16] who is a researcher at NORC was also valuable to our project; draw-
ing on personal experiences from research projects undertaken when she was based
in Australia, Hobson told us that “a noisy reef is a healthy reef ” which inspired us to
investigate this aspect and subsequently became a factor in shaping the conceptual
approach to sound design and soundtrack orchestration in Choral.
11.3 Project Development, Phase Two
A steady intake of information fostered the necessary processes of refinement and

metamorphosis, guiding the project’s vector away from didacticism toward a more
poetic direction. A project of this scale compelled us to undertake two different,

though complementary research tracks. The scientific data obviously had to be assim-
ilated and understood to inform our conceptualization of the project, but we also
needed to comprehend what kinds of hardware, software, and personnel resources
were needed to realize our vision, and to develop a budget and a team that could
feasibly support that goal. With the scientific side in full swing by the onset of
winter 2017, we accepted an invitation to participate in the Species/Biodiversity
Loss panel of the conference Why Do Animal Studies?: The Turn to the Quasi-,
Post-, Anti-, Non-, Para- [17] in April 2018, with our presentation “Choral, a work
in progress”. At minimum, it was a chance to get expert feedback from scholars
whose critical thinking and writing on non-human others and Anthropocene-centric
issues had been impactful on our own mode of inquiry and efforts. As is common in
these types of conferences, it was also a remarkable opportunity to be immersed in
the multidisciplinary confluence of scholarly and artistic projects that comprise the
field.
One of the strengths of how localStyle’s collaborative approach has deepened since
its founding in 2000 is the way we are able to pursue independent development tracks
in our working methods based on our individual skill sets and areas of expertise, while
maintaining a constant conceptually driven dialogue that revolves around the central
thrust of any specific project. Inspired by the hospitality of corals to create habitats for
other creatures, Novak began modeling, texturing, and animating individual corals—
devoting time to both individual colonies from various species as well as to detailed
hexacoral polyps—and to assemble a synthesized version of their biome utilizing
3D software tools (Autodesk Maya, Arnold Renderer, Pixologic zBrush) on a Mac
Pro workstation. In the conference, her slides incorporated digitally created images
that symbolically represented coral bleaching and the degradation of the ecosystem
(Figs. 11.2 and 11.3). Based on audience feedback, this imagery turned out to be
effective for attendees in the academic context of various presentations zeroing in on
challenges that ranged from justice, law, and ethics to postcolonialism to disruption
on a variety of planetary scales. These preliminary images thus played a crucial role
in our conceptual development of Choral, even though we ultimately decided to
tactically focus the installation’s mode of address toward one that emphasized the
Fig. 11.2 Study for Choral: Reef Ecosystem Degradation. 3D image: Marlena Novak, image
copyright ©M. Novak 2019 and reproduced by permission
11 Coral Voices 191
Fig. 11.3 Study for Choral: Zooxanthellae expulsion. 3D image: Marlena Novak, image copyright
©M. Novak 2019 and reproduced by permission.
hopeful side of human efforts to rescue coral. As with any environmental or climate-
related issue, the presence of the political dimension is never far from the surface,
and the operational question arises as to how people can be motivated to engage and
empathize with a topic so as to enable policy-making to occur in positive ways.
Shortly after meeting with the Northwestern University scientists, the project
implementation side of Choral began to accelerate. The organization that devel-
oped for the installation at 150 N Riverside Plaza placed Novak as co-producer,
director, project manager, lead 3D visual artist (concept, modeling, texturing, and
animation), and motion graphics conceptual developer; Yim was the co-producer and
composer/sound designer as well as being involved with the 3D and motion graphics
conceptual development. Clearly a work of this scope would have much to gain from
having project assistants; they would benefit from the experience, contribute to the
work in a multitude of ways, and be credited for their roles in a high-profile project.
In Spring and Fall 2018, Novak assembled a group of young artists who had
formerly studied with her at the School of the Art Institute of Chicago. At an early
stage, she compiled a collection of references as an encapsulated database to share
with team members. These comprised science-based pieces of journalism in print,
web browser, and video formats, organized into three main groups:
(1) how corals live,
(2) reasons for humans to care about corals, and
(3) factors threatening their existence.
The first group of references concerned coral taxonomy, structure, anatomy, habi-
tat, behavior, etcetera. It was important for everyone on board to assimilate some of
the basic information about these animals: coral polyps have a symbiotic relation-
ship with photosynthetic dinoflagellates; their survival depends on these algae that
provide between 90 and 98% of their nourishment; these zooxanthellae live under
the epidermis of each polyp; the polyp’s epidermis is coated with bacteria; and these
three organisms appear to benefit one another. Especially pertinent were key points
provided by the Coral Reef Alliance:
Coral reefs are often referred to as the medicine chests of the sea. A number of creatures found
on reefs produce chemical compounds that have been isolated for human applications—and
many more are yet to be discovered. Scientists have developed treatments for cardiovascular
diseases, ulcers, leukemia, lymphoma, and skin cancer, all from chemicals in reef plants
and animals. Other compounds reduce inflammation, kill viruses, and relax muscles. The
beautiful and fragile creatures of our coral reefs have the potential to make even greater
contributions to our lives by providing new cures for life-threatening diseases. More than
half of all new cancer drug research focuses on marine organisms. In addition, coral’s unique
skeletal structure has been used to make our most advanced forms of bone-grafting materials
[18].
In addition to the stresses of temperature-induced bleaching and ocean acidifi-

cation weakening their skeletal structures, other factors (many of which are under
direct human control) combine to threaten coral’s existence: in the natural competi-
tion between coral and seaweed, warmer water combined with fertilizer run-off gives
seaweed and other macro-algae the advantage; overfishing of species that ordinarily
keep seaweed in check by feeding on it exacerbates the stress; governmental policies
that revoke environmental protections for existing marine national monuments result
in more pollution from industrial exploitation such as mining and oil development;
even tourism can be a factor as products like many sunscreens are toxic when intro-
duced to the water. (The most recent evidence points to these products also being
toxic to humans.)
As work progressed, we held group screenings of selected documentaries to har-
monize our perspectives. Joining localStyle in preproduction concept development
were Sally Jo [19] and Malu Ayers [20], and project intern Shinuo Snow Xu [21].
As team assignments solidified, Ayers and Xu worked as 3D artists (modeling, tex-
turing, lighting, animation); Max Crider [22] built and animated 3D rigs for polyp
and coral animation in the close-up scenes. Nathaniel Gillette [23] joined us in the
Fall as a 3D and 2D artist and became motion graphics coordinator. Also in the Fall,
Mak Hepler-Gonzalez [24] joined the Choral team as assistant producer, rendering
manager and 3D artist involved in all aspects of 3D production, along with post-
production, conceptual, and technical development; he contributed several advanced
3D techniques for compositional layout, rendering, and animation that would serve
the project until its completion (Fig. 11.4).
11.4 Workflow
Due to having an artist’s residency abroad during some of this period, we set up a
shared online project folder and a blog to stay involved with each other’s progress;
11 Coral Voices 193
Fig. 11.4 Choral team members working at Unspecified Research Lab (top to bottom: Gillette,
Novak, Hepler-Gonzalez, Ayers, Xu) Photo Jay Alan Yim, image copyright ©J. A. Yim 2019 and
we also scheduled regular Skype meetings while away. Once back in Chicago, studio
sessions resumed. Our working method evolved to include collaboration on many
levels: in some cases, a single artist modeled, textured, lit, and animated their models;
at other times, they would hand off models they made for another team member to
texture, while another artist did the lighting and another executed the animation.
If one of us developed a new material that would be suitable for others to use, or
a lighting-set, or an animation technique that would be effective for each other’s
models, we shared these 3D elements. For the most part, our roles were fluid, unlike
production hierarchies where a single person works on modeling and another solely
engages with texturing, and so forth. Except for one artist who was only involved
with much of the rigging, we exchanged and reversed the majority of our roles. This
created a special sense of team effort and connection to the final piece. When one of
our valued members relocated to another part of the country, we continued to work
together via retrieval and development of their uploaded files. Choral would not exist
in its current state if not for the synergy of this solid collaboration.
We were fortunate to receive supplementary grant funding from the Illinois Arts
Council and a Faculty Enrichment Grant (SAIC) that supported aspects of the project;
several new computers were necessary to address the rendering time needed for the
unusually high resolution of each frame of animation. In addition to the previously
mentioned software programs (Maya, Arnold, zBrush), Allegorithmic Substance

Painter was used for texturing and Adobe After Effects was used for composit-
ing. Over the course of the project, two custom-configured PC workstations were
acquired by Unspecified Research Lab, which is localStyle’s studio facility. Sound
design/scoring took place on two Mac workstations. Throughout the project, as many
as eight laptops were used in conjunction with the desktop machines. Work continued
in a mixed Mac/Windows environment, tailored to take advantage of the capabilities
of each platform (Fig. 11.5).
Fortunately, we had designed our studio space to be flexible in usage, which
allowed a number of artists to work together comfortably and in close proximity to
the kitchen and seating areas where we could take breaks as necessary. We regularly
provided lunches and dinners, and team members ate together depending on the
work periods (Fig. 11.6). Sharing home-cooked meals, often from items grown in
our garden, built a closer sense of community; several times an artist from the team
would also contribute cookies or bread they had baked. Group meal conversations
sometimes addressed recent project achievements or trouble-shooting concerns, but
mainly it was a time to focus on positive topics.
Fig. 11.5 Choral team members working at Unspecified Research Lab (left to right: Yim, Ayers,
Hepler-Gonzalez, Gillette, Crider). Photo Marlena Novak, image copyright ©M. Novak 2019 and
11 Coral Voices 195
Fig. 11.6 Choral team members enjoying a lunch break (clockwise from upper right: Yim, Gillette,
Xu, Hepler-Gonzalez, Ayers). Photo Marlena Novak, image copyright ©M. Novak 2019 and
11.5 Compositional Design and Technical Challenges
There were a number of design challenges unique to this project. One of them entailed
the sheer physical size of the video display; with the tallest panels standing at nearly
22 feet and the full array spanning 153 feet in width, we needed to make the visual
elements narratively coherent even though it is impossible to stand back far enough
in the lobby to see all of the LED blades simultaneously from the center. It is quite
a different experience to visually compose a scene with digital models being made,
textured, and animated on 28-in. computer screens having 3840 × 2160 resolution;
we could assess the design in its entirety. Though we are fortunate to have a 4.9 m
(diagonal) projection screen in-house at URL, that is still dwarfed by the size of the
tallest blades (6.54 × 0.225 m) at 150 Media Stream. The physical scale impacted
team decisions regarding formal qualities such as repetition, proximity, motion, and
pacing, requiring care as to where visual elements would be located to create the most
dynamic mise-en-scène. Strategically, the array’s width helped direct the temporal
proportions of different passages in Choral, as the situation compelled the viewer to
move actively through the space to take in as many CGI corals as possible.
The second major factor in the design was working with the negative space that
is an integral component of the display: we had to be conscious of this site-specific
parameter to ensure that animated elements retained maximal legibility as they nego-
tiated the voids between panels. Though it was quite useful to have a TIFF mask
composited as a layer to preview image sequences on our computers, we appreciated
the difference it made to see and hear the draft versions of Choral in situ. In a sense,
it was visually somewhat like working with a reverse zoetrope in that the alternating
slits and opacities worked together to create an interrupted continuity.
LED display resolution was a corollary aspect of the size factor: 150 Media
Stream’s total pixel count was 15360 × 2160, the equivalent of four 4K screens
arranged in a horizontal panoramic view. Rendering moving image sequences at
high resolutions is intensive in terms of both time and computational resources.
We had non-exclusive access to a multiblade render farm, and non-exclusive access
to several banks of computers, in addition to the machines at URL. Emily Kuehn
[25]—who is an exceptionally resourceful staff colleague at SAIC—provided crucial
support and assistance in addressing the unique and precise Maya template for the 89
screens (which measure a variety of heights and widths), and optimizing the render
sequencing setup for the 16K display.
The acoustics of the building conditioned the approach taken to orchestrating the
electronic soundtrack and equalizing the mix: the presence of many hard, reflective
surfaces—stone, glass, metal—and the height of the atrium effectively disperses
much of the audio signal and makes it nonlocalizable. Architectural design at 150 N
Riverside Plaza produces acoustic results that are comparable in some significant
ways with the reverberant spaces often associated with the performance of large
choral works. This made the decision to base the soundtrack on Thomas Tallis’ forty
voice motet [26] an apposite one, both for these acoustic characteristics and because
the eight-channel audio system corresponded to the polychoral division of voices in
his score into eight separate vocal quintets. Additional motivations for choosing this
point of departure will be discussed below.
11.6 Choral Animation Summary
The introduction begins with waves on a dark sea (where the ocean is a digital
simulation). In the tradition of an origin myth, we see friendly bioluminescent polyps
ascend to take their places amidst a field of stars in a night sky filled with simulated
constellations (Fig. 11.7).
Stylized bubbles rise against the water to become stars; many are replaced by the
astral polyps. As the polyps wink out, the celestial field metamorphoses into a digital
wire frame that descends to form the contours of a seabed, gradually evolving into
more solid terrain (Fig. 11.8).
As an aqueous curtain rises, a richly populated reef full of many species of coral
is revealed, and the virtual camera pans at a deliberate pace across the ocean floor
(Fig. 11.9).
Being digitally modeled, our reef places all of the corals front and center to empha-
size their importance to the community of fish, other creatures, and marine plants that
11 Coral Voices 197
Fig. 11.7 Polyps rising. Photo Michael Salisbury, image copyright ©Michael Salisbury and
Fig. 11.8 Celestial polyps Photo credit: Michael Salisbury, image copyright ©Michael Salisbury
and reproduced by permission
Fig. 11.9 Seabed revealed. Photo Marlena Novak, image copyright ©M. Novak 2019 and
would comprise and inhabit this ecosystem in nature. Throughout our 3D animation,
the corals are singing when visible (Fig. 11.10).
The digital camera zooms in and slowly circles around several of the more charis-
matic colonies on this imaginary reef. Despite most coral polyps being individually
very small animals, in the aggregate, they play an outsized role in the formation
of reef ecosystems, which is our motivation for modeling them in detail for these
close-up views (Figs. 11.11 and 11.12).
The intended goal of these two sequences is to engage audience members with the
awe-inspiring beauty of coral: to reconnect them to personal memories of experienc-
ing coral in the sea or other environments. Hearing viewers remark, “We should go
back to the Shedd Aquarium” after seeing our piece in the lobby of 150 N Riverside
underscored the importance of making this kind of contact since Chicago’s Shedd
Aquarium has very good coral exhibitions with detailed information regarding the
stresses on coral ecosystems and how those risks will impact our lives if corals are
allowed to perish.
The following passage features many computer-generated tetrapods with growing
baby corals descending through the water (Figs. 11.13 and 11.14). These represent
the small concrete forms that researchers have designed to serve as ideal substrates
for newly attached larvae to grow upon. Outplanting tetrapods is one of several
complementary solutions currently under development and the Shedd is involved in
promoting this approach. As artists, we had hoped that an engaged public would
be curious to know more about what the tetrapods are and how they relate to coral
reefs; during the test screenings, we have been pleased that this sequence has sparked
people’s interest and initiated conversation.
11 Coral Voices 199
Fig. 11.10 Coral reef Photo Marlena Novak, image copyright ©M. Novak 2019 and reproduced
by permission
Next, the scene crossfades from objective potential to the metaphysical, as the
corals in Choral dance and demonstrate choreographic prowess and graceful coor-
dination. They use their bioluminescent qualities in place of costumes. The dance
aspires to usher in their miraculous spawning event (Figs. 11.15 and 11.16).
Finally, the magical moment arrives that happens once per year when coral
colonies spawn in synchronization with other colonies of the same species: eggs
and sperm are released by the millions to join in the water and produce new coral
larvae. For those species whose preference is to incubate their larvae inside the bod-
ies of their polyps, the mother polyps shelter their eggs and are fertilized by sperm
from male polyps; when the time comes, the larvae emerge from the mouths of their
mothers. Regardless of the method, the new corals and gametes rise in the water col-
umn toward the starry sky, swimming up to the surface as they are attracted to light,
and eventually falling back to the seabed where they will begin new colonies. Sym-
bolizing the cycle of regeneration, our animation is formulated to seamlessly loop
from the spawning sequence back to the ocean scene at the beginning (Figs. 11.17
and 11.18).
Fig. 11.11 Coral close-up view. Photo Jay Alan Yim, image copyright ©J. A. Yim 2019 and
11.7 Sound Design and Electronic Score Summary
A healthy reef is in reality a noisy place, thus, our narrative begins with a rhythmicized
evocation of that percussive sound world, gradually superseded by more sustained
and contemplative textures. When the seabed materializes and we see the reef replete
with coral colonies of many kinds, we hear the polyps singing polyphonically; it is
left to the viewer whether one chooses to interpret this as metaphor or fantasy: that
corals are the voice of the Anthropocene.
These non-human choristers are constantly present in the sound design of Choral
until the final gamete sequence fades back into the ocean scene of the opening
frames. Conceptually, the soundtrack began as a recomposition project, taking one
of the most celebrated choral works in the early music repertoire, Thomas Tallis’
1570s motet for forty voices “Spem in alium”, and applying an idiosyncratic version
of musica ficta [27] so pervasively as to relocate the harmonic world of the motet
to that of the present day. (At this point, it may be worth noting that this author
first encountered the Tallis score as an undergraduate composition student in 1977.
It was immediately elevated to my personal list of desert-island discs, and from the
time of initially becoming a professor of composition, has for decades since then
been recommended as a study score to my own students. It is truly a marvel of
counterpoint and spatialisation, arguably superior—particularly in terms of melodic
11 Coral Voices 201
Fig. 11.12 Active polyp close-up. Photo Marlena Novak, image copyright ©M. Novak 2019 and
independence—to Alessandro Striggio’s [28] own forty voice motet, “Ecce beatam
lucem”, which is typically seen as the precursor to and inspiration for the Tallis.) The
timbral world of the music is similarly updated to incorporate otherworldly vocal
sounds with electronic textures far removed from the sensibilities of the Renaissance.
In the spirit of technical inversion, all of the percussive sounds are triggered by the
rhythms of Tallis’ motet, whereas the smooth resonances that create the impression
of being in the ocean have been fashioned from underwater hydrophone recordings
of reefs, which are full of clicking sounds with short attack and decay times.
A variety of software tools contributed to the workflow: Finale for MIDI score
input and recomposition in a staff notation environment (due to my classical com-
positional training, staff notation offers more effective control over counterpoint
and harmony than the piano roll display in Live); Ableton Live as a MIDI track
host for softsynth instrument playback and recording (including Native Instruments’
Absynth 5); Sound Forge for time compression and as a plugin host for DSP and sound
design (transforming the hydrophone recordings, shaping samples for the descending
sounds in the tetrapod scene, spectral blurring of the vocal textures); Audacity for
timestretching; and ProTools for overall track assembly, synchronization with video,
and mixing. In the plugin department, Eventide’s Blackhole reverb and Unfiltered
Audio’s BYOME were invaluable.
Fig. 11.13 Tetrapod with growing coral close-up. Photo Jay Alan Yim, image copyright ©J. A.
Yim 2019 and reproduced by permission
Fig. 11.14 Tetrapods descending to ocean floor. Photo Michael Salisbury, image copyright
©Michael Salisbury and reproduced by permission
11 Coral Voices 203
Fig. 11.15 Dance sequence close-up. Photo Marlena Novak, image copyright ©M. Novak 2019
Fig. 11.16 Dance sequence, long view. Photo Jay Alan Yim, image copyright ©J. A. Yim 2019
Fig. 11.17 Spawning coral sequence close-up. Photo Marlena Novak, image copyright ©M. Novak
2019 and reproduced by permission
11 Coral Voices 205
Fig. 11.18 Spawning coral sequence long view. Photo Jay Alan Yim, image copyright ©J. A. Yim
2019 and reproduced by permission
The first (and most intensive) step was undertaking recomposition of Tallis’ motet
in Finale. Working in score view gave me the greatest degree of control as chromatic
modifications—with increasing deviations—were made to the original score. Garri-
tan Orchestra samples were used to record the basic choral textures, but these were
modified substantially in Sound Forge with granular spectral tools and other audio
processors. The MIDI data from the Finale file was exported to Live for further
development. In Live, the combined MIDI data for all forty voices of the complete
motet were used as a single clip to trigger a variety of softsynth instruments; each was
recorded and rendered as an 8.5 minute audio track. These tracks were synchronized
to the same start time in ProTools because the scoring strategy was somewhat uncon-
ventional: (1) to have the recomposed motet play uninterruptedly from beginning
to end, thus ensuring continuity, while (2) the electronic orchestration changed for
each animation sequence in order to best support the scene, via automation curves
when making the final mix. MIDI data was processed using some of the commonly
available utilities found in most modern DAWs: automation of pitch bend data made
the microtonal retuning of the harmonic series arpeggios in the final scene easily
achievable and straightforward, and a moderately complex deployment of an arpeg-
giator function in the techno-prog scene was used to create the swirling melodic
arabesques.
11.8 Conclusions
Exploring this topic through the use of computer-generated motion graphics and 3D
software has created an opportunity for us, along with our artist team members, to
look very closely at these remarkable animals in order to observe their attributes as
best as possible. But it didn’t stop at digital mimicry; using these tools compelled a
type of looking that resulted in awestruck wonder at the complexity and beauty of
coral’s functions. The privilege of having this enormous, custom-designed ultra-high
resolution LED display to present our project in a public space allows a wide audience
of diverse means and pursuits to engage with this subject. We hope that some portion
of the viewers will be motivated to consider what role they may themselves play in
fostering the sustainability of coral reefs, as scientists in the field have made it clear
that we must act now without hesitation to save this ecosystem from extinction.
References
1. 150 Media Stream: [Online]. https://150mediastream.com/. Accessed 7 June 2019

2. Zhou, Y.: [Online]. http://yugezhou.com/. Accessed 7 June 2019
3. Salisbury, M.: “Michael Salisbury” [Online]. https://msalisbu.com. Accessed 7 June 2019
4. Pr!ck: [Online]. https://vimeo.com/116508599. Accessed 7 June 2019
5. Scale: [Online]. https://vimeo.com/19933816. Accessed 7 June 2019
6. Bird: [Online]. https://vimeo.com/45934636. Accessed 7 June 2019
7. Schaffer, K., Lemos, B.G.: Obliterating thingness: an introduction to the “What” and the “So
What” of quantum physics (2019) [Online]. https://doi.org/10.1007/s10699-019-09608-5
8. Marcelino, L., Marcelino Group:. [Online]. http://www.civil.northwestern.edu/EHE/
Marcelinogroup/. Accessed 7 June 2019
9. Swain, T.: Halmos College of Natural Sciences and Oceanography Faculty and Staff Profiles
[Online]. https://works.bepress.com/timothy-swain/about/. Accessed 6 March 2020
10. PBS: Chicago tonight [Online]. http://www.pbs.org/video/chicago-tonight-local-scientists-
institutions-fight-save-coral-reefs/. Accessed 7 June 2019
11. Albright, R.: Sci. Am. Jan 2018 [Online]. https://doi.org/10.1038/scientificamerican0118-42.
Accessed 7 June 2019
12. Marine Conservation: [Online]. https://marine-conservation.org/what-we-do/program-areas/
coral-conservation/coral-reefs/. Accessed 7 June 2019
13. NOAA Ocean Service: [Online]. https://oceanservice.noaa.gov/facts/coral_medicine.html.
14. Hughes, T., et al.: Coral reefs in the Anthropocene (2017) [Online]. https://doi.org/10.1038/
nature22901. Accessed 7 June 2019
15. Swain, T., et al.: Coral bleaching response index: a new tool to standardize and compare
susceptibility to thermal bleaching (2016) [Online]. https://doi.org/10.1111/gcb.13276
16. Hobson, K.: [Online]. http://www.norc.org/Experts/Pages/kate-hobson.aspx. Accessed 7 June
2019
17. Why Do Animal Studies: [Online]. https://www.whydoanimalstudies.com/. Accessed 7 June
2019
18. Why Care About Reefs: [Online]. https://coral.org/coral-reefs-101/why-care-about-reefs/
medicine/. Accessed 7 June 2019
19. Jo, H.: “HeeJun Jo” [Online]. https://www.heejunjo.com/. Accessed 7 June 2019
20. Ayers, M.: “Malu Ayers” [Online]. https://www.maluayers.com. Accessed 7 June 2019
11 Coral Voices 207
21. Xu, S.: “Snow Xu” [Online]. https://snowxushinuo.com. Accessed 7 June 2019
22. Crider, M.: “Max Crider” [Online]. https://www.maxcrider.com. Accessed 7 June 2019
23. Gillette, N.: “Nathaniel Gillette” [Online]. https://www.nathanielgillette.com. Accessed 7 June
2019
24. Hepler-Gonzalez, M.: “Mak Hepler-Gonzalez” [Online]. https://makhepler-gonzalez.com.
25. https://www.emilyemily.me/
26. Spem in alium: [Online]. https://en.wikipedia.org/wiki/Spem_in_alium. Accessed 7 June 2019
27. Musica ficta: [Online]. https://en.wikipedia.org/wiki/Musica_ficta. Accessed 7 June 2019
28. Alessandro Striggio: [Online]. https://en.wikipedia.org/wiki/Alessandro_Striggio. Accessed
19 Sept 2019
Chapter 12
Cyberdreams: Visualizing Music
in Extended Reality
Jonathan Weinel
Abstract From the visual music films of the twentieth century to the Video Jockey
(VJ) performances seen at the latest electronic dance music festivals, there is an
extensive body of artistic work that seeks to visualize sound and music. The form
that these visualizations take has been shaped significantly by the capabilities of
available technologies; thus, we have seen a transition from paint to film; from
hand-drawn animations to motion-graphics; and from analog to digital projection
systems. In the twenty-first century, visualizations of music are now possible with
extended reality (XR) technologies such as virtual reality (VR), augmented/mixed
reality (AR/MR), and related forms of multi-projection environment such as full-
dome. However, the successful design of visual music and VJ performances using
XR technologies requires us to consider the compositional approaches that can be
used by artists and designers. To investigate this area, this chapter will begin with
an analysis of existing work that visualizes music using XR technologies. This will
allow us to consider the spectrum of existing design approaches, and provide a com-
mentary on the possibilities and limitations of the respective technologies. Following
this, the chapter will provide an in-depth discussion of Weinel’s practice-led research,
which extends from work exhibited at the Carbon Meets Silicon exhibitions held at
Wrexham Glyndŵr University (2015, 2017), and includes AR paintings, VJ perfor-
mances, and a VR application: Cyberdream VR. Through the discussion of these
works, the chapter will demonstrate possible compositional principles for visualiz-
ing music across media ranging from paint to XR, enabling the realization of work
that reinforces the conceptual meanings associated with music.
Keywords Music visualization · Visual music · VJing · Virtual reality ·

Augmented reality · Extended reality · Electronic dance music
J. Weinel (B)
London South Bank University, London, UK
e-mail: weinelj@lsbu.ac.uk

210 J. Weinel
12.1 Introduction
In 2019 the Coachella music festival featured an “interactive augmented reality (AR)
stage,” in which audiences could use a mobile application [1] to enhance the expe-
rience through various fantastical contents, which appeared superimposed on the
Sahara venue when viewed through a camera [2–4]. When viewed through the app,
audiences could see “space objects like planets, asteroids, and stars” [5] within the
performance space, providing digital visual content that complements the audience
experience of music. This is just one example of extended reality (XR) technologies
being brought into live performance spaces for electronic dance music. XR is an
umbrella term that encompasses augmented/mixed-reality (AR/MR), virtual reality
(VR), and more broadly, other associated technologies such as multi-projection 360-
projection environments and fulldomes, all of which are increasingly being used to
complement or visualize music at electronic dance music events, by VJs and digital
artists.
The use of these technologies points toward a future in which public performance
spaces for music are enhanced through immersive XR content. Music performance
spaces are already immersive—sounds, lighting, and communal experiences of dance
can all contribute toward generating powerful social experiences that are emotive
and meaningful [6]. Yet, while sound can fully engulf the audience as a spatial, aural
experience, visual accompaniments too can go further in surrounding and immersing
the audience. Through the use of XR, we can begin to conceive music performances
where eventually a mesh of synthetic 3D graphics engulfs the performance space,
creating hallucinatory computer graphics that visualize the music [7]. Yet, these
technologies also generate a variety of new issues that must be considered through
research.
The main purpose of this chapter, then, is to explore how audio-visual experi-
ences of music can be designed in XR. Specifically, I will discuss compositional
approaches that can be used to design these experiences. I will argue for an approach
in which designs in XR can provide visualizations of music that go beyond basic
audio reactivity, embodying similar forms of symbolism as those present in the music,
and enhance the audience experience by reinforcing meaningful resonances with the
music. Towards this aim, first I will review some examples of XR music visualiza-
tions and VJ performances, in order to evidence present activity in this field. Follow-
ing this, I will then turn to consider my own practice-led research creating artistic
works such as AR paintings, VJ performances, and a VR application: Cyberdream
VR. The exploration of these artifacts, performances, and software applications will
demonstrate compositional approaches for realizing music visualizations that cross
boundaries from analog media such as paint, into the digital; and from digital pro-
jections into immersive forms of XR. Through the course of this chapter, I therefore
aim to illuminate possible approaches and open new conversations about visualizing
music in XR.
12 Cyberdreams: Visualizing Music in Extended Reality 211
12.2 The Current State of the Art
Visual music is an established art-form with considerable history dating back over
a hundred years to the early color organs; works by artists such as Kandinsky; and
later the films of artists such as Len Lye, Oskar Fischinger, John Whitney, and others
[8]. The essential priority of visual music and related forms such as the psychedelic
light show [9] is the representation of sound and music through complementary
visual media. From the 1980s to present, the rapid expansion and democratized
home availability of computer and video technologies saw a significant growth in
the related forms of music visualizations, music video, and VJ performance [10]. Of
course, there are many possible journeys through this expansive area of work, and
it is beyond the scope of this chapter to provide an extended history. As explored
in a recent panel discussion that the author participated in at London South Bank
University [11], some VJs connect their work with visual music, yet inspiration may
also come from the wider sphere of music culture, motion graphics, film, and video.
For our purposes here, it will be sufficient to acknowledge that music visualization
is a rich and varied field, but also one that has been shaped significantly by new
digital technologies over the past century. It therefore comes as no surprise that
XR technologies are gradually being assimilated into the panoply of tools used for
visualizing music, yet with this, they generate specific new affordances and concerns,
which we will now discuss through a selection of examples.
12.2.1 Multi-projection VJ Performances
VJ performance typically involves an individual mixing live video as a complement

to electronic dance music performances, in an approximately equivalent manner to
the DJ [12]. In recent decades, VJ performance has developed to use multiple projec-
tions, and video mapping techniques, which allow video to be projected on custom,
irregular surfaces. At electronic dance music festivals such as Mo:Dem (Croatia,
2017), elaborate sculptures provide the stage design, on to which the VJ projects
video-mapped visuals. Besides the main stage, smaller projection screens are also
mounted in the trees (Fig. 12.1), and complement other aspects of the festival decor
such as ultra-violet canopies. Here the VJ projections are audio-reactive, responding
to the beat, but the designs also reflect otherworldly, alien or shamanic symbolism,
which complements the psychedelic themes of the music (as discussed in [13]).
Along similar lines, Burning Man (USA, 2016) was one of many recent festivals
to feature a fulldome theatre. Here, the fulldome provided an ad hoc movie theatre,
which the audience could enter to view 360° films such as Samskara [14], which
provides a cinematic experience based on the concept of a psychedelic journey anal-
ogous to an LSD experience. The film incorporates electronic music and relates to
the overall psychedelic theme and ethos of the festival. Elsewhere, video-mapped
domes have also been used for VJ performances; for instance, United VJs provide
212 J. Weinel
Fig. 12.1 Psychedelic decor with VJ projections among the treetops at Mo:Dem festival, Croatia,
28 July 2017. Photo credit J. Weinel
performances [15] and workshops [16] specifically addressing techniques for VJing
in fulldomes.
Besides fulldome, other custom multi-projection setups are often used. For
instance, the Resolution [17] series of events in London has utilized various projec-
tion configurations to provided extended immersive visuals. For instance, the series
presented Sim Hutchins, whose performances revisit 1990s rave nostalgia through
combinations of electronic music and projected visuals, on a 270° projection system
in the G05 venue [18]. Meanwhile, another event at G05 featured music by Bobby
Tank with a VJ performance by L’Aubaine (Fig. 12.2, [19]), who performed on a
360° projection system.
12.2.2 Augmented Reality Companion Apps
Mobile apps are often used to provide companion experiences at music events, for
instance Notting Hill Carnival mApp [20] provides useful features such as a map of
sound systems for Notting Hill Carnival. The Coachella app fundamentally fulfills
a similar function, providing digital marketing and informative features; however, it
also goes further with the incorporation of a ‘Coachella Camera’ with AR features.
This provides AR features similar to the popular app Snapcat, allowing the user to
take selfies and photographs with AR enhancements. However, for our purposes here,
Fig. 12.2 L’Aubaine performing visuals on a 6-screen projector setup at Bobby Tank Oxygen
EP release event, by Not Like That, G05, London, 2 February 2019. Photo credit Laurie Bender
(L’Aubaine)
more interesting is the capability for real-time (synchronous) experiences of location-

specific AR content, such as the space-themed 3D imagery described in the opening
of this chapter, which could be seen at the Sahara stage when viewed through the app.
Here the space-theme reinforces the identity of the Coachella festival, suggesting a
fun, exotic festival experience that seductively indicates escape from the everyday.
Elsewhere, AR is also being used to complement or visualize music in other ways.
The Kybalion album by Øresund Space Collective includes a mobile application
created with artist Batuhan Bintaş. The application can be activated by viewing the
illustrations of the album artwork through a mobile device, bringing to life the artwork
through AR/MR and VR computer graphics. Here the app allows the audience “to
not only listen to the songs but also to learn about the Hermetic teachings of Thoth
by interacting with the album artwork” [21]. In this regard it extends the conceptual
universe suggested by the space rock music, effectively allowing the audience to
enter the imaginative virtual world that the music describes. Bintaş sees the work
as revitalizing the album cover as an artifact to be enjoyed alongside the music,
and in this regard it could be understood as complementary to the vinyl revival
[22]. However, this piece can also be considered as a ‘cyberdelic’ (a portmanteau
of ‘cyberculture’ and ’psychedelics’) experience, using a term that was popularized
in the 1990s by Timothy Leary [23], and is now used by The Cyberdelics Society
[24, 25], of which Bintaş is an affiliate. Kybalion fits with the idea of cyberdelics
because it utilizes the illusory capabilities of the technology to produce a digital art
experience that is analogous to psychedelic hallucinations.
214 J. Weinel
12.2.3 Music Visualizations for Mixed Reality Headsets
Although we have noted the Coachella AR app as an example of XR aimed at the

dancefloor, mobile phones may be poorly suited as a means of enhancing immer-
sion in the live experience of music; indeed their use at concerts has proven con-
tentious for audiences [26–28]. An alternative approach could be to use MR devices
such as the Microsoft HoloLens or Magic Leap, which allow the viewer to wear
a holographic headset that superimposes 3D content over the surrounding natural
environment. Elsewhere, MR headsets have been used to enhance digital arts exhibi-
tions, by bringing animated visualizations to life. For example, the recent exhibition:
Leonardo da Vinci and Perpetual Motion: Visualising Impossible Machines [29]
used tablet devices and a HoloLens to provide AR/MR 3D holograms, to visualize
Leonardo da Vinci’s perpetual motion sketches as 3D computer graphics animations
in the gallery setting.
This area remains relatively unexplored for music visualization, but there is some
early work, such as Synesthesia, a HoloLens app that provides an MR music visu-
alization experience based on the generation of audio-reactive graphics [30]. It is
possible that technologies such as these could be brought into live music events.
However, the obvious current limitation is that the devices are too expensive to be
widely used in the music festival environment. Significantly, the viewing angle of
these devices is also relatively small [31], which combined with cumbersome head-
sets would be likely to have a significant impact on immersion during a live music
event. At the present time, MR music visualization would therefore be unsuited to the
music festival environment, though certainly we could see visualizations of music
in smaller scale settings such as gallery installations. As the technology improves
and decreases in cost, it may be more feasible to organize larger scale events where
audiences use MR glasses to view live visualizations of music. Alternatively, tech-
nologies such as the Holo-Gauze [32] provide possibilities for projecting Pepper’s
ghost holograms on invisible reflective materials, which can be viewed by audiences
without headset. A similar approach using Pepper’s ghost holograms has been uti-
lized in visual music performances by Carl Emil Carlsen for his work with Silicum
[33], and it is possible that future technologies using approaches such as these may
provide other ways to introduce MR into the dancefloor context.
12.2.4 Music Visualizations in Virtual Reality
VR is also gradually being incorporated at music festivals. For example, Psych-

Fi [34] provides immersive experiences at music festivals such as Boomtown Fair
and Sci-Fi London film festival. At the 2016 edition of the Liverpool International
Festival of Psychedelia, their app Dioynsia was included in the PZYK Gallery, a
multisensory arts installation designed to complement the festival experience [35].
Dioynsia provides a short journey into a hallucinatory landscape in VR, thereby
realizing the idea of a psychedelic trip through synthetic computer graphics and
sound [36].
Elsewhere, L’Aubaine has also created work that translates the aesthetics of her VJ
performances into 360° narrative piece, 360 Life #1, which explores “introspection
versus outrospection and the boundary between reality and surreality” [37]. Works
of this type can be situated in the gallery context, but are also sometimes featured
at VJ events; for instance, Fathomable’s Gnosis [38] is a cyberpunk VR experience
created by VJ Rybyk, which was featured at VJ London’s AV Depot event [39]. There
are also various other commercial VR experiences of music, which are relevant to
consider in this section. For example, Fantasynth [40] is a short VR experience which
provides a journey through a landscape of audio-reactive graphics, while The Wave
VR [41, 42] and Amplify VR [43] are other music platforms that aim to provide music
video or virtual concert experiences in VR. There are also various other bespoke
music-related VR music experiences, such as Fabulous wonder.land VR [44], a VR
experience based on wonder.land, a National Theatre musical created by Damon
Albarn, Moira Buffini, and Rufus Norris.
12.2.5 Discussion
This section has outlined a selection of examples where XR technologies have been
used to visualize music. It is significant to note that similar XR technologies underpin
many of these productions, since 360° production software and video game engines
such as Unity can be used to adapt XR experiences across multiple platforms. Thus, in
some cases, where a work is designed for VR, it can be realized for other formats such
as AR/MR or fulldome projection. The fluidity between these platforms is one reason
why it is pragmatic for our discussion here to look at the bigger picture regarding
the use of XR for visualizing music. Yet in considering various types of XR, we also
find that these technologies afford different forms of audience experience. In some
cases, multi-projection environments may be used to provide VJ experiences in the
electronic dance music context, extending the lightshow. Yet in festival environments,
XR technologies such as AR/MR and VR are finding new contexts, such as their use
to provide ‘side-shows’ that audience members may experience between the main
acts.
It is perhaps worthwhile to consider how XR technologies may impact on dance-
floor immersion. Dancefloor immersion is often characterized as arising from the
experience of losing oneself in communal experiences of dance and music [45]. Yet
Rietveld [46] argues that electronic dance music culture has undergone a shift from
dimly lit nightclubs and warehouse parties, toward visual spectacle, which can be
associated with the power structures of consumer capitalism. In her argument, elabo-
rate stage designs enhance the visual and redirect the gaze of dancers away from each
other, toward all-powerful ‘superstar DJs’. For Rietveld then, enhancing the visual
spectacle of events may lead to a negative effect on collective experiences of dance-
floor immersion. From this perspective, XR clearly carries a risk. When we consider
216 J. Weinel
examples such as the Coachella app, audiences are encouraged to direct their gaze
to a mobile device, shifting activity away from the dancefloor toward social media
interaction and the narcissistic taking of ‘selfies’. The negative impact of mobile
phones on dancefloor immersion is already recognized in popular music press—for
instance, a recent article in DJ Mag [47] even calls for a ban on mobile phones on
the dancefloor, referring to academic research by Henkel [48] that suggests a ‘photo-
taking-impairment’ effect on memory, underscoring the idea that such activities may
reduce the presence and immersion of individuals in real-world contexts.
However, XR may also be capable of immersing the participant in other virtual
spaces that are distinct from the dancefloor [49]. Where VR ‘side shows’ are provided
at festivals, these may provide virtual spaces that relate to the themes and symbolic
meaning of the event as a whole. In St. John’s [50] discussion, festivals and raves
provide liminal spaces of physical and social activity that are removed from the
everyday. While the dancefloor experience may be of critical importance for these
events, activity in these spaces is diffuse and encompasses multisensory experiences
in which aspects such as clothing, costumes, and conversation are also significant.
Here, XR may be complementary, since condensed experiences of digital content
away from the dancefloor may stimulate conversation, reinforcing the meaning and
immersion into the event as a whole.
12.3 Case Study: Projects Visualizing Music in Extended

Reality
Having outlined various examples related to the visualization of Music in XR, I will
now turn to consider my own practice-led research in this area, which includes work
across the areas of AR painting, VJ performance and a VR application: Cyberdream
VR. Notably, some earlier iterations of this work were presented and discussed in the
Carbon Meets Silicon exhibitions at Wrexham Glyndŵr University (2015, 2017).
Beginning with a brief outline of earlier artistic works, in this section I will provide a
personal journey through my creative practice, in order to demonstrate how the work
visualizes music in different ways across a variety of media, eventually moving into
XR territories. In doing so, I aim to illuminate some possible compositional strategies
for visualizing music in XR.
12.3.1 Background
My background is in electronic music and visual arts, and was significantly developed
through my Ph.D. [51], completed at the Keele University music studios. Here my
work focused on the composition of electroacoustic music based on altered states
of consciousness. In summary, this work seeks to design music that is analogous to
the form and structure of psychedelic hallucinations, through corresponding sonic

materials and structures. This resulted in a series of compositions that were released
on the Entoptic Phenomena in Audio 12” vinyl [52]; software tools that were used
to realize these compositions; and an audiovisual composition entitled Tiny Jungle
[53, 54]. These works organize sounds, and (for audiovisual works) visual materials,
in order to construct experiences analogous to hallucinatory journeys [55].
I later extended these ideas through further audiovisual compositions: Mezcal
Animations [56], Cenote Zaci [57] and Cenote Sagrado [58] are three fixed-media
visual music films that seek to provide synaesthetic experiences of electronic music
and abstract visuals, based on the concept of altered states of consciousness. These
were widely performed at international festivals for electronic music and visual
music, such as the International Computer Music Conference [59], Seeing Sound [60,
61] and others, and were included in Technoshamanic Visions from the Underworld
[62], a loop of collected audiovisual works presented at the first Carbon Meets
Silicon exhibition. Notably, these works use the technique of ‘direct animation’,
where 8 mm film is hand-painted, projected and digitized, and then combined with
other materials such as stop-motion animation and computer graphics. Around this
time at Wrexham Glyndŵr University, I also created Quake Delirium [63, 64], a
video game modification that seeks to represent a hallucinatory state in the form of
an interactive video game; and Psych Dome, an interactive installation for mobile
fulldome, in which participants wear an EEG headset that captures brainwaves, which
are used to affect parameters of an audio-visualization based on the visual patterns
of hallucination seen during altered states of consciousness [65].
12.3.2 Augmented Reality Paintings
While working at Aalborg University in Denmark, I created several new paintings,

which explored similar ideas to my earlier work [66]. For example, Vortex (2017) is
based on the concept of visual patterns of hallucination, providing a funnel image
related to Klüver’s [67] ‘form constants’ (honeycomb, cobweb, funnel, and spiral
forms seen during hallucinations). Alongside this work, I also began working with
the creative coding environment Processing, designing motion graphics sketches
related to altered states, while also drawing influence from demoscene art [68] and
the related VJ mixes [69]. I began experimenting with mixing video live to music
using the VJ software VDMX, combining direct animation with materials created in
Processing and footage made using other techniques such as stop-motion animation.
This resulted in Technoshamanic Visions from the Underworld II [70], a pre-recorded
video loop created by mixing video live to music by the Japanese psychedelic rock
band Hibushibire, which was presented at the Carbon Meets Silicon II exhibition
at Wrexham Glyndŵr University. The exhibition also featured Vortex and several of
my other paintings, which are essentially companion pieces that test similar visual
ideas as those I explore in the videos.
218 J. Weinel
Continuing to explore both painting and audiovisual composition in parallel, I

created a series of works that interpret music through synaesthetic, psychedelic forms
of visual art. Technically these works explore the use of flow techniques, airbrushing,
and digitally cut stencils. They incorporate other aesthetic influences from music via
the artwork of L.A. punk bands such as Excel [71], Suicidal Tendencies [72], and
hip-hop music via the artist Rammellzee [73]. For example Trip at the Brain (2017)
interprets a Suicidal Tendencies song of the same name as a pen sketch, which is
converted into a digitally cut stencil and rendered in airbrush. 31 Seconds (2017)
incorporates airbrushed lettering, referencing a sample from the jungle track Origin
Unknown ‘Valley of Shadows’ [74], and uses acrylic flow techniques and patterns
that reference the designs of rave collective Spiral Tribe. Bug Powder Dust (2017)
references the Bomb the Bass featuring Justin Warfield song ‘Bug Powder Dust’ [75],
which is based on the William S. Burroughs novel The Naked Lunch [76]. For the
latter piece, the painting uses airbrushed skeleton stencils reminiscent of Burroughs’s
‘shotgun paintings’, to provide a form of visual quotation (or ‘sampling’, to use a
music production metaphor) [77].
In many cases these paintings were created alongside the VJ work and vice versa,
and develop similar aesthetic ideas and symbolism across these forms. I began car-
rying out some initial experiments that integrate these practices, by video-mapping
my VJ content on to the paintings, thereby providing visual art with moving ele-
ments. Later, I created a series of three paintings that link the practices of painting
and VJing by incorporating printed stills from my VJ work as collage elements:
Enter Soundcat (2017), Soundcat S-101 (2017), and Soundcat 2000 (2017). These
paintings were later developed through the use of an AR app, which brings the still
images to life as VJ animations when the application is viewed through the mobile
application, thereby reinserting the moving image elements into the paintings [78].
The AR paintings provide symbolic interpretations of sound and music, utilizing XR
to link the physical media of paint with computer-generated motion graphics.
12.3.3 VJ Performances
My exploration of VJ performance began with improvizational jamming, in which

video loops created with direct-animation, stop-motion animation, and computer
graphics techniques were mixed live to various kinds of music including psychedelic
rock and electronic dance music DJ mixes. This allowed me to experiment with
different combinations of sounds and images. I eventually formulated this work into
a live DJ/VJ performance under the alias Soundcat [79], which consisted of a DJ
set utilizing various breakbeat music from the 1990s breakbeat rave era and beyond
(e.g., [80]). In 2018 this was performed at as part of audiovisual concerts for VJ
London at New River Studios, London (Fig. 12.3, [81]), and at a concert held at Tŷ
Pawb arts centre as part of the ACM Audio Mostly in Wrexham [82].
The visual materials for the Soundcat performance are based on my previous
explorations of psychedelia, while also drawing on graphics inspired by 1990s VJ
Fig. 12.3 Soundcat DJ/VJ performance at VJ London, New River Studios, July 2018. Photo credit
Laurie Bender (L’Aubaine)
mixes and demo-scene graphics [83–85]. I incorporate 3D tunnel effects and geomet-
ric animations; 3D scenes reminiscent of the ‘cyberdelic’ science-fiction landscapes
seen on fliers for mega-raves such as Fantazia or Dreamscape; dancing 3D figures;
scrolling patterns referencing acid house culture through smiley faces; detourné-
ments of the London Underground and Intel Inside logos; and other Discordian [86]
or absurdist imagery. Branching out into the area of ‘video music’ (in which video
samples or loops are used to create music, as exemplified by artists such as Addictive
TV, Coldcut or Eclectic Method), one section remixes video trailers from the Planet
of the Apes films to match the samples used in a track by Unkle (‘Ape Shall Never
Kill Ape’ [87]). During this period I also became interested in vaporwave (Tanner
[88]), an Internet music subculture which provides a surrealistic or hyperreal inter-
pretation of 1990s computer graphics and techno-utopian culture, and some sections
incorporate symbolic references to these forms through the use of computer software
user interfaces and related symbols or designs.
Using these materials, I created original music videos for all of the tracks that
I wanted to include in the VJ mix. These videos were mixed live in VDMX using
audio-reactive effects, layering, and synchronized looping techniques, all of which
were manipulated in real time using a MIDI controller (a Korg NanoKontrol). For
some sections, I used an Akai MPC Studio to rhythmically trigger video clips live in
synchronization with the music by improvising with the percussion pads. For a section
based on Equinox ‘Acid Rain V.I.P. (Breakage Final Chapter Mix)’ [89], I used the
tracker music sequencer Renoise to program a MIDI sequence in synchronization
with the drum track, which was then used to trigger closely synchronized 3D graphics
220 J. Weinel
within VDMX. For each song, I created a different VJ mix, which was performed live
in the studio, and recorded using a Blackmagic HyperDeck Shuttle. In some cases,
further video overdubs were carried out in order to provide additional layering of
visuals. This process resulted in a collection of original music videos for each music
track, which could then be used to create the final DJ/VJ mixes.
The final live performances were created using the DJ software Serato Scratch,
the MixEmergency video plugin, an Akai AMX mixer, and the Akai MPC Studio.
This allowed the music videos to be mixed in the same way a DJ mix would usually
be created, where changes to pitch can be made to synchronize the beat and blend
between music tracks. Visually, further composites were created as the tracks are
blended, which could also be further manipulated with effects in MixEmergency
(for example, linking EQ adjustments to color contrast). In addition, I used the per-
cussion pads of the Akai MPC Studio to trigger ‘one shot’ audiovisual materials,
which were layered as composites with the video mix. The resulting DJ/VJ mix
has some limitations in that various aspects of the visuals are pre-recorded, how-
ever, by carrying out intensive work beforehand (both in artistic and computational
terms), the approach allows for a highly varied and efficient end result. This approach
also prioritizes the mixing of audio and the overall structure of the DJ mix as the
focus of effort during live performances, which is an appropriate strategy for solo
performances where the music takes priority, and visuals are complementary.
12.3.4 Cyberdream VR
Cyberdream VR is a recent project that extends many of the approaches discussed

in electronic music, painting and VJing into XR using VR. The project is based
on the concept of a hallucinatory journey through the broken techno-utopias of
cyberspace, providing a surrealistic world of psychedelic rave visuals and vapor-
wave music. Cyberdream VR was created for Samsung GearVR, was adapted for a
VR cardboard version, and has been shown at various events including Cyberdelic
Incubator Melbourne [90], Sci-Fi London festival [91] and MIND: Past, Present +
Future/Cyberdelics/Remote Viewing [92].
The experience provides a short ‘fly-through’ (approximately 5 min long), con-
sisting of a series of scenes. The menu screen is based on the flier for the Fantazia
NYE 1991–1992 rave event, presenting a surrealistic virtual landscape with a large
face suspended over it. Upon entering Cyberdream VR, the viewer flies across a
chequerboard bridge surrounded by statues of strange creatures, entering a fractal
structure based on the Sierpinski triangle (Fig. 12.4). Next, we travel over a vast
infinity pool with broken manikin heads floating in it. Following this, the viewer is
suspended in a large room with airbrushed walls, which were created by digitally
scanning paintings made with a real airbrush. In this room, an animated effect creates
vortex patterns based on visual patterns of hallucination. In the next sequence we
fly through a virtual sky bombarded with pop-up windows; a pastiche of the John
Fig. 12.4 Still image from Cyberdream VR. Image J. Weinel (2019)
Carpenter movie They Live [93], the spam adverts are revealed as signals of capital-
ist control. The next scene depicts a virtual chequerboard island on which Atari ST
cursors (pixel art rendered as 3D graphics) bounce manically or lie derelict among
Grecian statues suggestive of techno-utopianism (also a vaporwave trope, see [94,
95]). After this, the next two rooms consist of cycling waves of brightly-colored
cubes with oscillating color patterns and sizes. These are based on the classic ‘plas-
mas’ of demoscene videos [96], which generate fluid animations using oscillating
patterns—here the technique is translated into 3D, giving an impression of being
inside the pixels of a computer monitor, while also subtly referencing the design of
the Windows ‘95 logo. Following this, the viewer enters another airbrushed room
(again, created using digitized hand-painted art), in which spherical objects move in
Lissajous figures. The final scene consists of a dark, chaotic room with bouncing stro-
boscopic arrows, and the text “the future is lost, crash the system, back to the tribes.”
This message is a comment on the loss of the techno-utopian futures once imagined
by cyberculture [97, 98], while calling for a dissolution and ecstatic recombination
of these digital structures. The comment ‘back to the tribes’ also playfully hints at
the idea of ‘technoshamanism’ (the use of technologies to access shamanic forms of
experience, see [99]) and references free-party rave culture (e.g., Spiral Tribe).
The soundtrack for Cyberdream VR includes short pieces of acid house, hardcore
rave, and vaporwave music. Just as the scenes of the project are essentially artistic
sketches, these pieces of music are audio sketches. Thematically both the visuals and
audio are related to, and reinforce the overall concept of the piece. Rave music sug-
gests the futuristic aspects of hardcore techno [100], while the vaporwave clips use
a plunderphonic approach (music made from existing audio recordings, see [101])
222 J. Weinel
by slowing down imperfect loops of corporate library music intended to enhance

productivity in the workplace. In this regard, the piece sonically mirrors the visuals
through combinations of symbolic elements from psychedelic rave cyberculture and
corporate computer culture. The overall result aims to elicit a broken, hallucinatory
vision of the techno-utopianism of cyberculture, revealing the artificial or hyperreal-
ity of these visions, while also hinting at the ecstatic possibility in the dissolution of
these structures. Described another way, the piece takes Douglas Rushkoff’s [102]
concept of the Internet as a hallucination, and attempts to visualize that hallucination
as a synaesthetic XR experience that allows the viewer to enter into the virtual worlds
suggested by rave culture and vaporwave music.
12.3.5 Discussion
The work I have outlined in this subsection spans over a decade of creative practice
creating work related to the concept of altered states of consciousness. In different
ways, these works represent hallucinations, and synaesthetic experiences of sound
and music through combinations of sound and visual art. One of the distinguishing
features of this work is that it prioritizes the visualization of music by focusing on the
symbols and conceptual meanings that are suggested by music, rather than the phys-
ical properties of acoustic soundwaves through audio-reactivity (though some parts
of the work do also include audio-reactive or closely synchronized elements). In this
regard, the work follows Danneberg’s view that music visualisers based primarily on
audio reactivity may be relatively uninteresting, because they render only simplistic,
readily apparent features of sound. Instead, he argues that composers should “make
connections between deep compositional structure and images” [103]. With my own
work, I interpret this ‘deep structure’ at a conceptual, symbolic level, where the visu-
alization becomes a means to unlock the imaginary spatial environments and visual
associations suggested by the music. While these symbolic visualizations can be
realized with static visual art, animated visuals provide a way for these to be realized
as time-based audiovisual media. XR technologies then provide a way to extend this
idea further still, by providing spatial portals into these synaesthetic virtual worlds.
12.4 Conclusions
The first half of this chapter explored various examples where XR technologies are
being used to visualize music. Through this discussion, we saw how XR provides
new possibilities for constructing immersive visual experiences that complement
music. These may extend the idea of the concert lightshow, or provide complemen-
tary ‘side’ experiences that reinforce the cultural meaning of these events. The latter
half of the chapter then discussed my own practice-led research, creating AR paint-
ings, VJ performances, and a VR experience related to music and altered states of
consciousness. These works broadly seek to elicit synaesthetic experiences of sound

and music through various forms of visualization. Through the exploration of these
works, I have demonstrated some possible approaches for visualizing music using
XR, and I also hope to have shown that XR technologies need not be approached as
novelty gadgets—but rather as means through which to extend fundamental artistic
concepts for visualizing music. The approach that I have emphasized here is one
in which XR visualizations do not lean heavily on audio-reactivity, but rather seek
to access deeper symbolic meanings, in order to manifest the imaginative worlds
suggested by music as synaesthetic immersive 3D spaces. The unique potential of
XR is to go through the portals into music that visual music paintings, films, and
VJing have so tantalizingly provided in the past. Now, it is possible for the listener
to enter into the music as an audiovisual space, and feel as if they are inside the
visual worlds suggested by music, or for the visual forms of music to spill out into
the concert hall or living room. Whether these technologies are used to visualize the
psychedelic music, rave music, and vaporwave discussed here, or other genres, the
potential is to radically transform the way we experience music.
References
1. Coachella: Coachella 2019 [mobile application for iOS and Android] (2019). Information
available online: https://www.coachella.com/mobile/. Accessed 21 April 2019
2. Dellinger, A.J.: Coachella built an AR experience for festival-goers, Engadget, 12 April
2019. Online: https://www.engadget.com/2019/04/12/coachella-cameras-augmented-reality-
sahara-tent. Accessed 21 April 2019
3. Emrich, T.: Augmented Reality Coachella Experience (2019) [YouTube video]. Online:
https://youtu.be/kY1Mr0b_I4Y. Accessed 21 April 2019
4. Melnick, K.: Coachella Debuts Its First Interactive AR Stage, VR Scout, 14 April 2019.
Online: https://vrscout.com/news/coachella-ar-interactive-stage/. Accessed 21 April 2019
5. Roettgers, J.: Coachella Goes AR With Immersive Stage Experience, Variety, 12 April
2019. Online: https://variety.com/2019/digital/news/coachella-ar-experience-1203188058/.
Accessed 21 April 2019
6. Weinel, J.: Inner Sound: Altered States of Consciousness in Electronic Music and Audio-
Visual Media. Oxford University Press, New York (2018)
7. Weinel, J.: Augmented unreality: synesthetic artworks and audio-visual hallucinations. In:
Grimshaw, M., Walther-Hansen, M., Knakkergaard, M. (eds.) The Oxford Handbook of Sound
& Imagination. Oxford University Press, New York (2019)
8. Brougher, K., Mattis, O.: Visual Music: Synaesthesia in Art and Music Since 1900. Thames
& Hudson, London (2005)
9. Weinel, J.: Improvising with Light: Nova Express Psychedelic Lightshow, OUPBlog,
14 December 2018. Online: https://blog.oup.com/2018/12/nova-express-psychedelic-light-
show/. Accessed 21 April 2019
10. Weinel, J.: Synesthetic overdrive. In: Weinel, J. (ed.) Inner Sound: Altered States of Con-
sciousness in Electronic Music and Audio-Visual Media, pp. 115–134. Oxford University
Press, New York (2018)
11. Rietveld, H.C., Allen, C.T., Bender, L., Pilkington, M., Weinel, J.: VJ Panel: Immersive Visual
Music, 15 May 2019. London South Bank University, London (2019)
12. Faulkner, M.: VJ: Audio-Visual Art and Vj Culture. Laurence King Publishing, London (2006)
13. St. John, G.: Global Tribe. Equinox, London (2012)
224 J. Weinel
14. Android Jones and 360art (2015–2016) Samskara [fulldome film]

15. VJSpetto: This is Fulldome VJ—United VJs [YouTube] (2013). Online: https://youtu.be/
dl3b5PBl87o. Accessed 26 May 2019
16. United VJs: VJ University (2019). Online: http://vjuniversity.com.br/. Accessed 26 May 2019
17. Resolution: Resolution Events London [Facebook event page] (2019). Online: https://www.
facebook.com/resolutioneventslondon/. Accessed 26 May 2019
18. 4:3: Sim Hutchins: ‘Druk Pak’|4:3 Music Video [YouTube] (2019). Online: https://youtu.be/
FtckhsaB5RA. Accessed 26 May 2019
19. L’Aubaine: L’Aubaine—Live visuals and videos in immersive environments [Vimeo video]
(2019). Online: https://vimeo.com/326374325. Accessed 26 May 2019
20. Nicomon: Notting Hill Carnival mApp: the official Carnival app (2018). Information available
online: https://www.notting-hill-carnival-mapp.com/. Accessed 21 April 2019
21. Bintaş, B.: Øresund Space Collective—Kybalion: A Cyber-Dimensional Way to Experience
Music. EVA London (Electronic Visualisation and the Arts) 2019, 8–11 July 2019, British
Computer Society, London (2019)
22. Moore, J.: Back in the groove: How vinyl rose from its sickbed to cap-
ture the eyes and ears of millennials, The Independent, 21 September 2017
(2017). Online: https://www.independent.co.uk/news/long_reads/vinyl-demand-lps-record-
store-day-a7952911.html. Accessed 21 April 2019
23. Leary, T.: Chaos & Cyber Culture. Ronin, Berkeley (1994)
24. The Cyberdelic Society: The Cyberdelics Incubator (2019). Online: https://www.
cyberdelicsociety.com/cyberdelics-incubator. Accessed 21 April 2019
25. Valentish, J.: Acid test: how psychedelic virtual reality can help end society’s mass bad trip,
The Guardian, 26 March 2019 (2019). Online: https://www.theguardian.com/technology/
2019/mar/26/acid-test-how-psychedelic-virtual-reality-can-end-societys-mass-bad-trip.
26. Beaumont-Thomas, B.: Phones at gigs can be annoying—but they must never be banned, The
Guardian, 3 December 2018 (2018). Online: https://www.theguardian.com/music/2018/dec/
03/phones-at-gigs-annoying-research-must-never-be-banned. Accessed 21 April 2019
27. Jenkins, S.: Should phones be banned at gigs? BBC News, 28 January 2018 (2018). Online:
https://www.bbc.co.uk/news/entertainment-arts-42851434. Accessed 21 April 2019
28. Thorpe, V.: Kendrick Lamar joins stars trying to keep concerts special, The Guardian, 10
February 2018 (2018). Online: https://www.theguardian.com/music/2018/feb/10/kendrick-
lamar-bans-phones-photographers-from-concerts. Accessed 21 April 2019
29. Birkbeck: New Exhibition Explores Leonardo da Vinci’s Impossible Machines (2019). Online:
http://www.bbk.ac.uk/news/new-exhibition-explores-leonardo-da-vinci2019s-impossible-
machines. Accessed 27 May 2019
30. Odom, J.: Synesthesia Adds Trippy Sound Visualization to the HoloLens, Next Real-
ity, 14 February 2017 (2017). Online: https://hololens.reality.news/news/have-you-seen-this-
synesthesia-adds-trippy-sound-visualization-hololens-0176215/. Accessed 21 April 2019
31. Heaney, D.: HoloLens 2’s Field of View Revealed, UploadVR.com, 25 February 2019 (2019).
Online https://uploadvr.com/hololens-2-field-of-view/. Accessed 21 April 2019
32. Holotronica: Holo-Gauze—Holotronica (2019). Online: https://stuartwarrenhill.com/holo-
gauze/. Accessed 27 May 2019
33. Carlsen, C.E.: Silicum 04: Audiovisual Performance (2018). Online: http://cec.dk/works/
silicium-04/. Accessed 26 May 2019
34. Psych-Fi: PsychFi | Turn On, Boot Up, Trip Out—#HackThePlanet (2019). Online: https://
www.psychfi.com/. Accessed 27 May 2019
35. Liverpool Psych Fest: PZYK 2020 (2016). Online: http://www.liverpoolpsychfest.com/2016/
08/pzyk-vr/. Accessed 26 May 2019
36. Dinehart, S.E.: Dionysia: A Cyberdelic Experience VR Demo Video [YouTube video] (2017).
Online: https://youtu.be/94ehDHthw7k. Accessed 21 April 2019
37. L’Aubaine: 360 Life #1 [Vimeo video] (2018). Online: https://vimeo.com/313546173.
38. Fathomable: Gnosis [virtual reality experience] (2018). Available on Steam: https://
www.oculus.com/experiences/rift/1242416849112952/. Information available online: http://
gnosisthegame.com/. Accessed 21 April 2019
39. VJ London: AV Depot #2 Interactive Labyrinth (2016). Information available online: http://
vjlondon.com/home/events/event/av-depot-2. Accessed 21 April 2019
40. Fantasynth: Fantasynth [virtual reality experience, multiple platforms] (2017). Information
available online: http://fantasynth.com/. Accessed 21 April 2019
41. WaveVR: The Wave VR (2018). Online: http://thewavevr.com/. Accessed 21 April 2019
42. Deahl, D.: Inside Imogen Heap’s Cutting-Edge VR Concert. The Verge, 23 August
2018 (2018). Online: https://www.theverge.com/2018/8/23/17769166/imogen-heap-concert-
oculus-vr-headset-thewavevr. Accessed 26 May 2019
43. Mbryonic: Amplify VR (2018). Online: https://mbryonic.com/portfolio/amplify-vr/
(Accessed 21 April 2019)
44. Productions: Fabulous wonder.land VR (2019). Online: https://59productions.co.uk/project/
fabulous-wonder-land-vr/. Accessed 26 May 2019
45. Hutson, S.R.: Technoshamanism: spiritual healing in the rave subculture. Popul. Music Soc.
23(1), 53–77 (1999)
46. Rietveld, H.C.: Journey to the light? Immersion, spectacle and mediation. In: Attias, B.A.,
Gavanas, A., Rietveld, H.C. (eds.) DJ Culture in the Mix: Power, Technology, and Social
Change in Electronic Dance Music. Bloomsbury Academic, New York (2013)
47. Shortlidge, C.: Is it Time for a Total Ban on Phones on the Dancefloor? DJ Mag, 17 April 2019
(2019). Online: https://djmag.com/content/it-time-total-ban-phones-dancefloor. Accessed 26
May 2019
48. Henkel, L.A.: Point-and-shoot memories: the influence of taking photos on memory for a
museum tour. Psychol. Sci. 25(2), 396–402 (2014)
49. Rietveld, H.C., Weinel, J.: Extended Dancefloor: Visualing Dancefloor Immersion in XR,
Play/Pause Symposium—Immersion|Dissonance, University of Birmingham, 22 May 2019
50. St. John, G.: Technomad: Global Raving Countercultures. Equinox, London (2009)
51. Weinel, J.: Altered states of consciousness as an adaptive principle for composing electroa-
coustic music [Ph.D. thesis]. Online: http://www.jonweinel.com/media/ASC_Commentary_
38.pdf. Accessed 26 May 2019
52. Weinel, J.: Entoptic Phenomena in Audio [12” vinyl]. Hardcore Jewellery (2014)
53. Weinel, J.: Tiny Jungle [visual music film] (2011). Online: https://youtu.be/i3kK9PG2XkU.
Accessed 26 May 2019
54. Weinel, J.: Tiny Jungle: psychedelic techniques in audio-visual composition. In: Proceedings
of the International Computer Music Conference 2011, University of Huddersfield (2011).
Online: http://hdl.handle.net/2027/spo.bbp2372.2011.066. Accessed 26 May 2019
55. Weinel, J.: Entoptic phenomena in audio: categories of psychedelic electroacoustic composi-
tion. Contemp. Music Rev. 25(2), 202–223 (2016)
56. Weinel: Mezcal Animations [visual music film] (2013). Online: https://youtu.be/
E6Rd201F8QA. Accessed 26 May 2019
57. Weinel: Cenote Zaci [visual music film] (2014). Online: https://youtu.be/x4UGq1LlyYU.
58. Weinel: Cenote Sagrado [visual music film] (2014). Online: https://youtu.be/q8XrEagszt8.
59. ICMC-SMC: Music Programme (2014). Online: http://www.icmc14-smc14.net/program/
conference-program/music_program.html. Accessed 27 May 2019
60. Seeing Sound: 2013 Movies (2019). Online: https://www.seeingsound.co.uk/seeing-sound-
2013/2013-movies/. Accessed 27 May 2019
61. Seeing Sound: 2016 Screenings (2019). Online: https://www.seeingsound.co.uk/2016-
screenings/. Accessed 27 May 2019
62. Weinel, J.: Technoshamanic Visions from the Underworld [artist statement]. ITA(15) Carbon
Meets Silicon Exhibition Catalogue, Oriel Sycharth Gallery, 8–11 September 2015. Wrex-
ham Glyndŵr University, North Wales, pp. 36–37 (2015). Online: http://glyndwrpix.co.uk/
carbonaugust2/mobile/index.html. Accessed 26 May 2019
226 J. Weinel
63. Weinel, J.: Quake delirium: remixing psychedelic video games. Sonic Ideas/Ideas Sónicas
3(2), 22–29 (2011)
64. Weinel, J., Cunningham, S., Roberts, N., Griffiths, D., Roberts, S.: Quake delirium EEG: a
pilot study regarding biofeedback-driven visual effects in a computer game. In: IEEE Pro-
ceedings of the Sixth International Conference on Internet Technologies and Applications
2015, Glyndŵr University, North Wales (2015). https://doi.org/10.1109/itecha.2015.7317420
65. Weinel, J., Cunningham, S., Roberts, N., Roberts, S., Griffiths, D.: EEG as a controller for
psychedelic visual music in an immersive dome environment. Sonic Ideas/Ideas Sónicas 7(14),
85–91 (2015)
66. Weinel, J.: Finding ‘the weird’ in psychedelic art, OUPblog, 18 January 2018 (2018). Online:
https://blog.oup.com/2018/01/finding-weird-psychedelic-art/. Accessed 26 May 2019
67. Klüver, H.: Mescal and Mechanisms of Hallucinations. University of Chicago Press, Chicago
(1971)
68. Polgár, T.: Freax: A Brief History of the Computer Demoscene. CSW-Verlag, Winnenden
(2005)
69. Weinel, J.: How Atari and Amiga computers shaped the design of rave culture, OUPBlog,
5 April 2018 (2018). Online: https://blog.oup.com/2018/04/atari-amiga-computers-design-
rave-culture/. Accessed 21 April 2019
70. Weinel, J.: Technoshamanic visions from the Underworld II. In: IEEE Proceedings of the
Seventh International Conference on Internet Technologies and Applications 2017, Wrexham
Glyndŵr University, North Wales, pp. 345–346 (2017)
71. Ross, S., Clements, D.: Excel 1979–1992. Paper Work NYC, New York (2018)
72. Clayton, R.: Welcome to Venice RXCX. Gingo Press/KYI, Los Angeles (2018)
73. Laz Inc.: Rammellzee: A Roll of the Dice [exhibition], 2nd October–8th December 2018
(2018). https://www.lazinc.com/exhibitions/1394/. Accessed 12 March 2019
74. Origin Unknown: Valley of Shadows [12” vinyl]. Ram Records (1993)
75. Bomb the Bass featuring Justin Warfield. Bug Powder Dust [12” vinyl]. 4th & Broadway
(1994)
76. Burroughs, W.S. (1959) The Naked Lunch. Reprint, Fourth Estate, London 2010
77. Riflermaker: Dead Aim: The Unseen Art of William S. Burroughs. Riflemaker, London (2005)
78. Soundcat VJ: Augmented Reality Paintings by Soundcat VJ [YouTube] (2019). Online: https://
youtu.be/TzIb0-mt10w. Accessed 26 May 2019
79. Soundcat VJ: Soundcat [Facebook page] (2019). https://www.facebook.com/SoundcatArt/.
80. Soundcat VJ: Soundcat—Breakbeat DJ/VJ Set [YouTube video] (2018). Online: https://youtu.
be/xuJP4asCr44. Accessed 26 May 2019
81. VJ London: July VJ Meet-Up and Showcase, 12 July 2018, New River Studios, London (2018).
Information available online: http://vjlondon.com/home/events/event/july-2018-vj-meet-up-
showcase/. Accessed 26 May 2019
82. Audio Mostly: Soundcat—Breakbeat DJ/VJ Set Exploring Psychedelia, 12 September 2018,
ACM Audio Mostly 2018: A Journey Into Sound [evening concert], Tŷ Pawb, Wrexham,
North Wales (2018). Information available online: https://audiomostly.com/2018/conference-
programme/. Accessed 27 May 2019
83. Dr. Devious and the Wiseman: Dance in Cyberspace [VHS]. Parade Video (1992)
84. Hex: Global Chaos [VHS]. Ninja Tune (1993)
85. Frost, M., Irwin, C., Jamieson, D., Simpson, P.: Future Shock. Prism Leisure Video (1993)
86. Koke, J., Pearcy, D., Hartsock, N.: Principia Discordia. Steve Jackson Games, Austin (1994)
87. Unkle: Ape Shall Never Kill Ape [12” vinyl]. Toy’s Factory (1998)
88. Tanner, G.: Babbling Corpse: Vaporwave and the Commodification of Ghosts. Zero Books,
Alresford (2016)
89. Equinox: Acid Rain V.I.P. Planet Mu (2006)
90. The Cyberdelic Society: Cyberdelics Incubator Melbourne [Facebook event page] (2019).
Online: https://secure.latest.facebook.com/events/293967014606877/. Accessed 27 May
2019
91. Sci-Fi London: VR Experiences [festival programme] (2019). Online: https://sci-fi-london.

com/2019/04/06/vr-experiences/. Accessed 26 May 2019
92. The Explorers Club/The Cyberdelic Society: MIND: Past, Present +
Future/Cyberdelics/Remote Viewing [Facebook event page] (2019). Online: https://sci-
fi-london.com/2019/04/06/vr-experiences/. Accessed 26 May 2019
93. Carpenter, J. (director): They Live. Universal Pictures (1988)
94. Chandler, S.: Escaping Reality: The Iconography of Vaporwave (2016). https://daily.
bandcamp.com/2016/09/16/vaporwave-iconography-column/. Accessed 12 March 2019
95. Chandler, S. (2016) Genre As Method: The Vaporwave Family Tree, From Eccojams to Hard-
vapour. https://daily.bandcamp.com/2016/11/21/vaporwave -genres-list/ (retrieved 12 March
2019)
96. Vandevenne, L.: Lode’s Computer Graphics Tutorial—Plasma (2004). https://lodev.org/
cgtutor/plasma.html. Accessed 12 March 2019
97. Turner, F.: From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network,
and the Rise of Digital Utopianism. University of Chicago Press, Chicago (2006)
98. Bennetsen, H.: From Counterculture to Cyberculture: The Legacy of the Whole Earth Catalog
[Youtube video] (2011). https://youtu.be/B5kQYWLtW3Y. Accessed 12 March 2019
99. Weinel, J.: Inner Sound: Altered States of Consciousness in Electronic Music and Audio-
Visual Media, pp. 90–93. Oxford University Press, New York (2018)
100. Fisher, M.: Ghosts of My Life: Writings on Depression, Hauntology and Lost Futures. Zero
Books, Alresford (2014)
101. Oswald, J.: Bettered by the borrower: the ethics of musical debt. In: Cox, C., Warner, D. (eds.)
Audio Culture: Readings in Modern Music. Continuum, New York (2004)
102. Rushkoff, D.: The Internet Is Acid, and America Is Having a Bad Trip, Medium, 4 September
2018 (2018). Online: https://medium.com/team-human/the-internet-is-acid-and-america-is-
having-a-bad-trip-fa0864c3ef1e. Accessed 26 May 2019
103. Dannenberg, R.B.: Interactive visual music: a personal perspective. Comput. Music J. 29(4),
28 (2005)
Chapter 13
Augmenting Virtual Spaces: Affective
Feedback in Computer Games
Stuart Cunningham, John Henry, and Jonathan Weinel
Abstract Computer games can be considered a form of art insomuch as they are
critiqued, revered and collected for their aesthetics in addition to their ludic qualities.
Perhaps most significantly, computer games incite a plethora of emotional responses
in their players as a deliberate and defining mechanism. However, unlike other forms
of traditional media and art, another key feature of games is their intrinsic interactiv-
ity, reliance upon technology and non-linearity. These traits make them particularly
noteworthy if one wishes to consider how art forms might respond and adapt to
their audience’s emotions. The field of affective computing has been developing for
several decades and many of its applications have been in the analysis and mod-
elling of emotional responses to forms of media, such as music and film. In gaming,
recent developments have led to an increasing number of consumer-grade biofeed-
back devices which are available on the market, some of which are explicitly sold as
‘gaming controllers’, giving rise to greater opportunity for affective feedback to be
incorporated. In this chapter, a review is provided of the affective gaming field. Specif-
ically, it is proposed that these developments give rise to interesting opportunities
whereby virtual environments can be augmented with player affective and contextual
information. An overview is provided of affective computing fundamentals and their
manifestation in developments relating specifically to games. The chapter consid-
ers the impact this biometric information has upon games players, in terms of their
experience of the game and the social connections between competitors. A number
of associated practical and technological challenges are highlighted along with areas
for future research and development activities. It is hoped that by exploring these
developments in gaming that the longer established forms of art and media might be
inspired to further embrace the possibilities offered by utilising affective feedback.
S. Cunningham (B) · J. Henry

Centre for Advanced Computational Science (CfACS), Manchester Metropolitan University,
Manchester, UK
e-mail: s.cunningham@mmu.ac.uk
J. Henry
e-mail: john.henry@mmu.ac.uk
J. Weinel
London South Bank University, London, UK
e-mail: weinelj@lsbu.ac.uk

230 S. Cunningham
Keywords Affective computing · Affective feedback · Computer and video

games · Emotion · Game design · Interactive media · Social gaming · User
experience
13.1 Introduction
The ability to engage with an audience on an emotional level is one of the defining
traits of many forms of art and media. The Cambridge English Dictionary defines
art as “the making of objects, images, music, etc. that are beautiful or that express
feelings” [1]. This supports the notion that the author of such works is able to express
emotion, be it their own or that of others, through a variety of mechanisms. One might
also take the view that the audience members, whilst being able to perceive the
emotion of the author, will also have their own response to the stimulus, which may
be consistent with the author’s emotional intent or not. Historically, this emotional
communication has been a largely one-directional affair: the author creates a piece
of work with its emotional intent and disseminates it in some way, the audience
experiences the art and has their emotional response to it, and this sequence may
repeat itself. However, there is traditionally no recourse for the emotional response
of the audience to feedback to the author beyond the writing of reviews, critiques,
blogs and other forms of response.
We propose that computer games can be considered a form of contemporary art,
much in the same way as video, film, animation and interactive audio-visual instal-
lations are now viewed. Certainly, games combine the requisite forms of media cited
by the dictionary definition: images and music that are visually pleasing. Games
are also recognised for their ability to engage an audience and provoke emotional
responses in their audience. Gee [2] advocates that the interpretation of their aesthet-
ics and the story aspects of games qualify them as art. Further, we note that games
are critiqued, reviewed, discussed and are points of cultural reference. Smuts [3]
shares this perspective, although notably recognises that not every computer game
may qualify for the label as being recognised as art. One characteristic that makes
games particularly compelling as an art form is the rate of change and development
with which the media evolves and its facility to integrate easily with other techno-
logical developments, both hardware and software based. Therefore, as a form of
interactive art, games are uniquely poised to trailblaze and be the proving ground of
new ideas and initiative, which may later be adopted by the wider artistic community
in all forms of practice. Melissinos [4] states that “…games are also the only form
of media that allows for personalizing the artistic experience while still retaining
the authority of the artist,” making the distinction that narrative and content of the
work can be tailored to the individual(s) playing. It is on these bases that we consider
games as being a platform to develop work that affords interactivity, in terms of
audience affect and immersion within the game. Further, we propose that this can
lead to develop and enhance a sense of community and connection between human
players in a game environment, who are remote to another.
13 Augmenting Virtual Spaces: Affective Feedback in Computer Games 231
Many years have now passed since Rosalind Picard produced the seminal work
that helped to define the field of Affective Computing and its associated challenges,
opportunities and potential risks [5]. Key to the domain are the three types of affective
technology Picard outlines where computers are able to: recognise; represent and
have emotions. Whilst the former of these is of greatest concern in this chapter, the
second element of representation of emotion should not be neglected. The field itself
has developed, at least on the front of academic and some industrial research, over this
time period. However, its mainstream applications in the hands of the audience are
still relatively few and far between with exceptions such as the formation of spin-out
company Affectiva1 from Picard’s MIT Media Lab. Perhaps this is unsurprising, as we
have seen the widespread adoption of other paradigm-shifting art and entertainment
technologies, such as virtual reality (VR), take a long time to reach the mainstream
and gain acceptance from publishers and audiences alike.
A common underpinning element of affective computing is the reliance upon
various sensors and mechanisms for the provision of biofeedback; the mechanisms
by which aspects of bodily function and response can be identified and manipulated
[6]. Whilst not an essential aspect of all affective computing systems, the use of
such technologies is an intuitive method that provides indicators of affective state
and responses to stimuli, often with the ability to operate in real-time. There are
distinct advantages of using such sensors. For example, subjects, in our case games
players, do not need to actively focus on providing feedback as they would with self-
report mechanisms, rather data can be collected passively and without distracting
attention from the medium itself. Additionally, sensors are able to provide objective
measurements, whilst asking for self-reports or other information may introduce
bias on the part of the subject. Whilst medical grade equipment is desirable in high-
resolution research scenarios, at a consumer level there are a range of technologies,
some of which can provide viable affective data [7–10] and be easily integrated or
added to gaming devices and apparatus.
We propose that affective gaming is a natural extension of the affective computing
field. Such an extension offers game developers, the opportunity to detect, respond
to and provoke emotions in players. Computer games, like many forms of art, media
and entertainment, offer their players the opportunity to be emotionally stimulated
and involved [11, 12]. One of the key features that separates games from other
forms of media-based entertainment is its broad non-linearity and its deep and varied
levels of interactivity. As such, affective gaming affords the possibility to create
gaming experiences that can not only provoke and influence their players’ emotional
responses, but that player emotions can also be used as an input to the game itself,
allowing it to respond to the player and provide a customised experience. This process
is generally referred to as affective feedback [13]. One clear example would be a game
that increases its level of difficulty to increase tension in the player, if it were to detect
that the current level of difficulty is not sparking the intended response.
The ways in which end users interact with technology is maturing and tradi-
tional mechanisms of button pressing, joystick moving and mouse clicking have been
1 https://www.affectiva.com/.
232 S. Cunningham
ubiquitous and effective. However, these technologies have become a legacy, partly
down to reluctance in the market to engage with alternate methods of input, such
as motion detection, speech and other modes of interaction. We suggest that, in the
coming years, the time will be right to examine both active and passive methods
of interaction with games and other digital media. There are multiple possibilities
to create experiences that are powerful, intelligent, customisable, adaptive, socially
enabling and responsive to their users. These developments represent exciting oppor-
tunities to create innovative entertainment and artistic works that enhance and evolve
the user experience.
13.2 Affect, Immersion and Engagement
In Inner Sound: Altered States of Consciousness in Electronic Music and Audio-

Visual Media [14], Weinel proposes a conceptual model that considers the ‘affective
properties’ of sound and audio-visual media as a central feature. Here affective
properties are considered as related to the properties of valence and arousal described
in Russell’s [15] circumplex model of affect. In games, sound and music may help to
elicit an emotional response from the player. Following Gabriellson and Lindstrom’s
[16] meta-study of music and emotion, typical features such as percussive sections
of music may be associated with high arousal, while sparser rhythmic sections may
relate to low arousal. Major modes may relate to positive valence, while minor modes
may suggest negative valence. While these are general correspondences that may not
apply in all cases, such typical associations can still be used by games designers to
reinforce the affective experience of games. We commonly see this through action
sequences that are scored with up-tempo music, or failure or melancholy sections
in game narratives that are reinforced with corresponding music. For Weinel, such
features help to provide an affective experience that is framed by representational
properties, and the overall experience provides a heightened state of multimodal
awareness that may encompass forms of experiential knowledge.
Of course, in the gaming context, affective states do not only relate to music.
Cajella’s [17] work on ‘player involvement’ breaks down immersion in video games
into the categories of kinaesthetic, spatial, shared, narrative, affective and ludic forms
of player involvement. While this model takes the affective experience as a distinct
component, it is surely a form of involvement that must emerge from all of the
others. For instance, kinaesthetic involvement describes engagement through control
of a game character, while spatial involvement relates to the virtual environments
provided by games. When a player controls an avatar that leaps through the levels of
a game, they are engaging kinaesthetically and spatially, but the feeling of excitement
that may develop from this also leads to affective involvement. Similarly, for players
who enjoy the shared involvement of multiplayer combat games, an added sense of
arousal comes from knowing that their own success or failure has a bearing on the
emotions of their human competitors. For games that develop complex narratives, as
with films, designers may aim to tell a story that has an emotional resonance with the
player. Lastly, ludic involvement also generates arousal, as players attempt to evade
losing lives or reaching a ‘game over’ state, which inevitably is disappointing. Hence,
we can see that in fact, all types of involvement feed into the affective involvement
of the player. Sound and music are by no means the only way through which affect
can be generated in interactive games, but these are often used to help reinforce and
heighten the affective experience of the player.
13.3 Social Presence and Connection
Weinel’s [14] work on altered states of consciousness also draws some further con-
nections between the affective properties of media and social or shared experiences.
This emerges through Rouget’s [18] seminal work on music and trance in various
ritual contexts such as those found in shamanic cultures. Rouget draws a distinction
between high-energy states such as trance, which could be understood as extreme
states of high arousal, and meditative states of contemplation (referred to in the orig-
inal text as ‘ecstasy’), which conversely may be thought of as low arousal in form.
Rouget explores a variety of ritual contexts in which these states may be used in
cultures across the globe, and comments on how music can be involved in structur-
ing and producing these states. Yet interesting, high-energy states such as trance are
also distinguished as social, involving loud noises, movement and other techniques
of sensory overload such as the spraying of liquids; while meditative states are char-
acterised as occurring in darkness, solitude and related to a reduction, narrowing or
focusing of the senses.
There has been little work done on considering video games as altered states of
consciousness, but clearly this is a popular form of media that perhaps is valued by
audiences for its capability to induce a sense of (secular) trance. Is it not possible that
as teenagers play popular games such as Fornite [19], the heightened state of excite-
ment they experience, which draws them out of the everyday and into the virtual,
is in fact something we might call a state of trance? Various interactive aspects of
the media conspire to produce a highly aroused, social experience, in which they are
immersed into imaginative virtual worlds of play. In these terms, it seems that there
is no shortage of high-arousal action games that aim to produce something we might
call a sense of trance. But what then, of more tranquil or meditative states? In fact,
there are also various games that specifically aim to induce states of meditation, such
as Guided Meditation VR [20]. This title is essentially the VR equivalent of a guided
meditation experience, which provides tranquil virtual locations with an instruc-
tive audio narrative. More broadly, other single-player games may seek to provide
tranquil, solitary experiences. For instance, ABZÛ [21] is a mysterious underwater
exploration game that de-emphasises ludic involvement in favour of kinaesthetic,
spatial and narrative involvement, as the player navigates mostly peaceful ocean
environments with a relaxing soundtrack. Certainly, games such as this could be
understood as offering experiences that are characterised as meditative in form.
234 S. Cunningham
13.4 Augmenting Virtual Environments
The field of Augmented Reality (AR) has developed significantly in recent years.
From the perspective of a user or consumer, it seeks to combine information from the
real world with information not visible in that environment using the standard array
of human senses [22]. Potentially, this additional information may be drawn from
imperceptible information that exists in that space or by including extra information
from another place: real or virtual. Thus far, the majority of AR systems have sought
to overlay a user’s everyday environment with additional information. For example,
an AR navigation application might overlay information about a car that is being
looked at, such as year of manufacture, engine size and so on. Alternately, a real
worldview might be overlaid with the user’s schedule for the day. There have been
several commercial attempts at developing AR systems of this kind, perhaps most
notably the Google Glass2 hardware project, as well as an array of smartphone apps
developed commercially or as research projects.
However, less common is the ability to apply the same principles to augment a
Virtual Reality (VR) space with information from the real world. For example, a
virtual tour of a building to be constructed might be presented by replicating the
current weather being experienced in the real world by the user. Notable consumer
devices that explore these possibilities are Sony’s PlayStation Camera [23] and
Microsoft’s Kinect for Xbox One [24]. These platforms have seen video games being
developed that insert, or respond to, a live camera feed image of the player in a game
environment, giving them the ability to interact by gesture with the game, hence
augmenting the VR environment. This builds upon the more established practice
of integrating audio, specifically player speech utterances, into game environments,
made popular in online multiplayer gaming. It is this latter variation of augmentation
that exemplifies some of our concerns in this chapter.
Our motivation particularly stems from the recognition that playing games in
virtual or online environments has many differences from their real-world equiva-
lents. For instance, there is a reduced ability to interact with fellow team members
and opponents. For example, there is a distinct absence of interaction through facial
expressions, gesture, posture and general body language. Even with the presence of
speech utterances and emoticons, it is challenging to determine accurate emotional
responses and exert forms of emotional intelligence, to determine if other players
may be angry, sad, scared, exhilarated and so on. As recognised by Picard [5] in
relation to other forms of electronic communication, such as email, this detracts
from the entertainment and immersion within the game and also from the ability for
players to successfully interact with one another. To address this, we propose that
these differences lead to opportunities to improve players’ sense of immersion within
a virtual environment and their social experience with other players in that environ-
ment. However, the drive for increased immersion and realism in games and virtual
worlds leads to causes for concern regarding the health issues associated [25] and
2 https://developers.google.com/glass/.
thus any experimental approach must be mindful of the ethical issues surrounding
such work.
The use of standard biofeedback is fairly commonplace, whereby real-time repre-
sentations of the underlying signals in question are presented to a person connected
to some form of sensor. Perhaps the most familiar manifestation of this would be
an electrocardiogram (ECG) machine in a clinical environment, where the person
connected to the device would be able to hear and see their heart rate. Similar body
sensors allow games to alter their virtual environments based on the user that is
playing them. Elaborating, emotion is quantifiable through sensor measurements. A
virtual game environment becomes dynamic, if it reacts to emotion. A strong case
study for such an application is Dynamic Difficulty Adjustment (DDA) in games.
A game such as those from Electronic Arts’ FIFA [26] series can generate an
enormous amount of frustration for players; with a number of players reporting they
quit playing the game because they felt it unfairly caused them to lose. In future
consoles, controllers could embed heart rate monitoring sensors. The data produced
could in turn generate a crude measure of emotion, calculated through computer
algorithms that translate raw sensor data to meaningful information. Therefore, the
level of emotional distress becomes the value for adjusting the difficulty in the game.
This adheres to the theory of flow by Csikszentmihalyi [27]. The channel of flow
exists between challenge and achievement. A game that dynamically alters its dif-
ficulty based on the player’s emotion, can help keep gameplay within the channel.
The Internet of Things (IoT) and serious games provide more detail and background
on this topic.
The ability for the affective information about players to be shared with third-
parties [28], such as in online multiplayer gaming environments, presents interesting
opportunities. Consider the illustrative example of a mock-up interface provided in
Fig. 13.1, where the virtual game environment has been augmented with data about
the human players controlling the in-game avatars. In this simplistic example, we
observe the presentation of raw biosensor data alongside a higher-level summary of
affective state. In this exemplar, raw heart-rate data in beats-per-minute is presented
numerically in a heart icon on the left of the pop-out window. In the middle is an emoji
icon, which might be derived by a range of different sensors or other mechanisms, but
with the intention of providing a summary representation of the player’s emotional
state. Finally, on the right side of the pop-out a lightning bolt, which fills up with
an orange colour, provides an indication of the galvanic skin response (GSR) data
values of the player. Although visual paradigms and designs have been adopted for
the purposes of communicating these concepts clearly in a written form, there is of
course no reason why this information could not be communicated in alternate mode,
such as sonification or tactile feedback.
236 S. Cunningham
Fig. 13.1 Multiplayer game mock-up with affective data visualisations. Image Copyright © Stuart
Cunningham 2019
13.5 The Internet of Things and Games
Research is helping bridge the gap between sensors and games. In 2015, Favorskaya
et al. [29] presented their book into smart serious games. The idea behind the term is
games that incorporate the benefits of the Internet of Things (IoT) with the applica-
tions of serious games. Existing research in the field investigates measuring student
engagement through the IoT and serious games [30]. Developing data algorithms
forms a key component for the integration of IoT with games. It is possible to exclude
them in games where sensors are limited to direct input. An example of such games
is Kinect Sports [31], where the Kinect acts as an input device for controlling a player
in the virtual environment. However, such games constrict their scope of application,
as the data could be interpolated into something grander and more meaningful.
However, this idea can become more abstract, and potentially, more beneficial
if we consider the combination of IoT with games technology. Henry et al. [32]
produced a framework that illustrates the backbone for merging serious games with
IoT, however as the framework did not focus on a case study, it is also valid for the
broader spectrum of game technologies. The potential for this line of research is near
unlimited.
Figure 13.2 illustrates the untapped potential for this academic domain in
the present day. This validates the theories and ideologies presented in this
chapter, and produces a stepping-stone to realising them. Research is continuing
Fig. 13.2 Potential applications for Henry et al. framework on serious games and the Internet of
Things
into understanding the relationship between all aspects of game applications and
harnessing their potential.
Research is emerging into combining IoT with Gamification applications. Crow-
ley et al. [33] proposed a system where users can report issues within their local
environment through gamification. Users would be awarded for contributing and
could send the information through their smartphones. Research by Islam et al. [34]
into advancements of IoT healthcare applications suggested wearables to bridge
patient engagement and public health information. In industry, smartwatches have
turned this concept into reality. Companies such as Apple utilise sensors to monitor
the activity of its user and promote healthier habits by awarding badges and forming
competitions with friends that own the same product. This is an example of gam-
ification and IoT in applications. The method for engaging users with their health
and maintaining their commitment is gamification, utilising game elements to an
application outside the definition of a game.
It is evident that games are much more than just entertainment software, and
this remains true for game technology. Research utilises game technology to solve
specific problems in domains such as business modelling and smart cities. Maines
et al. [35] introduced a third dimension for Business Process Model and Notation,
in an attempt to overlay the security requirements of a system in its design stage,
therefore increasing its security at implementation. This solution was developed
utilising the Unity 3D game engine. If Maine’s research were to introduce IoT, it
could assess existing physical and virtual systems and highlight vulnerabilities based
on current standards. Tully et al. [36], presented a methodology for rendering large
map data for crisis management. This application ties into the principles of smart
238 S. Cunningham
cities and provides perspective to the future tools that emergency services may use
to run more efficiently during a crisis.
When reflecting on the progress research is making in this domain, it becomes
evident that sensors and the IoT generate more than novel applications or theoretical
frameworks, but a new perspective. This perspective allows all existing research into
serious and traditional games to be re-examined with the inclusion of the IoT. The
following section describes the technical challenges developers of ‘smart serious
games’ solutions face today, and how they may be tackled in future.
Regarding the combination of IoT and serious games, the biggest technical chal-
lenge is heterogeneity. As the number of manufacturers producing Internet intercon-
nected solutions increases so does the competition, driving prices down and encour-
aging exploration. However, this competition generates segregation in the industry
and causes development to be cumbersome. There are devices that only work within
a manufacturer’s ecosystem, such as Samsung’s ecosystem of Smart appliances,
devices that require bespoke bridges to connect with the Internet, and devices with
bespoke online portals.
The only method for tackling the increasing segregation in the Internet of Things
is standardisation. Wireless charging is an example for the effective use of standardi-
sation. Without the Qi standard [37], devices would require their own type of wireless
charger. In such a scenario, large technology companies such as Apple and Samsung
would develop their own versions of wireless charging that would only be compatible
with their ecosystems. Therefore, wireless charging in cars or in embedded surfaces
would become challenging.
13.6 Measuring Affective Response
Sykes and Brown [28] conducted a small-scale investigation to determine how the
affective response of players would be influenced by increasing game difficulty in
a remake of the classic arcade game Space Invaders [38]. Their study employed a
sample of ten participants who each played the game at three randomised levels of
difficulty. During each period of gameplay, the pressure with which players pressed
the analogue button to move their game character was recorded. The authors argue
that the pressure of button presses may give an indication of the player’s level of
emotional arousal, drawing upon earlier findings in the field of sentics [39]. The
results obtained gave an indication that at the hardest difficulty level, the pressure
applied was significantly different than at the easy and medium levels. As an early
work these results are compelling, although a larger sample would be desirable along
with more rigorous statistical analysis of the data. There is doubt about the ability
of touch universal method to communicate human emotion [40], although for most
games players this may be an intuitive and appropriate scenario and might be one
that could be learned for each individual over a period of time. Nevertheless, the
method employed is interesting, especially since the authors justify the use of button
pressure measurement as being something that can be employed in game controllers
with analogue buttons, meaning that more elaborate equipment and setups can be
avoided. This is hugely beneficial given that the control devices, such as gamepads,
touchscreens, buttons and joysticks, are the main physical interaction mechanisms
that the player has with a game.
Following the convention of making use of game controllers for affective feed-
back, Bacchini et al. [41] attempted to integrate several biofeedback mechanisms
into a common gamepad design. As such, they augmented an existing console style
control pad with sensors designed to detect GSR, skin temperature, pressure/force
and heart rate statistics via photo plethysmography (PPG). Thus, the device itself is
one that is already well accepted in the gaming community and presents minimal
disruption to the ergonomics of the typical gamepad. Their work was undertaken with
the specific design intention that such a device would lend itself to affective gaming
applications, which might be used for a variety of purposes, including entertainment
as well as serious game situations. The gamepad was evaluated using a linear hor-
ror game named Death: Unknown [42], which was an independent game available
freely to download. The device itself showed promise in being able to successfully
detect scare events in the game. Data obtained in the reported trial of 31 participants
shows promise in being able to successfully detect expected scare events. However,
the analysis makes limited to no use of additional sensor data and is largely focused
upon use of GSR data.
Simões et al. [43] developed a serious game to help individuals with Autistic
Spectrum Disorder, measuring the levels of independence and performance in the
context of a serious game designed to simulate the performance of an everyday task—
planning a bus route, making the journey and interacting with others along the way.
The key metric to measure anxiety in this scenario was via the use of electrodermal
activity (EDA). It is interesting to note that the VR game itself attempts to mimic real
life (to a reasonable degree), rather than obtusely stimulate emotional responses.
Kivikangas and Ravaja [44] report upon the affective responses encountered in
users playing against one another in a multiplayer scenario compared with playing
against the computer. Measurements of emotion were recorded using electromyo-
graphy (EMG) and EDA sensors. As such, there was no direct affective feedback
loop returned to the game nor was there specific measurement of pre-defined game
mechanics designed to provoke emotion. Rather, the authors were evaluating the
effect of the outcome of playing the game (winning or losing) and how the relation-
ship between each player and their opponent (whether they were friends, strangers or
a computer-controlled character) impacted their emotions. Their study involved 33
participants and used the game Duke Nukem Advance [45], a version of the classic
first-person shooter game adapted to be played on a portable gaming console; the
GameBoy Advance. The findings showed that positive affect occurred every time the
player won a game, but also when they lost. The positive responses to victory became
stronger when playing against another human, rather than a computer-controlled
character. One intriguing anomaly is that the authors note “…a curious effect of
negative response to victory over a friend, for which we presented numerous pos-
sible explanations, although we cannot say which one would be the most plausible
240 S. Cunningham
Fig. 13.3 Game controller equipped with bio-sensors (points 39a and 39b) in Sony Patent [46]
one.” Broadly, the work is supportive of the concept that relationship is an important
factor in the intensity of emotional response in players.
An early study of how affective feedback can be used to influence in-game actions
used a simple competitive two player racing game, appropriately entitled Relax-to-
Win [13]. The notion of incorporating biofeedback has also received attention from
a number of large games console manufacturers, with a notable example being the
patent held by Sony [46] that depicts various interface and game control devices
that have notably been fitted with bio-signals, shown in positions 39a and 39b in
Fig. 13.3, designed to be used as inputs to the game. The patent goes on to discuss a
range of such signals and systems that may be used to generate data to be sent to the
game or interactive software. Such an approach is flexible and avoids over-reliance
upon a particular type of signal. The validity of skin response equipment may be
brought into question under certain gaming circumstances, especially in demanding
scenarios where intense movement of the muscles, usually in the hand, is required
[28].
13.7 Suggestions for Future Work
13.7.1 Measuring the Efficacy of Affective Games
An initial piece of research to be done would examine the effects of incorporat-

ing biofeedback within networked multiplayer video games. This research could
use a participatory study to demonstrate the effects of biofeedback technologies on
the experience players have of the video game, particularly with regard to their immer-
sion, sociability, experience and overall enjoyment of the game. This would make use
of a range of bio-sensors attached to players of the game. For the purposes of a start
point, we suggest that indicators of players’ ECG, EDA and electroencephalography
(EEG) are selected.
The research could provide a simulated game environment that displays biofeed-
back information about competitors on the computer screen, much like the mock-up
presented in Fig. 13.1. Suitable methods of visualisation and sonification of this
data may be used, which would enable participants to perceive the data in an acces-
sible manner appropriate to the video game medium. Participants would play the
video-game in a head-to-head or collaborative scenario, comparing the video game
experience without biofeedback to the one with biofeedback (one group of partici-
pants will play with biofeedback, another group will play without). As such the study
would broadly follow the approach of a randomised controlled trial. Both subjective,
self-report data, using instruments such as the GEQ outlined earlier could be used to
capture player experience alongside other objective measures such as playing time,
success metrics in the game itself, video footage of each player and so forth.
13.7.2 Affective Games and Altered States of Consciousness
In previous work, several prototypes were created which explored the use of
consumer-grade biofeedback headsets as controllers for interactive artworks and
games. One of these: Psych Dome [47] utilised a NeuroSky MindWave to control
the generation of a visualisation and sound in a mobile full-dome. While the design
of the sound and visualisation were based on the concept of altered states of con-
sciousness, using biofeedback allowed properties of the simulation to be linked to
the brain activity of the person seeing it. This type of interaction is largely non-
volitional (that is, the individual tends not to be able to control their brain in such
a way that they can choose to influence the signals in a particular way), but instead
provides a form of ‘passive’ interactivity that can enrich the sense of interactivity.
In a related project, Quake Delirium EEG [48], this passive interaction was again
provided for an adapted video game, which aimed to simulate an intoxicated state
of hallucination, for which graphical and sound properties were linked to the EEG
signals. These projects are based on practice-led research, in which prototypes are
developed, but the prototypes can then use empirical research approaches to measure
the experience of participants. Eventually such approaches could lead towards the
possibility of ‘altered states of consciousness simulations’, as proposed by Weinel
[14].
In relation to the latter, it has been shown that there are significant therapeutic
benefits that can arise from altered states of consciousness facilitated by drugs such
as MDMA, LSD or psilocybin (e.g. Mithoefer et al. [49]; Bogenschutz and Johnson
[50]). Technologies such as VR have been widely touted as capable of simulating
real-world experiences that could have therapeutic applications—could VR also be
242 S. Cunningham
used to effectively simulate an altered state? Some early projects such as Guided
Meditation VR [20] hint at this possibility, yet there are also good reasons to be
sceptical about the benefits such experiences can actually provide when compared
with real meditation practices. What is therefore needed is research that takes bespoke
designs aimed to simulate altered states of consciousness through games or VR, and
measures the response of players, in order to demonstrate the extent to which these
designs are effective. Such research can then feed back into the generation of new
designs, perhaps allowing us to create virtual experiences that are tailored towards
particular kinds of perceptual experiences, eventually simulating altered states of
consciousness.
13.7.3 Measuring the Social Aspects of Multiplayer Affective

Games
Research on the level of connection and social presence [51] between players in
multiplayer gaming scenarios is underrepresented when compared to the amount
of information available about the single player experience. This is especially true
when considering the range of sensor data that affective gaming systems can provide.
Ekman et al. [52] recognised this, defining the metric entitled physiological linkage
and postulated that it may be associated with the social presence experienced by
players. The concept itself is simple to understand: the physiological measures of
players are likely to display patterns of synchronicity or correlation to one another,
meaning that it “…can be used as a measure of the intensity of the interaction between
participants” [52].
As explained earlier, when dealing with affect in multiplayer games it would
be highly desirable to examine how social elements of the gaming experience con-
tribute to player affect and, in turn, how the feedback of a player’s affective data might
influence other players. Accurately determining multifaceted concepts of game user
experience such as social presence, immersion and flow can be tough when using
only biosensors. Similarly, mechanisms to verify the affective responses predicted
from these sensors are also desirable. As such, a challenge exists in being able to mea-
sure these complex phenomena of the human condition. It is here where self-report
mechanisms become useful and there are several good examples of tools to measure
many of these concepts. In their work, which also highlighted objective measures for
immersion, such as eye-tracking, Jennett et al. [53] developed a series of self-report
questionnaires to be issued to games players. Factors they recognised as signifying
immersion in players included: affect; cognitive involvement and detachment from
the real-world. Contributing to the level of immersion experience were also factors
produced by the game itself, namely, that of challenge and control. Whilst Jennett
et al.’s work deals with measuring the experience of the individual, it was not pur-
posely intended to examine social aspects and connection with other human players
in the game.
One of the most commonly employed tools in the field of computer games research
is the Game Experience Questionnaire (GEQ) and its component instrument, the
In-game GEQ [54]. The tool integrates aspects of: competence; immersion; flow;
tension; challenge and affect. Additionally, it offers a social presence module, which
is designed to “…investigates psychological and behavioural involvement of the
player with other social entities…” [54], where these social entities may be other
human players or computer-controlled characters. The social presence element of
the GEQ consists of a total of seventeen questions that participants rate using a five-
point Likert scale. These questions provide scores under three themes of: empathy;
negative feelings and behavioural involvement. The first two of these have a strong
resonance in affective gaming, leading to useful indicators of how well an affective
game might be communicating the emotional state of other players (empathy) and
how this information might impact the player observing such information (nega-
tive feelings—incorporates elements of influence from others as well as a sense of
revenge or retribution). Lastly, behavioural interaction, seeks to determine how the
in-game actions of other players would change what the player in question did in the
game, as well as providing some indication of how meticulously players observed
one another. Consequently, the GEQ appears to be an effective mechanism for the
subjective evaluation of experience in affective multiplayer games. Its social presence
module certainly takes a step in the right direction to gauge the nature of interaction
and influence between players, but it would be useful to consider other instruments
that attempt to measure how connected or ‘close’ players feel to one another dur-
ing gameplay. A solution to this may be to append the In-Game Relatedness items
from the Player Experience of Need Satisfaction (PENS) questionnaire [55]. Such a
hybrid approach, combining overlapping aspects from a range of game experience
instruments is advocated by the work of Denisova, Norden and Cairns [56].
13.8 Ethical Considerations
There are ethical concerns relating to the combination of serious games and IoT.
Games in traditional forms store little information regarding their players—com-
monly scores, an alias or their real name. The advances in mobile gaming have
transformed the data that a game can access. Numerous games require you to sign
into a social media account. Once you have signed-in, the company has access to
personal information about the player and a list of the player’s friends. Introducing
the IoT only increases the amount of information games would retain about its users.
As discussed, an IoT enabled game could record a player’s heart rate to dynami-
cally alter a virtual environment, or help a player progress in it. The idea of a game
company holding such personal data could become uncomfortable to players; how-
ever, there are arguments to counter this perspective too. As younger generations are
introduced to technology from an increasingly early age, their perception of shar-
ing data could be different in comparison to generations that were introduced to
technology at an older age.
244 S. Cunningham
Irrespective of the discomfort users may have, there is an irrefutable fact that
games of this nature will need to store very sensitive information on its users. This
raises concerns regarding security. The General Data Protection Regulation (GDPR)
law in Europe [57] is a conscious effort to increase liability on developers regarding
storing personal data securely. This pressure increases the priority of securing new
interconnected systems and the data that flows within it. However, there are still
many challenges that the Internet of Things faces regarding security, many of which
stem from the lack of standardisation. Therefore, when embedding IoT with Games,
these challenges will relate to games too.
Associated with the work discussed here, which proposes to utilise affective data
and bio-sensor signals as input to game environments that may be socially oriented,
are concerns over psychological effects. Consider the first research activity proposed
earlier, for example, which would involve the visualisation and sharing of bio-sensor
information in collaborative or competitive games between human players. Certain
parallels might be drawn with aspects of the Stanley Milgram’s controversial experi-
ment [58] into behaviour and obedience. If one player of a game is competing against
another and sees that their opponent’s heart rate has elevated to abnormally high-
levels, will they attribute this situation to themselves and would guilt and blame be
attributed to them should their real-life opponent suffer cardiac problems? Perhaps
the answer is that the sensor attached to their opponent is simply faulty, but without
knowing this, the player might still suffer anxiety or low mood as a consequence.
One rebuttal to these fears may be that multiplayer gaming is not a new concept and
so simply presenting this information is only making visible a phenomenon that may
have been occurring already. Nevertheless, there is a duty of care to be considered
once this information has come to light. As recently pointed out in an article by
Tidsdale, “…video gaming has been claiming victims for decades” [59] and with the
enhanced sense of immersion, social connection and presence that we propose here,
we would be wise to approach research in this field with due care and diligence.
References
1. ART | meaning in the Cambridge English Dictionary, Dictionary.cambridge.org. [Online].

Available: https://dictionary.cambridge.org/dictionary/english/art. Accessed 30 May 2019
2. Gee, J.P.: Why game studies now? Video games: a new art form. Games Cult. 1(1), 58–61
(2006)
3. Smuts, A.: Are Video Games Art? Contemporary Aesthetics, vol. 3, no. 1 (2005)
4. Melissinos, C.: Video Games Are the Most Important Art Form in History, 22 Sept
2015. [Online]. Available: https://time.com/collection-post/4038820/chris-melissinos-are-
video-games-art/. Accessed 27 June 2019
5. Picard, R.: Affective Computing. The MIT Press, Cambridge, MA (2000)
6. Schwartz, M.S., Andrasik, F. (eds.): Biofeedback: A practitioner’s Guide. Guilford Publica-
tions, New York, USA (2017)
7. Maskeliunas, R., Damasevicius, R., Martisius, I., Vasiljevas, M.: Consumer-grade EEG devices:
are they usable for control tasks? PeerJ 4(e1746) (2016)
8. Parak, J., Korhonen, I.: Evaluation of wearable consumer heart rate monitors based on photople-
tysmography. In: 36th Annual International Conference of the IEEE Engineering in Medicine
and Biology Society (2014)
9. Wright, S.P., Hall Brown, T.S., Collier, S.R., Sandberg, K.: How consumer physical activity
monitors could transform human physiology research. Am. J. Phys. Regul. Integr. Comp.
Physiol. 312(3) (2017)
10. van den Broek, E.L., Westerink, J.H.: Considerations for emotion-aware consumer products.
Appl. Ergon. 40(6), 1055–1064 (2009)
11. Rouse, R.: Game Design: Theory & Practice. Jones and Bartlett Publishers, Burlington, MA
(2004)
12. Oxland, K.: Gameplay and Design. Pearson Education UK, London (2004)
13. Bersak, D., McDarby, G., Augenblick, N., McDarby, P., McDonnell, D., McDonald, B.,
Karkun, R.: Intelligent biofeedback using an immersive competitive environment. In: Designing
Ubiquitous Computing Games Workshop at UbiComp (2001)
14. Weinel, J.: Inner Sound: Altered States of Consciousness in Electronic Music and Audio-Visual
Media. Oxford University Press, New York (2018)
15. Russell, J.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)
16. Gabrielsson, A., Lindstrom, E.: The Role of Structure in the Musical Expression of Emotions,
in Handbook of Music and Emotion: Theory, Research, Applications. Oxford University Press,
Oxford (2012)
17. Calleja, G.: In-Game: From Immersion to Incorporation. MIT Press, Cambridge, MA (2011)
18. Rouget, G.: Music and Trance: A Theory of the Relations Between Music and Possession.
University of Chicago Press, Chicago (1985)
19. Epic Games (2017) Fortnite
20. Cubicle Ninjas (2016) Guided Meditation VR
21. Giant Squid (2018) ABZÛ
22. Feiner, S.K.: Augmented Reality: a new way of seeing. Sci. Am. 286(4), 48–55 (2002)
23. Sony, PlayStation Camera | More Ways to Play | PlayStation, Sony Interactive Entertainment
Europe Limited, [Online]. Available: https://www.playstation.com/en-gb/explore/accessories/
playstation-camera/. Accessed 30 June 2019
24. Microsoft, Kinect for Xbox One | Xbox.com. Microsoft (2016). [Online]. Available: http://
www.xbox.com/en-US/xbox-one/accessories/kinect. Accessed 30 June 2019
25. Parkin, S.: Death by Video Game: Tales of Obsession from the Virtual Frontline. Serpent’s Tail
(2015)
26. E.A. Inc, FIFA Video Games—Official EA Site, [Online]. Available:https://www.ea.com/en-
gb/games/fifa. Accessed 30 June 2019
27. Csikszentmihalyi, M.: Flow: the psychology of optimal experience. Acad. Manag. Rev. 16(3),
636–640 (1991)
28. Sykes, J., Brown, S.: Affective gaming: measuring emotion through the gamepad. In: CHI’03
Extended Abstracts on Human Factors in Computing Systems (2003)
29. Favorskaya, M., Sharma, D., Jain, L., Howlett, R.: Advances in Smart, Multimedia and
Computer Gaming Technologies. Springer International Publishing, New York (2015)
30. Henry, J., Tang, S., Hannenghan, M., Carter, C.A.: Measure of student engagement for serious
games and IoT. In: Edutainment 2017: E-Learning and Games (2017)
31. Microsoft Game Studios (2010) Kinect Sports
32. Henry, J., Tang, S., Hanneghan, M., Carter, C.A.: Framework for the integration of serious
games and the Internet of Things (IoT). In: Henry, J., Tang, S., Hanneghan, M., Carter, C.
(eds.) A framework for the integration of serious games and the Internet of Things (IoT). 2018
IEEE 6th International Conference on Serious Games and Applications for Health (SeGAH)
(2018)
33. Crowley, D.N., Breslin, J.G., Corcoran, P., Young, K.: Gamification of citizen sensing through
mobile social reporting. In: 2012 IEEE International Games Innovation Conference (2012)
34. Islam, S.R., Kwak, D., Kabir, M.H., Hossain, M., Kwak, K.S.: The internet of things for health
care: a comprehensive survey. IEEE Access 3, 678–708 (2015)
246 S. Cunningham
35. Maines, C.L., Llewellyn-Jones, D., Tang, S., Zhou, B.: A cyber security ontology for BPMN-
security extensions. In 2015 IEEE International Conference on Computer and Information
Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure
Computing; Pervasive Intelligence and Computing (2015)
36. Tully, D., Rhalibi, A., Carter, C., Sudirman, S.: Hybrid 3D rendering of large map data for
crisis management. ISPRS Int. J. Geo-Inf. 4(3), 1033–1054 (2015)
37. Van Wageningen, D., Staring, T.: The Qi wireless power standard. In: Proceedings of 14th
International Power Electronics and Motion Control Conference EPE-PEMC (2010)
38. Midway (1978) Space Invaders
39. Clynes, M.: Sentics: The Touch of Emotions. Anchor Press, Harlow, UK (1977)
40. Trussoni, S.J., O’Malley, A., Barton, A.: Human emotion communication by touch: a modified
replication of an experiment by Manfred Clynes. Percept. Mot. Skills 66(2), 419–424 (1988)
41. Bacchini, P., De, T., Barbosa, A.: Emopad: an affective gamepad. Int. J. Comput. Appl. 102(15),
5–12 (2014)
42. Johansson, J. (2013) Death: Unknown, Entyro
43. Simões, M., Bernardes, M., Barros, F., Castelo-Branco, M.: Virtual travel training for autism
spectrum disorder: proof-of-concept interventional study. JMIR Serious Games 6(1), e5 (2018)
44. Kivikangas, J., Ravaja, N.: Emotional responses to victory and defeat as a function of opponent.
IEEE Trans. Affect. Comput. 4(2), 173–182 (2013)
45. Take-Two Interactive (2002) Duke Nukem Advance
46. Weising, G.: Biometric interface for a handheld device. Washington, DC: U.S. Patent and
Trademark Office. Patent U.S. Patent No. 9,557,814. (2017)
47. Weinel, J., Cunningham, S., Roberts, N., Roberts, S., Griffiths, D.: EEG as a controller for
psychedelic visual music in an immersive dome environment. Sonic Ideas/Ideas Sonicas 7(14
) (2015)
48. Weinel, J., Cunningham, S., Roberts, N., Griffiths, D., Roberts, S.: Quake delirium EEG: a
pilot study regarding biofeedback-driven visual effects in a computer game. In: The Sixth
International Conference on Internet Technologies and Applications (ITA 15) (2015)
49. Mithoefer, M., Wagner, M., Mithoefer, A., Jerome, L., Doblin, R.: The safety and efficacy
of 3,4-Methylenedioxymethamphetamine-assisted psychotherapy in subjects with chronic,
treatment-resistant post-traumatic stress disorder: the first randomized controlled pilot study.
J. Psychopharmacol. 25(4), 439–452 (2010)
50. Bogenschutz, M., Johnson, M.: Classic hallucinogens in the treatment of addictions. Prog.
Neuropsychopharmacol. Biol. Psychiatry 64, 250–258 (2016)
51. Biocca, F., Harms, C., Burgoon, J.K.: Toward a more robust theory and measure of social
presence: Review and suggested criteria. Presence: Teleoperators Virtual Environ. 12(5), 456–
480 (2003)
52. Ekman, I., Chanel, G., Järvelä, S., Kivikangas, J.M., Salminen, M., Ravaja, N.: Social inter-
action in games: measuring physiological linkage and social presence. Simul. Gaming 43(3),
321–338 (2012)
53. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and
defining the experience of immersion in games. Int. J. Hum Comput. Stud. 66(9), 641–661
(2008)
54. Ijsseltein, W., de Kort, Y., Poels, K.: The game experience questionnaire: Development of a self-
report measure to assess player experiences of digital games. Technical University Eindhoven
(2008)
55. Rigby, S., Ryan, R.: The Player Experience of Need Satisfaction (PENS) Model. Immersyve
Inc (2007)
56. Denisova, A., Nordin, A.I., Cairns, P.: The convergence of player experience questionnaires. In:
Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (2016)
57. Regulation (EU) 2016/679 Of The European Parliament and of the Council of 27 April 2016
on the protection of natural persons with regard to the processing of personal data and on the
free movement of such data, and repealing Directive 95/46/EC General Da (2016)
58. Milgram, S.: Behavioral study of obedience. J. Abnorm. Soc. Psychol. 67(4), 371–378 (1963)
59. Tisdale, A.: Gaming In Virtual Reality Could Be the Very Real Death of You, Vice Media LLC,
8 June 2016. [Online]. https://www.vice.com/en_uk/article/4w5g7d/gaming-in-virtual-reality-
could-be-the-very-real-death-of-you-911. Accessed 4 June 2019
Part V
The Convergence of Digital Design,
the Arts, Computing,
and the Environment
Chapter 14
Chandini (A Bride for the Moon)
Rachel Davies and Daniel Saul
Abstract Chandini (A Bride For The Moon) is an art/science project led by R&D
Studio which explores the dream of India’s ambition to be the fourth nation to make
a soft landing on the moon. In collaboration with a dancer, musician and scientists,
poetic visual and performative metaphors are developed that represent the progress
of Indian society in the technological age. This chapter describes the ongoing project
from the authors’ perspective; how they collaborated, in response to different oppor-
tunities and changing circumstances, the obstacles they encountered and how the
public engaged with their artworks at events and festivals between 2017 and 2019 in
the UK and India.
Keywords Filmmaking · Choreography · Art/science · Collaboration ·

Documentary · Projection mapping · Outdoor arts · Space · Lunar landing ·
Moon · India
14.1 Introduction
Chandini (A Bride For The Moon) is an art/science project led by R&D Studio [1]
that explores the hopes and dreams of India’s ambition to be the fourth nation to make
a soft landing on the moon. Working in collaboration with artists and scientists they
develop poetic visual metaphors to consider notions of progress to Indian society in
the technological age.
R&D Studio is a coming together of two artist filmmakers; Rachel Davies and
Daniel Saul. They describe here how their artistic responses to a subject have evolved
creating various iterations of the project over a two-year period.
Both artists have experience of working with Indian documentary subjects and
dancers. Rachel with short Channel 4 dance films, collaborations with Mavin Khoo
R. Davies (B)
Kingston University, Kingston, UK
e-mail: r.davies@kingston.ac.uk; rachel@racheldavies.co
D. Saul
Royal College of Art, London, UK
e-mail: d.saul@arts.ac.uk; dan@rachelanddaniel.co
252 R. Davies and D. Saul
(Khooyile) [2] and Akram Khan (Loose in Flight) [3] which toured internationally
with the British Council [4]; and Daniel’s film The Morris Jelly House of Fashion
(Channel 4) [5] which drew on his personal experience of being from an Anglo-Indian
family originating from Calcutta.
The artists remember when India was characterised in the UK as an indigent recip-
ient of foreign aid. However recently India has positioned itself on the global stage
with a symbolic ascendency to become a self-proclaimed leader in space technology.
This project reflects on these contrasting contemporary global perceptions of an India
which has been transformed by technology yet in many ways whose traditions have
stayed the same.
For Chandini (A Bride For The Moon) R&D Studio worked with Hemabharathy
Palani a dancer/choreographer [6] and TeamIndus [7] a private aerospace company,
both based in Bangalore, to produce short films, choreography, performances, pro-
jection mapping and installations that engaged audiences through broadcast and
performances in the UK and India.
The wider project began as a response to an open call from the British Coun-
cil/Big Dance Shorts in 2017 [8]. The commissions were seeking dance and film
collaborations between the UK and India to mark 70 years of Indian Independence
from Britain, a complex relationship from the outset. It was then that R&D Studio
decided to focus on a small private company of mainly young people who were late
entrants to the Google’s 2007 Lunar X-Prize; a $30 million international competition
open to non-governmental agencies to land a robot on the Moon, have it travel 500
m, and broadcast high resolution captured images back to Earth [9].
TeamIndus [10] presented themselves in a markedly different way from more
grandiose national Space agencies. They cast themselves as young dreamers working
to creatively inspire other young minds in India. When we initially contacted them
we said, “We want to make a film about your rover, but using contemporary dance.”
They replied “So cool!”
Simultaneously we began a search for an Indian dancer who could translate a
science story into a personalised performance. Emma Gladstone (Artistic Director
Dance Umbrella) [11] recommended Hemabharathy (Hema) Palani. Hema was based
in Bangalore, as were TeamIndus. From this point our themes developed. We discov-
ered that TeamIndus’ robot, as often with ships or craft, was referred to as ‘she’ by
the team who made her. TeamIndus co-founder Sheelika Ravishankar told us “When
I received your proposal the first thing I asked was, is your dancer female?” Hema
told us that the one-way nature of the robotic moon mission reminded her of a com-
mon Indian female experience; when a young woman moves from her birth family’s
home to the new home of her husband, it is sometimes said that “She will only ever
return as ashes.” A parallel story began to emerge. We would track the journey of
an Indian robot on the moon echoed by an Indian woman’s journey on Earth. Our
filmic mise-en-scène became Bangalore; once a small colonial outpost, now rapidly
transforming into an Asian information technology-driven megalopolis. A city con-
taining both ancient tradition and a world-beating aerospace industry. Our project
became a container for exploring a dialogue between these two ideas. Figure 14.1
14 Chandini (A Bride for the Moon) 253
Fig. 14.1 R&D Studio 2018. Evocation of the cosmos created in milk droplets
shows a frame of an animation rendered in development workshops, and eventually

used in large-scale projection within live performances in Bangalore and Dorset.
14.2 A Changing Real-World Context
One of the challenges of documentary-making is access. The subject matter of the

Indian Space programme could have presented us with several obstacles. ISRO [12],
the governmental space agency responsible for nearly all of India’s activities in space,
are not known for being an open access organisation and we had heard rumours that
they were very unlikely to allow foreigners to visit them. We made several attempts
to contact ISRO and received no response.
Finding TeamIndus was a blessing as they were wonderfully open and generous
and once in India facilitated our visits thoroughly and gracefully. However, we did
not know if we would be able to find empathy between our very different professional
disciplines. We spent a lot of time preparing our questions and equally allowed our
interviews to unfold in a relaxed and unhurried atmosphere. When we first met
TeamIndus in 2017 many teams from other countries had dropped out of the Lunar
X-Prize and TeamIndus were ranked as favourites among the surviving four. It had
been a white-knuckle ride with many delays and last minute deadline extensions.
In January 2018, TeamIndus announced that they were scheduled to launch their
spaceship in March–April that year. However, it was not to be. TeamIndus had to
cancel their contract with ISRO (Indian Space Research Organisation), their launch
provider [13]. It was a heart-breaking moment for the team. Google announced the
closure of the Lunar X-Prize without a winner. However, it was clear that the idea
had been timely and had stimulated imagination in the private sector. A slew of new
international missions to the moon were announced during this time, by national
agencies such as NASA and ISRO and also many private companies around the
world.
In March 2019 the ‘new space race’ dramatically intensified. In a speech at NASA’s
Marshall Space Flight Center in Huntsville, Alabama Vice-President Mike Pence
announced, in the most nationalistic terms, the U.S. government’s challenge to NASA
to “return astronauts to the moon within the next 5 years.” Also saying, “The United
States must remain first in space in this century as in the last. Not just to propel
our economy and secure our nation, but above all because the rules and values of
space, like every great frontier, will be written by those who have the courage to
get there first” [14]. Also in March, Narendra Modi declared India to be the ‘Fourth
Space Superpower’ after ISRO successfully destroyed one of its own satellites with a
missile launched in space. Creating an estimated 6,500 pieces of space debris (in an
exact echo of the opening sequence from the film ‘Gravity’) Modi announced a day
of national celebration and called the action something that would “impress India’s
rivals” [15]. In this year too the Trump administration announced the creation of a
5th branch of the United States armed forces called the ‘space force’ [16]. In April
2019, Israel’s Space IL, also a competitor in Google’s Lunar X-Prize, successfully
launched the world’s first private moon mission on an unusual orbital trajectory. They
proved just how challenging lunar expeditions could be as the craft unexpectedly
crash-landed on the moon’s surface [17].
On 31 May 2019, TeamIndus tweeted that they had successfully partnered with
American company Orbit Beyond and were once gain hoping to launch in 2020 as
part of NASA’s CPLS programme [18]. On 15 July 2019, ISRO was set to launch its
most complex lunar mission and hoping to make India only the 4th nation to soft-land
on the moon. They cancelled due to technical difficulties only 56 min before launch
[19]. The mission successfully launched 2 weeks later, but tragically communication
with the lander was lost just before touch down when the craft was reported to be
just 2.1 km from the lunar surface [20]. As of writing the craft is presumed to have
crash-landed, demonstrating perhaps, together with the Israeli attempt, the scale of
the technological challenge involved in reaching earth’s closest neighbour.
The dramatic twists and turns of the story have meant that our project also has had
to be fleet of foot, able to respond quickly to an ever-changing back-story. Within each
development, and beside the grandiose and gung-ho statements of some players, we
feel that our story offers up an Indian female human perspective, the ‘small dream’
of hope, set against the backdrop of an escalating international space race.
14.3 Short Film
Ek Choti si Asha (A Small Dream) Dance film, 4 min, Channel 4 Television 2017
The overall project had to meet considerable challenges: creative, technical and
organisational, with many aspects that we had not tried before. The collaborations,
though exciting, presented us with potentially difficult paths to negotiate. We first met
Hema in Verbania north Italy, to watch her perform her own choreographic pieces
in a dance festival. We were hugely impressed by her work, delicate and edgy at the
same time.
The first iteration of the project was a short film commissioned for Channel 4
television; Ek Choti si Asha (A Small Dream) [21]. Our collaboration, including
costume design, was developed across continents and time zones via Skype and
WhatsApp. It was necessarily quick as we had limited time to develop the work.
We designed a container for Hema to contribute to, sending video storyboards to her
whilst she sent rehearsal clips of ideas and movements. Decisions were made quickly
and the planning was precise. We attempted to present the audience with a deliberate
conceit—that the inclusion of one narrative thread would be appropriated to explore
another. In this case, the voices of young Indian space engineers would be featured
describing how their robot would cope with physical challenges and obstacles on
her perilous journey. This audio would be set against imagery of an Indian female
traveller negotiating the busy streets of Bangalore; people, animals, traffic and the
male gaze; the combination of voice and choreography allowed us to explore parallels
between the Earth and Moon stories without literal space or moonscape being shown.
Further to this idea, we decided to incorporate TeamIndus’ use of the feminine
pronoun and request that they always refer to the robot as ‘she’. Interestingly at
this point the two young male engineers warned us “You won’t get an emotional
response from us. We are engineers and technicians.” They then proceeded to make
recordings that were infused with obvious emotion as they described the robot that
they had been creating for the past six years.
We continued our remote video dialogue with Hema, sending her animatics (edited
voice-over with improvised visuals) with Hema responding with small sections of
dance recorded in her studio. In this way the ideas and the interplay between words,
dance and imagery evolved and a larger narrative emerged.
The specific aspects of the mission meant that the robot rover, named ‘Ek Choti
si Asha’ (E.C.A. for short), meaning ‘a small dream’ in Hindi, would only survive
whilst her solar panels could be in direct sunlight. One lunar day is equivalent to 14
earth days. ECA would travel from Earth to Moon with her batteries switched off.
She is only powered on when her wheels touch lunar soil. From this moment the
engineers would be able to control her movements on the Moon from the Earth for
a fortnight [22].
For us it meant we could give Hema a structure for her choreography. The young
character ECA would begin asleep in a black space then wake and start her journey
outdoors (Fig. 14.2). A dawn to dusk framework in the 4-min film would inform the
remaining narrative that also included the entire lifespan of the character. At the end
Fig. 14.2 R&D Studio 2017. Ek Choti si Asha. Opening shot in Bangalore residential street, where
Hema starts her walk. https://vimeo.com/240144999
of the day, she faces the future with the knowledge that her end is imminent. Hema
introduced tropes from marriage rituals [23] and developed her female character’s
narrative, conceived of her as a contemporary Indian woman, yet bound by tradition
and family. Simply told, she would leave home, full of hope as a young woman
embarking on married life, yet she faces anxiety when she realises that her freedoms
must change forever and she can’t go back.
The storyboard and animatic followed this format, building mini-scenes or chap-
ters based on different technical challenges that the engineers described in detail.
Individuated sections were based on how to move the robot in low gravity, how to
negotiate obstacles such as rocks and gradients, and how to remain safe from very
fine electro-static moon dust. These factors meant that the engineers estimated they
would be able to move ECA no faster than 5 cm/min. In response Hema’s first task
was technical too—how to choreograph a walk so slow it resembled time-lapse. It
was pleasing when we filmed tests and rehearsals outdoors that Hema’s choreogra-
phy appeared to resemble slow motion, nicely belied by the figures around moving
at normal speed.
This first iteration enabled us to start building a film language that blended doc-
umentary interviews with choreography and interwove a journalistic story with a
fictional character. Ground rules were established and a simple narrative framework
was conceived. For all the constraints of time and distance the simple structure
worked very well and all three artists were satisfied with the result.
For Hema the project also became expansive. During 2018 we began working in
the context of a larger framework beyond the short film and a departure into live
performance.
14.4 Outdoor Performance
A Little Big Dream. National Gallery of Modern Art, Bengaluru January 2018
In January 2018 the British Council in India was in the process of concluding a year
of cultural events marking 70 years of Indian independence. Luke Jerram’s Museum
of the Moon sculpture [24] was brought to Bangalore; a 7-m diameter spherical
sculpture printed with high-resolution NASA photographs of the Moon’s surface.
Hemabharathy Palani and R&D Studio were invited to make a performance
beneath this artwork on the night of the ‘super moon’ on the 31 January 2018 outside
the National Gallery of Modern Art [25].
The performance outdoors under the enormous moon sculpture immediately gave
a sense of scale and suggested a duality of strength and fragility: a hardy robot built
to withstand other worldly forces and yet tiny against the vastness of space. Hema’s
choreography included a team of about 20 dancers who made a chorus surrounding
her. She, however, was still alone, marooned on an island in the centre of an artificial
pond, visually separated from the other performers. The audience were given an
image of a lone explorer supported by a big team who were nevertheless separated
from her and less and less able to control her as she journeyed further away.
Luke Jerram’s Moon and the central performance were both reflected perfectly in
the still water around her, a material existing in a non-solid state; an ethereal inverted
echo transposed within another atmosphere and space. Pre-recorded video close ups
of Hema’s movements projected into the decorative masonry and structure of the
museum echoed her live performance, yet were strangely remote and disconnected
too. The engineering team growing ever distant echoed a fading connection with
birth parents. These key images expanded our metaphorical lexicon and represented
several ideas at once (Fig. 14.3).
14.5 Objects and Projections
Residencies: Leverhulme & 101 Outdoor Creation Space Spring/Summer 2018

Hema was invited by Ballet Rambert to develop new work during a residency spon-
sored by Leverhulme [26]. Together with theatre producers Fuel [27], we started
to develop ideas for a larger outdoor show, both in Rambert Studios London, and
by securing a further ‘Seedbed’ residency at 101 Outdoor Arts Creation Space near
Newbury [28], a large rehearsal space intended for artists to develop outdoor work
at scale.
In order to expand the film story into a live performance; narratively, choreograph-
ically and visually, this more open-ended collaboration presented more challenges
in terms of balancing our creative voices with our different cultural backgrounds,
artistic training and preferred methodologies.
We wanted to further develop the work as site-specific location-based, working
outside to enable extreme contrasts in scale, the tiny robot in amongst the vast cosmos.
Fig. 14.3 R&D Studio 2018. A Little Big Dream. https://vimeo.com/256314451
During these residencies, we conceived of making a wordless visual theatre show

that would bring an Indian dimension to 2019, the year of the fiftieth anniversary of
Apollo 11.
During the Leverhulme residency we made a series of physical experiments,
inventing mechanisms to create a sequence of images to carry a basic narrative. If
we were aiming to tell a parallel story between a robot travelling from Earth to
Moon and a woman moving between houses on Earth then perhaps simple contrasts
could serve as starting points. We experienced a break-through when we began to
play with objects and projectors in a darkened space. Choreography could be very
small, just hands and small objects, while projections could be huge and immersive
suggesting space or other worlds. We imagined how to combine content made with
high technology with other images made with domestic objects, playing with large
and tiny projections, with the body and with objects on a table-top (Fig. 14.4).
During the 101 Residency, to increase our emotional range we invented a second
character, a ‘mission controller’ who would appear on stage both as a live musician
and a kind of director, tracking Hema’s character (who we called ‘Chandini’—trans-
lating as ‘a bride for the moon’), and arranging small domestic objects on a table-top.
For example, a cup of tea would become a surface for small focused projections that
would be re-filmed and re-projected on a large scale thus suggesting immense objects
or vast landscapes. Technical and visual experiments such as this were combined with
narratives, expanded to take place on three stages: ‘Earth’, ‘Mission Control’ and
‘Moon’. The action would take place live in an outdoor setting in front of an audi-
ence but also filmed and projected on three large screens. The Earth and Moon stages
would also have cameras directly above looking down, and the Moon stage would
be covered with water.
Fig. 14.4 R&D Studio 2018. Table Top Moon. Leverhulme residency film. https://vimeo.com/
266294389
We considered the potential use of a drone above the performer and audience, used
to stream live images of Chandini’s journey to a large-scale outdoor audience. We
took drone footage above Greenham Common around 101 Creation Space which,
when treated, appeared like an alien landscape. (This footage was later used for
subsequent stage iterations.)
Via digital interactive tools a nuanced relationship between Chandini and her
maker/controller/family became a further theme explored. Chandini arrives with a
live camera attached to her head, whose feed is received and augmented by the
mission controller and streamed onto the big screen for the audience.
Interactive sensors attached to the dancers’ wrists created a cascade of ceremonial
floating flowers symbolising marriage vows (Fig. 14.5).
We mapped out a partially visualised narrative for a full-length outdoor piece in
the form of a diagramatic triptych film incorporating spoken and labelled description
with footage gathered from both residencies. Chandini Animatic (Fig. 14.6).
14.6 Choreography
Chandini—Work in Progress Ranga Shankara Auditorium Bangalore, Attakkalari

Interim Festival, 2 February 2019
Having focussed on technical possibilities the previous year, it was now time to
develop the choreography. Hema worked on the identity and story of the character
‘Chandini’ with several days of devising alone before our arrival in Bangalore. The
Fig. 14.5 R&D Studio 2018. Overhead image of Hema with augmented animated flowers
Fig. 14.6 R&D Studio 2018. Chandini Animatic. Triptych diagramatic film. https://vimeo.com/
297342932
approaching public showing focused our energies and brought together the choreo-
graphic and visual aspects of the piece. Hema’s choreography comes from working
closely with music and we collaborated with PK a young Bangalore-based musi-
cian who formed a series of digital soundscapes. In rehearsal sessions we mapped a
story of Chandini’s journey on the moon; from the dream of her landing, adapting to
her unforgiving environment physically and psychologically, her fears and struggles
with a perceived feminine role, and her ultimate realisation that she is facing her new
world alone.
In the theatre film, animation sequences (e.g. Fig. 14.1) were projected onto dark
surfaces and in ultra-wide-angle, bathing the whole auditorium and extending beyond
the stage, exploring the impact of moving images as a vast spatial environment into
which the small Chandini struggles to find her place (Fig. 14.7).
Fig. 14.7 R&D Studio 2019. Chandini—Work in progress. 20 min dance performance. https://
vimeo.com/320478323
14.7 Festival
A Small Dream at Dorset Moon Festival June/July 2019

R&D Studio was invited to propose ‘Under The Moon’ ideas for ‘Dorset Moon’
Festival in summer 2019, a special one-off festival to coincide with the 50th anniver-
sary of the Apollo 11 moon landing. Dorset Moon was curated by three partners in
Dorset; Activate, producers of the Inside Out Festival, Dorset, Bournemouth Arts by
the Sea, and b-side [29].
Using three spectacular settings we created A Small Dream [30] to showcase the
story of ‘Chandini’ in three chapters: Launch, Journey and Arrival, comprising two
outdoor performances and a video installation across 3 consecutive weekends in July
2019.
Each show was different and each was intended to work alone. By viewing all
three the audience were able to track the story of a tiny female robot on a perilous
400,000-km journey. Families, walkers, beach lovers and culture vultures encoun-
tered a perambulatory dance through seaside gardens, films projected to look like
stained glass windows and a final performance in a fort built out into the sea. The
festival was the project’s most ambitious outing so far and once again we diversified
our approach, developing and extending our metaphors.
Hearing that ISRO’s Chandrayaan robot mission was planned for the same summer
we also updated Chandini to embody her perspective within this current bullish and
intensifying commercial race.
14.8 Live Art
Launch Bournemouth beach and St. Peter’s Church 28–30 June 2019
Chandini’s Dorset journey began on Bournemouth beach. It was the first sunny
weekend of the season and the hottest day of the year thus far. People thronged to the
beach. In amongst them squatting close to the pier Chandini gazed out to sea, a Go-
Pro camera strapped to her head, recording her view. She gathered sand in her hands
and made a ritual offering, a puja, to the sea. Slowly standing up Chandini began to
pick her way across the sand, her slow cautious progress echoing that which might be
taken on an alien surface. The imagery of the packed beach, as seen from Chandini’s
headcam, is extraordinary. The camera picks up a strange world in which people are
gingerly adapting everyday movements to an unfamiliar environment. Both staring
and pretending not to look, the pleasure-seeking beachgoers are strangely unclothed
and vulnerable. Chandini slowly passes through crowds, up close and intimate, pro-
voking curiosity with a slow and mysterious approach towards Luke Jerram’s moon
installed in St. Peter’s Church nearby [31].
After performing another puja at the door and taking off her shoes and headcam,
Chandini enters the church to make a ten-minute performance under the moon includ-
ing drawing the orbital path of ECA’s journey to the moon in a blue sand rangoli on
the floor of the nave.
After the performance audience members were asked, why do you think she is
going to the moon? Echoing the question, why are ‘we’ as a species heading back to
the moon after 50 years?
The main form of this piece was not drawn from dance or theatre but from Live
Art. Durational, interactive and placed in the real world this medium situated our
character again in a new context and with a new relationship to an audience who may
not even know they are witnessing an artwork (Figs. 14.8 and 14.9).
14.9 Video Installation
Journey Sherborne Abbey [32] 5–7 July 2019

We next encounter Chandini’s journey installed in the magnificent eighth century
Sherborne Abbey [32] in video projections in the shape of stained glass windows.
In the quiet devotion of the ancient space, film imagery (such as a turmeric hand
clap in slow motion and the cosmos as suspended milk droplets) was combined
with the beach point of view scenes from Chandini’s Bournemouth headcam, and
embedded into the crenulations of a ‘blind window’.
Headphones played a musical score peppered with audience responses captured
during the week, with the aim of subtly drawing viewers into current conversations
about space exploration.
The technique used here was projection mapping. Two projectors carried synced
video imagery mapped into eleven complicatedly shaped windows. The venue echoed
Fig. 14.8 R&D Studio 2019. A Small Dream: Launch. Hema’s slow walk
Fig. 14.9 R&D Studio 2019. A Small Dream: Launch. Hema’s headcam POV
Fig. 14.10 R&D Studio 2019. A Small Dream: Journey
Hema’s ritualistic and ceremonial interpretation of the scientific narrative. We almost

literally created a window onto another world (Figs. 14.10 and 14.11).
14.10 Outdoor Dance Theatre
Arrival Nothe Fort, Weymouth [33] 12–14 July 2019

The final part of the Dorset Moon trilogy was the largest and most ambitious chapter.
The setting of Nothe Forte, a nineteenth century military fort built on a promontory
with sea on three sides gave a grandeur to our performance and we consciously
wanted to provide a spectacle for the 300+ audience.
Fig. 14.11 R&D Studio 2019. A Small Dream: Journey (seen through arch)
We placed the audience up on the ramparts looking down into the fort where Luke
Jerram’s Museum of the Moon was suspended from a crane. A stage was built just
off centre beside the moon and we immersed Hema’s live performance in a huge
projection extending 27 m across the circular floor of the fort (Fig. 14.12).
Finally arriving on the moon Chandini appears, small and fragile, in the centre of a
cosmos projected at scale over the sunken grounds of the amphitheatre-like fort. The
audience looks down on her as she wakes in an alien world surrounded by danger.
Chandini confronts and struggles with her new environment whilst we realise she is
ultimately abandoned.
Being the grand finale to our constructed story developed over the three week-
ends this piece was more akin to theatre. Using rehearsed choreography, projection
mapping onto a square stage and within a larger elliptical surface, lighting and a
live camera effect, the techniques were interwoven into the narrative. In addition to
combat the prevailing sound of a windy environment, the audience wore wireless
headphones so they could hear the soundtrack including its subtleties and speech.
This performance presented large challenges as we moved from the medium of
film to large-scale outdoor site-specific theatre and a variety of technical difficulties
had to be overcome. The greatest challenge, however, was narrative; how to convey
our emotional story also based on the current real lunar Indian mission, in the context
of a family-based festival marking the 50th anniversary of Apollo 11. We tackled
this with soundtrack; the performance begins with a cacophony of sounds and voices
Fig. 14.12 R&D Studio 2019. A Small Dream: Arrival. View of Nothe Forte performance from
ramparts
including current bold statements of ambition from national agencies, contrasting

this later with the voice of TeamIndus’ Sheelika Ravishankar describing the diurnal
cycle on the moon alluding to the emotional journey of the character Chandini.
14.11 Documentary
The Vyomanaut 30 min film October 2019

Figure 14.13 shows a frame from the film Vyomanaut.
Whilst making the choreography in India we began to record and document our
process behind the scenes (Fig. 14.14). We also interviewed Hema about her feel-
ings towards the work and also about her life in general. We felt as if we had an
extraordinary resource in her. Fiercely independent and fearless, she did not have
an academic background and doesn’t consider herself an intellectual. Not from an
artistic family her first love was sports. Yet we found her to be a gifted creative and
naturally articulate. She talked about her parents, how her father was supportive of
everything she did but how her mother approved of her traditional performances but
had refused to see any of her contemporary work. Almost constantly on tour, travel-
ling the world and performing extraordinary edgy works that she had created herself,
Hema is a cultural warrior steeped in Indian tradition and mythology. It was apparent
that Hema herself embodied the same layers and complexities that the Indian space
programme itself contained: simultaneously ground-breaking at the leading edge of
innovation and entirely shaped by ancient culture.
Fig. 14.13 R&D Studio 2019. The Vyomanaut. Hema creating soundtrack. https://vimeo.com/
327502293
Fig. 14.14 R&D Studio. The Vyomanaut. Super slow-motion shot of Hema’s feet landing in milk
(the moon). https://vimeo.com/327502293
The original conceit, the parallel story of a robot on the moon and an Indian woman
journeying on earth, remained intact even though the film subtly shifts attention
to our choreographer herself. In fact the metaphors deepened. Our documentation
film became a portrait of Hema interwoven with a further exploration of the space
mission. We visited TeamIndus again, with Hema, and gaining more insights into
their evolving story, explored with them the parallels between their endeavours and
Hema’s creative world. Also, being appreciative of Hema’s love for speed, we made
another visual encapsulation for the film: in contrast to her slow-motion walk, we
filmed her dynamically running through Bangalore’s streets.
The film allowed us to develop themes of speed, freedom for women in India, fam-
ily, nature and science, wrapped up in a human portrait. It also exists as a document of
our entire collaborative process from beginning to end, a video accompaniment to this
chapter and a creative documentary film in its own right, incorporating different tex-
tures of fly-on-the-wall documentary to highly rendered visual images (Fig. 14.14).
It is currently being completed and will be pitched to dance film/science festivals in
2020/21.
14.12 Conclusions
When we began making the Chandini project it was already a hybrid creature, mixing
the worlds of contemporary dance, documentary film and animation. But we hadn’t
planned for it to diversify into so many different forms. All three of our individ-
ual practices are predicated on a playful intermingling of ideas and techniques and
what we hope is healthy disregard for the traditional silos that some art forms find
themselves constrained within. It is therefore unsurprising that our collaboration has
become a multi-headed creature.
We would argue that this process has not been schismatic but rather the opposite.
The original playful conceit of the parallel story remains a constant, as does the
overarching narrative of a one-way journey.
Observing the engineers at TeamIndus search for innovative solutions to technical
problems made us consider how adopting an expansive and exploratory approach
could push our work forward in unexpected ways [34] and furthermore to ask—what
is the purpose of creative research?
By responding to layers within a story in the real world our iterations were driven
by form following content, through deliberate meanderings across genre, form and
technique, enabling us to examine the facets of our diamond in different lights, a
challenging and illuminating process.
References
1. R&D Studio. www.rachelanddaniel.co

2. Khooyile. Channel 4/Walk Clements (1999). https://vimeo.com/8150040, https://www.bfi.org.
uk/films-tv-people/4ce2b835a4d26/
3. Loose in Flight. Channel 4/Walk Clements (1999). http://www.akramkhancompany.net/
productions/loose-in-flight-2000/
4. Mitra, R.: Auto-ethnography and loose in flight. Akram Khan: Dancing new interculturalism.
Palgrave (2015). https://www.palgrave.com/gp/book/9781137393654
5. Morris Jelly House of Fashion. http://film.britishcouncil.org/the-morris-jelly-house-of-fashion
6. Hemabharathy Palani. http://www.attakkalari.org/index.php?page=dancers

7. ET Rise [online]. https://economictimes.indiatimes.com/small-biz/startups/how-axiom-
research-labs-has-emerged-as-indias-first-first-private-aerospace-company/articleshow/
57101170.cms
8. British Council. https://uk-india.britishcouncil.in/big-dance-shorts-india. Big Dance http://
www.bigdance.org.uk/big-dance-shorts-india/
9. Google Lunar X Prize. https://lunar.xprize.org/prizes/google-lunar
10. The Hindu. https://www.thehindu.com/todays-paper/tp-national/Indian-team-in-lunar-rover-
competition/article14678319.ece
11. Dance Umbrella. http://www.danceumbrella.co.uk/
12. ISRO. https://www.isro.gov.in/
13. News Minute. https://www.thenewsminute.com/article/end-team-indus-mission-moon-
contract-isro-cancelled-report-74403. Jan 2018
14. NASA. https://www.nasa.gov/press-release/nasa-administrator-statement-on-return-to-moon-
in-next-five-years. March 2019
15. India Today. https://www.indiatoday.in/india/story/mission-shakti-narendra-modi-full-speech-
1487838-2019-03-27. March 2019
16. USA Today. https://eu.usatoday.com/story/news/politics/2019/08/29/donald-trump-space-
force-closer-reality-new-space-command/2149265001/. Aug 2019
17. BBC News. https://www.bbc.co.uk/news/science-environment-47879538. April 2019
18. QZ. https://qz.com/1633918/americas-first-private-moon-lander-will-be-engineered-in-india/.
June 2019
19. Business Standard. https://www.business-standard.com/article/current-affairs/experts-say-
chandrayaan-2-delay-was-pre-emptive-mission-not-a-failure-119071500600_1.html. Sept
2019
20. BBC News. https://www.bbc.co.uk/news/world-asia-india-49615665
21. Ek Choti si Asha—A Small Dream. Random Acts Channel 4. https://randomacts.channel4.
com/post/167939666571/a-small-dream-hemabharathy-palani-in-2018-india
22. R&D interview with Shileeka Ravishanka at TeamIndus, 9 July 2017
23. Gaye Halud. https://steemit.com/colourchallenge/@eshani/turmeric-beautiful-significance-in-
indian-weddings-2017712t202722301z
24. Luke Jerram. https://www.lukejerram.com/
25. National Gallery of Modern Art, Bangalore. http://ngmaindia.gov.in/ngma_bangaluru.asp
26. Leverhulme Trust. https://www.leverhulme.ac.uk/research-fellowships
27. Fuel Theatre. https://fueltheatre.com/
28. Creation Space. https://www.101outdoorarts.com/
29. Dorset Moon Festival. https://dorsetmoon.com/
30. A Small Dream. https://activateperformingarts.org.uk/a-small-dream-hemabharathy-palani-
and-rd-studio
31. Chandini at St Peter’s Church Bournemouth. https://www.alamy.com/stock-photo/
hemabharathy.html
32. Sherborne Abbey. https://www.sherborneabbey.com/
33. NotheForte Weymouth. https://nothefort.org.uk/
34. TeamIndus blog. https://medium.com/teamindus/teamindus-the-next-phase-d133629e0430
Chapter 15
Digital Moving Image Installations
and Renewable Energy: 1994–2018
Chris Meigh-Andrews
Abstract In the period between January and April 1994, while I was artist in Res-
idence in Digital Imaging in the School of Visual Arts, Music and Publishing at
Oxford Brookes University, I developed Perpetual Motion, a gallery-based installa-
tion presenting a computer animation powered via a wind turbine. This early work
initiated a series of installations presented within a “white cube” gallery setting and
outside in the landscape in which renewable energy systems were integral to the
themes and functioning of the work and to the ethos and concerns of my approach to
working with moving image and sound technologies. This chapter traces the devel-
opment of a significant body of work produced spanning a twenty-five-year period
from this initial project to some of my most recent installations, discussing my ideas
and intentions; describing the functioning and operation of the work; and identify-
ing my influences, context and approach as well as the challenges and issues that I
encountered.
Keywords Renewable energy · Solar energy · Wind turbines · Video art · Video
installations · Video sculpture · Site-specific installations · Computer-generated
images
15.1 Introduction
Throughout the period that I have been engaged in making moving image and sound
installations, which includes all the works under discussion, I have aspired to produce
works which are temporary and ephemeral. In every case, the installations involved
the use of readily available objects and materials that were assembled to perform
and function for a brief and finite period of time, usually simply for the period of
the exhibition. The installations were subsequently disassembled and dismantled;
the equipment and materials were reused or recycled whenever possible. This ethos
is central to the work and to my intentions and continues to underpin my approach
Chris Meigh-Andrews (B)

University of Central Lancashire, Preston, UK
e-mail: meighandrews@btinternet.com

272 Chris Meigh-Andrews
to working with renewable energy and to my engagement and use of technological

systems within my work in general.
15.2 Perpetual Motion
Perpetual Motion (Fig. 15.1) was the first installation I made using a computer to
produce and present a moving image. Previously images within my work had been
originated using analogue video techniques, although I had increasingly made use
of digital effects and non-linear editing systems during the post-production since the
mid-1980s. Perpetual Motion was conceived for a “white cube” gallery space and
intended to be viewed within what I considered at the time to be a sculptural context.
As with my previous sculptural video installations Eau d’Artifice (1990–93), Stream
Line (1991–94) and Cross-Currents (1992), I hoped the work would be encountered
and interrogated conceptually, by which I mean that I intended visitors to become
engaged with an exploration of the elements and components of which the work was
constructed and to trace the “logic” of its presentation and functioning to reach an
understanding of the ideas and concerns of the work. Over time and with hindsight,
I have come to recognize that this idea was both problematic and presumptive, but
Fig. 15.1 Perpetual Motion (1994): Installation for computers, wind machine, wind turbine, video
projector and video monitor. Funded by Southern Arts, Oxford Photography and the Ontario Min-
istry of Culture & Arts, Canada. (Visiting Artist Award). Copyright © Chris Meigh-Andrews,
2019
15 Digital Moving Image Installations … 273
during this early period, I believed my work was sufficiently grounded in funda-
mental ideas and concerns as to be universally understood and decoded from “first
principles”.
A basic description of Perpetual Motion is as follows: a digital animation of
a flying kite was produced using “Adobe Photoshop” and “Macromind Director”
(animation software) and displayed as a continuously repeating loop on a ceiling-
mounted DC powered CRT video monitor which was connected to a 12 V battery on
continuous charge via a small wind turbine. The turbine was subjected to a continuous
blast of air produced by a large industrial fan at the far side of the gallery, which in
turn was plugged into a conspicuously located mains socket. A second computer-
processed video sequence of swaying grass was projected onto the gallery floor via
a ceiling-mounted video projector.
This work was subsequently exhibited in three different venues: Oxford Brookes
University and the Saw Gallery of Contemporary Art, Ottawa in 1994 and Castlefield
Gallery, Manchester in 1996 with the same basic configurations and components,
although each time requiring minor modifications due to the differing architectural
circumstances and logistics of the venue. In all three iterations, it was of central
importance that the wind machine produced a sufficient flow of air to turn the turbine
and provide enough current to maintain the charge to the 12 V battery powering the
CRT monitor displaying the kite sequence. This flow of air was also a significant
physical presence in the gallery, both in terms of the sound and the force of the airflow,
contributing to what I considered at the time to be the “sculptural” experience.
In her perceptive discussion of the installation for the exhibition catalogue, the
curator Lowena Faull identified many of the key ideas and tensions behind the work:
Perpetual Motion uses space in an intelligent way- using the dimensions of the gallery to
create a sculptural installation which poses a series of relationships between the viewer,
the physical presence of the objects and the technologies at work in the piece. There is a
flow of the viewer’s imagination as s/he makes associative leaps between the wind machine
driving the turbine and the image of the kite on the monitor. Meigh-Andrews has left creative
gaps in his work, so that the audience is left to create a simple technical narrative- how it
all works- and to create a narrative of meaning within the work itself…..the effect is both
meditative and engaging….This is a circuit of energy, imagination and visual representation
through which the artist comments on the representation of landscape, the desire of both art
and science to represent and imitate the natural world, to force from its disorder, structure,
to take its structures and create a more perfect replica. If there is an implied synergy /
dependency / inspiration between nature and the machine- there is also an implied critique
of that relationship [1].
15.3 Fire, Ice & Steam
In 1995, I was given the opportunity to develop a new work for the Middlesbrough
Gallery while engaged in an artist’s residency in Cleveland, North Yorkshire. The
installation of Fire, Ice & Steam (Fig. 15.2) occupied three linked spaces in the
Middlesborough Art Gallery, and although following on from Perpetual Motion, I
Fig. 15.2 Fire, Ice & Steam (1995): Installation for solar panels, halogen lamps and LCD screen.
Funded by Northern Arts and the UK Foundation for Sport and Art. Copyright © Chris Meigh-
Andrews, 2019
was keen to continue developing new work featuring renewable energy components,
only one of the rooms in the exhibition featured solar panels. As the focus of this
chapter is on the development of my work with renewable energy systems, I will
restrict my description to this aspect of the installation. The display in this room
consisted of four framed and wall-mounted photovoltaic panels, connected to a 12 V
“deep-cycle” battery powering a miniature LCD screen, displaying a repeating time-
lapse forward/reverse video loop of melting and refreezing ice cubes. The melting
and reforming ice was arranged to represent the written phrase “that time”. Each
of the four walls of the room containing a framed solar panel was lit by powerful
spotlights timed to switch on and off alternately, so that the light would sequentially
rotate in a clockwise manner around the space across the day during the period while
the gallery was open. The inspiration for this three-part installation was related to
the industrial heritage of the region, and I was keen that the work was understood to
be a representation of the complex and dynamic relationship between energy, light
and time.
This work provided me with my first opportunity to explore the potential of solar
panels as a sculptural element within a gallery installation and provided the initial
inspiration for further research. However, although the challenge of developing a
work using the energy produced by photovoltaic panels in Fire, Ice & Steam led to
further ideas, other installations took precedence (Vortex:1995 and Mind’s Eye:1996,
see http://www.meigh-andrews.com/installations), and it was not until 1998 that I
obtained the opportunity and the funding to develop Mothlight, a new installation
involving renewable energy.
15.4 Mothlight
My starting point for Mothlight (Fig. 15.3) was an apocryphal story about the ori-
gins of the term “bug in the system” to describe problems associated with com-
puter programming. According to the version of the story I came across, an early
prototype computer system at the computer lab at M.I.T. in Boston developed an
operational fault. Because the room-size machine’s electronic valves needed to be
kept cool during the long hours of calculation, all the windows in the lab housing
the machine had to be kept open and during the night flying insects had entered the
room, adversely affecting the operation of the computer. The engineer’s log for that
particular day included a description of the fault as being due to “bugs” in the appa-
ratus, and according to the story the log had even included the insect in question—a
moth, carefully preserved between the pages! The flying insects in Mothlight were
computer-generated, intentionally making what I considered to be an ironic refer-
ence to this bit of scientific history, alluding to the uneasy relationship between the
natural and the technological worlds.
Fig. 15.3 Mothlight (1998): Installation for solar panels, halogen lamps and CRT Monitors),
Funded by North West Arts, with support from the University of Central Lancashire. Copyright ©
C. Meigh-Andrews, 2019
Fig. 15.4 Mothlight, Rich

Women of Zurich, London
1998. Copyright © Chris
Meigh-Andrews, 2019
The decision to produce a computer-generated artificial moth was inspired by

this story, but it was also in keeping with my approach to representation in previous
works. (For example Eau d’artifice (1990), an electronic fountain created from sep-
arately recorded elements playing out over a simulated “day”; Streamline (1991),
an artificial stream made of nine separate but related sequences pieced together to
form a “narrative”, or the previously mentioned Perpetual Motion (1994) in which
kite, clouds, sky and grass were electronically combined to suggest a landscape.)
In these installations, themes of conflict and interrelationships between the natural
and artificial—“the made and the born”; between technology and nature—were at
the heart of much of my video work during this period. This review, published in an
Italian issue of “Flash Art”, identifies this relationship between the natural and the
artificial, suggesting an engagement with the transmutation of matter, and picks up
on both my fascination with creating parallels between the flow of matter and the
flow of thought that had been inspired by my interest in the ideas of the physicist
and philosopher David Bohm (1917–1994) [2].
In his research, (Meigh-Andrews) grew increasingly interested in natural images transfig-
ured through a series of manipulations which create an artificial, alchemical world. The
installations of the 90s focus, lastly, on a fundamental topic – the physic flux and its paral-
lelism with the mental flux, and the possibility that one activates the other. The installation
at Calci plays on this twofold aspect – some halogen lamps illuminate four solar panels
which feed some monitors which generate some moths. The whole process is an infinite cycle
that the spectator mentally builds through a linear series of logical passages. The thinking
flux also establishes a connection among spatially discontinuous elements. Mothlight puts
forward another characteristic also present in other works of the artist – the search for
contradiction, specifically the “ironic” exploitation of alternative energy. The viewer that
has patiently reconstructed the path of energy cannot miss the fact that the solar panels are
fed by the halogen lamps [3].
My video installation work at this time involved both sculpture and the moving
image in interrelationship. In Mothlight, I sought ways to highlight the interdepen-
dence of the elements which were the core of the work—the repeating cycle of
the fluttering moths and the suspended solar-powered video screens illuminated by
the halogen lamps were all connected to form an interrelated cycle of meaning. The
light was an important theme in this piece—illuminating, powering and conceptually
connecting the images and objects within the work.
As stated above, my interest in the relationship between technology and nature
was a major concern. In Mothlight the use of “renewable resources” was intended
to be subversive. The solar panels were not used to generate electricity but to act as
passive conductors which were transducing light from the domestic mains power-
point. In my thinking at the time, I felt that by inverting the “conventional” appli-
cation of renewable energy with electricity, I was serving the poetic rather than the
technological.
The structure of Mothlight was an attempt to make a work that suggested both
balance and movement, and I attempted to make the mobility of the linked elements
clearly visible insofar as was practically possible. The various elements which con-
stituted the work were held in a physical balance—the solar panels, the illuminating
lamps, and the video screens were arranged in counterbalance, in an attempt to make
the physical balance echo the conceptual balance of the interrelated elements.
The whole work was also of course a play on the idea of a “mobile”, and I
wanted to make reference to the image of the moth as an illusion of mobility. The
animated insect was tied to a predictable and endlessly repeating flight path, tethered
as much as the various functioning physical elements of the work were constrained
by the trailing cables and its relationship to the forces of gravity, and this illusion
of movement was at the heart of the piece. The video monitors were placed in such
a way that the viewer accepted (even if she/he knew better) the possibility that the
moth was flying across the gallery space—flitting from one suspended screen to the
other, and this was reinforced by a panning soundtrack and by the movement of the
animation through the illusory space of the TV screen. I also observed that it might
be possible to see the fluttering moth sequence as a reference to the flickering origins
of the film and video image itself and had aspirations that the work could be read
as a set of “nested” illusions—starting with the flickering origins of moving-image
technology and outwards through the illusion of movement via the mechanics of
perspective (both sound and picture) to the illusion of the mechanical mobility of the
sculptural form of the entire installation.
Mothlight was exhibited on three separate occasions: the Museum of Natural His-
tory in Pisa, Italy; the Glass Box Gallery at the University of Salford, Manchester;
and the Rich Women of Zurich, Hatton Garden, London. Each installation was pre-
sented in the same basic configuration, but with minor modifications to accommodate
the architectural and logistics of the different venues. Each gallery required some
adaptations and adjustments—some technical, others more practical. For example,
the Natural History Museum, located just outside Pisa in Calchi, was exhibited in a
space with an ancient ceiling and required a special fixing to be installed to support
the combined weight of the installation. In the Glass Box Gallery, the work could
only be viewed from outside the gallery through large windows on four sides of the
space. As the work was completely contained within this space, the buildup of heat
from the halogen light sources caused a significant voltage drop from the solar panels
necessitating the hiring of a portable air conditioning unit. At the Rich Women of
Zurich, the length of mobile support arms had to be modified to fit the gallery space.
In 2001 a variation of this installation was developed for the 291 gallery in Hack-
ney, London, replacing the CRT monitors with data projectors displaying the moth
flight sequences onto suspended sheets of translucent glass. The images of the moth
were sent via infrared to the projectors, and the transmitters were powered via the
solar panels, which I considered at the time added a further conceptual level to the
work, making a feature of the video signal “flying” through space from its source
to reach the screen. The scale of Mothlight II was more substantial and the greater
ceiling height in the 291 gallery provided an opportunity for a more monumental
installation, while retaining its ephemerality.
15.5 For William Henry Fox Talbot (The Pencil of Nature)
In 2002, I had the opportunity to realize a project that I had been considering for
several years. In response to an invitation to make a new work for “Digital Responses”
a group exhibition at the Victoria and Albert Museum, London, I proposed to install a
solar-powered live video camera at Lacock Abbey, in Wiltshire—the former home of
the pioneering scientist and inventor William Henry Fox Talbot. Electricity produced
from a solar panel was harnessed to power a digital video camera focused on the
large latticed window which had first been photographed by Fox Talbot in August
1835. The image from the camera was composed to exactly reproduce Fox Talbot’s
pioneering “photogenic drawing”, the world’s earliest surviving photograph. This
digital facsimile was relayed via an ISDN phone line to the V & A, the resultant
“live” digital image of the window presenting a full-size image of the window in
“real time”. This digital replica of the oriel window at Lacock was presented in
a special display beside an original copy Fox Talbot’s book, The Pencil of Nature
(Longman, Brown, Green & Longmans, London, 1844), the worlds’ first book to
be illustrated with photographs (Fig. 15.5). Researcher and curator Vince Dziekan
described his encounter with the work when it was installed at the Victoria and Albert
Museum in his essay “Distributed Aesthetics and the Tele-image”:
Fig. 15.5 For William Henry Fox Talbot (The Pencil of Nature), (2002): Site-specific installation
for solar panel, video camera, ISDN phone line, and data projector.) A site-specific installation for
“Digital Responses”, Victoria & Albert Museum, London, Sept-Oct. 2002. Curated by Paul Cold-
well. Technical Consultants: David Dorrington (Internet) and Richard Monkhouse (Solar energy).
Funded by The University of Central Lancashire, The Arts & Humanities Research Board and the
London Institute with sponsorship from Canon UK, Solar Century PLC, with support from British
Telecom, Bow Arts Trust and The National Trust. Copyright © Chris Meigh-Andrews, 2019
At first, this fleeting projection could just as easily be dismissed as a case of the morning light
outside being cast through the windows lining the length of this narrow gallery. Upon closer
inspection, however, the shadow play seemed uncannily to re-enact the exact characteristics
of this famous photographic image. To refer to this digital image as a representation seems a
somewhat inadequate description in that the image gently playing on the wall surface I was
facing involved the direct transmission of the light passing at that very moment, not through
the windows in the very room in which I was standing, but through the actual windows of
Lacock Abbey in Wiltshire, near Bath in the west of England. Titled, For William Henry Fox
Talbot (The Pencil of Nature), the work was an exact re-composition of Fox Talbot’s famous
‘photogenic drawing’, here captured by a solar-powered digital camera and relayed ‘live’
via an ISDN phone line to the gallery in South Kensington, where it was presented at actual
size in ‘real time’ [4].
With this installation, I wanted to imply the complex web of interrelationships

between art, technology, light, time and physical space and reference the origins
of photographic imaging and the nature and significance of light and vision and
its relationship to the flow of communication systems and to the interconnecting
of two geographically separate sites. The work sought to build on ideas developed
in previous installations such as Perpetual Motion and Mothlight which presented
“renewable energy” as a metaphor. In this new work, the daylight at the site of
Fig. 15.6 The author setting

up the video camera, Lacock
Abbey, Wilshire. Copyright
© Chris Meigh-Andrews,
2019
the abbey passing through the historic and culturally significant window at Lacock
Abbey set the entire work in motion, reflecting my intention to present an installation
in which the past, present and future were linked electronically, geographically and
conceptually.
15.6 Interwoven Motion
At the edge of a wooded area in Grizedale Forest, Cumbria, overlooking Coniston

Water in the English Lakes, a large tree was temporarily equipped with four video
surveillance cameras arranged in a circular formation around the trunk at the height of
approximately eight metres. The images produced by the four cameras were relayed
via a switcher to a weatherproof LCD video display mounted at the base of the tree.
The speed and direction of the camera image flow was determined by the velocity
and direction of the wind. The entire system was powered by a wind turbine extended
beyond the height of the forest canopy and four solar panels which were mounted
within the tree itself (Fig. 15.9).
Since beginning this period of work to develop renewable energy installations, all
of the works I had developed were necessarily intended to be for interior locations,
but for some time I had wanted to make an outdoor video installation that responded
to its environment [5]. Working with video artist Catherine Elwes and engineer Dr.
Fig. 15.7 Richard

Monkhouse and the author
on the roof of Lacock Abbey.
Copyright © Chris
Meigh-Andrews, 2019
John Calderbank in the early 1990s, I had conducted a period of research into the
feasibility of building a permanent outdoor video sculpture for the Chiltern Sculpture
Trail at Cowleaze Wood in Oxfordshire, although this early project did not proceed
beyond the report stage [6].
The notion of constructing an outdoor video installation in the landscape con-
tained many of the contrasting and contradictory aspects that I enjoyed working with
at the time and which continue to some extent in my most recent works. Interwoven
Motion juxtaposed the natural and the artificial and made use of technology which
was intended for interior use placed outdoors. I was interested in finding ways to
contrast the strength and fragility of the technology with the durability and vulnera-
bility of the tree it was temporarily connected to and the landscape it was part of. I
was also interested in highlighting and contrasting different notions of temporality,
permanence and impermanence. The specific video images produced by the instal-
lation were in themselves of no direct consequence—they were simply part of a flow
of very subtly changing ephemeral moments. For me, the relationship between the
light and the wind was at the core of the work. The light and wind provided the
Fig. 15.8 Interwoven

Motion, Grizedale Forest,
Cumbria, (2004). Copyright
© Chris Meigh-Andrews,
2019
source of the images both in terms of the generation of the electrical power which
supported the video and electronic apparatus and in terms of the direct physical and
visual experience which become part of the work. (Day/night, ambient light and the
movement of clouds, and foliage, the changing weather conditions, etc.)
It should also be noted that the work itself, like the image-sequences it pro-
duced, was transient. The components which constituted the work were temporarily
clamped to a living tree for a period of ten days. The various bits of inexpensive
technology—wind turbine, solar panels, video cameras, image switcher, LCD video
display, cabling, etc.—were temporary modifications, which, once removed, left no
trace. During the period in which the prototype installation was functioning, it was
left running night and day for as long as the technical systems remained operational.
Designed to be self-powering as long as the weather conditions provided sustaining
light and wind, the installation was equipped with two large-capacity rechargeable
batteries capable of powering the installation for approximately 72 hours. Located on
Forestry Commission land, it was relatively inaccessible except via an unpaved road.
From the distance, the solar panels and the wind turbine would certainly have aroused
the attention of curious by-passers. However, the casual visitor coming across the
installation would find no explanation or context for the piece, what it was, why it
Fig. 15.9 Interwoven Motion, (2004): Outdoor site-specific installation for four cameras, wind
turbine and solar panels. Commissioned for “ITEM”, Funded by the National Endowment for
Science, Technology and the Arts (NESTA) and Arts Council, England for The Foundation for
Art and Creative Technology (FACT) Liverpool, with support from Grizedale Arts, Wind and Sun
Ltd. and the Forestry Commission. Consulting Engineer, Dr. John Calderbank. Copyright © Chris
Meigh-Andrews, 2019
was there, or what purpose it might have. Visitors were free to respond (or not) and
to offer up their own explanation for its existence.
The location of the prototype outdoor video piece at Lawson Park was significant,
as the site was on land once owned by John Ruskin, the influential Victorian English
writer and critic. Ruskin’s passionate enthusiasm for the landscape of this area is well-
documented, not least in his published lectures and prolific diaries. His detailed and
evocative descriptions of the ceaselessly changing views of the “Old Man” above
Coniston Water and of the cloud formations and vivid skies provided me with a
compelling sense of this dynamic landscape—and provided me with a title for the
work:
From the west the wind blows fiercely towards you out of the blue sky. Under the blue space
is a flattened dome of earth-cloud clinging to, and altogether masquing the form of, the
mountain, known as the Old Man of Coniston. The top of that dome of cloud is two thousand
eight hundred feet above the sea, the mountain two thousand six hundred, the cloud lying
two hundred feet deep on it. Behind it, westward and seaward, all’s clear; but when the
wind out of that blue clearness comes over the ridge of the earth-cloud, at that moment
and that line, its own moisture congeals into these white—I believe, ice-clouds; threads,
and meshes, and tresses, and tapestries, flying, failing, melting, reappearing; spinning and
unspinning themselves, coiling and uncoiling, winding and unwinding, faster than eye or
thought can follow: and through all their dazzling maze of frosty filaments shines a painted
window in palpitation; its pulses of colour interwoven in motion, intermittent in fire,—
emerald and ruby and pale purple and violet melting into a blue that is not of the sky, but of
the sunbeam;—purer than the crystal, softer than the rainbow, and brighter than the snow
[7].
Ruskin’s vivid perception and appreciation of the lakeland landscape provided me

with a powerful connection to the cultural history of the site and was an important
element within the context of the work, deeply connected to a sense of its location. At
that time, I saw this project as the first step towards the goal of creating a landscape
installation that was an integral part of the landscape in which it was sited, responding
directly to and in relation to its location, which remains an aspiration to this day [8].
15.7 Resurrection
The subsequent invitation to produce and exhibit an installation incorporating renew-

able energy was once again within a gallery context and I took the opportunity to
develop a companion piece to the recently completed Interwoven Motion. A descrip-
tion of the elements and structure of the installation may help to visualize the work
(Fig. 15.10).
Fig. 15.10 Resurrection,

(2005–06): Site-specific
installation for solar panels,
DVD player and data
projector. Commissioned for
“Digital Discourse”, curated
by Vincent Briffa, St. James
Cavalier Centre of Creativity,
during the Commonwealth
Heads of Government
Meeting, (CHOGM),
Valletta, Malta. Funding
from the British Council and
support from the University
of Central Lancashire.
Copyright © Chris
Meigh-Andrews, 2019
A dead tree, complete with roots (approximately twenty feet high), was cut in
half, the root end mounted in the centre of the floor at one end of a rectangular
gallery space. The upturned tree and roots were lit by halogen lamps, casting strong
shadows on the opposite gallery wall. Over forty individual miniature solar panels
were arranged irregularly on the roots and connected in series, the wires grouped
and bundled and flowing down the trunk and in a cluster along the floor towards the
centre of the gallery. The clustered cables were connected via a junction box and
through a regulator to a twelve volt battery positioned in the centre of the space. The
battery was connected in turn to a DC powered DVD player.
The top half of the tree was mounted in the centre of the ceiling pointing down-
ward, its branches reaching down towards the floor with numerous small rectangular
sheets of heavy white paper fixed to the branches, arranged to resemble leaves. A
data projector mounted on the ceiling at the opposite end of the gallery was fed a
continuously cycling pre-recorded video sequence of the original living tree, com-
plete with leaves stirred by a fresh breeze. The sound of the wind in the leaves filled
the space and the projected image created a strong silhouette of the upturned tree on
the gallery wall.
This new gallery installation, entitled Resurrection, drew directly on the experi-
ence of building Interwoven Motion, bringing both the technological and the natural
elements back into the “white cube” gallery space to create a companion piece. The
living tree of the Grizedale project was conjured up in a revivified form within the
interior of the gallery.
The video images of the fluttering leaves in Resurrection presented a record of a
previously living existence, recreated via technology. The electrical energy used to
bring the resurrected tree back to life was transformed within the gallery space from
electricity to light and back again and the shimmering leaves were experienced as
both light reflectors and light receptors, the solar panels as both surrogate leaves and
transforming technology.
Resurrection was developed from the knowledge gained through the development
and making of previous works and shared many of the same conceptual concerns (and
components). In retrospect, it was a refinement of my approach to previous gallery-
based installations made prior to Interwoven Motion because it offered gallery visitors
a more pronounced sculptural experience. The technical and functional aspects of the
work were more integral to the spatial arrangement of the sculptural components.
In this sense, Resurrection was more akin to my aspirations with Mothlight and
Perpetual Motion and my earlier non-renewable installations such as StreamLine in
which I was conscious of the relationship to the architectural space of the gallery and
intent on achieving a balance between the metaphorical “function” of the technical
components and a perceptual (haptic) experience within the exhibition space.
15.8 Sunbeam
The environmental burden of electronic equipment, in terms of resources, manufacture,

energy use and waste has become a matter of concern for artists working with this equipment.
Some have developed works that variously recycle old technologies or decline to take their
power from the grid. Thus, Chris Meigh-Andrews’ SunBeam uses a solar-tracking array
(photovoltaic cells that follow the sun’s arc during the day) to power a night-time projec-
tion of processed images from NASA’s Solar Dynamics Agency, the content and the process
matching. The piece appeals to a cultural history of solar energy and the symbolic power
of plants, like the daisy and the sunflower, that track diurnal rhythms in their behavior and
growth [9].
I had been fascinated by the large tracking solar array installed adjacent to the
School of Dentistry at the University of Central Lancashire in Preston ever since
they had been installed in 2009. This solar array was able to track the sun as it
moved across the sky during daylight hours, substantially increasing its efficiency
[10]. During discussions with Dr. Robert Walsh, director of research at the Jeremiah
Horrocks Institute for Astrophysics and Supercomputing who was seeking ways to
publicize the institute’s work with NASA’s Solar Dynamics Observatory, I suggested
that we harness the energy from the solar array to produce a series of evening pro-
jections of the high-definition images of the sun produced by the observatory. My
proposal was to draw energy collected during the day to project images of the sun
back onto the surface of the array itself in a reversal of the process. Tests during
the development stage of the project revealed that this would require covering the
panels with a reflective material during the evening to provide an image of suffi-
cient brightness to be visible. The project would also require a high-powered data
projector and local police clearance to allow the images to be projected from the
building across a busy road from the array. For the screen, I ordered a custom-made
PVC sheet large enough to cover the surface of one of the two solar panels and
this was installed using a large hydraulic lift. At dusk on the evening of each event,
the covered solar array was rotated to face the university building directly across
the road in which a Christie high-definition video projector was installed on the
third floor. Edited time-lapse high-definition video sequences were projected onto
the array on four consecutive nights, drawing considerable crowds and providing a
spectacular demonstration of the power and majesty of the sun and of the university’s
collaborative research initiatives (Fig. 15.11).
My own aspirations for this work extended beyond this and were related to ideas
and concerns that link it to my previous renewable energy installation projects.
As Charlie Gere, professor of media theory and history at the Lancaster Institute
for the Contemporary Arts at the University of Lancaster, has observed, SunBeam
brought me closer to my conceptual goal of producing a technological artwork which
attempts to integrate the source of its energy with the images it presents, to celebrate
the harmonious relationship between light, energy and the fluid nature of matter in
general:
In SunBeam Meigh-Andrews now perhaps realises what the earlier works hinted at, an
artwork which both represents the prodigious energy of the sun and performs its effects by
Fig. 15.11 SunBeam, (2011): Site-specific digital projection event. (Tracking Solar array, data
projector.) Produced in collaboration with the Jeremiah Horrocks Institute for Astrophysics and
Supercomputing, UCLan and the Solar Dynamics Observatory, (NASA). Scientific Advisor: Dr.
Robert Walsh, with assistance from Dr. Stephane Regnier, David Henckel and Michael Dorricott.
With support from Astley Hire. Video editing of solar sequences: Cinzia Cremona. Copyright ©
Chris Meigh-Andrews, 2019
using that energy to make the representation possible…That the energy harvested during
the day can then be used to make an artwork possible beautifully encapsulates (Georges)
Bataille’s notion of art as a form of general economy exemplified in the sun itself. The system
that harnesses the sun’s extraordinary power for straightforward and restricted uses, such
as supplying energy to the university and to the national grid, is ‘detourned’ to produce
a work of art, or in other words something useless according to the restricted economy of
reciprocity and exchange. This is, perhaps, the very definition of art itself. [11]
15.9 Aeolian Processes
Aeolian Processes I and II (Fig. 15.13) were unusual within my body of work in that
renewable energy was harnessed to produce sound and neither installation included
any visual imagery [12]. However, both works followed my general approach of
seeking to produce a series of visual connections enabling the functioning of the
work to be decoded by following the operation of the elements from which the work
is constituted; hence the subtitle of the 2nd version of this work “Box Revealing
Fig. 15.12 SunBeam. Installing the temporary screen on the solar tracker, UCLan, Preston, May,
2011. Copyright © Chris Meigh-Andrews, 2019
the Sounds of its Own Making”, which makes a reference (and homage) to Robert
Morris’ 1961 sculpture [13].
Both installations had a definite physical presence and the various technical com-
ponents that produced the sounds were all visible and involved elements of movement
and change. As with all the installations discussed in this chapter, I was interested in
making a temporary sculptural object in which the visitor engages with the function-
ing and operation of the work in order to make sense of it. With Aeolian Processes,
I aspired to make something which was simple, direct and compatible with the land-
scape ethos of its location. Both installations were comprised of similar elements
and operations, although Aeolian Processes II is a refinement and development of
the earlier work.
Aeolian Processes I was commissioned for a group exhibition to be sited in a
large urban park, and consisted of a large wind chime suspended within the interior
of a glasshouse which was being continuously activated by the breeze from a small
domestic fan. A microphone placed close to the wind chimes picked up the sounds
which were then amplified, electronically manipulated and relayed to an outdoor
speaker installed on the roof of the glasshouse. All the electrical components were
powered via solar panels which were also installed on the roof.
Aeolian Processes II (Box Revealing the Sound of its Own Making) used the same
basic configuration of components as the previous version, but given that the work
Fig. 15.13 Aeolian Processes I (2013): Outdoor solar-powered sound installation, commissioned
for “Art in Your Park”, Highfields Park, Nottingham, curated by Theresa Caruana, with funding
from Arts Council, England. Aeolian Processes II (Box Revealing the Sound of its Own Making)
(2014), “La Lune: Energy Producing Art”, Long Reef, New South Wales. Curated by Allan Giddy
with funding from the Environmental Research Initiative for Art, College of Fine Arts, University
of New South Wales, Australia. Copyright © Chris Meigh-Andrews, 2019
was commissioned for a sculpture trail intended to be viewed after dark, the interior
of the glasshouse was illuminated. As the work required a more substantial power
supply, the solar panels were located on a separate structure which was positioned
adjacent to the glasshouse. The electronics were also refined; an echo device was
incorporated into the sound processing and an improved amplification of the wind
chimes enabled the sound to be heard from a greater distance.
In 2015, I began developing a series of small-scale sculptural installations under
the generic title of “Impossible Objects”, motivated by an interest in making works
which would not require a specific commission, purpose, venue or dedicated gallery
space, and with no major funding. Although these pieces have much in common
with earlier works discussed in this chapter in that they often incorporate or feature
renewable energy components in order to make connections to the theme of flow,
they are also more directly centred on notions of “process” as a central concern.
They are “Impossible Objects” not because they cannot exist (as they clearly do),
but because they make use of or refer to a process that contains a contradiction
or presents an “impossible” idea. They are representations of a state or situation
that cannot be achieved, except through the processes and agency of art. In this
respect, I have been influenced in part by the Mono-ha works of the Korean artist Lee
Ufan, in which there is an encounter between different materials—“a relationship
of tension” in which the work is the site of the encounter. In common with my
approach to my large-scale installations, all of these “Impossible Objects” are hybrid
installation/sculptures made using domestic technology; temporary assemblages are
made using readily available materials and equipment. Most involve moving images
and some have sound. Although there are at the point of writing five in the series,
only two are relevant to the themes in this chapter.
15.10 Impossible Objects
Intended to fit onto a tabletop, Impossible Object Number 1 consists of a symmetrical

arrangement of two solar panels top lit by two anglepoise lamps (Fig. 15.14). The
panels provide a continuous charge to a 12 V battery which provides the electrical
energy to drive a small electric motor geared to turn a crank operating a miniature
music box mechanism which repeatedly plays the first few bars of the tune from
(John) Lennon’s “Imagine”. The work employs a similar ironic reference to that
of my earlier large-scale installations Mothlight and Mothlight II, i.e. the notion
that solar energy (or more accurately renewable energy as a whole) will solve the
current environmental crises. Although the work is perhaps more directly critical
when compared to the earlier works, it is also more whimsical.
Similarly, Impossible Object Number 2 (Fig. 15.15) continues this more directly
cynical perspective with respect to the misplaced optimism regarding the role of
renewables to “save the planet”. Physically, the work is more of a sculpture than an
Fig. 15.14 Impossible Object Number 1: Imagine (No Pollution) (2016–17). Anglepoise lamps,
solar panels, battery, electric motor, music box. Funding: First Site Collectors Group Bursary,
(2015). Copyright © Chris Meigh-Andrews, 2019
Fig. 15.15 Impossible Object Number 2: Blue Sky Thinking (2018–19). LCD screen, media
player, wind turbine, battery and electric fan. Copyright © Chris Meigh-Andrews, 2019
installation. A black 92 cm × 3 cm × 8 cm wooden beam leans against a gallery wall

at a 45 degree angle, with one end sitting on the floor. At the bottom end nearest the
floor, a compact domestic electric fan creates the air current to turn a toy wind turbine
connected to a battery powering a media player connected to a miniature LCD video
monitor, which is arranged so that it must be viewed through a magnifying glass.
The monitor displays an endlessly cycling video sequence of a full-size wind turbine.
The video presented on the screen was recorded using a rotating mount locked to
the speed of the turbine rotors, so that the surrounding landscape appears to rotate
while the turbine blades remain static. As with all the other installations discussed in
this chapter, both works are conspicuously plugged into the gallery’s mains power
supply.
These two small-scale sculptures clearly draw on the ideas, techniques and
approach of the earlier installations, and although they are less ambitious in scale
and execution, they were intended to be more direct in their intention and content.
They might be understood as maquettes, as it would be conceivable to create larger
versions of both of these pieces, and at a larger size they might be more challenging
to experience in a gallery or outdoor setting and perhaps even more effective. Certain
elements would need to be reconfigured and the formal arrangements would require
revising and modifying, and in an upscaled format both of these “sculptures” would
almost certainly, in my view become “installations”.
Fig. 15.16 Impossible Object Number 2: Blue Sky Thinking (2018-19), detail. Copyright © Chris
Meigh-Andrews, 2019
15.11 Conclusions
Although all of the works discussed in this chapter inevitably have a level of critical
engagement with the impending current environmental crises, and is therefore rele-
vant to any reading of the work, in very few cases was this aspect my sole motivation
or even my central concern. The photovoltaic panels and wind turbines in my instal-
lations and sculptures have been employed for their ability to transform or transduce
energy from one form to another; from light or flowing air into electrical energy. In
all of these installations, I have attempted to create or make reference to the flowing
movement of matter and its parallel to the fluid movement of time. In relation to
this, I wanted to find ways to create an awareness of the process of perception that
takes place in the mind of the viewer while she/he actively engages with the work
and participates in its potential to exist and/or function.
It seems as if there is something to be said about the terms I have used to describe
the various works. I have written about “installations” because I feel that most of the
work I have developed that makes use of or features renewable energy systems are
works which require the visitor or viewer to engage in a kind of perceptual action—
reading or following a line of thinking and making links and connections between
elements to reach a conclusion or arrive at an understanding. This could be seen as

a kind of narrative activity or at least a time-based process of reading and making
connections. A sculpture, on the other hand, requires that it is seen and understood—
or at least perceived “all at once” as an object, and then perhaps unravelled and
deconstructed, after the initial encounter.
At the point of writing this, I have tentative plans for further work to extend or
continue my line of enquiry which may perhaps result in new or further successful
developments. However, I believe the context of this work has changed considerably
since I began making installations which included renewable energy components.
The environmental issues have been brought into much sharper focus, as there is a far
greater public awareness of the danger to the environment posed by the use of fossil
fuels. For example, in my earliest work visitors were not always able to immediately
recognize the function of the solar panels as transducers of electrical energy, but
now these objects are so commonplace that they are not perceived as remarkable or
intriguing and their potential as a symbolic or poetic device has been considerably
eroded. This shift in consciousness makes the environmental dimension of the work
too dominant, and weakens its impact and potential, and this requires that I either
accept this and move on, or try to discover a new level of signification and relevance
for them in my future work.
References
1. Faull, L.: Digital meditation: imagine technology as art. Back/Slash, Castlefield Gallery,
Manchester, Sept–Nov (1996)
2. Bohm, D.: Wholeness and the Implicate Order. Routledge and Kegan Paul, London (1980)
3. Leoni, C.: Flash Art (Italy), Summer (1998, translated from Italian by Cremona, C.)
4. Dziekan, V.: Distributed aesthetics and the tele-image. Remote, Plimsol Gallery, Centre for the
Arts, Hobart, Australia, June (2006)
5. In this aspiration I have drawn on two seminal artist’s films, both made in the 1970’s: Michael
Snow’sLa Region Centrale and Chris Welsby’s Seven Days
6. Panorama, a feasibility study, Meigh-Andrews, Elwes & Calderbank, Arts Council of Great
Britain (1993)
7. Ruskin, J.: Lecture (1876)
8. Gilman, C.: Interwoven motion: steps towards a semi-permanent outdoor, self-powered video
installation in research. In: Gilman, C. (ed.) The Itemisation of Creative Knowledge. FACT
and Liverpool University Press (2006)
9. Cubitt, S.: Angelic ecologies. Millennium Film J. 58 (Fall, 2013)
10. https://www.uclan.ac.uk/news/assets/annual_review_2011.pdf. Accessed 29 May 2019
11. http://clok.uclan.ac.uk/4903/1/Sunbeam%2C%20Gere.pdf. Accessed 31 May 2019
12. Although I have worked predominantly with video, I have made other sound installa-
tions, notably “For John Cage” (2001), “End Repeat”, The Center for Contemporary Art,
Tallinn, (http://www.meighandrews.com/installations/1996-2001/for-john-cage) and “Impos-
sible Object Number 1” (2016)
13. Morris, R.: Box with the sound of its own making. https://www.metmuseum.org/art/collection/
search/689665. Consulted 07 June 2019
Chapter 16
From Digital Nature Hybrids to Digital
Naturalists: Reviving Nature
Connections Through Arts, Technology
and Outdoor Activities
L. Edwards, A. Darby, and C. Dean
Abstract This work considers how the arts and technology in combination can stim-
ulate connections in heritage gardens, and also nurture care for non-human nature.
The chapter divides into two overlapping parts. The first part describes and critiques
the design of Digital Nature Hybrid artifacts for interpreting gardens and exploring
nature. The second part builds on the first by showing how the challenges presented
by the Digital Nature Hybrids stimulated the design of Digital Naturalist workshops.
It shows the value of combining arts, digital technologies and outdoor activities
to support active engagements with non-human nature and to inspire the develop-
ment of knowledge and skills needed to attend to natural environments. Research
through design underpins the way of working and the project uses a critical approach
toward technology, to guide the design decisions. One of the insights is the value that
adopting this critical approach has in shaping both processes and designs.
Keywords Digital nature · Digital Naturalist · Digital design · Digital Nature

Hybrids · Design lens · Research through design · Technicity · Focal things ·
Focal practice · Commoditization · Instrumentalism · Non-human nature ·
Cultural horizon · Hegemonic structures
Received on September 13, 2019. Revised on September 30, 2019. This work was funded by
EPSRC’s Digital Economy programme (RCUK Grant EP/G037582/1), EPSRC, an EPSRC Senior
Fellowship (Grant EP/P002285/1, The Role of Digital Technology in Understanding, Mitigating
and Adapting to Environmental Change) and an ESRC Impact Acceleration Account award (for
Digital Naturalists: The Lost Sounds).
L. Edwards (B)
School of Computing and Communication, Lancaster University, Lancaster, UK
e-mail: liz.edwards@lancaster.ac.uk
A. Darby
Lancaster Institute for Contemporary Arts, Lancaster University, Lancaster, UK
C. Dean
Edge Hill University, Ormskirk, UK

296 L. Edwards et al.
16.1 Introduction
Digital and other technologies play an increasing role in human life and have become
the dominant paradigm in the industrialized world. Technologies shape our move-
ment and interactions, for example, the way we walk through streets, head down,
checking our phones. Each technology favors particular senses and pushes and pulls
at them for attention. The values and assumptions that are built into technologies
are osmosed into society and become accepted as norms by both individuals and
wider society, hence shaping the cultural horizon. Once they have been normalized,
technologies reproduce themselves because people take them for granted and are
conditioned to notice and value the benefits they bring.
Despite widespread acknowledgement of technology’s contributions, many writ-
ers over the decades, from Simmel [1, 2] and Marcuse [3] to Feenberg [4–6] and
Kahn [7, 8] have expressed concern at the effect of technology on humans and non-
human nature. Some draw attention to the impact of technicity (the prevalence of
and reliance upon technology) on the human psyche, others draw attention to the
framing of the world that prevails in this paradigm. Common themes are alienation
from the self, from others and from the natural world, and a tendency to perceive
every thing and every person as a resource to be harnessed and operationalized, and
an inability to see this happening from within the system.
The implications are particularly evident in human relationships with the natural
world. People spend more time in communion with their digital devices and less
time in nature, leading to a loss of familiarity, awareness and care for the natural
environment. As people’s care for a place dwindles, protection of that environment
declines, and people care even less, leading to an exponential cycle of deterioration.
Fortunately, the cycle can be broken, and the cultural horizon can be shifted, for
the benefit of humans and non-humans. Feenberg [4–6] argues that technical codes
can be reshaped through a democratic, participatory approach. His work builds on
Marcuse’s [3] which looks to artists to critique and confront systems and structures.
This chapter discusses digital designs which draw on these ideas, embracing crit-
icism of technologies in order to design differently. The aim of these designs is to
revive nature connections and re-energize the culture of amateur naturalists, to ensure
that the detailed local knowledge needed to safeguard environments and biodiver-
sity is not lost forever. The designs range from Digital Nature Hybrids intended to
stimulate senses and raise awareness of gardens and plant heritage to workshops
that meld arts, computing and nature encounters to raise levels of engagement and
connection to the natural world. These workshops bring together experts from the
different domains, and each iteration responds to location, participants and inputs
from the particular partners. Beyond the individual workshop, the project’s aim is to
strengthen a collaborative community of artists, designers, technologists, environ-
mentalists and educators to excite people about exploring their local area and the
wildlife around them.
16 From Digital Nature Hybrids to Digital Naturalists … 297
16.2 Background
The Industrial Revolution caused major changes to the economy and society that cas-
caded down to individuals’ everyday life. People moved from the countryside into
cities, becoming geographically separated from the natural world. The character of
work changed. Agricultural work had flexed in time with natural rhythms of the day,
across the seasons, but machine processes exerted a fixed, relentlessly unwavering
pattern, increasingly uncoupled from rhythms in the natural world. Previously people
made whole things, from start to finish, but the compartmentalized structures neces-
sitated by machine processes fragmented production, which meant workers carried
out discrete, repetitive tasks. Workers no longer saw the workflow from the start to
finish and this had consequences. They lost the autonomy to manage their time, they
became deskilled, and as a result, were less invested in the production process.
Simmel contended that workers experienced a sense of disconnection in work that
affected their identity and extended to their sense of self. The intensity of industrial
life distracted and bombarded humans [1] to the point where they had no mental
space for reflection and they withdrew emotionally from work and other aspects of
life, including the natural world. The narrative of disconnection was later revisited
by Marcuse [3] in “One Dimensional Man”, which described how technological
structures overwhelmed life, to the point where anything that did not fit the dominant
technological worldview was squeezed out. In this setting, every thing and every
person came to be viewed as a resource to be harnessed and operationalized. This
created a situation where people were unwittingly trapped and consumed in their
never-ending pursuit of a lifestyle, associated with technological progress, which
was promoted through advertising and marketing.
Some, like Heidegger [9, 10], have asserted that technology has a particular charac-
ter that determines these behaviors and it is so pervasive that retreat from technology
is the only way to halt disconnection from the natural world, alienation from one’s
self and damage to the psyche. Heidegger argues that technology induces a mindset
in which everything is seen as “standing reserve”, a resource to be consumed, and
the essence of things is lost or diminished by this mentality. This perspective implies
technology is an external force that acts independently of society to determine partic-
ular behavior and values. It has since been shown that technology isn’t acting alone
in determining behavior but is heavily influenced by society [4–6]. Crucially this led
to the possibility that digital technologies could be imbued with a different set of
values, prompting alternative behaviors.
This chapter discusses research that explores the use of a critical approach to
the design of digital technology for interpretation of gardens and the natural world
in support of connection to place and non-human aspects of nature. In Marcuse’s
critique of technological society [3], he framed artists as outsiders, with the distance
and perspective necessary to uncover and challenge the pervading conditions that
limit human life. Some fields of art and design have become part of the (hegemonic)
structures that perpetuate the cycles of desire that Marcuse criticized, exemplified
by “affirmative design”, which serves the needs of industry without challenging its
structures [11, 12]. Critical design, by contrast, asks questions about society, its
values and structures and critiques culture through the medium of design. This role
has traditionally been the domain of art, so it has been argued that this pushes design
toward artistic practice [12]. Dunne and Raby have been drivers of critical design,
through projects such as Foragers, which asks questions about humans’ futures in
the time of food shortage and overpopulation [13]. In recent years speculative design
and more specifically design fiction, which challenge the status quo or ask “what if”
questions about potential futures that have gained traction.
This chapter describes work that is derived from the same theoretical roots as
critical design but diverges in approach. Where critical design critiques society and
speculates on futures through designed artifacts, the work discussed in this chapter
attempts to design differently, within the parameters of the current world, by using a
design lens that keeps the perceived problematic aspects of technology to the fore,
while making conscious efforts to address these problems. This approach focuses on
the implications of design process as well as design outputs.
The remainder of the chapter discusses practice-based research conducted by the
author that explores the use of a critical approach to the design of technology in
the context of environmental communication. The research which began as the lead
author’s thesis research [14] in 2013, and is ongoing as post-doctoral research, can
be divided into three phases: a short initial phase of exploration and prototyping;
a five-year collaboration with a team of gardeners at a National Trust property in
Nottinghamshire; and development of thesis findings as a research strand within a
program researching digital technologies and environmental change. The final phase
has a primary focus on Morecambe Bay, Lancashire but its reach has expanded
across North West England. The research marries the lead author’s background as
a Secondary School Geography teacher and a Higher Education researcher, lecturer
and multimedia designer.
16.3 The Design Lens
The design lens draws on recurrent discourse found in critiques of technology, as well
as themes that are specific to particular theorists. Some of the original arguments were
made prior to the ubiquitous presence of digital technologies, though they retain their
relevance. Describing all the design principles and the theories that underpin them is
beyond the scope of this chapter, but the following example shows the development
of the design lens.
16.3.1 Technologies, Things, People and Context
One of the starting points is that technologies differ substantially in character from
non-technological “things.” Devices, (Bormann’s term for technologies) [15–17],
are designed to be ubiquitous, so they can easily be used anywhere. The assets
they provide are foregrounded, while the controlling mechanisms are hidden away.
Without much thought or effort, they become absorbed into the background of our
lives. The freedom devices bring comes with a hidden cost. Devices are (intentionally)
detached from their context, in order that they can be used anywhere, by anyone but
this means they are not rooted in a culture and its values [15–18]. Technological
devices are a means of accessing/unlocking resources, be they reserves of fuel or
information and as a consequence, they frame everything as a commodity to be
consumed. Heidegger calls this “standing reserve” [9] and he believed that this lens
prevents us from seeing things in their own right.
By contrast, it is argued that (non-technological, handmade) “things” provoke a
different response, because a thing is of its world and is connected to that world
by a myriad of social, historical and material threads. Things are indebted to the
people, place, environment, history, culture, knowledge and traditions that brought
them into being and continue to sustain their life [19, 20]. When humans encounter
a thing, they are momentarily “interrupted” by it. They pause to notice and reflect
on it and the world that brought it into being. They see the thing in and of itself [20].
Where technology’s lens reveals everything to be a raw material to be harnessed by
humankind, in an ongoing, never-ending process, things illuminate the world that
surrounds them. Things are seen through their use, but technologies ultimately dis-
appear in their use [19]. Spinosa and Dreyfus [21] suggest that if technical devices
could behave more like other “things”, humans could be drawn into a positive rela-
tionship without themselves being turned into a standing reserve and losing their
capacity to be world disclosers. World disclosers [22] is a term used by Heidegger
to describe how things in the world become meaningful through everyday human
practices. Equipment has a purpose and when someone uses the equipment for the
given purpose, it gives an identity to that person [23]. Hence peoples’ identities and
sense of self are tied to things and practices.
16.3.2 Focal Things and Focal Practices
This connects to the concept of focal things and focal practices [15, 16]. The term
derives from the word “focus”, which formerly referred a hearth or fireplace, which
Borgmann used to illustrate the concept [15]. In the past, a fire was the focal point in
a home, and people were brought together around it. It was the focus around which
families gathered, shared news, told stories, played games and connected with one
another. Maintaining the fire took effort, exertion and learned skills, from cutting
and collecting the wood, to cleaning the hearth and setting the fire. The fire and
its activities create sensory experiences, from the smells of the burning wood to
the feeling of creeping warmth around a room or muscles fatigued from using an
ax to split logs. Focal things like fires form a point of convergence for family and
community life, and a social world revolves around them. The world is increasingly
revealed, as the focal practice is habituated.
This is contrasted with technologies which require minimal effort, skill and
engagement [15, 16]. Switching on a central heating system does not provide the
same level of engagement and heat distributed through radiators allows a family to
spread out across different rooms. Central heating gives the liberation of privacy and
instant heat, at a cost.
16.3.3 Commoditization
Feenberg [4, 5] suggests that there are ways to challenge the tendency toward techno-
logical de-worlding that results in wholesale commoditization. Rather than stripping
away to reduce something to a raw material (termed Primary Instrumentalization),
he advocates incorporation of the social into the technical so that technologies reveal
more than commodity. Secondary Instrumentalization integrates the social environ-
ment, through systemizations that disclose connections to the world, and valuative
mediations that provide space for societal values to inform designs.
In pre-industrial times when things were made in the place where they were used,
the production networks were short and local cultural values and priorities were
integrated and carried through production. In more recent times, technical networks
involved in making digital devices are longer, often spanning the globe, and so the
societal and cultural meanings and values that ground things and give them meaning
beyond that of commodity may be lost along the way [5]. That said, longer production
networks potentially involve more people and present increased opportunities for
social interventions [5].
Feenberg argues that more voices are needed in the design process to represent a
greater range of values and counter the control exerted through technical systems, a
concept he called democratic rationality [4, 5]. From this perspective, technologies
need to be open enough that they can be shaped or reconfigured by users to reflect
their priorities. In this way the social is inevitably part of the design.
16.4 Using the Critical Lens to Establish Design Guidelines
Turning theory into practical guides to shape design is subjective because different
designers will interpret and emphasize the theory in different ways. This subsection
presents some selected design guides from the author’s interpretation of the theory
above.
Technological devices require little effort and skill so designs should be effortful
and encourage skill development.
Traditionally technology designers have determined what a design is and how it
is used. This potentially reinforces power inequalities and excludes the input of users
who will be affected by design choices. So, technologies should be flexible enough
for wider society and users to influence and reshape designs.
Technologies slip into the background offering easy access to commodity which
means there is no pause for interruption and reflection, so we lose sight of things,
their world and ourselves. Consequently, designs or design process should create
pauses and interruptions that nudge people into noticing things afresh and attending
to them with more intent.
Digital technologies don’t require deep engagement or connection to community
or place so digital technologies should support focal practices and integrate social
dimensions.
Technologies have been criticized for creating distance and abstraction. To counter
this, designs should be rooted in context by, for example, medium, materials, pro-
cesses and values. In order for a design to connect to its world, production networks
need to be short to stay close to local culture and values.
Sensory stimuli connect people to the world of a thing. Smell, touch and taste are
particularly associated with the immediate body space surrounding a person, so they
are associated with immediacy and intimate connection. In contrast, visual stimuli
are believed to create a sense of distance, between the seer and the seen [24]. Hence
designs should encourage more intimate experiences and activate multiple sensory
receptors.
16.5 Artifacts
These and other criteria informed the design of a series of objects for personal use
and for interpretation of a public garden, including the Nature Meditation Egg, the
Rhubaphone and Audio Apples [14, 25–28].
The Nature Meditation Egg (Fig. 16.1) was the first artifact made within the
initial exploration and prototyping phase. It was designed for personal use by the
lead author to explore the relationship between theory and practice in her role as
designer–researcher. The Nature Meditation Egg was designed to encourage its user
to notice the natural world. This involved the user audio recording the non-human
Fig. 16.1 Nature Meditation Egg. Image copyright © J. Lindley 2015 and reproduced by permission
world on a walk in their local area. During meditation, the user would hold an egg-
shaped wooden object which contained a sensor, microprocessor and audio player.
When held with focus for a period of time, the audio from the walk would begin to
play. The aims were to stimulate memories of the outdoor experience with the audio,
and through the act of recording and listening draw attention to the natural world at
a fine level of detail. The wooden egg was turned from local wood by a local joiner,
though the digital components were manufactured elsewhere.
The Rhubaphone and Audio Apples were made as digital hybrid interpretation
for a public walled kitchen garden, to encourage people to notice it in new ways
and support connection to place. The Rhubaphone (Fig. 16.2) was created to bring
to attention the individual cultivars in the National Rhubarb Collection, grown in
the garden. The Rhubaphone held stems of rhubarb side by side so that differences
in size and color were more obvious. Holding a stem activated an audio recording
about that variety, its history and cultural significance made by the Head Gardener.
The smell of the cut rhubarb and touch of the stem added extra sensory layers to the
experience.
The Audio Apples took people into an Apple orchard, that held part of the Regional
apple tree collection, but the design also spoke to the past and present of the garden,
through audio stories, diary entries, audio recollections and poems recorded by gar-
deners and volunteers (Fig. 16.3). The wooden Audio Apples hung from branches
encouraging visitors to step off the path to reach and pluck them from the tree. Once
plucked the different voices of gardeners and volunteers “spoke” to the visitors,
through a variety of different kinds of recordings, intended to light a spark in the
listener.
The Rhubaphone and Audio Apples used wood from the site and the apples were
turned by a member of the team. The heritage of the garden was incorporated through
motifs that referenced the garden’s past. The design process was shaped to fit the
working patterns in the garden and seasonal shifts within the garden. The iterative
design process involved feedback and reflections from the Gardens team, and later in
the collaboration, a volunteer learned to maintain the digital interpretation artifacts
Fig. 16.2 The Rhubaphone in use. (Left) Image copyright © A. Johnson 2015 and reproduced by
permission. Rhubarb cultivars. (Right) Image copyright © L. Edwards 2019
Fig. 16.3 Audio Apples in use. (Left) Audio Apples in the orchard. (Right) Image copyright © L.
Edwards 2015
and ultimately began to develop content and new ideas. Hence ownership over the
artifacts started to shift from researcher–designer to the Gardens team. The software
used was open source so that code wasn’t locked down and could be adjusted or
completely rewritten. Content was stored on SD cards that were familiar to people
from everyday use in cameras and phones. This meant people working in the garden
could control content with only limited technical expertise. Where possible compo-
nents were used that could be replaced to extend the life of the design or uncoupled
by the team to create new designs. However, through the iterative design process
it was found that pre-made boards incorporating microprocessors and sensors were
more robust and easier for laypeople to manage. The design and development of
these interpretation artifacts is expanded upon in Edwards [14].
16.6 Reflections on the Digital Nature Hybrid Designs
Responses to the interpretation and its impact on visitors, volunteers and staff were
collected through interviews, field notes and observations. The lead author (in her role
as designer–researcher) made notes reflecting on design iterations as part of a research
through design [29, 30] approach. She participated in the life of the garden and
documented and reflected on experiences, conversation and observations in relation
to the underpinning theory and practice in notes, audio recording and images made on
site, or after visits. The effectiveness of the designs, in conforming to design criteria
was evaluated with reference to these reflections as well as theory-driven thematic
analysis of interviews with the Gardens team and visitors.
The digital interpretation was positively received. Many visitors commented that
they had not previously realized there was more than one variety of rhubarb or they
hadn’t thought about rhubarb varieties. One of the gardeners said that the process of
creating the Audio Apples helped the team realize the value of their stories, and also
let them see how their colleagues experienced the garden. Both Audio Apples and
Rhubaphone prompted a pause in people’s passage through the garden, where they
stopped and attended to the installations.
The multisensory digital artifacts used touch, smell and sound to encourage a sense
of intimacy with the artifact and place. The design criteria resulted in a particular
aesthetic, that emphasized materials from the site. It produced technologies rooted
in place and not easily used elsewhere. For example, the Rhubaphone incorporated
heritage varieties only grown in the garden. Once cut, the rhubarb would begin to
wilt and over time it would be unusable. This meant the Rhubaphone could only be
used properly within the range of the garden, a characteristic that prioritizes the local
and particular.
Overall, the designs cohered with the design lens and had strong contextual rel-
evance. The design processes were a product of place, fitting daily and seasonal
patterns in the garden, and respecting the routines and practices of the team. At some
points during the collaboration, design and maintenance activities were carried out
in a shed in the garden in order to stay connected to the place and enable ongoing
input from gardeners and volunteers. This was a response to Feenberg’s comments
about the benefits of short production networks [5].
Material, software and technological choices were made with the gardeners and
volunteers in mind. The question of what would be compatible with their working
practices and what they might be able to use independently was kept in the fore-
ground when making design decisions. This enabled a transition toward designs that
were initiated by the team, though the team members who managed the digital inter-
pretation artifacts felt more time was needed to build skills to gain more control of
the software. The fact the designs were open meant their content could be driven
by the team from the ground up, so that their priorities were centered. Stories and
content could emerge from the garden.
The interfaces and interactions were described as “fun” and “novel” and they
interrupted users, as intended, triggering memories and prompting questions. The
use of “actual rhubarb” brought the Rhubaphone to life.
16.6.1 Challenges Arising Through Use of the Critical Lens
Although the unfamiliarity of the interfaces caused people to pay attention, it also
created barriers to use. The Audio Apples definitely didn’t slip easily and effort-
lessly into the background and this was off-putting for some visitors and therefore
potentially problematic for the organization.
The technologies were designed to reveal the world of the garden, its values
socially, historically and culturally but there were tensions as the interactive instal-
lations could be critiqued as turning the things in the garden into commodities to be
consumed by visitors. This conflicts with the aims of the design lens.
The digital interpretation artifacts responded to many of the criteria drawn from
the theoretical lens, but they didn’t achieve some of the objectives in relation to
nurturing connection to place. The digital interactions created a pause, but often
this was fleeting and superficial and didn’t require enough active engagement from
participants to really prompt reflection on place. They didn’t offer many opportunities
for the kinds of extended, embodied engagements that create sensory surprises and
experiences which change relationship to place and build emotional connection.
16.6.2 Response to the Challenges Arising Through Use

of the Critical Lens
The Nature Meditation Egg was designed by its user (the lead author) and the process
of designing, creating and using the egg added to a richer understanding of place.
The relationship to place was stimulated by the act of design. The process required
effort, skill acquisition and significant time spent in a natural environment, which
are some of the characteristics of focal practices.
These insights from the Nature Meditation Egg about how the creative process
prompted reflection and seeded rich emotional relationships to the land influenced
the next phase of work. The realization that the act of designing the artifacts unlocked
understanding of place resonated with Heidegger’s writings about the nature of
humans as world disclosers, who illuminate worlds through practices and make vis-
ible the conditions on which things in the world depend. Making artifacts or stories
in the garden required the creator to give time to the process and attend to the place.
It involved sensory and experiential engagements that prompted emotional connec-
tions and a relationship to place. It isn’t possible to control placemaking because
relationships to place are individual and intangible but there is an argument that some
conditions are conducive to nurturing connection. Active engagement appeared to
play a significant role, and this insight echoes findings from earlier environmental
education projects like Ambient Wood [31].
Reflecting on the initial Digital Nature Hybrid artifacts drove a new research phase
about how digital technologies might support active engagement and exploration
toward greater awareness of and connection to non-human nature and environments.
This led to the inception of the Digital Naturalist program, which began in the
public garden where the original work took place but has subsequently been further
developed in a range of settings.
16.7 Digital Naturalists
The Digital Naturalist program comprises a series of workshops that combine arts
activities, with digital making, computer programming, nature walks and fieldwork
activities to reveal new perspectives on a place and rekindle naturalist skills. Cur-
rently, there are two workshops in the Digital Naturalist program, Digital Boggarts
and The Lost Sounds. The program builds on the insights gained from the Digital
Nature Hybrid artifacts. The pilot project (Digital Boggarts) which began in 2016
extended the collaborative work with the National Trust gardeners and with writ-
ers who had developed content for the Audio Apples. The second project (The Lost
Sounds) has run in multiple locations across the North West and Midlands with a
variety of audiences.
16.7.1 Background
The number of amateur naturalists has declined with successive generations [32]
and children spend less time playing outside ranging across smaller territories than
in previous generations [33]. Consequently, less than a quarter of children regularly
use their “local patch of nature”, which is half the number in their parents’ genera-
tion [34]. The Digital Naturalists workshops were designed to explore how digital
technologies might be used to stimulate more sustained and effortful interactions in
natural environments, with particular attention to sensory awareness. The workshops
slow people down in order that they spend time in a place, while the activities aim
to tune people into noticing details. The nature walks are led by local experts from
environmental organizations in order to raise awareness of local groups. Participants
may be more likely to get involved with these groups, as a member or volunteer, if
they have made personal contact with someone in the organization.
In the program, digital technologies are used to help people experience the natural
world afresh by revealing things that may be otherwise hidden and by supporting
the acquisition of traditional skills that bring a new perspective on the natural world.
The aim is to inspire a new generation of amateur naturalists and citizens who have
increased awareness of their local environments.
Thinking through making is used to focus attention, consolidate knowledge and
reflect on a place. The modular structure enables the workshops to be adapted for
the specific contexts and audiences. Versions of the workshops have been created for
formal and informal education and public engagement settings.
As the workshops have progressed, the importance of sharing the knowledge
and products resulting from the workshop, through an exhibition, has become more
apparent. The exhibition extends the reach of the program to the wider community
beyond workshop attendees and also presents opportunities for workshop participants
to act as/or be experts.
16.7.2 Digital Boggarts
Boggarts are folkloric creatures, like hobgoblins, that are often associated with a
particular location, for example, the Farndale Hob from North Yorkshire [35]. In
effect they are spirits of a place. Traditional tales about Boggarts can be found across
the United Kingdom making them a useful conduit for talking about place, nature
and history across the country. The pilot project, Digital Boggarts, drew attention to
microclimates and planting within a public garden, and the spirit or feel of the place
experienced as a result of these factors (Fig. 16.4).
The project/workshop offered an opportunity to explore a small part of the garden
in detail and build an intimate personal relationship with it. This study of a garden
at micro level threw into relief the contrasting character, (or spirit) of differing areas
within the garden to make these places present to the workshop participant. The
Digital Boggart was a manifestation of the kind of creature that might live in that
microclimate, a response to the question “Who would live in a place like this?”
At first a wilder setting might seem more suitable than a formal, managed public
heritage garden, but many people in the UK have access to a private or communal
garden or yard, and gardens are often the most immediate point of contact with
non-human nature. The public garden was far larger than a domestic garden but the
scale made it easier to identify different microclimates. One of the aims was that
participants might start to notice the microclimates and the spirit of different areas
within their own gardens or communal spaces. Another aim was to encourage new
visitors to the public garden.
Twelve children aged 9–10 from a primary school located close to the public
garden took part in the 1.5-day workshop, some was delivered in school, while
the major part was delivered in the garden. In school, prior to the garden visit, the
children learned to program Micro:bit computers to sense temperature, humidity and
direction. The day in the garden began with a guided walk and mapping activity, led
by gardeners. This was followed by time exploring the garden with sensors to identify
microclimates.
Using information collected in the garden as inspiration, the children worked in
groups with a writer to make a fictional story about a Boggart that lived in an area of
the garden. They performed the story in the garden and made an audio recording. The
children created Boggart creatures from clay, beans, seeds and materials found in
the garden, so that the character was rooted in the garden. Each Boggart was given a
digital dimension in the form of a Near Field Communication (NFC) tag embedded
Fig. 16.4 Images from the Digital Boggarts workshop: sensing humidity, story making and a Digital
Boggart incorporating Near Field Communication tag. Image copyright © A. Darby 2019
in its body. When the Boggarts were placed on top of upturned flower pots (that
housed hidden NFC readers and speakers), the story created by the group played
aloud. In effect, the children created a piece of artistic digital interpretation inspired
by their experiences and exploration in the garden. Interpretation is commonly used
within museums, heritage sites and cultural organizations as a kind of communication
that happens in conjunction with direct, firsthand visitor experience. Traditionally, it
presented the voice of the expert, in a range of forms but there has been a move toward
a more collaborative approach to include more voices alongside that of the trained,
experienced interpreter [36, 37]. The idea behind this is the inclusion of excluded
histories and the recognition of multiple perspectives on place. Back at school the
children made a display of their interactive Digital Boggarts to share stories of the
garden with their wider community.
16.7.3 The Lost Sounds
The Lost Sounds encourages people to pay attention to the birds in a place through
their calls and songs (Fig. 16.5). The name has a twofold meaning. It refers first
to the increasing number of birds on the International Union for Conservation of
Nature “Red List” of species at risk of extinction. In 2015, 67 of 244 species were
on the Red List, compared with 52 species, 6 years before in 2009 [38, 39]. This
includes birds such as the Curlew, Nightingale, Merlin and Mistle Thrush. As their
numbers decline, their songs and calls are lost to our localities and over time they fade
from consciousness. The Lost Sounds is as much about the lost skill of recognizing
birdcalls and the songs that are “lost to us” (as the number of amateur naturalists
decline) as it is about the loss of particular species.
Fig. 16.5 The Lost Sounds interactive birdcall prints. (Left) Image copyright © L. Edwards
2019. The Lost Sounds mini exhibition. (Right) Image copyright National Trust Images 2019 and
The Lost Sounds [40] has been run in formal education settings at primary schools
and a nursery and informal education settings including community festivals and
events run in partnership with environmental organizations/charities.
It was originally designed for children in Year 5 targeting Key Stage 2 learning
outcomes in computing, art and design, design technology, music and science. It has
since been adapted for children in Year 3 and 4, nursery children and family groups.
It is also being reconfigured for after school Code Clubs as a way of blending natural
history and environmental science into the program. We anticipate compatibility
between the structure of Code Clubs and Digital Naturalist workshops that may
inspire the development of additional workshops.
The workshop has a similar structure to the Digital Boggarts, involving time
spent outside, combined with arts and technology. Specifically, the project includes
a guided nature walk, birdwatching, sound recording and editing, listening skills,
printmaking, circuit building and computer programming with Scratch running on
Raspberry Pi. As before, the aim is to build naturalist skills including careful obser-
vation and listening. The full workshop takes 2 or 3 days but shortened one-day and
drop-in versions of the workshop have been run in settings where the longer format
is unfeasible. This necessitates a smaller range of activities, but the workshop always
involves art with technology.
Ideally the workshop begins with a guided walk led by a bird expert or expert in
the local environment, but in extreme weather timings may shift. Participants watch
for birds with binoculars and take turns with a shotgun microphone and digital audio
recorder to listen and make recordings at intervals on the walk. The aim is to use the
digital equipment to tune people into their environment and make them more aware
of the birdcalls around them. The directional capability of shotgun microphones
helps participants to isolate the calls of individual birds, but they don’t require the
same degree of precision as parabolic microphones. Sight is the dominant sense for
most humans [24, 41–43] while sound is omnidirectional and dynamic making it
less precise and harder to locate without a visual cue. Locating sound can require
practice and effort. Using the microphone gives the holder a temporary “superpower”
of enhanced hearing that makes it easier to notice sounds that are initially unfamiliar.
As one becomes able to identify individual calls, it is easier to pick them out without
the microphone.
After the walk participants draw one of the birds found in the area from pictures,
sometimes a bird they have seen or heard. The aim is to look carefully and closely
observe the detail of the bird. The drawings (etched into foam) are turned into prints,
made with conductive ink. In shorter workshops, participants are taught to recognize
selected calls and then pin their print to an interactive sound sculpture, which plays
the call of the bird when the print is touched.
In the full workshop, participants either edit their own recordings or the ones made
available under a Creative Commons license. They make a circuit with a capacitance
sensor attached to the input and output pins of a Raspberry Pi computer and learn to
program it to play appropriate calls when prints are touched.
After these workshops, the participants’ prints and edited audio are transferred to
an installation comprising: Bare Conductive Touch Boards, speakers and fabric print
of a location visited during the workshop. The installation acts as a mini exhibition
of the outputs of The Lost Sounds workshop and it encourages involvement from the
wider community beyond workshop participants, (for example, parents and other
pupils in a school, or National Trust visitors or local residents). The interactive
soundscape installation has been designed in a form that is easy for external partners
to manage.
16.8 Reflections on Digital Naturalists
Both strands of the Digital Naturalist program, Digital Boggarts and The Lost Sounds
have been well received by participants, host organizations, teachers and parents in all
settings. Focused time spent in place, and time spent reflecting on non-human nature
is built into the design of the program, so this inevitably results in a more significant
experience for participants than that created by digital nature interpretations alone.
The combination of arts, computing and outdoor experiences is particularly potent
because the digital technologies like the audio recorders enable the participant to
experience the environment differently and reveal things that it would otherwise be
harder to sense. The digital technologies don’t supplant the traditional noticing skills
but awaken them, whether that is noticing sunny and shady parts of a garden or using
directional microphones to pick out the sounds of individual birdcalls. The digital
technologies are intended to encourage people to spend time outside in a focused
way. This doesn’t have the value of habitual focal practices, but it does require effort
while also contributing to skill acquisition that can change a person’s relationship to
a place and potentially their relationship to technologies.
Creative practices require different kinds of engagement. The drawing encourages
the participant to look carefully and notice the difference and the printmaking asks
people to think about the character and signs that communicate the essence of a
bird species. To write the Boggart stories, people have to think imaginatively about
the space which goes beyond recalling facts. The story outcomes may start from
jumping off points in a world that is close to the reality of the garden and then shift
into realms of fantasy, but both have an impact on the memory of the children and
the connections they make to the garden. A translation occurs as the participants
turn what has been shared with them into something that they own, and that process
involves the investment of the participants.
The layered activities provide different hooks for individuals. Some respond to
the making activities, while others are drawn to computing, but the combination is
intended to make each activity more appealing to a wider audience. The intention is
to consider how this interdisciplinary approach affects perceptions, particularly of
technology, computing and environment.
One of the most significant outcomes has been the impact of the sound installation
that creates a mini exhibition of project work. These have been left in schools or
National Trust properties for days or weeks after a workshop to open the project out
to wider audiences who did not participate directly in the workshops. In schools,
teachers and children talked about the pride of showing their creations to others
and sharing the birdcalls they had learned. The fact that the fabric image was of
the place where they had spent time outside birdwatching and recording also made
it more meaningful. Showing the work to others in the community, for example,
at Community Days can start conversations about birds in the area and how to get
involved with projects to protect their habitat.
The role of environmental and creative experts is also a key feature of the workshop
design. Local environment experts reveal what is hidden or unnoticed and when this
is paired well with technologies that also make the invisible visible (or audible),
they complement each other powerfully. Creative experts like writers or printmakers
introduce ways of seeing that may also unlock new dimensions of non-human worlds.
These inspire the active engagements that build care for the non-human world. The
creative activities create a space for conversations and questions about ecology and
action. Beyond the participants, the workshops bring together organizations and
individuals with different competencies who have an interest in the environment,
and this has the potential to strengthen networks and seed collaborative action.
The design criteria used at the outset emphasized that the designs should fit the
place in which they will be situated. In the earlier Digital Nature Hybrid interpreta-
tion, this included using materials from the site and designing artifacts with the local
team. This presents a challenge for the Digital Naturalists workshops because they
move from place to place. For example, the materials for the exhibition stand cannot
come from the location unless the stand is continually remade.
Nevertheless, the Digital Naturalist workshops have been designed to give promi-
nence to place and to respect individual contexts. For example, the Digital Boggarts
incorporate materials from the garden in which the Boggarts “live”. The fabric back-
drop for the Lost Sounds is reprinted with a site-specific photo for new workshop
locations. Working with experts from a site and planning the activities with partners
who know the location well is another way of ensuring that care for the place is
embedded, and that the use of digital technology is appropriate. The combination of
arts, outdoor activities and computing is important because at different times each
pulls place into focus. That means there is a tension when the shortened versions of
the workshops are run because an element is lost that may weaken the connection
to place. However, the shortened workshops have prompted bookings for the longer
workshops.
One of the ways that Digital Naturalists responds well to the original design lens is
that the workshops encourage attention, endeavor and the development of skills. The
commitment required in time and effort makes these engagements potentially more
meaningful than the brief encounters with the Digital Nature Hybrid interpretation.
In both cases, the participant’s intent and motivation will affect their experiences.
16.9 Conclusions
Technology has been criticized because of its instrumentalizing tendency, and the
implications of that tendency, for example, the increasing separation of humans from
non-human nature that results in a lack of care that puts the environment under threat.
It also includes the impact on humans’ sense of self and well-being. By responding
to the criticism directed at technologies through design criteria that referenced the
criticism directly, the projects aimed to refocus and reshape priorities and design
processes. The simple act of setting the criteria was important for foregrounding the
questions, “How are we using digital technologies?” and “What impact is this having
on humans and non-humans?”
One of the significant impacts of this lens was the cascade created by prioritizing
individual places and rejecting the idea that “one size fits all.” Attending to place
meant the design process had to respect local ways of working including the rhythms
of the place, for example, seasonal and daily working patterns. This in turn influenced
digital technology choices and other material decisions which ultimately influenced
physical designs and sensory interactions. By attending to place within the design
process, the designer–researcher (lead author) became more connected to the places
in which she designed. This led to the insight that making in a place and in response
to a place could foster a connection and a realization which drove the second phase
of research.
The initial Digital Nature Hybrid designs went some way to revealing places and
showing non-human nature in a new light. It demonstrated how digital technologies
could be rooted in context and culture and how they could amplify particular sensory
stimuli to resonate with the sense of place. This in itself had value, for designers,
organization and visitors.
However, the engagements with the Digital Nature Hybrids were brief and too
superficial to prompt significant change or connection, unless the people concerned
were already primed for change or connection. Analyzing the limitations of the
Digital Nature Hybrids with respect to the design criteria showed that effort, focus,
skill and social connection that made focal things and practices meaningful were less
present in the Rhubaphone and Audio Apples and so the question of how to build in
these attributes became central to the second phase.
The subsequent projects created deeper, richer encounters with natural environ-
ments and showed the power of combining artistic activities with technological activ-
ities. The activities required participants to use their technologically amplified senses
and creative skills to attend to and reflect on the world.
Using technologies to kindle traditional skills and practices shows one way that
technologies might contribute to re-energizing the culture of amateur naturalists and
nurture care for non-human nature.
Acknowledgements The work would not have been possible without partners and participants in
schools, arts and environmental organizations. I would particularly like to thank the gardeners and
volunteer gardeners at National Trust who contributed to this work.
References
1. Simmel, G.: The Metropolis and Mental Life. Blackwell Publishing, Oxford (2004)
2. Frisby, D., Simmel, G.: Key Sociologists, 2nd edn. Routledge, London and New York (2002)
3. Marcuse, H.: One Dimensional Man: Studies in the Ideology of Advanced Industrial Society.
Routledge & K. Paul, London (1964)
4. Feenberg, A.: Transforming Technology: A Critical Theory Revisited. Oxford University Press,
New York (2002)
5. Feenberg, A.: Between Reason and Experience: Essays in Technology and Modernity. MIT
Press, Cambridge, MA (2010)
6. Feenberg, A.: Critical Theory of Technology. Oxford University Press, New York (1991)
7. Kahn, P.H.: Technological Nature: Adaptation and the Future of Human Life. MIT Press,
Cambridge, MA (2011)
8. Kahn Jr., P.H., Severson, R.J., Ruckert, J.H.: The human relation with nature and technological
nature. Curr. Dir. Psychol. Sci. 18(1), 37–42 (2009)
9. Heidegger, M.: The Question Concerning Technology, and Other Essays. Harper & Row, New
York (1977)
10. Heidegger, M.: Only a god can save us: the Spiegel interview (1966). In: Sheehan, T. (ed.)
Heidegger: The Man and the Thinker, pp. 45–68. Routledge, London and New York (2017)
11. Dunne, A., Raby, F.: Critical design FAQ [Online]. http://dunneandraby.co.uk/content/bydandr/
13/0
12. Gonsher, L.: Beyond Design Thinking: An Incomplete Design Taxonomy [Online]. http://
www.cd-cf.org/articles/beyond-design-thinking/
13. Dunne, A., Raby, F.: Designs for an overpopulated planet: Foragers (2009) [Online]. http://
dunneandraby.co.uk/content/projects/510/0
14. Edwards, E.: Designing contextually relevant digital interpretation for a public garden,
HighWire Centre for Doctoral Training, Dept. Computing and Communications, Lancaster
University, UK (2019)
15. Borgmann, A.: Technology and the Character of Contemporary Life. University of Chicago
Press, Chicago (1984)
16. Borgmann, A.: The moral complexion of consumption. J. Consum. Res. 26(4), 418–422 (2000)
17. Borgmann, A.: Orientation in technology. Philos. Today 16(2), 135–147 (1972)
18. Borgmann, A.: Reality and technology. Camb. J. Econ. 34(1), 27–35 (2010)
19. Edwards, J.C.: The thinging of the thing: the ethic of conditionality in Heidegger’s later work.
A Companion to Heidegger, pp. 456–467 (2005)
20. Heidegger, M.: Building dwelling thinking. Poetry, Language, Thought, vol. 154 (1971)
21. Dreyus, H.L., Spinosa, C.: Highway bridges and feasts: Heidegger and Borgmann on how to
affirm technology. Man World 30(2), 159–178 (1997)
22. Heidegger, M.: Being and time. A Revised Edition of the Stambaugh Translation. Suny Press
(2010). First published (1927)
23. Spinosa, C., Flores, F., Dreyfus, H.L.: Entrepreneurship, Democratic Action, and the Cultivation
of Solidarity: Disclosing New Worlds. MIT Press, Cambridge, MA (1997)
24. Rodaway, P.: Sensuous Geographies: Body, Sense and Place. Routledge, London and New
York (2002)
25. Edwards, E., Coulton, P., Darby, A., Chiasson, M.: Planting and tending digital-nature hybrids
in a walled kitchen garden. Des. Issues 33(3), 65–78 (Summer, 2017)
26. Edwards, L., Coulton, P., Chiasson, M.: The inclusions afforded by a value-led design process
for technology-nature hybrids in a walled kitchen garden. In: Proceedings of 11th European
Academy of Design Conference, Paris, France (2015)
27. Edwards, L., Coulton, P., Chiasson, M.: Walking off the garden path: a design journey. In:
Proceedings of Research Through Design Conference, Cambridge, UK (2015)
28. Edwards, E., Coulton, P., Chiasson, M., Darby, A.: Growing an interpretation ecology in a
walled kitchen garden. In: Proceedings of the 2014 Companion Publication on Designing
Interactive Systems. ACM, Vancouver, Canada, pp. 65–68 (2014)
29. Frayling, C.: Research in art and design. R. College Art Res. Pap. 1(1) (1993/94) [Online].
http://researchonline.rca.ac.uk/384/3/frayling_research_in_art_and_design_1993.pdf
30. Durrant, A.C., Vines, J., Wallace, J., Yee, J.S.: Research through design: twenty-first century
makers and materialities. Des. Issues 33(3), 3–10 (Summer, 2017)
31. Rogers, Y., Price, S., Fitzpatrick, G., Fleck, R., Harris, E., Smith, H., Randell, C., Muller, H.,
O’Malley, H.C., Stanton, D. et al.: Ambient wood: designing new forms of digital augmentation
for learning outdoors. In: Proceedings of the 2004 Conference on Interaction Design and
Children: Building a Community. ACM, pp. 3–10 (2004)
32. Moss, S.M.: Natural Childhood. National Trust, London (2012)
33. Louv, R.: Last Child in the Woods: Saving Our Children from Nature-Deficit Disorder.
Algonquin Books, New York (2008)
34. Pretty, J., Angus, C., Bain, M., Barton, J., Gladwell, V., Hine, R., Pilgrim, S., Sandercock,
S., Sellens, M.: Nature, childhood, health and life pathways. Interdisciplinary Centre for
Environment and Society Occasional Paper 2009-02. University of Essex, UK (2009)
35. The Farndale Hob [Online]. http://www.farndalefamily.co.uk/Other/The%20Farndale%
20Hob.htm
36. Silberman, N.A.: Heritage interpretation as public discourse: towards a new paradigm.
In: Marie-Theres, A., Bernecker, R., Britta, R. (eds.) Understanding Heritage, pp. 21–33.
DeGruyter, Berlin (2013)
37. ICOMOS: The ICOMOS charter for the interpretation and presentation of cultural heritage
sites. Technical Report. International Council on Monuments and Sites, Charenton-le-Pont,
France (2008) [Online]. http://icip.icomos.org/downloads/ICOMOS_Interpretation_Charter_
ENG_04_10_08.pdf
38. Hayhow, D.B., Burns, F., Eaton, M.A., Al Fulaij, N., August, T.A., Babey, L., Bacon, L.,
Bingham, C., Boswell, J., Boughey, K.L., Brereton, T., et al.: State of Nature 2016. The
State of Nature partnership (2016) [Online]. https://www.rspb.org.uk/globalassets/downloads/
documents/conservation-projects/state-of-nature/state-of-nature-uk-report-2016.pdf
39. Eaton, M., Aebischer, N., Brown, A., Hearn, R., Lock, L., Musgrove, A., Noble, D., Stroud,
D., Gregory, R.: Birds of conservation concern 4: the population status of birds in the UK.
Channel islands and Isle of Man. British Birds 108, 708–746 (2015)
40. Edwards, E.: The Lost Sounds (2019) [Online]. http://www.thelostsounds.org/
41. Tuan, Y-F.: Space and Place: The Perspective of Experience. University of Minnesota Press
(1977)
42. Lewis, N.: The climbing body, nature and the experience of modernity. Body Soc. 6(3–4),
58–80 (2000)
43. Ingold, T.: Culture on the ground: the world perceived through the feet. J. Mater. Cult. 9(3),
315–340 (2004)
Part VI
The Use of Virtual Reality
and Augmented Reality to Extend
Creativity, Reach, and Engagement
in the Arts
Chapter 17
Virtual Reality Holography—A New Art
Form
I. Pioaru
Abstract Virtual reality holography (VRH) is a new art form that synthesizes the
qualities of traditional hand drawing with the unique features of holography and
virtual reality (VR) art. As a new art-making tool, it offers the possibility to develop
artistic ideas and concepts that could not be materialized with any other medium,
whilst from a practical point of view it has the unique ability to showcase VR artworks
external to VR space, without the need for a headset. Starting with a brief historical
overview of holography art and VR art, this chapter presents an exploration of the
VRH medium and the artworks which were generated. The various contexts in which
these artworks are disseminated are presented, and the dialogues arising from the
material specificity of VRH as a new medium are discussed. VRH is envisaged to
extend the reach of audience engagement.
Keywords Virtual reality holography · Digital holography · New media · Virtual

reality art
17.1 Introduction
Virtual Reality (VR) art-making tools have become increasingly popular amongst
artists. However, VR art is still far from ubiquitous, mainly because of the high cost
of purchasing a VR system and the space required to run it. At the same time, VR
artworks are fundamentally difficult to view or exhibit outside of VR, as they have a
functionality that makes them impossible to reproduce in any other medium without
losing some of their essential features. For example, 3D printing, due to its particular
constraints, only allows for a relatively narrow range of models to be produced [1].
Online galleries which allow the viewer to explore VR projects using a standard
monitor, flatten the three-dimensionality of VR artworks, as is the case with any 2D
representation of a volume [2].
I. Pioaru (B)
Wrexham Glyndwr University, Wrexham, UK
e-mail: ioanapioaru@geola.eu

318 I. Pioaru
Digital holography, despite its own limitations, can constitute good support for VR
imagery, offering a high fidelity of representation and preserving the most important
properties of such imagery.
The advantage of bringing together these relatively new media is that it allows an
artwork produced in VR to be displayed and viewed easily in either a public gallery
or a private space, by a much larger number of people simultaneously, without the
constraint of requiring viewers to wear a cumbersome headset in order to preserve
the 3D appearance.
17.2 Overview of Holography as an Artistic Medium
In simple terms, a hologram is a photographic recording of a light field, on either film

or glass, resulting in a seemingly 3D image which can be seen with the naked eye.
Holography can be either analog, when it captures something which exists in reality,
or digital, when it displays computer-generated objects or scenes. Both types of
holograms require a special, dedicated illumination system in order to be viewed, in
the absence of which they appear as dark, indistinct, flat surfaces. There are a number
of techniques and optical illusions which are commonly and mistakenly referred to as
‘holography’, such as Pepper’s Ghost or various stereoscopic and lenticular displays.
Although holography practitioners will reject these as mere tricks, the general public
is perhaps more familiar with such techniques than with holography itself, due mainly
to the fact that they are often employed in funfairs, in theatre and in other on-stage
acts. Arguably the most notorious use of Pepper’s Ghost was Tupac Shakur’s virtual
appearance at Coachella Valley Music and Arts Festival, in 2012, alongside Dr. Dre
and Snoop Dogg [3].
Although primarily a scientific field, holography has a significant potential for
being used as an art medium and, as a consequence, the domain appeals equally to
artists and scientists. Since its invention by Denis Gabor in the 1940s, holography has
drawn the attention of a number of artists, amongst whom Salvador Dali is perhaps
the best known [4]. Many of these artists have dedicated their career to the medium
of holography, for example, Margaret Benyon [5] or August Muth [6], whilst others
have tried to incorporate it in, or add it to, their existing practice, as in the case of
Moysés Baumstein.
Catalogues, photographic documentation and generally any type of literature on
holography don’t really do justice to a medium that doesn’t lend itself easily to 2D
reproduction. However, articles on holography written by practicing artists, curators
or critics are useful for understanding how holography was, and is, perceived and
experienced first-hand.
Two volumes of the Leonardo art and science magazine dedicated to holography
art [7] comprise numerous interviews and articles where holography art practitioners
amongst whom Margaret Benyon, Andrew Pepper and Paula Dawson express their
dissatisfaction with how the art world receives their work.
17 Virtual Reality Holography—A New Art Form 319
Perhaps the most striking concept that comes from these texts, and others written
later, is that holography and the art establishment still don’t feel comfortable with one
another. The medium of holography has encountered constant criticism and dismissal
from the art establishment. The main points made against it being vulgarity of colours,
awkwardness of display and lack of a proper critical and conceptual vocabulary. This
is revealed by D. Tulla Lightfoot in a study titled ‘Contemporary Art World Bias in
Regard to Display Holography’ [8], where the author interviews several curators
and directors of art galleries in New York, seeking to understand their views on
holography art and how they justify the constant rejection of this medium by the
establishment. The main conclusion of the study is that major museums and galleries
don’t necessarily dismiss the medium itself—they base their rejection on the status
of the artist, claiming that, if an artist who has previously gained recognition by
working with other media decided to take up holography, they, the art venues, would
not hesitate to show their work. If this is the case, one might infer that holographers
simply face the same problems as any other artist approaching a venue seeking
representation; with the added difficulty of having chosen a medium that hasn’t
really managed to prove its aesthetic or conceptual value, as yet.
Nevertheless, there seems to be an ongoing struggle amongst holography practi-
tioners to be accepted and valued or sometimes even considered by major art venues,
in spite of long years of sustained efforts and obvious developments, both in the
field of holography as a whole and within the individual practices of different artists.
During a presentation at the International Symposium for Display Holography 2018,
August Muth, one of the most successful artists working with holography today,
confessed that when approaching a gallery to propose a collaboration, he simply
steered away from using the word ‘hologram’ in reference to his artworks. This was
to avoid the immediate negative reaction and consequent rejection which are usually
triggered by the mere mention of this word.
It must be acknowledged that the medium still has glaring deficiencies and limita-
tions that make holographic artworks unconvincing and could be the reason for recur-
ring negative responses. For example, common limitations mentioned are brightness
and colour balance, the complexity of arranging good replay illumination, blurring
in the rear of a scene, vertical non-uniformity of colour, digital pixelization and the
lack of a good archival quality photographic recording material.
Perhaps one of the reasons why many attempts at making holography art have
failed, is that the practice of holography bears no clear line of demarcation for
where the science ends and where the art begins or vice versa. In order for a holo-
graphic product, and any new media artwork to be successful from both points of
view—technical/scientific and artistic—this distinction needs to be acknowledged
and embraced.
Some of the most successful examples of holographic art come from practitioners
who have understood the importance of limiting the role of the artist and that of the
scientist in order to allow each of them to contribute optimally in this equation.
A collaborative approach between artists and scientists is needed to enable the
two fields to intersect in a successful way that leads to further progress. New media
often come with the temptation to use their novelty and specificity as an objective
320 I. Pioaru
in itself. As a result, practitioners with a background in art are unable to produce

artworks of satisfactory technical quality, whilst technicians or scientists with no
artistic background, seduced by the visually enticing qualities of the media, produce
trivial objects of no real artistic value. When working with a difficult medium such
as holography, it is important that both the artists and the scientists understand the
limits of their knowledge, and by using each other’s expertise and providing constant
feedback, work together towards taking the technology to a level that allows it to be
acknowledged by the establishment.
17.3 Virtual Reality Art
To clarify the terminology, VR art is a type of artistic content created directly inside
the VR environment using an application designed specifically for the purpose of
art-making, and not computer-generated content presented in VR.
VR art-making applications such as Tilt Brush or Gravity Sketch are impressive
tools for creating digital 3D imagery, offering great new possibilities for art and the
creative industries. They stand out amongst other VR applications due to the fact that
they allow users to generate virtual reality content, rather than simply consuming
it. This unprecedented combination of painting and sculpture, not bound to physi-
cal laws, with its lack of spatial limits and highly intuitive interface, offers a very
satisfying immediacy that materializes gestures instantly.
With its impressive variety of brushes often accompanied by special visual and
sound effects, Tilt Brush was designed with the intention of offering the user not
just art tool but a fully immersive VR experience, entertaining in and by itself.
On the other hand, Gravity Sketch, with its focus on creating geometry in the VR
environment, has proved a real game changer by providing a level of editability
and precision much superior to Tilt Brush, features which appeal greatly to design-
oriented creatives. More recently, the team behind the VR animation tool Tvori
aim to create a powerful, immersive and intuitive tool for crafting visual stories,
which anyone can use, regardless of their previous experience with either 2D or 3D
animation [9].
Currently an increasing number of artists have expressed their interest in this
medium and have started adding VR art-making to their ‘tool box’. However, because
VR art is still in its infancy, and also because of the difficulties associated with
showcasing these productions outside of VR space, the works realized in this medium
are only accessible to the wider public via online 2D galleries [2].
17.4 The Creation of a New Artistic Medium
The most important achievement of holographic imagery and what truly differentiates
it from other media is its capacity to visually expand space without resorting to the
traditionally accepted convention of treating a 2D surface as an interface for 3D

content. In seeking to redefine maximalism as a type of art which uses a minimum
amount of space to deliver a maximum amount of information or content, holography
becomes a good example of an efficient use of the ‘space of art’, in the sense that it
employs a flat surface to display a volume, rather than the actual 3D space usually
taken up by a physical object or scene.
17.4.1 ‘Van Gogh by Pioaru’
By using this unique quality of holography in conjunction with VR art, there is an

opportunity to not only bridge the gap between the virtual and the real, but also to
develop a new form of art that could be unique in itself.
Therefore, a digital hologram was created from a VR artwork. The project was
implemented using Google Tilt Brush and an HTC Vive headset and, since it was a
reinterpretation of a Van Gogh self-portrait, it was entitled ‘Van Gogh by Pioaru’.
In order to transfer a VR project onto a holographic format (and, to my knowledge,
the first time anyone had attempted such a transfer), attention was focused principally
on the technological aspects of the process, and the choice of subject was a secondary
consideration.
Part of the process of transferring the VR project to holography consisted of
using the FBX file format for exporting the scene. However, due to various software
incompatibilities, only the volumes were imported from TB into 3ds Max, whilst
the colour and texture information was lost. This meant that the scene had to be
recolored in 3ds Max. In addition, when imported, the brush strokes were reduced
to plain, ribbon-like meshes. All this resulted in significant differences between the
original VR scene and the final hologram, which was printed using a commercial
digital holographic printer from the data set (Fig. 17.1).
17.4.2 ‘Spectral Figures’
In spite of all technical difficulties, Van Gogh by Pioaru was a promising first step.
The artwork was well received at a public demonstration. However, there were some
constraints and these were addressed in a follow-on project, ‘Spectral Figures’ which
made adjustments to the limitations of the transfer process. This consisted of a series
of black and white portraits of artists and philosophers (Fig. 17.2). These were drawn
in black and white in order to reduce the risks associated with recoloring the scene
(described in the previous section). In order to maintain a degree of similarity to the
models, several photographic portraits of the subjects were imported into Tilt Brush.
These were sourced online and taken from different angles, and used as reference
images for the artworks, with the objective of building into the models the correct
volumetric structure. Thin black strokes were used set against a white background in
322 I. Pioaru
Fig. 17.1 A snapshot taken in Tilt Brush (left) and a photograph of the final hologram (right).
Copyright © I. Pioaru 2019
Fig. 17.2 ‘Spectral Figures’ series—a collage of snapshots taken in Tilt Brush. Copyright © I.
Pioaru 2019
order to simulate the characteristics of drawing on paper. As a consequence, possibly

the most striking aspect of the resulting images is that, although effectively they are
sculptures, every viewing angle gives the illusion of a 2D line drawing [10].
This decision brought the project a step closer to a preferred artistic media, i.e.
drawing, and particularly to ‘sculptural drawing’. This is a concept which has been
approached in several recent projects and it refers to expanding the characteristics
of drawing to create sculptural objects, as part of an exploration of the boundaries
between 2D and 3D imagery. Figure 17.3 is a visualization of a project which best
illustrates this concept: Insectarium, 2012—a sculptural drawing installation—fol-
lowed by two close-up photographs (Figs. 17.4 and 17.5) of one of the boxes which
make up the installation.
Fig. 17.3 ‘Insectarium’ by Ioana Pioaru—sculptural drawing installation, 2016. Copyright © I.

Pioaru 2019
Figs. 17.4 and 17.5 ‘Insectarium’—sculptural drawing installation, close-up. Copyright © I. Pioaru
2019
324 I. Pioaru
The pipeline for creating a hologram from a VR project is relatively straightfor-

ward. In order to generate the image data required by the digital holographic printer
for a single-parallax hologram, the VR project needs to be exported as a Filmbox
(FBX) file—a feature initially implemented into Tilt Brush to allow artists to share
their creations online. This is a format that makes it possible for digital content to
be manipulated across a variety of digital creation software. The incompatibilities
between Tilt Brush and 3ds Max were addressed by simplifying the scene and using
a minimum of colours—effectively just black and white. This reduced the time of
post-processing significantly, the main remaining task being the setup of the cam-
era for image rendering. This is achieved by programming an automated camera in
3ds Max, a process described by David Brotherton-Ratcliffe and Hans Bjelkhagen
in their book ‘Ultra-realistic Imaging’ [11]. This method produces the required per-
spective views that can be pixel swapped to generate the data necessary for the digital
holographic printer.
The data was sent to Geola for printing on their digital holographic printer and
a series of 30 cm × 40 cm colour reflection holograms was produced using a sil-
ver halide emulsion. The holograms, when properly illuminated, produce a good
impression of the original VR subject (Fig. 17.6).
Digital holograms may generally either be ‘Full-Parallax’ or ‘Single-Parallax’.
‘Single-Parallax’ holograms are also sometimes referred to as ‘Horizontal-Parallax-
Only’ holograms. In general, the ‘Full-Parallax’ hologram is to be preferred as it
presents the most general and most faithful 3D experience of the original light field
to the viewer. The viewer will be able to see both to the left and to the right of an
image, as well as properly perceive the image from above and below. The viewer
will also be able to approach the image and observe it from close-up; in brief, a
Fig. 17.6 ‘Spectral figures #4’, single-parallax hologram (three photographic views). Copyright ©
I. Pioaru 2019
full-parallax hologram will relatively faithfully reproduce the original 3D light field
from the active side of the hologram.
Single-Parallax or Horizontal-Only Parallax holograms remove the vertical par-
allax from the light field. As the viewer moves their head up and down, the 3D image
will appear to simply tilt up and down. Since human eyes are horizontally separated,
this type of hologram still looks very realistic to most people. The tilting of the image
is fairly well accepted by the viewer and interpreted as a natural movement of the
image itself. The largest drawback with single-parallax holograms is that the viewer
cannot approach too closely to the hologram without image distortion becoming
excessive.
There are several advantages of the Single-Parallax hologram. The first is the
digital image data is usually some hundreds of times smaller than the data required
for full parallax holograms. This means that whilst single-parallax holograms can
be calculated on a normal laptop or PC, full-parallax holograms often need either
a reasonably large network of PCs or a small supercomputer. The second principal
advantage of the single-parallax hologram is that it can be illuminated much more
easily than the full-parallax hologram leading to brighter images and deeper in-focus
scenes.
Nevertheless, full-parallax holograms stand out as being far superior in that they
convey the 3D reality of the drawings in a much more efficient and successful manner.
The missing vertical parallax is in the single-parallax holograms acts to destroy the
illusion of the image integrity that these sculptural drawings seek to embody. The
single-parallax holograms are brighter and they can be illuminated by arrays of
lights, making them potentially much brighter than the double parallax holograms.
However, this does not really compensate for the critical loss of the vertical parallax.
Fully un-apertured double-parallax holograms, giving the maximum field of view,
is useful if the hologram is to give the best illusion of reality, printed on Silver Halide
material. However, this may inevitably lead to a relatively poor brightness at replay,
even when powerful LED lights are used for illumination. Improving this brightness
is an important ‘next step’. Potentially, photopolymer could be used instead of Silver
Halide, but a few companies are currently offering this choice. Other solutions would
be to use more powerful laser diodes for the illumination. The best possible result
would be expected if dichromated gelatin glass plates were to be used in the printer.
17.4.3 Tilgate Forest
The starting point of this project was a homonymous ink drawing first created in
2018. This was an opportunity to explore a theme that was different from previous
VRH artworks, which had been exclusively portraits. In addition to this, since the
landscape is rather underrepresented in holography this choice of theme seemed
likely to produce an interesting result.
A photograph of the original was imported into Tilt Brush to use as a reference
image (Fig. 17.7).
326 I. Pioaru
Fig. 17.7 Tilgate Forest by Ioana Pioaru, original ink on paper drawing, 2018. Copyright © I.
Pioaru 2019
Several factors identified in the initial drawing pointed to the possibility of struc-
turing a 3d model of this scene as a diorama: the way in which the elements of
the composition are clearly individualized, grouped and layered at varying depths;
the relative difference in scale of these elements; and the amount of graphic detail
used for rendering each layer. Thus, whilst drawing the scene in VR, the composi-
tion was distorted by making objects in the distance much smaller and closer to the
main viewpoint than they would be in an accurate scale model of the real landscape.
Figure 17.8 shows the scene from the main viewpoint and Fig. 17.9 shows what it is
as seen from above.
The compressed depth also offered several advantages holographically, primarily
that of limiting the angular resolution required to resolve the rearmost parts of the
scene. This is a particularly important consideration in light of the fine brush strokes
employed. Another advantage is geometric: due to the ‘window’ nature of a hologram,
scene components nearer the glass are viewable from a wider angle of view before
being obscured by the edge of the hologram. Thus, compressing the scene makes the
compositional layout more robust to viewer movement.
The use of the brushes roughly followed the same method employed for previous
projects, in that the sense that the brush used for the white areas had to be different
from that used for the black areas.
The resulting ‘sculptural drawing’ was close enough to the original ink drawing
and quite convincing as a 3D depiction of a landscape, in terms of the parallax effect.
However, pure black and white images such as ‘Spectral Figures’ and ‘Tilgate Forest’
represent particularly challenging subjects for holography, straining its capabilities
with respect to brightness and colour balance. Additionally, whilst the maximum
angular resolution of digital holography itself is state of the art (thousands of views),
Fig. 17.8 ‘Tilgate Forest’: snapshot taken in Tilt Brush. Copyright © I. Pioaru 2019
Fig. 17.9 ‘Tilgate Forest’: scene viewed from above. Copyright © I. Pioaru 2019
current replay illumination techniques reduce this by an order of magnitude. As

a result, although the depth compression used in ‘Tilgate Forest’ was an effective
illusion in its own right, it was insufficient to fully mitigate the blurring imposed on the
background of the scene. This somewhat compromised the pen-drawing appearance,
the very fine brush strokes seemingly blending together to create a charcoal-like
effect (Fig. 17.10).
328 I. Pioaru
Fig. 17.10 ‘Tilgate Forest’: photograph of the hologram. Copyright © I. Pioaru 2019
17.4.4 Heterotopia
The project titled ‘Heterotopia’ is also based on an earlier work, a two-colour reduc-
tion linocut print, part of a larger series (Fig. 17.11). This combined architectural
structures and pseudo-realistic machinery, in the ‘techno pop’ style characteristic of
recent art practice, where the stark, rigid appearance of the machines and buildings
contrasts with the rawness and vividness of the colours and the cartoon style. Grav-
ity Sketch seemed much more appropriate a tool, as it offers the features necessary
for achieving this type of imagery: the possibility to generate and edit basic shapes
and volumes, a certain degree of precision and perhaps most importantly, a grid and
world axes/coordinates system that the objects can be aligned to.
As the original artwork was a flat, frontal representation of the architec-
tural/‘machinic’ structure, adding depth was the first obvious step necessary in the
creation of the 3D model of the subject. Consequently, a roof-type surface and a
lateral wall were constructed and filled with graphic detail, so that in the hologram,
the roof—an element that is not conceptually essential to the composition—would
be revealed and concealed by the viewer moving their head up and down. The added
lateral wall was intended as a ‘transition area’ combining architectural features with
‘machinic’ details (Fig. 17.12).
The post-processing necessary to prepare the data for the holographic printer was
different and much more complex than those required by previous projects, and was
therefore done in collaboration with Tal Stokes from Geola Technologies, whose
knowledge and expertise were essential (Fig. 17.13).
Fig. 17.11 ‘Heterotopia’ series: first print (left) and colour permutations digital mock-up (right).
17.5 Presenting Virtual Reality Holograms to the Public
The ‘Van Gogh by Pioaru’ hologram was first presented in 2017 at the Cyberworlds
International Conference in Chester. During the 2018 Focus Wales Festival, it was
on display in one of the exhibitions organized at the Ty Pawb Gallery in Wrexham.
The ‘Spectral Figures’ VRH series was presented for the first time at the 11th
International Symposium for Display Holography which took place in Portugal in
2018. On the same occasion, it was part of the holography exhibition held at the
Museum of the City of Aveiro, titled ‘Art in Holography: Light, Space and Time’
[12]. In 2019, during the Artists’ Open House event held within the Dulwich Festival,
‘Spectral Figures #1’ was exhibited together with some recent artworks made in other
media.
330 I. Pioaru
Fig. 17.12 ‘Heterotopia’: collage of three views of the Gravity Sketch model. Copyright © I. Pioaru
2019
Fig. 17.13 ‘Heterotopia’:

photograph of the hologram.
‘Tilgate Forest’ and ‘Heterotopia’ were presented at the SPIE Photonics West
conference which took place in San Francisco in February 2019, together with one
of the pieces from the ‘Spectral Figures’ series.
On each occasion, the work was well received both by the general public and by
artists and scientists attending the events and was deemed highly innovative, partic-
ularly the concept and method of combining traditional drawing with the medium of
digital holography.
In August 2019, a selection of the VR holograms will be included in a group exhi-

bition which will look at the artistic drive being immersed by the image, organized
at the Centre for the Holographic Arts in New York.
17.6 Discussion
For artists and researchers interested in ultra-realistic imagery, it may be important

to understand how these relatively new art-making media, holography and virtual
reality sculptural drawing, which share the ephemeral nature of their content, can be
employed together and how they can highlight and enhance each other’s potential.
Major companies developing VR technology are striving to make their devices and
products more affordable, lighter, wireless, more intuitive and to offer the possibility
to swap between AR and VR. This will inevitably impact on how artists make,
experience, and understand art. On the other hand, it may not be necessary to find
ways of ‘exporting’ productions of VR to other platforms, simply because the best
way to experience them will be inside VR. Nevertheless, due to the challenges and
difficulties associated with such ambitious projects, VR technology is still far from
ubiquitous.
In this transition period, it is therefore relevant to examine how the intrinsic gap
between the virtual and the real can be bridged. As a stage in developing Tilt Brush,
Google has recently initiated an Artist in Residence programme [13], emphasizing the
importance of an active collaboration between artists and scientists in understanding
what steps are necessary to take VR art-making to the next level. Because of this,
many artists are expected to use VR as an instrument to develop their practice, and
this highlights the need to find ways to exhibit their creations.
VR holography is currently dependent on commercial digital holographic printing
services. These services take the processed VR data set and generate the physical
hologram. Perhaps the main problem for the practicing VRH artist today is that
commercial holographic printers usually print on plastic film. To achieve the flatness
required for the display of the artwork, this film must be laminated onto either a
plastic or glass sheet using an optical glue. Unfortunately, even if the visual result
is usually excellent using this technique, the process itself is associated with poor
archival properties. This is a significant problem for the artist and indeed for the wider
art collecting industry. The situation is slowly changing in this respect and digital
glass hologram printers are starting to appear. In the EU, Geola, in addition to offering
large holographic prints (up to 1.5 m × 1 m) on film now offers a service for printing
some types of smaller glass holograms (up to 15 cm × 15 cm) on photoresist. The
Centre for Ultrarealistic Imaging at Glyndwr University in Wales is also planning
on recommissioning its large glass printer (capable of printing up to 1.2 m × 1.2 m)
later this year.
Another problem with commercial holographic printing is the availability of small
hogel sizes. Currently, Geola offers a 0.8 mm hogel with its film printers although its
new glass service can in principle goes down to 100 microns. ‘Hogelization’ or the
332 I. Pioaru
visual pixelation caused by larger hogel sizes is a distracting feature in the context
of VR holographic art. It is a particular problem with full-parallax holograms where
the nature of the image invites the viewer to view the artwork from all distances
including from very close proximities. Reducing the hogel size by a factor of two
does however increase print time by a factor of 4. And with a 30 × 40 cm × 0.8 mm
hogel hologram usually printing in around 3–3 h, commercial printers are not keen
to reduce the hogel size dramatically. Nevertheless, technology is again coming to
the rescue here with the latest generation of digital printers now forecast to offer
printing speeds between 120 and 180 hogels per second—a significant improvement
over typical current rates of 30 hogels per second.
A further problem encountered with VR full-parallax holography is image bright-
ness. This is usually related to the preferred use of Silver Halide emulsions in the
hologram printing industry. Sculptural VR drawings can be particularly susceptible
to this brightness problem if there is too much white (or coloured) background and
too few black drawing lines.
A way around the brightness issue is to use the technique of drawing with a white
pen and a black background. This effectively routes the light energy available from
the hologram into the drawing lines. Because the drawing lines usually occupy a
much smaller solid angle in total than the background, the relative brightness of the
lines can be much higher. Another effect of this technique is that when the hologram
is switched on, the only thing which changes is the image lines appearing in front
of a black background. This is in contrast to the case of drawing with a ‘black’ pen.
In this case, when the hologram is switched on, the background changes from black
to white. A further advantage of this technique is that hogelization should be less
perceptible through the bright pen lines when compared to their perceptibility through
a white background. ‘White’ is also the hardest colour to achieve in holography and
large areas of white tend to often show up discoloration. Discoloration should be
minimized by both the geometry and brightness of the lines.
17.7 Conclusions
The primary conclusion of this paper is that digital full-parallax full-colour hologra-
phy provides an appropriate and useful means of visualizing 3D art created in VR.
Whilst digital holography is not without its problems, it is currently the only medium
that affords simultaneous group viewing of 3D virtual works without equipment such
as glasses or head-mounted devices. This is an important consideration in the con-
text of an art gallery. The artist should therefore be encouraged to consider its flaws
as simply an inherent constraint of the medium and understand how to work with
the medium and how to use it to its best effect. In this regard, VR holography is
no different from any other medium. Perhaps the principal difference is the highly
technical nature of holography and the burden that this places on the artist.
The next step is to present these VRH artworks in the context of an art gallery. Here
they could be brought to the attention of a wider audience and also art experts who
could assess their relevance within the landscape of contemporary art. Hopefully, the
dissemination of VRH artworks would then result in more art practitioners deciding
to work with it and finding it stimulating, useful and inspiring.
References
1. Fantasygraph, 3 Rules to avoid support structures when 3D printing (2016). https://pinshape.

com/blog/dotm-fantasy-graph-3d-printer-support-structures/
2. https://sketchfab.com
3. Dodson, A.: The strange legacy of Tupac’s ‘hologram’ lives on five years after its
historic Coachella debut (2017). https://theundefeated.com/features/the-strange-legacy-of-
tupacs-hologram-after-coachella/
4. https://spie.org/news/spie-professional-magazine/2014-january/dali-in-holographic-space?
SSO=1
5. http://www.members.optusnet.com.au/mbenyon/index.htm
6. http://www.augustmuth.com/new-page
7. Journal edition dedicated to holography, Leonardo 22(3&4) (1989)
8. Lightfoot, D.T.: Contemporary art world bias in regard to display holography: New York.
Leonardo 22(3&4), 419–423 (1989)
9. Feltham, J.: Tvori’s latest update makes VR animation even better (2018). https://uploadvr.
com/tvoris-latest-update-makes-vr-animation-even-better/
10. Short video showing the progress of one of the author’s drawings in VR. https://www.instagram.
com/p/BxRyXLfnN1K/
11. Bjelkhagen, H., Brotherton-Ratcliffe, D.: Ultra-realistic Imaging: Advanced Techniques in
Analogue and Digital Colour Holography, pp. 357–368. CRC Press, Taylor & Francis Group,
Boca Raton, London, New York (2013)
12. Video by Narcis Rovira of the artworks in the exhibition ‘Holography: Light, Space and Time’
and the Margaret Benyon retrospective. https://vimeo.com/277902286
13. https://www.tiltbrush.com/air/artists/
Chapter 18
In Darwin’s Garden: An Evolutionary
Exploration of Augmented Reality
in Practice
Alan Summers
Abstract This chapter discusses the rapid developments in augmented reality and
mixed reality technologies, from a practitioner’s perspective of making the aug-
mented reality sculptural work In Darwin’s Garden. From its conception in 2012, to
its exhibition at Carbon Meets Silicon II in 2017, the advances in augmented reality
technology led to an interplay between the goal of the creators and the technological
realisation of that vision. The art, design and technology involved, generated a reac-
tive process that was mired in external influences as the accessibility to augmented
reality became commercially valuable and subsequently restricted. This chapter will
be of interest to anyone who wants to understand more about the possibilities, tech-
nologies and processes involved in realising mixed reality practice and about the
commercial culture that supports it.
Keywords Augmented reality · Sculpture · Extended realities · Transmediation ·

Embodiment · Virtuality
18.1 Introduction
The artwork In Darwin’s Garden was developed by the artist Chris Meigh-Andrews
with the collaboration and assistance of Rowan Blaik, Head gardener at Down House,
and the author, a design educator and researcher at the University of Chester, UK.
Chris Meigh-Andrews’ art practice considers the resonance of place, by using
moving image to discuss temporality and the relationships to space, of history and
natural forces. He produces site-specific installations that examine spatio-temporality
using technological devices. In Darwin’s Garden explores the historical resonance of
the old mulberry tree in Charles Darwin’s home, Down House in Kent. The tree was
there when Charles Darwin lived in the house and as Chris explained in an interview
for the Leonardo Electronic Almanac:
A. Summers (B)
University of Chester, Chester, UK
e-mail: a.summers@chester.ac.uk

336 A. Summers
is a living link to the past, and through that connection there is a tangible nexus with the
history of ideas, to science, and to the development of theories that have profoundly shaped
our sense of what it is to be human [1].
The artwork comprises of two related works where the mulberry tree is central
to the viewer’s experience. In the first work, the three-dimensional structure and
environment are digital; viewed on a screen or recently explored using a virtual
reality headset. The second work is a physical sculptural form, with an augmented
digital object at its centre.
The digital representation of the tree used in both works was formed from photo-
graphic imagery that included a year of time-lapse photography collected from four
cameras placed in the garden of Down House. This body of time-lapse photography
was the catalyst for the development of a digital 3D form. Chris Meigh-Andrews
discussed how “3D visualisation seemed a natural extension and progression from
previous work, and deeply related to the fact that the mulberry tree is itself a growing,
living organic structure” [1].
The mulberry tree is now in decline and requires human intervention through the
use of man-made structures to support the weaker branches (Fig. 18.1). In Darwin’s
Garden uses full-size facsimiles of these structures in both forms of the artwork.
In the augmented sculptural form, these create a physical space that the digital tree
inhabits and the viewer engages with this physical space, exploring the digital form
within it. The collaborative challenge was how to implement this augmented reality
sculpture, combining both digital and physical elements in order to facilitate an
immersive experience for the viewer.
The completely digital form of In Darwin’s Garden is a three-dimensional struc-
ture set in an environment that can be viewed on a web browser, and more latterly this
has been explored using a virtual reality headset. Within a web browser, this work
is viewed as a window into a digital space, seen as another reality. The viewer can
move around this digital space using a keyboard or mouse, but their physical reality
is separated through the implicit knowledge that the artwork is a reality contained
within the screen. In a virtual reality format, the viewer stands in front of the structure
with the real world expunged from their sight. Virtual movement is by a handheld
controller but the viewer’s body does not move due to hardware and safety restric-
tions. The virtual reality experience is exactly that, a virtual reality that disappears
when the headset is removed and the viewer returns to the real world.
These web and virtual reality (VR) implementations, while effective in their own
right, do not offer what the use of augmented reality (AR) brings to the experi-
ence. Part intention, part necessity, the AR version of In Darwin’s Garden brings an
embodiment to viewing the artwork that is worthy of independent discussion. The
technical journey to this augmented realisation is also of interest as the technology’s
rapid development posed interesting questions of remediation across realities. This
chapter discusses AR and its use within a project that combined physical and digital
structures, theoretical notions of digital and physical space and also details of the
technical elements of AR for art practice. There is also the relationship between
practitioner and technology and the evolution that occurs over time, as both evolve
18 In Darwin’s Garden: An Evolutionary Exploration … 337
Fig. 18.1 The mulberry tree. Copyright © C. Meigh-Andrews 2019, reproduced by permission
while developing the project; that is an important thread running through this chapter.
My background is in industry and education, with a design practice that researches
the understanding of digital space and notions of place, often working with com-
puter animation and game engines. The approach to this collaboration was from the
aspect of both designer and researcher into the possibilities of digital realities. The
development of AR was, and still is, resolving the design and theoretical language
needed to discuss its application. In 2011 Papagiannis likened AR to early cinema
338 A. Summers
…in its infancy, when there were as yet no conventions. AR, like cinema when it first
emerged, commenced with a focus on the technology with little consideration to content,
marked as secondary [2].
The rapid and continued development of AR means it will take time before there
are clear artistic conventions, for as Pearson states technologies are “not developed by
the artistic community for artistic purposes but by science and industry to serve the
pragmatic or utilitarian needs of society” [3]. This is reflected in the artistic devel-
opment of In Darwin’s Garden, which was at the mercy of commercially focused
technological developments. But, as Malina states when discussing art forms enabled
by the computer, “since contemporary culture is being driven by contemporary sci-
ence and technology, one of the roles of the artist is as “coloniser” of the technology
for artistic end” [4]. Chris Meigh-Andrews stated this was an extension to his previ-
ous work—it was definitely an extension to my knowledge of designing commercial
applications for digital realities—so with the artistic vision discussed, we set out to
explore the possibilities for an augmented sculpture.
18.2 Augmenting Reality
To augment reality is to blend another reality with a person’s physical reality, so

both realities are perceived as a single seamless environment. In the most common
and current context the phrase “augmented reality” is thought of as blending digital
imagery with the surrounding environment. It is often discussed as a technological
development and not as a theory of blending a different reality with the real world.
The most identifiable forms of augmented reality are looking at a mobile phone screen
and seeing digital objects blended with the live camera feed, or more recently using
smart glasses that directly overlay the digital elements onto the user’s field of view.
Yet the theoretical concept of augmented reality relates to more than a modern digital
development and has a broader historical context in art and design practice. There
are studies of artworks from Hellenistic times that superimpose painted realities onto
interior walls, in a form designed to create the optical experience of both artwork and
real world as a single reality [5]. The scientific development of perspective during the
Renaissance led artists to the development of trompe l’oeil, mixing real and painted
realities. Neither trompe l’oeil nor digital augmented realities require the viewer to
lose their sense of reality; both blur the optical boundaries between what is real and
what is not.
Renaissance theories of perspective are important discussions for the future of AR
as the Albertian perspectival system that actualised trompe l’oeil is still driving the
Cartesian logic of today’s computer space and subsequent digital realities. McGuirk
and Summers argue that augmented reality technologies bring “psychological and
even philosophical concerns with regard to those [Renaissance] perspectival systems
that underpin these technologies” [6]. And recommend the investigation of other
forms of perspective for AR such as ocular, floating-point or multi-perspective forms
that are less Western specific. Artists do not rely solely on Renaissance theories of
perspective, so it may now be time for the artistic “colonisers” of AR technologies
to explore other theories for developing practice and as Papagiannis discusses, not
just remediate the current contexts and forms of AR but break out from the current
environment AR is discussed in [2].
18.3 Extended Realities
The discussion of augmented reality in its digital form must be understood in the
context of other digital realities as technology continues to drive this area forward and
create new forms. A catch-all term used for this area is Extended Realities, commonly
abbreviated to XR. It refers to all forms of combined real and virtual environments
including augmented reality (AR), virtual reality (VR) and mixed reality (MR).
If a spectrum is considered, where a real environment and a completely virtual
environment are at opposite ends, then the range in between these two endpoints
is where realities are mixed in different ratios. This is Milgram’s Reality-Virtuality
Continuum [7], a concept to aid in the creation of a taxonomy for mixed realities.
The possibility is that this range may not be discrete but continuous and it has since
been extended in another dimension using mediality. In order to be clear for this
discussion, we shall briefly consider what defines virtual, augmented and mixed
realities.
Virtual reality is a completely immersive reality that replaces the user’s real-world
environment with a simulated one. A headset covers the user’s eyes, headphones cover
their ears, then visual imagery and audio immerse the user in a digital environment.
There may also be elements of haptic feedback to enhance the immersion.
Augmented reality, in its digital form, is where the real world is overlaid with
computer-generated imagery with the intention to create the illusion of a single
seamless environment. In order to avoid limiting AR to specific devices, Azuma [8]
stated three characteristics for AR:
1. Combines real and virtual;
2. Interactive in real time;
3. Registered in three dimensions.
These were written before mobile phones became powerful enough to run AR
technologies and so were prescient in their thinking.
Mixed reality is where virtual objects can interact with real-world objects in a
form where the user can act on either, or both, with them having an appropriate
reaction. A mixed reality headset uses sensors to map the real objects and track their
movements within the field of view so digital objects can appear attached to real
objects. The headset can recognise hand movements and the user can manipulate a
digital object as if it were in the real world.
340 A. Summers
Extended realities will continue to develop and so definitions may be redefined

by future technological advances.
18.4 The Drivers Behind the Development of Augmented

Reality
Compute-driven augmented reality was first explored in 1968 by Sutherland [9]

but remained a specialist area until the early 2000s when increases in processing
power in affordable mobile devices enabled AR to become a viable technology with
which to engage users. The development of a variety of AR software development
kits (SDK’s) for mobile devices meant developing AR applications became steadily
more accessible to anyone who understood a certain level of coding. Companies
released SDK’s free to developers, but this meant the developer was locked into that
company’s technology [10].
An alternative to using an SDK is the use of augmented reality browsers. Devel-
opers register with the browser company and can then create AR experiences by
uploading digital files to the company’s cloud service. The browser will then down-
load these files to a user’s device. This business model locks both developer and user
to the specific AR company and their browser technology.
The use of AR browser applications has enabled graphic designers to create printed
material containing images that play video or show three-dimensional objects when
the reader uses the specific browser on their phone. With this development’s ease
of use, an array of print-based AR advertising was created as a way of attracting
attention to products and appealing to mobile user’s curiosity to try AR experiences.
The monetization of AR as an advertising format instigated the development of an
array of applications and SDK’s and the developers of mobile devices took note.
Metaio GmbH was an augmented reality development company that started in
2003 and was used in the initial prototypes for In Darwin’s Garden. They provided
an AR web browser and development kit for programming AR applications for com-
puters, web and mobile devices. In May 2015, it was reported that Apple had bought
the company [11] and Metaio announced all products and subscriptions were to be
discontinued. Developers using Metaio had to find another way to serve augmented
reality experiences to their users before the deadline of December 2015. In Septem-
ber 2017 Apple released iOS 11, their mobile operating system for iPhones and
iPads, that included support for augmented reality development. Their application
programming interface (API), called ARKit, allows third-party developers to build
augmented reality applications that can take advantage of the device’s functional-
ity and processing power. The hardware developers of mobile devices were now
directly supporting the development of augmented reality on their products through
their operating systems.
18.5 Augmenting Art
In contrast to the evolving monetization of AR, artists recognised the opportunities

for their practice. In October 2010 Sander Veenhof and Mark Skwarek created the
WeARinMoMA exhibition in the Museum of Modern Art (MoMA), New York. This
was not an official MoMA exhibition, but as the exhibition’s website states an inva-
sion showcasing the “radical new possibilities and implications Augmented Reality
is bringing to the cultural and creative field” [12]. Using the Layar augmented reality
browser application [13], an Internet connection and the global positioning system
(GPS) on their phones, visitors were able to see digital objects inserted into MoMA’s
gallery spaces. Since then this form of unauthorised intervention has resulted in a
range of dialogues between artists and exhibition spaces, exemplifying the conflict
and opposition to traditional conventions both Pearson and Papagiannis discussed
[2, 3]. MoMA is still being used in this way with the 2018 MoMAR project target-
ing permanent displays in the museum for AR artwork interventions [14]. On the
WeARinMoMA website, Veenhof and Skwarek added a cheeky “PS The MoMA is
not involved yet” [12], and while MoMA has never responded they do now have at
least one artwork in their collection that uses AR, Martine Syms, Incense Sweaters
& Ice, 2017, [15].
18.6 In Darwin’s Garden: Producing the Digital

Environment
Production on In Darwin’s Garden started with the web-based digital environment,

as this was the less experimental production process. It was intended that digital
elements from this web format could then be used in the creation of the second
format, the augmented reality sculpture. Both artworks could be split into component
parts of the tree and the supporting structure. The tree would be digital in both, but
the supporting structure would be digital in the web format and physical in the AR
format. The relationship between the component parts and their true to life scale to
the viewer was deemed crucial to the experience.
…the old tree is now in its decline, with man-made structures supporting some of its branches.
We would like to make full-size facsimiles of these structures to use as a foil to the virtual
image of the tree and develop a work that would enable visitors to explore the tree in virtual
space and time—Meigh-Andrews [1].
The viewer of either experience should encounter a full-size tree and supporting
structure. This should not be a perfect digital representation of the tree in three-
dimensional space, as this would not carry the empathic connection of standing next
to a living, dying tree. The viewer in this experience has to feel they are up close to
a very specific mulberry tree, physically stepping around the framework supporting
the old tree, getting close to the trunk and looking up through the canopy to the sky.
342 A. Summers
At the time production started there was experimental photo software, such as
Photosynth [16], that could crowdsource photos of a landmark and then build a digital
photo cloud simulation of the landmark from them. This approach was a precursor to
photogrammetry, now used in the 3D scanning of real objects, where multiple photos
are captured and the software extracts information from these to build an exact three-
dimensional representation. An exact replication was not the intention for In Darwin’s
Garden, but a representative photographic form where the viewer builds their own
mental image of the real tree through the photo cloud approach offered interesting
possibilities. This format also had links to David Hockney’s two-dimensional photo
collages—referred to by the artist as “joiners”—but in a three-dimensional form that
would be at the actual scale of the subject matter.
Using 3D modelling software, the tree structure was assembled as a photo cloud
comprising of single planar shapes with a photo applied to each. These photo planes
were arranged in such a way as to create the abstracted form at the scale of the real
tree. The physical framework supporting the tree was modelled so as to link with the
tree in a manner representative of the real-world site. The time-lapse images were
placed as clouds of photo planes around the structure in locations that were spatially
representative of the camera locations (Fig. 18.2).
The main technical consideration for this design approach was the quantity and
quality of photographic images, more images equate to a larger file size and an
increased download time. The photos included the time-lapse imagery meaning a
single photo plane would contain 20 time-lapse photos, played in a looping sequence,
Fig. 18.2 The web-based form of In Darwin’s Garden. Copyright © A. Summers 2019
and there were 20 photo planes for each of the four cameras. In 2012 the average
speed in the UK was around 12.0 Mbit/s so the first version of the application at
86 Mb would take just over 1 min to download. This was considered to be too long.
To reduce file sizes, a consideration of how close a viewer will get to a single
image was determined. It was expected that a viewer would be able to walk up to
and into the lower areas of the tree structure. This meant that a single image in the
lower section of the tree could fill the viewer’s screen. Therefore, any image at this
lower level must be the same resolution as the screen the viewer is using. A standard
resolution, at that time, for playing a standalone application on a computer monitor
was 1024 by 768 pixels.
Photo planes in the middle of the tree are at a height above the viewer’s eye line so
could be a lower resolution, as no single image will ever fill the viewer’s screen. The
photo planes at the top of the tree were bigger in scale, but fewer in number, in order
to create areas of foliage and blue sky. These needed to be at the screen resolution
as these would again fill the viewer’s screen when looking upwards.
The author’s previous design practice had explored the analysis of how trees
move in the wind, applied to game environments, where wind and weather affected
how a player interacts with an environment. Wind movement was explored for In
Darwin’s Garden using an algorithm to create a random direction and strength of
the wind. Photo planes fluttered emulating foliage in the wind and added something
to the presence of the tree structure, but it was considered too much of a simulation;
ironically it did not feel natural in the digital environment. On testing it was left
running over 24 hours, only to discover an issue in the wind algorithm meant that the
photo planes moved imperceptibly over time. The whole tree structure would move
across the space, effectively walking out of the environment. A key development
point was to remember to leave each iteration running for a reasonable length of
time, as it might in a gallery situation.
Experimenting with programming the photo planes to face the viewer meant that
an element of movement was present but not overpowering. As the viewer moves,
the photo planes overlap and intersect while rotating to face the viewer. This kept a
sense of physicality and real-world movement within the abstracted foliage.
In its web-based form, In Darwin’s Garden explores space and temporality sur-
rounding the notion of place. The viewer can see Down House in the background
locating them in the space of the garden. Within that space the tree’s supportive frame
is treated as a physical barrier the viewer must move around, while the photo planes
of the tree offer no resistance to movement and the viewer can move through them.
The photo planes containing the time-lapse imagery hold the location of the real
time-lapse cameras making the viewer spatially aware of the image of the garden
space and the real-world tree. This web version was finalised and uploaded to the
IDG web site [17] and in August 2012 presented via Leonardo Electronic Almanac’s
digital media exhibition platform [18].
344 A. Summers
18.7 Producing the Augmented Reality
18.7.1 Production Process
At this point in production with digital assets assembled, the consideration of how to
transfer the conceptual vision to an augmented reality experience began. Transmedi-
ation across forms of extended realities is not a direct process, as virtual reality offers
complete immersion which brings greater levels of control over the viewer’s experi-
ence than augmented reality, where the viewer can see their physical surroundings.
The transmediation of In Darwin’s Garden was not a simple transfer of digital assets;
it meant overcoming technical issues while working with the dialogue between digi-
tal and physical forms, leading artist and designer on an explorative journey into the
possibilities of augmented reality. Design enquiry alongside technical investigation
is required for each of the component parts in an AR experience and this is a useful
way to break down and reflect on the process. Those component parts are
1. Viewing Device: this runs the application and requires a screen and camera.
2. Augmented Reality Application: to be installed on the viewing device.
3. Digital Asset: to be displayed on the viewing device.
4. A Positioning Target: often an image but may also be a physical object or a GPS
location that triggers and positions the display of the digital asset.
The principal for an AR application is that when launched it accesses the viewing
device’s camera, displaying the camera feed on the screen of the device. When the
camera is pointed at a target image the application recognises this image and displays
the digital object superimposed onto the camera feed. The digital object holds a fixed
spatial relationship with the target image, so when the device is moved around the
target image, the display shows the viewer to be moving around the digital object.
18.7.2 Viewing Device
The ability to position the work in any indoor or outdoor location was important
for the development of In Darwin’s Garden. In an indoor gallery, a viewing device
can be supplied with a preinstalled application. Using a specific device means any
application can be fully tested to ensure it works properly. Devices can be updated
and interesting developments in technology, such as advances in augmented and
mixed reality glasses, can be explored.
In an outdoor context, it is unlikely that a viewing device can be left securely at
a site, so the viewing device has to be the mobile phone the viewer is carrying. The
application will need to be developed for a range of devices and registered with the
appropriate application stores. Downloading the application will be limited by the
viewer’s network transfer speeds and data allowance so its file size will be a factor in
any viewing experience. If a person encounters the work outdoors and is not carrying
a suitable device, the engagement with the sculpture will only be through its physical
form.
18.7.3 Application Development
To use a pre-existing augmented reality browser would mean any viewer first has to
download it from the Internet, then the digital object is downloaded while the browser
is running. This is potentially the fastest development process and advantageously
uses a proven and tested browser. But it brings in the extra stage for the viewer of
dealing with the AR browser and its brand, which can act as a barrier to the process
of engaging with the artwork. If an application is built specifically for the work it
will contain all the information needed and is the only item to be downloaded. The
disadvantage is that the application will need to be developed, tested and certified in
order to be distributed by an application store. Also instructions on where and how
to download any application must be visible near the installation. Augmented reality
applications and SDK’s for building applications are constantly being developed and
improved upon but a useful starting point is A Comparative Analysis of Augmented
Reality Frameworks Aimed at the Development of Educational Applications [19].
The decision was made to use the Metaio software tools to develop the augmented
reality sculpture. This provided access to Junaio, a free mobile AR browser appli-
cation for iOS and Android devices, along with software development kits for pro-
gramming, PC, web and mobile augmented reality applications. The Metaio Cloud
stored content online and the Metaio Creator software was a very good drag and drop
creation system. Metaio also organised the insideAR conferences at the forefront of
technological developments in this field. This suite of software gave flexibility in
terms of choice of device, application development or a ready to use AR browser,
and an offline or cloud-based approach. Metaio also supported the Epson Moverio
BT-200 Smart Glasses that contain the processing power of a smartphone and the
ability to superimpose the device screen upon the field of view of the user. These
offered exciting possibilities for AR experiences so the development of the AR form
of In Darwin’s Garden began using Metaio tools, smart devices and Epson’s Moverio
glasses.
18.7.4 The Digital Object
There are various forms of digital object that can be displayed using AR; a two-
dimensional graphic in vector or raster form, a video file, or a three-dimensional
digital object which may be static or animated. Images, text and video are two-
dimensional objects that can be mapped directly onto a target object to overlay it, or
programmed to appear above it and always face the camera. A digital object might be
346 A. Summers
“smart” meaning it can continually access device functionality, such as GPS location,
to provide a constantly updated flow of information.
In 2013, AR advertising was starting to become more commonplace. AR was
being used to play videos over car advertisements in magazines and to show digital
replicas of the advertised car [20]. Three-dimensional objects can sit directly on or
over the target object aligned to face a specific direction and also animated. A major
factor in building digital objects to use in AR is file size. The larger the file size,
the longer its download will be and the more likely the user will move on before
the AR experience gets started. In the web version of In Darwin’s Garden, the size
of the digital asset for the tree was 36 Mb; this meant a minute or more download
time over a 3G mobile network. To explore file size options different iterations of the
digital asset were created by adapting the number of photo planes, the resolution of
the photos and the number of time-lapse images in each animation. The augmented
experience required a level of transmediation led by file sizes also considering a
viewer’s accessibility to an Internet connection, the viewing context in terms of
location, and the active engagement with the augmented reality experience.
Firstly, the time lapse images were relocated to be in the tree structure amongst
the images of foliage. A physical reason for this was the viewer might not be able to
walk across a gallery in order to see the images as in the web version. Importantly
they were now embedded in the experience of the tree, like fruit amongst the foliage.
The viewer could recognise that some photo planes contain a sequence of landscape
images as opposed to images of tree foliage. These became objects that draw the
viewer into the digital space, so that they move in the real world around and through
the physical framework. An interesting time-lapse image is where a portrait of the
artist appears because Chris Meigh-Andrews was working on the camera as it took
a photo, this creates the chance that a viewer will encounter the artist within the AR
experience.
In the augmented experience, the backdrop photos of the house were removed
because they show a fixed landscape behind the digital tree that is not there within
the real-world context; this would break the viewer’s immersion. Instead the view
only holds the digital tree and behind it the viewer’s actual location appears, situating
this digital form in the real landscape with the support structure. This means other
viewers can be seen as part of the view, situating them in both the digital and physical
realities, and true to scale against the digital tree (Fig. 18.3).
18.7.5 Target and Positioning
The most basic problem that limits the immersiveness of an augmented reality expe-
rience is that of registering the digital asset correctly over the target [8]. Advances in
the technology, needed to register a digital file in the correct position and hold it there,
have made a huge impact on user immersion in extended realities. Early AR systems
relied solely on images, known as fiducial markers, with the developer registering
each target image to display an appropriate digital file. The markers placed in the
Fig. 18.3 The augmented digital tree structure seen to scale. Copyright © A. Summers 2019
environment were often black and white block patterns which could be distinguished
in terms of direction to the camera and angle of view. These attributes could then be
used to place the object on the target image with the correct spatial and perspectival
relationship to the camera. Developments in image processing and camera resolution
in mobile phones has meant that photographic images can now be used as targets,
although contrast and asymmetry in a target image are important in order to calculate
direction and angle of view.
Another positioning system used is the device’s global positioning system (GPS).
This is reliant on the efficiency and stability of the GPS signal and limitations occur for
indoor locations where GPS information may be hard to determine. If GPS tracking
is unstable then small changes in location can make the digital object appear to jump
around in front of the viewer.
Metaio, then subsequently Apple’s ARKit, can use an object as a target. This
requires the AR application to know the shape of the object in order to calculate
direction and angle of view. Developers create separate applications dedicated to
scanning objects for use as a target.
348 A. Summers
Advances in environment recognition from camera feeds has resulted in the ability
to position a digital file using markerless tracking. With this system, the application
recognises the planar surface in the camera view and places the digital file onto this
surface. This means there is no need to prepare an environment with markers as the
digital file will locate itself and move around on any flat surface in front of the viewer.
The first iterations of In Darwin’s Garden used the GPS tracking function within
the Metaio software to align the digital tree form within the physical frame. Using the
application outdoors, the digital structure jumped around due to issues of tracking and
an inability to receive accurate location information. The sculpture is 5 m by 3.2 m
and as the viewer walked around the sculpture, the GPS tracking was inconsistent.
When the errors in tracking caused the digital tree to move only a few centimetres
this was deemed acceptable, but when the digital tree would suddenly move a number
of metres it immediately broke the immersion, as it appeared to jump outside of the
physical structure. Further tests in a gallery space found that some devices might not
be able to receive any GPS location information. After various tests, it was decided
an image-based target was to be used as it was the most stable form of tracking.
An issue with positioning a large digital object is the tracking of that object when
the viewing device moves off the tracking image. As the viewing device pans up the
tree structure, the camera loses sight of the target marker and the digital structure
would become unstable. This is also an issue where a digital object moves out of the
field of view but the user will expect the digital object to still be there upon returning to
view that part of the environment. This is commonly referred to as extended tracking
and has been important in developing the possibilities for AR for dealing with more
than a single digital asset in the user’s environment.
During the development of In Darwin’s Garden, Metaio developed object tracking
and provided an application for scanning 3D objects. A problem was the large scale
of the physical structure, as this technology was geared towards scanning small
objects such as toys. Experiments with scanning a smaller maquette of the physical
framework worked at the scale of the maquette but it was found that this could not
be scaled up to match the size of the sculpture.
A development with the Metaio software was the ability to upload a digital model
of the object that could be used for tracking a real object. This proved successful
in recognising the large framework, but only if the whole framework could be seen
within the camera view. If the viewer was too close to the physical frame, then the
tracking could not recognise a component part in order to position the digital asset.
If the viewer could be directed to approach the sculpture from a certain direction,
where the camera would have a full view of the structure, then this was a viable
method of tracking.
The experimentations in tracking eventually led to the use of a single image
marker placed in the centre of the framework as this offered the most consistent
stability during the developmental stages of production (Fig. 18.4).
Fig. 18.4 CAD render of the

modular framework showing
the circular image marker.
Copyright © A. Summers
2019
18.8 Exhibiting In Darwin’s Garden
In 2015 a modular framework for the physical structure was constructed. Five steel
units were fabricated to hold a wooden framework together that can support the two
“A” frames that lean inwards as if they are supporting the real mulberry tree. The
structure had been modelled in CAD, (Fig. 18.4), in order to create the necessary con-
struction diagrams and this also meant an exact digital object file could be extracted
to be used for tracking within the Metaio software.
In early November of 2015, the augmented reality sculpture was installed outside
the Chester Contemporary Art Space for testing. The devices chosen to run the
application where iPads and the Epson Moverio BT-200 Smart Glasses. Using a
Metaio application, the iPads picked up the structures’ shape and the digital tree
form would hold within the structure with minor stability issues. Using the Epson
Smart glasses, the lower resolution camera feed proved to be able to track the object
but only in a good light. When marker-based tracking was used, the tree structure
tracked with reasonable stability on both devices. Then in December 2015 Metaio
revoked all licences for their software, as they had reportedly been bought by Apple
in May of that year [11]. This stopped all applications from working and there was
a certain irony that In Darwin’s Garden, acting as a form of an archive for the dying
mulberry tree, had itself become obsolete within 3 years of its conception.
There followed a period of 6 months of testing other SDK’s and applications as
commercial products competed to fill the gap in the market left by Metaio. The SDK
from Vuforia was used to create a marker-based tracking system for In Darwin’s
Garden, but the possibilities of object tracking were not available.
350 A. Summers
Fig. 18.5 Using the iPad holder to view In Darwin’s Garden. Copyright © Wrexham Glyndwr
University 2019, with permissions
In September 2017, In Darwin’s Garden was installed as part of the Carbon Meets
Silicon 2 exhibition at the Oriel Sycharth Gallery, Wrexham. For this installation, the
smart eye glasses were not used as these could not be secured at the site. Instead iPads
were placed in specifically designed plywood holders that slot into charging stations.
These holders were robust, having handles at both ends to allow two hands to hold
the device (Fig. 18.5). They were designed to give confidence in handling the device
and remove the fear of holding, or dropping, an expensive electronic device. This
appeared to change the nature of the user interaction as users were more confident
holding these larger objects. The users were quick to step into the physical space
and interact with the digital elements and each other. The interconnection between
the physicality of the real world and the nuances of the digital world appeared to be
enhanced by giving the viewer confidence in their handling of the device required to
engage with the digital reality.
18.9 Conclusions
Throughout the development of In Darwin’s Garden, there has been the need to
react to the advancements of augmented reality technology, as well as the disap-
pearance of that technology when it became commercially valuable. This shifting
dialogue between technology and artwork raises interesting questions of transmedi-
ation and archival. Do we keep developing an artwork until the technology facilitates
the vision, or accept current technological limitations and compromise that vision?
In Darwin’s Garden is complete, yet for each new installation it is expected that
the digital assets will need to be embedded in new devices because of software and
hardware developments.
In its gallery iteration, it became apparent that there is an embodiment which
occurs between the viewing device and user that facilitates a more complete inter-
action with In Darwin’s Garden. This embodiment is inherent in the physicality of
holding the plywood iPad holders with both hands and moving around and through
the sculpture as these devices appear to push the digital foliage out of the way. This
may be in part due to the true to life scale of the digital element that facilitates a tacit
understanding within body movement and the path the viewer takes exploring the
digital tree. This embodiment between user and viewing device may also be due to the
connection we have with our personal smart devices. We are confident in positioning
ourselves and our smart device in order to get the best photo. Subsequently, we have
a familiarity with the spatial connection between a camera view and targeting objects
of interest within it.
The smart glasses used in developing In Darwin’s Garden were found to have a
narrower field of view than human vision, meaning a frame appeared to clip the edges
of the digital tree. This broke viewer immersion, as the frame appeared to float at a
constant distance away from both viewer and object. As this technology develops to
allow wider fields of view this disconnect should reduce, but in this instance these
devices did not facilitate a truly immersive experience for the AR sculpture.
In the end it was the physicality of holding a framing device in a form where
the user can twist and angle it with their body, hold it out or look in close, adapting
the screen view to their preference, that was found to be the most intuitive form for
the exploration of an augmented sculpture of this size. Personal smart devices can
facilitate this intuitive exploration, so for external installations of In Darwin’s Garden
it will be important for viewers to download the application in order to engage with
the experience using a familiar device.
In conclusion, there has been a constant tension between developing the aug-
mented artwork, utilising technological advances and adapting to setbacks as access
to technology was withdrawn. With each advance there was a temptation to be led by
technology, but the transmediation of the vision, not the application of technology,
was the essential driver throughout the development process.
References
1. Dziekan, V.: In Darwin’s Garden temporality and sense of place. Live Vis. Leonardo Electron.
Alm. 19(3) (2013)
2. Papagiannis, H.: The role of the artist in evolving AR as a new medium. In: ISMAR-AMH ’11:
Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality–
Arts, Media, and Humanities 2011, pp. 61–65 (2011). https://dl.acm.org/doi/10.1109/ISMAR-
AMH.2011.6093658
3. Pearson, J.: The computer: liberator or jailer of the creative spirit. Leonardo 1(Suppl. Issue),
73–80 (1988)
352 A. Summers
4. Malina, R.F.: Digital image: digital cinema: the work of art in the age of post-mechanical
reproduction. Leonardo 3(Suppl. Issue), 33–38 (1990)
5. Wyler, S.: Roman replications of Greek art at the Villa della farnesina. Art Hist. 29(2), 213–232
(2006)
6. Summers, A., McGuirk, T.: Albertian perspective and augmented reality: lessons from Panof-
sky. In: Proceedings of the Seventh International Conference on 2017 Internet Technologies
and Applications (ITA), Wrexham, pp. 355–359 (2017)
7. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on
the reality-virtuality continuum. Telemanipulator Telepresence Technol. 2351 (1994)
8. Azuma, R.T.: A survey of augmented reality. Presence Teleoper Virtual Env. 6(4), 355–385
(1997)
9. Sutherland, I.E.: A head-mounted three dimensional display. In: Proceedings of the December
9–11, Fall Joint Computer Conference, Part I, New York, NY, USA, pp. 757–764 (1968)
10. Metaio offers free mobile AR software kit, Worldwide Computer Products News, 9 Dec 2011
11. Apple Buys Metaio, a German augmented-reality software maker, Stock Watch (2015)
12. Augmented reality art exhibition MoMA NYC (guerrilla intervention) [Online]. http://www.
sndrv.nl/moma/
13. Van der Klein, R., Boonstra, C., Lens-Fitzgerald, M., Layar, M.: Amsterdam, Blippar (2009)
14. MoMAR. MoMAR [Online]. http://momar.gallery/
15. Syms, M.: Incense Sweaters & Ice (2017)
16. Microsoft Photosynth. Microsoft (2008)
17. Summers, A., Meigh-Andrews, C.: In Darwin’s Garden, In Darwin’s Garden [Online]. http://
www.darwinsgarden.org.uk/
18. LEA Digital Media Exhibition Platform: In Darwin’s Garden, Leonardo Electronic Almanac,
1 Aug 2012
19. Herpich, F., Guarese, R., Tarouco, L.: A comparative analysis of augmented reality frameworks
aimed at the development of educational applications. Creat. Educ. 08, 1433–1451 (2017)
20. Nissan augmented reality. Total immersion (2010)
Chapter 19
Creation of Interactive Virtual Reality
Scenarios as a Training and Education
Tool
Rinat R. Nasyrov and Peter S. Excell
Abstract A novel approach to the creation of realistic training scenarios for safety-
critical industrial applications is presented. This is based on virtual reality techniques
but extended by the incorporation of a range of options for interactivity, which per-
mit the trainee to take actions in realistic ways in the simulated environment. These
actions may include potentially dangerous errors, with realistic consequences simu-
lated, but in complete safety. Any real environment may be simulated both visually
and functionally in the virtual environment. An innovative feature is the use of virtual
buttons displayed on the image of the user’s hand, thus avoiding the need for acces-
sories such as haptic gloves. The system enables trainee specialists to gain realistic
operational experience without the anxieties of causing damage in a real environ-
ment, but it is also relevant to a wide range of applications where rich interactivity
is needed.
Keywords Interactivity · Learning · Safety-critical · Scenarios · Training · Virtual

reality
19.1 Introduction
It is probable that the majority of uses of modern Virtual Reality (VR) technologies
are for entertainment. However, VR technologies also routinely allow people working
in safety-critical industries to undergo training in a realistic environment, without
the risk of hazards such as injury or equipment damage. Electrical power distribution
is an example of a major safety-critical industry and a need arose to create a virtual
training simulator for power substation operators, who have to be ready to react
R. R. Nasyrov
Department of Power Electrical Systems, Institute of EPE, National Research University
for Power Engineering “MPEI”, Moscow, Russia
e-mail: nasirov.rinat@gmail.com
P. S. Excell (B)
Wrexham Glyndŵr University, Wrexham, UK
e-mail: p.excell@glyndwr.ac.uk
354 R. R. Nasyrov and P. S. Excell
rapidly, correctly and safely to a wide range of routine and emergency situations.
The work presented here discusses the development of an innovative VR training
system for such cases, but the principles established could be used in entertainment
or creative applications as well.
The target need arose from an approach by the Russian power industry, for which
over 30% of faults and blackouts have historically been caused by errors during
switching [1] and hence the improvement of operators’ training is a priority: it is
probable that the situations in other industries and in other industrialized countries
will show significant similarities.
19.2 Limitations of Current Two-Dimensional Simulators
Many simulators exist for training of industrial personnel, but they mainly rely on
two-dimensional screen displays. There is no doubt that the underlying algorithms are
sophisticated; for the power substation operator case they typically include: control
of simulated operation order with both routine and non-routine events; estimation
and recording of the operators’ decisions made during such simulated events; rapid
evaluation of the parameters of the modelled system.
Against this, such simulators have deficiencies, such as: a two-dimensional dis-
play of the main control room and substation equipment does not give a realistic
scenario to develop skills; routine normal operations tend to be perfunctory and with-
out deep insight for the trainees; navigation around the installation is not a realistic
representation of the real situation.
Given the powerful impact of three-dimensional simulations now routinely avail-
able for entertainment applications, it was felt that the development of such tech-
nology could deliver a major improvement in training simulators. A VR simulator
for industrial operatives would be vastly more meaningful, although there would
need to be significant improvements in the quality of interaction, in comparison with
entertainment-oriented VR devices.
19.3 Three-Dimensional Approaches to Scenario

Simulation
19.3.1 3D Technologies
There are two main three-dimensional training technologies: 3D Helmet and 3D

CAVE (cave automatic virtual environment). Both of them can be used as forms
of VR simulator and both deliver complete immersion in the virtual environment.
However, the CAVE demands a relatively large amount of space: it leaves the user
unfettered by a headset, but nonetheless has significant deficiencies of realism: in
19 Creation of Interactive Virtual Reality Scenarios … 355
particular, the user only sees a 2D projected image, albeit on a surrounding wall.
Thus, if the space is limited or true 3D is essential, the 3D Helmet technology is
preferable.
The 3D CAVE consists of a cube-shaped space, normally with a volume of at
least 8 m3 , with display screens surrounding the user on all sides [2]. The user may
work with virtual objects with hands or a joystick, but it is difficult to display these
realistically since the surrounding screens are basically two-dimensional; further,
the user’s mobility is limited by the cube perimeter. For 3D Helmet VR technology,
two small displays are mounted in front of the user’s eyes so that they can view the
simulated environment [3]. The user then has the opportunity to work with credible
3D representations of virtual objects, either with their hands or a joystick.
19.3.2 Modelling Requirements for Appropriate

Environments
A primary need is a 3D model of the environment: this must have a good degree of
verisimilitude in both its visual and spatial similarity to the real thing, but it must
also have functional similarity in the operation of relevant switches, indicators, etc.
The verisimilitude of a 3D model leads the user (operator) to develop the skills of
operation of the selected environment during the training, such that after completion
they will not need to adapt to the real environment, due to the realism of the VR
interaction.
Spatial analogue representations of objects and the environment are very impor-
tant: operators need to know the time needed to move from one point to another in
the space concerned and this can be critical in emergency scenarios. Further, the need
for functional similarity in the 3D model is paramount and the model must react to
the user’s actions in the same way as in the real environment.
19.3.3 Scenarios of Training
The scenarios of training have to include both regular and emergency cases: regular
operations refine the skill of routine switching and understanding of the basic oper-
ation on the station. Emergency scenarios need to start in the same way as regular
training but then there will be the insertion of an emergency at a time and of a nature
that must be unpredictable to the trainee, who then has to take correct decisions as
fast as possible [4].
The user can choose between training mode and test mode. If they choose training
mode, they can select either emergency or regular training and the type of scenario. In
test mode, the user does not know what type of training scenario they will encounter:
it will be a random choice of the simulator.
19.4 Implementation of Virtual Reality Simulator
A substation of typical Russian design was chosen to be simulated. This contained:

six 110 kV overhead line connections; six 35 kV overhead lines and one 35 kV
cable line connection; thirty 10 kV cable line connections; outdoor switchgear for
110 kV; outdoor switchgear for 35 kV; indoor switchgear for 10 kV; three three-
phase transformers. The area of the prototype was about 18,000 m2 . This diversity
of equipment enables many scenarios to be implemented.
The digitalization of the prototype has three main stages, as follows [3]: acqui-
sition of a cloud of object points for the prototype hardware; mesh calculation and
optimization; texture calculation and optimization. The object point cloud becomes
a skeleton model in the virtual space: the surfaces of the prototype objects are then
given appropriate textures and the whole 3D model is optimized.
Creation of training scenarios is a specialized issue dependent on the applica-
tion. For the substation operator case, the regular scenarios included taking out of,
and putting into, service the following: transformers; circuit-breakers; disconnec-
tors; overhead lines. For the emergency cases, the following scenarios were chosen.
1. Taking circuit-breakers out of service due to: emergency powering off of trans-
former; emergency voltage transformer fuse failure; emergency current transformer
explosion. 2. Taking an overhead line out of service due to emergency collapse of
column insulation of disconnector. 3. Taking a sulfur hexafluoride circuit-breaker
out of service due to emergency gas depressurising. 4. Bringing a disconnector into
service due to an emergency earth fault on an overhead line. The totality of these
scenarios covers about 90% of typical emergency operations in a substation.
19.5 Technical Description of the Simulator
The game engine part of the simulator is based on Unreal Engine 4 [5] (Epic Games,
2014). A software model of the chosen prototype was written in the C language,
enabling the latency to be reduced to less than 10 ms, which is essential for a con-
vincing experience. The VR headset used an umbilical cable to link to the host
computer because there is no current wireless link technology that can transmit two
channels of HD video plus three channels of gyroscopic information and a USB-3
channel (for the forward-looking camera—see below) with acceptable latency.
There are three main stages of the creation of the training system (Fig. 19.1):
preparatory stage, software creation, system operation.
The preparatory stage consists of the selection of the target environment, carrying
out 3-D scanning of the real objects in that environment, creation of a library of 3-D
models of objects and equipment in the environment, and finally documentation and
refinement of the scenarios (Fig. 19.2).
Stages of simulator
creaon
Soware creaon:
- Training simulator logical System operaon:
Preparatory stage: part: capabilies and - Documentaon
applicaon; creaon;
- Creaon of 3-D equipment
model library; - 3-D shell (engine) - Maintenance;
- Documentaon and capabilies/applicaon; - Instrucng on
refinement of training - Integraon of the 3-D shell training procedures;
operang scenarios. into the training simulator; - Commissioning.
calibraon of
interworking interface.
Further development Further development

without implementer’s involving implementer’s
parcipaon parcipaon
Fig. 19.1 Creation stages of VR training system
To create a functioning simulator, it is necessary to combine the technical functions

of the modelled system (the simulator logical part) with the capabilities of modern
3-D shells (graphic engines). The integration of these two subsystems is discussed
in Sect. 19.6.1.
19.6 Software Implementations
19.6.1 Functional Simulations and Graphics Engine
In principle, the system consists of two subsystems that constantly interact and
exchange information. The first subsystem is the logical part of the simulator: this
performs the technical algorithm that describes the real behaviour of the simulated
system and its response to operator actions, including button pressing, switch knob
turning, cable unplugging and, above all, mistakes.
The second subsystem is a graphical 3-D shell (graphics engine) responsible for
the visual presentation of the substation model. The connection between the two
Environment selecon
3-D scanning
Environment modelling
Elements of 3-D
Wring a soware modeling and their
3-D model library
model of the scenario interoperability
for equipment/
for the simulator with the simulator
objects
logical part logical elements
Simulated model tesng
Simulator commissioned into

operaon on the basis of the
environment model
Fig. 19.2 Component preparation for the developed system
subsystems is event-based and is performed through the interaction interface. The

protocol is thus that the simulator logical part proceeds to a new state calculation
for the system only after receiving the appropriate command from the 3-D shell, this
resulting from the user having performed some action that triggers the need for a
new response. Following this, the simulator logical part evaluates the action taken
and, if it causes a linked visual change in the environment display, sends a command
to the 3-D shell to display the corresponding change.
The main training station acts as a server, to perform functions of administration,
programming and information storage. The trainee interacts with the 3-D shell using
a specific set of controllers connected to their personal computer. There are three
options for the sets of controllers, with increasing levels of functionality:
1. Minimum: the user can effect changes in the simulated equipment using a min-
imum set of peripheral devices—a monitor, keyboard and mouse are needed,
connected to a computer.
2. The optimal set of controllers consists of a virtual reality helmet with gyroscopic
position sensing and a forward-looking camera to track the position of the user’s
head, arms and body.
3. Maximum: differs from the optimal by adding an omnidirectional treadmill,

which allows the trainee to move around in a virtual environment, ensuring the
maximum presence effect, and consequently the effectiveness of training.
19.6.2 Operating Modes
The system can operate in two basic modes: functional preparation mode and train-
ing mode. The functional preparation mode consists of environment preparation
(see Fig. 19.2): the equipment models are created, connection diagrams loaded and
training scenarios are recorded in the corresponding databases on the server.
The training mode is configured for two types of system operation: a multiplayer
mode and an individual training mode. A schematic diagram of the operation in
multiplayer training mode is shown in Fig. 19.3. In this mode, the personal computers
of the staff in training are connected to a single local area network which includes
the main training station, which is an automated workstation for the instructor. There
are servers for the 3-D shell and simulator logical part at the main training station,
providing communication between training stations during group training sessions, as
well as a complete database covering training scenarios and the library of equipment
in the environment. Administration and programming functions are possible from
the main training station. Programming involves entering training scenarios into
the simulator logical part using the “Form editor” application, as well as further
operational adjustments as required by the scenarios created.
The trainee
The main Training Training Training

training station station №1 station №2 station №..N
Hardware Hardware Hardware Hardware
3-D shell 3-D shell 3-D shell 3-D shell

server Required substation Required substation Required substation
scene and scene and scene and
equipment modules equipment modules equipment modules
library for training library for training library for training
Simulator Simulator Simulator Simulator

logical part logical part logical part logical part
server Substation wiring Substation wiring Substation wiring
diagrams and diagrams and diagrams and
training scenarios training scenarios training scenarios
Fig. 19.3 The training mode system structure, when the main training station is a separate supervisor
Basic individual training can be conducted on a dedicated PC: the simulator

application is installed at the trainee’s workplace; a model file of the trainee’s or
instructor’s choice is then loaded. Depending on the configuration, model files can
be located on a workstation or on a server with access to model files using the CIFS
protocol.
19.7 Creation of Scenarios
Analogous to a computer game, the graphics engine can create an arbitrary environ-
ment into which the trainee can enter when using the VR headset. However, as stated
above, to reduce latency to less than 10 ms, as is essential for a realistic experience,
the use of high-level language and packages was avoided and instead the model was
entirely written in the C language. For the prototype that was developed, the client
required a realistic model of a high-voltage substation, both the outdoor high-voltage
equipment and the indoor control room and low-voltage equipment. All of this had
to have simulated functionality, including moving parts, to cover all of the requested
training scenarios.
To create the environment model simulating a real example of a substation, about
1200 working hours were required, occupying six people, a modeller (mesh creator),
two programmers to create mathematical models, one unifying programmer and two
electrical power engineers. The three-dimensional information was derived partly
from scale drawings and photographs and partly from three-dimensional laser scans.
The results show an excellent degree of verisimilitude (Fig. 19.4). Further images
showing the realism that was achieved in modelling of the outdoor hardware are
given in Figs. 19.5, 19.6 and 19.7.
The same techniques were used in modelling of the interior of the control room
and low-voltage switch room (Figs. 19.8 and 19.9), although much more functional
detail had to be included here, such that switches could be turned, buttons could be
pressed, lights could come on and even plugs on flexible cables could be pulled out
in the virtual environment. Figure 19.10 shows the detail that was incorporated for
this purpose: the plugs shown may be pulled out of their sockets in the virtual world
and all of the switches, buttons, and lights function in it as well.
For the full VR experience, the equipment shown in Figs. 19.11 and 19.12 is
required. The volume of the region that can be accessed is determined by the placing
of the motion sensors attached to the tripods, but a cube with 3 m side length is
usually adequate, especially if an omnidirectional VR treadmill is available [6]: in
fact, the treadmill would allow a smaller volume to be used, e.g. a cube with side
length of 2 m.
Figure 19.13 shows the front view of the headset, which has a forward-looking
infrared camera attached to its front: this is used to collect the imagery of the user’s
hand so that the system can determine what the hand is doing and it can also be used
to implement some virtual controls on the hand (see below). Figure 19.14 shows the
pair of stereoscopic views that are generated in the two eyepieces of the headset,
Fig. 19.4 Top: photograph of the real substation that was simulated. Lower: modelled equivalent
(differences are mainly due to a slightly different viewing angle)
Fig. 19.5 3-D model of 110 kV outdoor switchgear yard
Fig. 19.6 3-D model of a 110 kV sulfur hexafluoride circuit-breaker
including a representation of the user’s hand reaching to press a button. Figure 19.15
shows the system in use: the user is wearing the VR headset and reaching out to
grasp one of the plugs attached to a flexible cable (Fig. 19.10). The left image shows
the view that the user experiences: the virtual image of the hand is shown extracting
the plug from its socket in the switchgear.
Figure 19.16 is a diagrammatic recreation of the “virtual buttons” that are made
to appear on the left hand of the user in the virtual view. The cube below the wrist
Fig. 19.7 3-D model of a 110 kV disconnector
Fig. 19.8 3-D model of the main control room
is touched when it is desired to change the view significantly, for instance, to move
from the control room to the outside switchyard in the combined representation of the
substation. The virtual button near the left-hand thumb is used to advance the user’s
virtual standing position forward in a way rather similar to Google Streetview. The
virtual button in the palm of the left hand has been provided for future functionality
but is not currently used.
Fig. 19.9 3-D model of 10 kV indoor switchgear group
Fig. 19.10 Detailed 3-D functional model of 10 kV circuit-breaker
There are no virtual controls on the right hand, but the system detects the index
finger pointing forward as a tool to press buttons, while a grasping action using all
of the fingers is seen as a tool intended to turn a control knob or grasp a plug.
Fig. 19.11 The appearance of the simulator’s core elements
Two tripods for motion sensors
Electrical outlet
3 meters Construction
height 3 meters
Rod for 3D helmet

3 meters
Computer
Electrical outlet
Monitor to display VR to those

who are not in 3D helmet
Triple electrical outlet
Table
Fig. 19.12 Plan layout of the simulator’s elements

Fig. 19.13 Trainee operator interacting with a 3-D model: the forward-looking camera is noticeable.
The marker on the index finger was optional in an early version, but not now needed
19.8 Outcomes and Conclusions
A three-dimensional stereoscopic software representation of an industrial installation

was created, including a large range of virtual interactive functions.
The implementation was designed to exploit a virtual reality headset with
enhanced functionality, including gyroscopes to determine the movement of the head-
set and a forward-looking infrared camera which was used to detect the position of
the hands and fingers of the user, thus avoiding the need for haptic gloves. Further,
virtual buttons were located on the hands of the user in the virtual reality space, thus
avoiding the need for hardware control buttons. The implementation also prioritized
extremely low latency in the display, to give maximum reality in the simulation. To
ensure this, the headset was connected to the controlling computer by an umbilical
cable, as no standard commercial wireless link would have sufficient capacity and
Fig. 19.14 The synthesized stereoscopic view seen by the user shown in Fig. 19.13
Fig. 19.15 A user interacting with the system. Right: the user is wearing the VR headset and
reaching out to grasp one of the plugs in Fig. 19.10. Left: the view that the user experiences: the
hand is shown extracting the plug from its socket
Fig. 19.16 The virtual button controls provided on the image of the left hand, as detected by the
forward-looking camera
adequately low latency. In addition, the code representing the simulated environ-
ment was written in C to avoid latency issues that could be caused by higher level
programming approaches.
The simulated environment in the prototype was an industrial plant (electricity
substation), created with great attention to verisimilitude. Excellent realism was
achieved and this included functional button switches and electrical connectors, all
of which could be operated on by the user’s own hands without the need for haptic
gloves or other technical adaptations. The industrial environment simulated was
dictated by the primary sponsor, but an arbitrary range of alternative environments
can be envisaged. A potential disadvantage is the large amount of specialist labour
needed to create the simulation: 1200 person-hours in the present case, not including
work on assembling the hardware.
The system functioned completely according to specifications and has been
adopted for the training of substation operators by a major electrical power distribu-
tion company in Russia. A particular advantage of a virtual reality training simulator
in an application of this type is the avoidance of any danger resulting from errors
made by the operators during their training. The system has been tested with a num-
ber of trainee operators and results have been successful, in the sense of perceived
realism and relevance of the training. Work is now proceeding to implement other
environments.
Acknowledgements and Note The authors are grateful to the PJSC Interregional Distribution Grid
Company of Central Russia for funding and technical support to develop the VR simulator.
This chapter is an expanded version of a presentation originally given at the 2018 European
Conference on Technology-Enhanced Learning [7].
The copyright of all the images in the figures belongs to R. R. Nasyrov.
References
1. Russkih, A.A., Nasyrov, R.R.: Electrical Safety System, pp. 14–21. PCS, Moscow (2015). ISBN
978-5-905485-81-7
2. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., Hart, J.C.: The CAVE: audio visual
experience automatic virtual environment. Commun. ACM 35(6), 64–72 (1992). https://doi.org/
10.1145/129888.129892
3. Thompson, J.I.: A three dimensional helmet mounted primary flight reference for paratroopers.
Thesis. Jason I. Thompson, Second Lieutenant, USAF. AFIT/GCS/ENG/05-18. Department of
The Air Force, Air University, Air Force Institute of Technology (2005)
4. Nasyrov, R.R., Suleimanov, I.R., Churkin, A.I., Pilyugin, A.V., Marchenkov, D.V.: Switching
training simulator based on virtual reality. Electrichestvo (Electricity) 3 (2016)
5. Epic Games. Unreal Engine 4. Epic Games Inc, Cary NC, USA (2014). https://www.
unrealengine.com/en-US/
6. Leatham, J.: VR omnidirectional treadmills making gains towards full immersion and car-
dio. VR Fitness Insider website, Boise, ID, USA (2018). https://www.vrfitnessinsider.com/vr-
omnidirectional-treadmills-making-gains-towards-full-immersion-and-cardio/
7. Nasyrov, R.R., Excell, P.S.: New approaches to training of power substation operators based
on interactive virtual reality. In: Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H.,
Elferink, R., Scheffel, M. (eds.) Lifelong Technology-Enhanced Learning. EC-TEL 2018.
Lecture Notes in Computer Science, vol. 11082. Springer, Cham (2018)
Part VII
The Future of Interdisciplinary
Research
Chapter 20
Interdisciplinary Research
and Development—Opportunities
and Challenges
Rae Earnshaw
Abstract Discipline boundaries have been systematically developed and established

over the last century which reflect some of the perceived differences between the sci-
ence and engineering disciplines and those in the arts and humanities. However,
both internal and external factors can result in new disciplines which can arise at the
boundaries between existing disciplines, rather than within them, and may combine
different aspects of these disciplines. Some of these may be produced by exter-
nal drivers such as information technology, the Internet and Big Data. It has been
argued that innovative and creative approaches that cross disciplinary boundaries
are more effective in producing new information and new knowledge. The current
scope and priority of interdisciplinary research can be obtained by analyzing the
funding programmes of national and international bodies and agencies. Some fund-
ing programmes are looking for quicker and more effective solutions for complex
grand challenge problems. Analyses of complex information systems may require
the crossing of departmental boundaries in order to generate new knowledge. It is
recognized that obstacles to effective and sustained interdisciplinary collaborations
have the potential to arise unexpectedly. Also, social and cultural factors often come
into play when new initiatives are proposed. Mechanisms to grow interdisciplinary
research such as internal funding, networking, and training are considered. Potential
barriers to progress are also noted. Ongoing issues that remain to be addressed are
summarized.
Keywords Discipline domains · Collaboration · Creative approaches · Cultural

factors · External drivers · Complex information systems · Research metrics
R. Earnshaw (B)
Department of Computer Science, Faculty of Engineering and Informatics, University of
Bradford, Bradford, UK
e-mail: r.a.earnshaw@bradford.ac.uk
St John’s College, Durham University, Durham, UK
Faculty of Art, Science and Technology, Wrexham Glyndŵr University, Wrexham, UK

374 R. Earnshaw
20.1 Introduction
In 1959 Snow identified differences between the science and engineering disciplines
and those in the arts and humanities [1]. This appeared to be due to a variety of factors
including tradition, vocabulary, ways of working, and contributions to society, all of
which could be different in different disciplines and circumstances.
Critchley [2] proposed that Snow had diagnosed the emergence of two cultures
because of the loss of a common framework of understanding. Scientists and engi-
neers favoured advancement of society through technology and industry, whereas
the arts and humanities preferred intellectual and literary endeavour.
However, Gould took an opposing point of view and emphasized the commonali-
ties between science and the humanities [3]. In 1963, Snow appeared to take a more
optimistic view about the relationship between science and arts [4].
These views stem from an era when 5% or less of the population participated in
higher education, and a faculty member’s role was not subject to detailed perfor-
mance management controls by their institution. The freedom in the academy pro-
vided ample opportunity for new ideas to be developed and explored. Research grants
were relatively easy to obtain (there were fewer applicants), and there was opportu-
nity for regular publication of research results in refereed journals or monographs.
Discipline boundaries had been fairly systematically developed and established over
the previous century and these were generally respected. In addition, there was ample
new knowledge to be discovered within the existing discipline domains.
This chapter provides a landscape view of interdisciplinary research and devel-
opment and examines the areas where future progress may be made. The ways in
which the creative arts relate to this view are detailed in Sect. 20.6 of this chapter.
20.2 Development of New Disciplines
However, in the last half-century new disciplines have arisen and there has been an
acceleration of the pace of change. For example, computer science developed princi-
pally out of mathematics in the UK because of the associated interests in numerical
analysis and the requirement to use automated methods to perform the required cal-
culations. In the USA, computer science developed principally out of engineering
due to the interests in hardware and manufacturing computing devices. However,
before they emerged as separate academic disciplines with their own departments
and faculty members, they did exist along the boundary of the existing disciplines
of mathematics and engineering, respectively. The primary driver for the develop-
ment of computer science came from outside the academy in the form of the vacuum
tube computer (1940) [5], the transistor (1948) [6], and the silicon chip (1961) [7].
National governments appreciated the strategic significance of the technology not
just for scientific and business computation but also for their application to a wide
20 Interdisciplinary Research and Development … 375
variety of disciplines and problems, and so invested significantly in computing facil-

ities for the academy and national research laboratories. Shockley, Bardeen, and
Brattain were jointly awarded the Nobel Prize in physics in 1956 [8] for the develop-
ment of the transistor. Therefore, in its initial stages, computing was seen as part of
the natural sciences, at least in its hardware aspects. This is still the case with regard
to quantum computing, and publications from this area can appear in the journal
Nature [9], whereas those from other areas of computer science do not.
Therefore, new disciplines can arise at the boundaries between existing disci-
plines, rather than within them, and may combine different aspects of these dis-
ciplines. Oceanography is a further example of an interdisciplinary discipline as
it covers all aspects with respect to the ocean, which includes its natural compo-
nents (seawater, living creatures, and the sea bed); its currents and movements (fluid
dynamics); and modelling of the overall system (such as by finite element analy-
sis). Similarly, cognitive science draws on interdisciplinary aspects of psychology,
linguistics, philosophy, interaction, and computer analysis.
Shneiderman [10] details how innovative and creative approaches that cross disci-
plinary boundaries are more effective in producing new knowledge. In particular, he
advances the case for combining applied and basic research and a new paradigm for
interdisciplinary collaboration that puts engineering and design on an equal footing
with basic science. Thus even within well-established disciplines there are significant
opportunities for new approaches.
Pressures on funding of universities in recent years have resulted in competi-
tion between departments and between institutions. There is competition between
departments for students and competition between institutions for research funding
and league table positions. It also produced competition between disciplines that
resulted in science and engineering arguing that they were better for students and
society because they imparted skills and generated jobs, whilst arts and humanities
subjects were less useful in these respects. In some cases, it resulted in national
bodies advising students against studying arts subjects, which in turn forced some
of these academic departments to close because their institutions could no longer
afford them if they could not pay their way. Senior academics argued strongly for the
value of arts and humanities [11, 12]. However, this overall antipathy between arts
and sciences which has been somewhat artificially created has in some cases acted
against furthering the important collaborations that are needed. Therefore obstacles
to effective and sustained interdisciplinary collaborations have the potential to arise
unexpectedly. The participants need to have the perspectives and resources to be able
to deal with them.
Further information on the advantages and disadvantages of interdisciplinary
research are detailed in [13].
376 R. Earnshaw
20.3 Acceleration of the Pace of Change
Friedman [14] identifies three principal components that are accelerating change
(i) Market
(ii) Nature
(iii) Technology
The market includes the effects of globalization and political tribalism. Nature
includes climate change. Technology includes the Internet and the increasing power
and reducing costs of computational facilities. All of these act as powerful forces
and they act in parallel. Their combined effect can cause severe disruption. Society
needs to be able to adapt and respond to the changes that these forces represent. Many
disciplines and applications use the Internet as a delivery service, to access Cloud
storage, a computational resource, or a marketing medium, and therefore become
subject to its accelerating effects. One Internet year is said to be equivalent to seven
calendar years [15]. One effect of the Internet has been to break down traditional
barriers. Formerly research teams were co-located with their departments, but today
the Internet can be used to facilitate collaboration across departments and countries
with equal facility. This can result in a greater sharing of ideas and results. Therefore
the potential for interdisciplinary collaboration is substantial, and the Internet can
act as an external driver of change in the same way that the silicon chip inaugurated
the computer revolution.
Many institutions and organizations are increasingly recognizing the need to ana-
lyze large data sets. This can arise in the context of experiments, sensors, simula-
tions, and the user data gathered by social sciences and social media companies.
In the academy, such data is often concentrated in a centralized Data Center, with
associated expertise on hand to advise on data collection and data analysis. They
are often resourced centrally by the academy as they are regarded as an investment
to support all disciplines. Such central resources also offer graduate courses (e.g.,
Masters, Ph.D.) in the various areas of Big Data and its applications. Thus, Data
Centers can have the direct effect of encouraging and facilitating cross-disciplinary
collaborations. Therefore, the rise of Big Data, rather like the rise of the Internet, has
become a significant internal and external driver which forces researchers involved
in this area towards interdisciplinary collaborations.
It is clear that interdisciplinary working is not just a matter of the right structures
and appropriate facilitation. Social and cultural factors also come into play [16].
20.4 Interdisciplinary Research Funding and Research

Priorities
An indication of the scope and priority of interdisciplinary research (IDR) can be

obtained by analyzing the funding programmes of national and international bodies
and agencies. Included in this analysis are the National Science Foundation (NSF)
in the USA, the European Union, and the UK funding agencies.
20.4.1 National Science Foundation (NSF)
NSF states the priority it gives to interdisciplinary research in terms of advancing

scientific discovery and extending the fields of knowledge [17–19]. It also supports
interdisciplinary training in order to enable the recipients to be able to address current
problems in innovative ways [17, 18].
The following challenges were identified [18]:
• Collaboration across people with different backgrounds and cultures
• Extra time needed to build consensus and learning new methods
• Traditions and policies tend to allocate resources to traditional areas
• Professional societies could assist more by active support of interdisciplinarity
• Ensure the peer review process including interdisciplinary researchers.
Potential lessons from industry and national laboratories were as follows [18]:
• Industrial and national laboratories are generally organized to address the
problems
• Management is more top-down than in the academy
• There is potential for greater collaboration between the academy and industry.
20.4.2 European Union
A report to the European Union (EU) on interdisciplinary research in the context of

ERA and Horizon 2020 identified a number of reasons why this should be a priority.
This included the following [20]:
• Faster solutions are needed for grand challenge problems
• Engage in problem-driven approaches
• Exploit disruptive innovation
• Bridge the gap between research communities.
378 R. Earnshaw
The report also outlined the potential effects of interdisciplinary research [20]:
• Trigger innovation
• Add value to the disciplines
• Solve complex real-life problems
• Bring about discoveries outside traditional disciplines
• Change established research perspectives and paradigms
• Achieve impact.
The following enablers were identified for interdisciplinary research and devel-
opment [20]:
• Resourcing of people, time, and space
• Providing targeted funding
• Developing interdisciplinary skills and practices
• Triggering of appropriate social and cognitive dynamics between researchers
• Ensuring undergraduate students receive a grounding in different disciplines
• Facilitating an understanding of different scientific vernaculars.
The EU has also explored the reasons for the support of interdisciplinary research
in disruptive technologies [21].
20.4.3 UK
Research England was set up as a Council of UK Research and Innovation by the

Higher Education and Research Act 2017 [22]. It exists and alongside the other
existing Councils (the seven Research Councils [23] and Innovate UK [24]). One of
its functions is to evaluate the extent to which interdisciplinary research and devel-
opment. It includes including cross-disciplinary, multi-disciplinary, interdisciplinary
and transdisciplinary activity.
A number of reports on interdisciplinary research (IDR) have been undertaken by
various bodies in the UK. Each body has particular interests to do with their mission,
so each report is oriented to their objectives. However, taken together the reports
• highlight a range of barriers and incentives for interdisciplinary research (IDR)
in the UK
• explore the challenges associated with assessing interdisciplinary research
• provide some real-world examples of ‘what works’ in supporting an IDR culture
in today’s higher education research institutions [25].
These are now summarized.
20.4.3.1 The British Academy
The Executive Summary of ‘Crossing paths: Interdisciplinary institutions, careers,

education and applications’ by the British Academy noted the following [26]
• Requests for evidence identified a broad support of interdisciplinary research
• Essential for addressing complex problems and global social challenges
• Enhances understanding of the separate disciplines.
20.4.3.2 Global Research Council
A Survey Report for the Global Research Council (GRC) in 2016 noted the role and
importance of IDR, the right conditions for establishing interdisciplinary working;
assessment, evaluation, and measurement; careers, training, and recognition; and
produced a number of recommendations including the following [27]:
• Better sharing of best practice
• The identification of grand challenge problems
• The provision of funding support over adequate time frames
• The provision of physical and social space
• Fair review processes
• Appropriate end of project evaluation metrics
• Training and capacity building
• Improving awareness of the value and importance of IDR.
20.4.3.3 Review of the UK’s Interdisciplinary Research
A review of the UK’s interdisciplinary research using a citation-based approach was

undertaken for HEFCE and MRC by Elsevier. In order to assess interdisciplinarity,
the diversity of article bibliographies was studied in eight comparator countries for
the period 2009–13. The principal conclusions were that interdisciplinary activity
was growing in the UK; it has an international collaboration component, and the
academy was a principal contributor [28]. It also noted a lower citation impact for
interdisciplinary research.
20.4.3.4 Landscape Review
A landscape review of interdisciplinary research in the UK identified some potential

areas where new stakeholders could promote suitable conditions for interdisciplinary
research to develop [29].
380 R. Earnshaw
20.4.3.5 Case Study Review of English Higher Education Institutions

(HEIs)
A Case Study review by Technopolis identified a number of ways of organizing IDR

such as [29]
• Co-location of researchers
• Researcher networks across subject areas, departments or faculties
• Researcher-led (‘bottom-up’) and/or strategic institutional (‘top-down’)
approaches
• A thematic or generic focus for IDR
• Support for high-quality research in general, not specifically for IDR.
It also suggested ways of growing IDR such as seed corn funding, network-
ing events, improving the skills base by appropriate Masters courses and Ph.D.
programmes [29].
20.4.3.6 Team Science in Biomedical Research
A report to the Academy of Medical Sciences in 2016 summarized the characteristics

of team science including benefits and challenges, reward and recognition, and career
progression [30].
An assessment of progress was done in 2019 [31].
20.4.3.7 Arts and Humanities Research Council
The Global Challenges Research Fund (GCRF) is a 5-year £1.5bn fund and a key
component in the delivery of the UK Aid Strategy: tackling global challenges in the
national interest. The funding was for 1 December 2018 onwards [32]. The objective
includes interdisciplinary research excellence.
20.5 Interdisciplinary Research and Development

in the Creative Arts
20.5.1 Introduction
The continued increase of processing power, data storage, telecommunications band-

width, and display screen resolution provide greater opportunities for handling new
types of users and applications. It also increases the ubiquity and facility of compu-
tational devices in terms of mobility and user interfaces, whether utilized directly by
designers, audiences, users, or embedded in the application environment.
Art and design have a long history in antiquity. They have shaped the values,
social structures, communications, and the culture of communities and civilizations.
The direct involvement of artists and designers with their creative works has left a
legacy enabling subsequent generations to understand more about their skills, their
motivations, and their relationship to the wider world, and to see it from a variety
of perspectives. This in turn causes the viewers of their works to reflect upon their
meaning for today and the lasting value and implications of what has been created.
Some historical examples of art and design were able to use semi-automated
methods for creation, particularly where large areas of a canvas or model needed
to be filled in. However, it was only with the advent of modern technology that the
advantages of harnessing digital techniques were able to be exploited. One of the
earliest examples was the Architecture Machine [33] designed and implemented at
the Massachusetts Institute of Technology (MIT). The objective was to enable digital
technology to assist the user with design tasks, particularly those at large and small
scales, where it was known that designers had particular challenges and difficulties. In
addition, the computer was able to store data and reproduce designs, thus facilitating
the speed-up of the iterative process towards a final design which met the objectives
of the designer and the requirements of the client [34]. It also enabled the aesthetics of
the design to be seen at the design stage within the wider content of the environment
in which it was to be placed. This was subsequently extended using virtual reality
technology to enable prospective users of the building to perform ‘walk-throughs’
within the created space and understand how it would work in practice in terms of
logistics, to ensure it was fit for the purpose. Materials and implementation costs
could also be optimized at the design stage.
More recent examples of artists and designers interacting with technology include
the use of the iPad to produce sketches of scenes which were subsequently grouped
into a montage to give a large wall display containing multiple images [35]. Mobile
phones with high-resolution screens are being used to produce art works and com-
municate them via social media networks [36]. Art installations have also harnessed
modern technology both to process information and to display it. Such environ-
ments have proved useful in engaging users and visitors with real-time images and
interactive art.
Collaborative design has enabled the sharing of information across digital net-
works to produce designed objects in virtual spaces. Augmented and virtual reality
techniques can be used to preview designs before they are finalized and implemented.
Ancient and modern art and design environments illustrate the design and implemen-
tation processes involved, and the opportunities for collaborating and interacting with
other artists and designers.
There are also increasing opportunities for artists to use their skills and exper-
tise to illuminate the latest discoveries in science and technology [37, 38]. There is
increasing interest in extracting meaning from very large data sets, which in turn
requires effective methods of analysis and presentation of the results. Science and
technology are also able to contribute increasingly to the arts, either as a compo-
nent of the artistic output, or part of the methodology used to produce the output.
In addition, the traditional boundaries between arts and technology are becoming
382 R. Earnshaw
blurred due to the way computing technology is being embedded into the everyday
environment in a seamless way, and the use of social media which enables a greater
degree of involvement and sharing by the community. Social media can open up new
dimensions of interaction and participation in both the arts and the sciences [34].
20.5.2 Contributions of Technology to the Creative Arts
Technology contributes to the creative arts by providing tools, interfaces, facilities

for collaboration, access to data, as well as a range of software for a variety of
applications. Such facilities may also include artificial intelligence and machine
learning techniques. Telecommunication networks enable artists to participate in
communities and collaborate with other artists on joint projects. Exhibitions can
be displayed on the Internet and allow local and remote audiences to participate
and interact with the project and each other. Such environments can also be used
for a variety of applications including simulation, planning, theatre rehearsal, and
entertainment, such as computer games [39–42]. Excellent artistic skills are important
for the generation of the scenes for effective gameplay.
Artistic works may use digital methods to produce digital art [43]. Computer
games may be considered as a particular kind of digital art as they use computer
technology and associated software to produce the game.
20.5.3 Contributions of the Creative Arts to Technology
The creative arts have made contributions to Renaissance Teams, the design of tech-
nology, and the effective incorporation of technology into the wider environment.
Artists have been able to contribute to scientific analysis and enable new results to
be produced [44, 45]. Such Renaissance Teams can stimulate creativity and produce
new knowledge [46] in way that would be very difficult, if not impossible, for a single
artist to do on their own. SciArt is a term used to describe the artistic contributions
that can be made to scientific investigations [47].
Technology companies give increasing attention to the aesthetics, usability, and
engagement of the devices that they create (e.g., Apple [48, 49]).
An example of the analysis of cultural artefacts by computer is the Digital
Michelangelo Project at Stanford University [50, 51]. Its objectives were to advance
the technology of 3D scanning and to create a long term digital archive of significant
cultural artefacts in Italy. These archives were made generally available over the
Internet.
20.6 Methodologies for Identification and Assessment

of Research
A number of methods to evaluate research are being used by institutions and national
bodies. These include journal quality (such as by impact factor and internationality),
paper citation rates, impact factors, H-indices of the individuals and departments, and
altmetrics. The UK Research Excellence Framework (REF) also has international
representation on its evaluation panels in order to provide externality, and to seek to
align internal review processes with externally accepted standards.
A number of difficulties and challenges arise in seeking to extend these methods
of assessment within single disciplines to research delivery across a number of dis-
ciplines. Identification and assessment of interdisciplinary research are summarized
in a consultancy report [52], which noted that different metrics can produce different
results [53].
20.7 Conclusions
20.7.1 Ongoing Issues
Although interdisciplinary research has been identified as a priority by NSF, EU, and
the UK, and resources have been identified to give grant and infrastructure support,
the following ongoing issues remain to be fully addressed
(1) Strong discipline silos are still the norm in higher education and graduate
research and there seems to be little inclination to change apart from a rela-
tively small number of institutions such as Arizona State University [54–56]
and Pohang University of Science and Technology (Postech) [57].
(2) External examiners who adjudicate on Ph.D. theses would not normally have
interdisciplinary experience and expertise sufficient to give an appropriate and
fair evaluation of an interdisciplinary thesis.
(3) Graduate students can feel that an interdisciplinary project may be too risky,
and also may have an uncertain outcome.
(4) Budget structures in higher education are often ring-fenced around specific
disciplines. It can also be difficult to allocate part of the budget across an internal
boundary.
(5) Interdisciplinary research appears to be mainly stimulated by external drivers
such as information requirements or technology developments. These drivers
often require new ways of working. This results in interdisciplinary research
operating in reactive mode rather than seeking to advance proactively according
to a detailed agenda.
384 R. Earnshaw
(6) Moving away from a traditional discipline into a new area can still be perceived
as risky due to the inertia in the academy, the difficulty of applying research
metrics to new areas, and uncertainty about the future of the new discipline.
(7) Increasing competition for grant funding and funding within institutions, can
force review bodies to be more conservative and only allocate funding to well-
established areas, when such funding is limited.
20.7.2 Enablers for Collaboration Between the Arts

and the Sciences
Traversing the boundaries between disciplines can be a challenge. However, recent

developments have brought about a number of enablers. Firstly, the Internet has pro-
vided a greater ease of communication across disciplines and between researchers.
It has also flattened organizational hierarchies to some degree, which in turn has
reduced the traditional barriers between existing disciplines. Secondly, the model
of Renaissance Teams [58] has proved useful in bringing together interdisciplinary
groups of researchers with complementary expertise sets and defined goals. Thirdly,
there have been significant successes in the application of information technology to
the arts and humanities (for example, in the digitization of ancient manuscripts and
artefacts). Fourthly, art and design have been successfully applied in support of the
creative processes involved in generating new technology products and services, and
in the presentation of scientific results. Fifthly, the increasing amounts of data being
generated by research projects have resulted in the establishment of Data Science
Centers within the academy, with teams of faculty and researchers brought together
from various disciplines. Sixthly, grant funding bodies are increasingly focussing on
larger initiatives often traversing different disciplines and countries (e.g., European
Union grants) so require personnel to work together on advanced research and devel-
opment. In addition, creative and artistic cooperation has demonstrated the benefits
for all parties. All these factors make interdisciplinarity more the norm for the future
and can generate a level of momentum and validity which provide reassurance and
support for new researchers who wish to enter the area.
References
1. Snow, C.P.: The Two Cultures. Cambridge University Press, Cambridge, UK (1959)
2. Critchley, S.: Continental Philosophy: A Very Short Introduction, p. 49. Oxford University
Press, Oxford, UK (2001)
3. Gould, S.J.: The Hedgehog, the Fox, and the Magister’s Pox. Harmony Books, New York City,
NY (2003)
4. Snow, C.P.: The Two Cultures: And a Second Look: An Expanded Version of The Two Cultures
and the Scientific Revolution. Cambridge University Press, Cambridge, UK (1963)
5. https://www.webopedia.com/DidYouKnow/Hardware_Software/FiveGenerations.asp
6. https://www.cedmagic.com/history/transistor-1947.html
7. http://edition.cnn.com/2004/TECH/12/27/explorers.silicon/index.html
8. https://www.nobelprize.org/prizes/physics/1956/summary/
9. https://www.nature.com/authors/author_resources/about_npg.html
10. Shneiderman, B.: The New ABCs of Research: Achieving Breakthrough Collaborations, p. 336.
Oxford University Press, Oxford, UK. ISBN 9780198758839 (2016)
11. https://www.timeshighereducation.com/opinion/a-dangerous-downgrade-undervaluing-the-
arts
12. https://www.thebritishacademy.ac.uk/news/diversity-subjects-essential-national-prosperity-
post-brexit-warns-british-academy
13. Earnshaw, R.A.: State of the Art in Digital Media and Applications, Chapter 3, pp. 21–28.
Springer, Switzerland. ISBN: 978-3-319-61408-3 (2017). For more information see: http://
www.springer.com/gb/book/9783319614083
14. Friedman, T.J.: Thank you for Being Late: An Optimists Guide to Thriving in the Age of
Accelerations. Penguin, London (2017)
15. Clark. J.: Netscape Time, p. 262, St. Martin’s Press, London (1999)
16. Bruns, P.: http://researchdevelopment.byu.edu/wp-content/uploads/2012/12/Full_Report_
Creating_Interdisciplinary_Science.pdf
17. https://www.nsf.gov/od/oia/additional_resources/interdisciplinary_research/
18. Committee on Facilitating Interdisciplinary Research, Committee on Science, Engineering,
and Public Policy.: Facilitating Interdisciplinary Research, p. 2. National Academies, National
Academy Press, Washington (2004). https://www.nap.edu/read/11153/chapter/1#xxvi, https://
www.nap.edu/catalog/11153/facilitating-interdisciplinary-research
19. https://www.nsf.gov/od/oia/additional_resources/interdisciplinary_research/definition.jsp
20. https://ec.europa.eu/research/openvision/pdf/rise/allmendinger-interdisciplinarity.pdf
21. https://ec.europa.eu/futurium/en/digital4science/why-should-interdisciplinary-high-risk-
research-disruptive-technologies-be-supported
22. https://re.ukri.org/about-us/
23. https://www.ukri.org/about-us/our-councils/
24. https://www.gov.uk/government/organisations/innovate-uk
25. https://re.ukri.org/research/interdisciplinary-research/
26. https://www.thebritishacademy.ac.uk/interdisciplinarity
27. https://www.globalresearchcouncil.org/fileadmin/documents/GRC_Publications/
Interdisciplinarity_Report_for_GRC_DJS_Research.pdf
28. https://webarchive.nationalarchives.gov.uk/20170712122715/, http://www.hefce.ac.uk/pubs/
rereports/Year/2015/interdisc/
29. https://webarchive.nationalarchives.gov.uk/20170712122426/, http://www.hefce.ac.uk/pubs/
rereports/year/2016/interdis/
30. https://acmedsci.ac.uk/policy/policy-projects/team-science
31. https://acmedsci.ac.uk/file-download/52394024
32. https://ahrc.ukri.org/funding/apply-for-funding/archived-opportunities/gcrf-interdisciplinary-
research-hubs/
33. Negroponte, N.: Toward a theory of architecture machines. J. Arch. Educ., March (1969)
34. Earnshaw, R.A.: Art. Design and Technology. Springer. ISBN: 978–3-319-58120-0 (2017).
https://www.springer.com/gb/book/9783319581200
35. How David Hockney became the World’s foremost iPad Painter, Wired. https://www.wired.
com/2013/11/hockney/
36. https://en.wikipedia.org/wiki/Coldwar_Steve
37. Miller, A.I.: Colliding Worlds: How Cutting-Edge Science is Redefining Contemporary Art.
W. W. Norton and Company (2014)
38. Wilson, S.: Art + Science Now: How Scientific Research and Technological Innovation are
becoming key to 21st Century Aesthetics. Thames & Hudson (2012)
39. Virtual Amusement Park (VPARK). http://cordis.europa.eu/project/rcn/46555_en.html
386 R. Earnshaw
40. Joslin, C., Molet, T., Magnenat-Thalmann, N., Esmerado, J., Thalmann, D., Palmer, I.J., Chilton
N., Earnshaw, R.A.: Sharing Attractions on the Net with VPARK. IEEE Comput. Graph. Appl.
IEEE Computer Society 21(1), 61–71 (2001). https://doi.org/10.1109/38.895134
41. Virtual Interactive Studio Television Application using networked graphical supercomputers
(VISTA). http://cordis.europa.eu/project/rcn/32297_en.html
42. Flerackers, C., Earnshaw, R.A., Vanischem, G., Van Reeth, F., Alsema, F.: Creating broadcast
interactive drama in a networked virtual environment. IEEE Comput. Graphics Appl. 21(1),
56–60 (2001). https://doi.org/10.1109/38.895133
43. https://en.wikipedia.org/wiki/Digital_art
44. Cox, D.J.: Using the supercomputer to visualize higher dimensions: an artist’s contribution to
scientific visualization. Leonardo 41(4), 391–400 (2008)
45. Cox, D.J: Collaborations in art/science: Renaissance teams. J. Biocommun. 18(2):15–
24 (1991). https://www.ncbi.nlm.nih.gov/pubmed/1874707, https://muse.jhu.edu/article/
243508, http://www.leonardo.info/isast/articles/popper.html, http://link.springer.com/chapter/
10.1007%2F978-3-642-46954-1_28, https://datascience.lanl.gov/data/papers/Pursuing.pdf
46. http://swiki.cs.colorado.edu/CreativeIT/248.print
47. http://cargocollective.com/artscienceclimatechange/SciArt-The-confluence-of-art-and-
science-in-conveying-the-uncertaintie
48. Yan, J.: Analysis of apple equipment in the product design thinking. In: International Con-
ference on Advances in Management, Arts and Humanities Science (AMAHS 2016), Atlantis
Press, 2016 (Open access article under the CC BY-NC license (http://creativecommons.org/
licenses/by-nc/4.0/))
49. https://www.fastcompany.com/1665375/the-6-pillars-of-steve-jobss-design-philosophy,
https://innovationmanagement.se/2017/06/22/lessons-from-apple-on-why-aesthetic-
innovation-is-important/, https://www.cleverism.com/apple-product-design/
50. https://graphics.stanford.edu/papers/dmich-sig00/dmich-sig00-nogamma-comp-low.pdf
51. https://graphics.stanford.edu/projects/mich/
52. https://www.digital-science.com/resources/digital-research-reports/interdisciplinary-
research-methodologies-identification-assessment/
53. https://digitalscience.figshare.com/articles/Digital_Research_Report_Interdisciplinary_
Research_-_Methodologies_for_Identification_and_Assessment/4270289
54. Crow, M.M., Dabars, W.B.: Designing the New American University, p. 360. Johns Hopkins
University Press, Baltimore, MD (2015)
55. https://newamericanuniversity.asu.edu/
56. Crow, M.M., Dabars, W.B.: Interdisciplinarity as a design problem: toward mutual intel-
ligibility among academic disciplines in the American research university. In: O’Rourke,
M.R., Crowley, S.J., Eigenbrode, S.D., Wulfhorst, J.D. (eds.) Enhancing Communica-
tion and Collaboration in Interdisciplinary Research, pp. 294–322. Sage Publications, Los
Angeles (2013). http://methods.sagepub.com/book/enhancing-communication-collaboration-
id-research, http://wwwhome.postech.ac.kr/web/eng/ecad_01_11
57. http://wwwhome.postech.ac.kr/web/eng/etc_01?p_p_id=EXT_BBS&p_p_lifecycle=0&p_
p_state=normal&p_p_mode=view&_EXT_BBS_struts_action=%2Fext%2Fbbs%2Fview_
message&_EXT_BBS_messageId=14790
58. Cox, D.J: Collaborations in art/science: Renaissance teams. J. Biocommun. 18(2), 15–24
(1991). https://www.ncbi.nlm.nih.gov/pubmed/1874707
Further Reading
59. Sharp, P.A., Cooney, C.L., Kastner, M.A., Lees, J., Sassisekharan, R., Yaffe, M.B., Bhatia, S.N.,
Jacks, T.E., Lauffenburger, D.A., Langer, R., et al.: The Third Revolution: The Convergence
of the Life Sciences, Physical Sciences, and Engineering, p. 34. Massachusetts Institute of
Technology, Washington, DC (2011)
60. Frodeman, R., Klein, J.T., Mitchum, C.: The Oxford Handbook of Interdisciplinarity. Oxford
University Press, New York, USA (2010)
61. Hage, J., Meeus, M.T.H.: Innovation, Science, and Institutional Change Oxford. Oxford
62. Holley, K.A.: Understanding interdisciplinary challenges and opportunities in higher education.
In: Ward, K., Wolf-Wendel, L. (eds.) ASHE Higher Education Report, pp. 1–131. Jossey-Bass;
Wiley Periodicals, San Francisco, CA (2009)
63. Klein, J.T.: Creating Interdisciplinary Campus Cultures: A Model for Strength and Sustain-
ability. Jossey-Bass, San Francisco (2010)

2020 Book TechnologyDesignAndTheArts-Opp

Uploaded by

Copyright:

Available Formats

2020 Book TechnologyDesignAndTheArts-Opp

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2020 Book TechnologyDesignAndTheArts-Opp

Uploaded by

Copyright:

Available Formats

What is the topic of discussion in the document?

What is the topic of discussion in the document?

What are some of the authors and editors mentioned?

What are some of the authors and editors mentioned?

Springer Series on Cultural Computing

Rae Earnshaw · Susan Liggett ·

ISSN 2195-9056 ISSN 2195-9064 (electronic)

Dr. Jon Peddie

Leeds, UK The Editors

Part I A Panoramic View of the Field

Part II Facilitating Communication Between the Arts, Technology,

Part III Interactions Between the Arts and Data

8 Interspeciﬁc Interactions: Interaction Modes Between Sound

Part IV Audio Visual Installations to Generate Collective Human

Part V The Convergence of Digital Design, the Arts, Computing,

Part VI The Use of Virtual Reality and Augmented Reality

19 Creation of Interactive Virtual Reality Scenarios as a Training

Part VII The Future of Interdisciplinary Research

About the Editors

Prof. Rae Earnshaw is Professor of Electronic

Visual Computing at the University of Bradford which

Dr. Susan Liggett is Reader in Fine Art and Associate

Prof. Peter Excell is Emeritus Professor of

Prof. Daniel Thalmann is a Swiss and Canadian

Daniel Buzzo University of the West of England, Creative Technology Lab,

Ardeshir Osanlou Centre for Ultra-Realistic Imaging, Wrexham Glyndwr

Keywords Forms of knowledge · Scientific revolution · Cultural context ·

1.1 Interactions Between Disciplines

© The Author(s) 2020 3

1.2 Relationships Between the Arts and the Sciences

1.3 Historical Developments in the Relationship Between

1.3.1 Computer Arts

1.3.2 Computer Animation

Making images move has been a preoccupation of computer technologists from

1.4 Digital Media

Collaboration in the production of digital media may involve a variety of discipline

1.5 Digital Humanities

1.6 A New Renaissance?

1. Computer Arts Society. https://computer-arts-society.com/

7. Ideas before their Time. Symposium summary. http://www.technocultures.org.uk/

Keywords Practice-based research · Practice-led research · Practice as research

2.2 Artistic Research and the Academy

It is useful to decode some of the terminology surrounding the arts in academia to

feelings, intuitions, and opinions.

2.3 Knowledge Production

Practice-based, practice-led, practice as research (PaR), artistic research, and arts-

2.4 Cross-Disciplinary, Interdisciplinary,

Bourdieu notes that on-going privileges of positivistic and instrumentalist approaches

4 Arts-based inquiry uses artistic expression as data for inquiry.

2.5 Art, Collaboration, and Technology

Susan Liggett and Mike Corcoran

Abstract The role of the exhibition in overcoming interdisciplinary communication

Keywords Curatorial model · Communication · Exhibition · Interdisciplinary ·

S. Liggett (B) · M. Corcoran

© The Author(s) 2020 25

3.2 Defining Interdisciplinary Communication

3.2.1 Defining Interdisciplinary

Before we identify interdisciplinary communication challenges, we must first

3.2.2 Defining Communication

In order to precisely identify communication challenges, it is helpful to adopt a sim-