Lived experience in human-building interaction (HBI): an initial framework

1 Introduction

The field of archaeology is continually uncovering new evidence of human-made settlements and architectures dating as far back as at least nine millennia. Humankind has incorporated interactive elements in these architectures throughout the centuries, such as the infamous ancient Greek theatrical machinery for scene changes, mediaeval castles with drawbridges to control visitor access, Renaissance moving statue automata, and Victorian moving staircases which were the predecessors of escalators. In recent times, interactivity in architecture has made leaps through the evolution of digital technology. Rapid technological advancements in sensors, actuators, and microprocessors find their ways into the built environment, and alter its design and our interactions and experiences with it in unprecedented ways. Through this technological transformation—and much like smart devices—built environments are becoming intelligent entities that quantify one's self (Margariti et al., 2023), inform on living conditions through ambient displays (Snow et al., 2019), optimise energy consumption (Chen et al., 2021), change their shape to accommodate human needs (Wiberg, 2018; Nguyen et al., 2022), and carry out other mundane tasks. Acting in a timely imperative manner, the research domain of human-building interaction (HBI) studies methods and design applications for the convergence of computing and architecture through the integration of computational intelligence with the built environment and looks at social, spatial, and ethical implications of such interfaces on everyday life and human experiences, as well as ways that people interact with such environments.

As is apparent from the journal's invited topic and the growing literature, there is an ongoing interest and enthusiasm for the rapidly evolving research area of HBI. The convergent field uniquely entwines architectural space with interactive technologies and attracts interest primarily from architecture, computer science, human-computer interaction (HCI), civil engineering, and media architecture. The definitions of HBI are therefore broad and diverse, stemming from various disciplines (e.g., Nembrini and Lalanne, 2017; Alavi et al., 2019b; Becerik-Gerber, 2019), dictating that the fields have a lot to learn from one another. Attempting to lay down the grounds for HBI as a research program, noteworthy pursuits presented entwinements between architecture and HCI (Alavi et al., 2016a,b), mapped different dimensions of the HBI field in relation to HCI (Nembrini and Lalanne, 2017) and more recently, defined potential research directions and a future agenda towards common goals in concerted efforts with other disciplines (Alavi et al., 2019a,b; Becerik-Gerber et al., 2022). In our work as HCI researchers with backgrounds in interaction design, architecture and computer science, HBI interfaces manifest as augmented physical environments, large-scale physical artefacts (such as furnishing, interior, or architectural elements), or tangible user interfaces (TUIs). These interfaces are designed to meet contextual needs, engage people in bodily movement, and promote social exploration of the built environment in a playful manner.

Aside from the growing interest, experience was identified as one of the main research directions for HBI (Becerik-Gerber et al., 2022), yet little has been done to provide a holistic view of the concept from both constituting fields and shed light on the lived experience of such environments. Studies on HBI addressing experience stemming from architecture and engineering fields, look at experience from a positivist angle, as part of human comfort in indoor environments (Jazizadeh et al., 2014; Lee et al., 2021). On the other hand, HBI studies stemming from HCI on experience are few; case studies on the experience of HBI artefacts are either in controlled lab environments (Nguyen et al., 2022) or only temporarily deployed in the real world (Houben et al., 2017). Only a fraction of studies investigate lived experience, and those who do, do so on commercial applications of smart buildings (e.g., Margariti et al., 2023). Therefore, HBI studies on the topic of lived experience are few and far between.

To address this research gap and provide a holistic approach to the experience of HBI artefacts, we delve into HCI and architecture literature on experience to discover experience-defining dimensions and theories which underpin our suggested framework of lived experience in HBI. This framework is further elaborated through a case study of a tailored HBI artefact which targets placemaking and noise reduction. Specifically, our study assessed young children's interactions and experience with the interactive artefact in relation to social interactions, appropriation, embodied action, and conflict. Our framework aims to guide HBI designers and researchers alike when assessing the lived experience of an HBI artefact by building on a set of three dimensions and providing five aspect pairs to capture the range of human experience with an HBI artefact.

In the sections that follow we initially review literature and previously conducted studies related to HBI interfaces and human experience, we lay the foundation of experience in both HCI and architecture fields and identify areas of convergence and differences. As a following step, we provide our case study in which we assessed experiences with an HBI artefact, by presenting its objective, methodology, analysis, findings, and reflection. In Section 4 we detail our proposed framework and its use underpinned literature, relevant theories and findings from our case study. Finally, we critically reflect on our suggested framework in terms of its implications for HBI and the related fields, and its strengths, limitations, and future directions.

2 Outlining experience

In this section, we explore theories, models, and core concepts on human experience that will help us lay the foundation for our experience framework and place the lived experience of HBI artefacts in an interdisciplinary context. We chose theories and models that are well known as fundamental approaches, empirically proven, or previously evaluated in literature. It is in the nature of the term ‘experience' that there exists no single, well-scoped definition in either architecture or HCI research. Thus, we initiated our literature research with widely-cited, peer-reviewed and review-oriented papers of both research fields. Our intent is to lay out the conceptual foundations of literature (frameworks and models) focused on experience from both disciplines and the interdisciplinary dialogue between them.

To do so, we initially offer a brief overview of HBI work and its brief intersection with the notion of experience. Secondly, in the subsequent sections (Sections 2.2 and 2.3) we delve into experience and its attributes from the point of view of the two fields that HBI largely stems from, HCI and architecture.

2.1 Experience in HBI

HBI has its roots as far back the 1960s and 1970s, with the rise of cybernetics and early forms of computing infrastructure integrated with architecture. Over the decades, the fields of HCI and architecture have been studying and working in this intersection in parallel, with limited collaboration efforts. In recent years, HCI and architecture writings have highlighted the benefits of cross-pollination among the two domains. As other scholars support, HCI skill-sets can be usefully applied in architecture design projects (Krukar et al., 2016) and building programs (Bratton, 2008) and, in the same vein, HCI can learn from architecture (Ingram, 2009; Wiberg, 2015). Specifically, HCI scholars have looked at the practice of architecture, its long tradition and tacit knowledge opposed to interaction design and proposed aspects such as responsibility, aesthetics, and cultural relevance that interaction designers could learn from Ingram (2009). For example, scholars suggested benefits of an architectural perspective on digital technologies (Wiltse and Stolterman, 2010) and ways HCI tools could be of use for the practice of architecture (Verma et al., 2017). From architecture's standpoint, scholars borrowed methodology and methods from HCI, such as personas, scenarios (Doktor Olsen Tvedebrink and Jelić, 2018) and wizard of Oz (Achten and Kopriva, 2010). In addition, architects have called for an interaction design practice that melds architecture with other design disciplines to generate new visions of the world (Steenson and Scharmen, 2011). Dade-Robertson (2013) suggest technologies should be explored through joint projects, collaborations and discussions among architects and HCI practitioners.

In a recent article by Becerik-Gerber et al. (2022), interdisciplinary HBI experts have identified human experience as one of the primary research dimensions of HBI, underlining the need for understanding the complexity of individualised human experiences and the ways these experiences are shaped by the built environment and vice versa. In HBI research, pursued by architecture and engineering scholars, experience is used in building performance research referring to occupants' indoor comfort in relation to energy consumption (Jazizadeh and Becerik-Gerber, 2012). This line of research is dominated by positivist views that consider experience to be measurable and address it in a number of quantitative studies investigating comfort (Jazizadeh et al., 2014; Lee et al., 2021; Dongre et al., 2022) and its impact on human performance and satisfaction as well as other contributing factors related to the well-being of building occupants. In contrast, HBI research deriving from the field of HCI tends to have a broader user-centred perspective (Alavi et al., 2016b) and it is slowly but surely expanding. The few qualitative studies on lived experience following this perspective examine the views, concerns, and future aspirations of occupants of buildings using quantified-self technologies (Margariti et al., 2023) or smart building appliances (Mitchell Finnigan and Clear, 2020) but do so by looking at commercial applications. As there is no framework yet to set grounds on the experience of HBI interfaces, these studies draw on approaches from other fields.

Evidently, since HBI is still a domain in development, case studies, methods, and theories that address human experience are scarce and the field has yet to holistically address the constituting aspects of lived experience when interacting with a deployed, physical HBI artefact. Therefore, we deem important to introduce an experience framework that can guide HBI designers and researchers alike in assessing lived experience in an HBI context. In an attempt to sketch HBI's research scope (Alavi et al., 2016a, 2019b), introduce a schema based on the interrelated dimensions of physical, social, and spatial. While the schema by Alavi et al. (2019b) is used to conceptually map the broader grounds of HBI as a research program and situate the different HBI spaces and research topics, we see these three dimensions as the constituting elements of experience with an HBI artefact. As the authors suggest, these three dimensions (physical, social, and spatial) should not be seen in isolation or a crosswise manner from one another and we propose that they are entangled in a way that they are all present, inter-relational and inseparable from one another in each and every HBI manifestation. Our experience framework adopts and builds on these three dimensions and further elaborates on them based on literature in the subsections that follow.

In these subsections, we look at literature on experience from both HCI and architecture fields. Specifically, we review theories and approaches in HCI and architecture by looking at how experience as a concept was shaped within each field.

2.2 The foundations of experience in HCI

Experience is a very prominent term elevated to a buzzword in HCI research and practice. In this subsection, we establish the foundations of experience in the field and how it was shaped throughout the years.

In the early years of HCI, which are often referred to as the human factors era or the first HCI wave (Harrison et al., 2007), usability was a prominent aspect of computer systems research; emphasising functionality, efficiency, and effectiveness in performing work tasks, with little consideration for the user experience of these interfaces. Computers back then were difficult and disruptive. As HCI entered its second wave of classical cognitivism and information-processing (Bodker, 2006), the Scandinavians recognised the computer-caused social disruption and reacted with participatory design (PD) (Zimmerman, 2011). As the field evolved and computers became more accessible and pervasive in this second HCI wave, researchers and practitioners identified the importance of designing computational systems and interfaces that were not only usable but at the same time provided a positive and engaging user experience, entering the user-centred design (UCD) era (Zimmerman, 2011).

Cognitive scientist Norman (1988) popularised the term ‘user experience' in his book ‘The Design of Everyday Things' stressing the need to consider human-centred design principles and to create products that were intuitive, aesthetically appealing, and user-friendly. This work paved the road for the influential concept of user experience (UX) in the 1990s and 2000s; with the prolific use of the term in the world of HCI and design. HCI's ‘experiential turn' marked its third wave (Harrison et al., 2007) which demonstrated the importance of subjective perceptions in shaping engagement and satisfaction. It introduced a more humanist agenda looking at what it was like for people to use such technology. This experiential turn coincided with technological developments in the area of ubiquitous computing (UbiComp) (Weiser, 1998) where technological artefacts started to become ever-present and embedded in people's lives providing richer experiences. These shifts underscored the importance of gaining a deeper comprehension of people's experience of technology.

Scholars and practitioners in HCI grappled with the seemingly nebulous and elusive notion of experience by introducing a plethora of definitions, theories, and classifications. Initial work on UX was rather programmatic (Bargas-Avila and Hornbk, 2011) and sought to build the foundation for this new area (Pucillo and Cascini, 2014). Some of this work used UX to refer to usability with some attention to hedonic or emotional features (Hassenzahl, 2004). Progressively, writings such as the seminal book ‘Technology as experience' by McCarthy and Wright (2004) provided a more holistic view on experience. For example, McCarthy and Wright (2004)'s framework explored the role of aesthetics, emotions, and engagement in user experience. According to them, ‘We don't just use technology; we live with it' (McCarthy and Wright, 2004, p.). However, something was amiss from this perspective on experience. Cognitive models defined design as problem solving and solution driven although the primary challenge in design lies in truly understanding the essence of the problem at hand (Mattelmäki et al., 2014).

In their reflecting endnote on their initial 2004 article, ‘Making Sense of Experience', Wright et al. (2018) highlight the missing ingredient in understanding experience: empathy. They emphasised the need for going beyond merely knowing the user to actively listening to their stories and understanding them. Attention in design practice and research shifted towards the sensory and bodily aspects of existence, fostering a sense of curiosity and exploration. As a result, following design practice's footsteps, HCI research introduced ample methods, approaches, and frameworks, such as work by Mattelmäki and Battarbee (2002), Koskinen et al. (2003), Horst et al. (2004), Mcdonagh (2004), and Freedberg and Gallese (2007), which placed empathy at the forefront of experience. This turn popularised the term ‘experience design' (Shedroff, 2001) and promoted a line of work in HCI where design was experience-centred (Blythe et al., 2006; Wright et al., 2008), and emphasised the importance of culture, emotion, and lived experience. Empathy though was not deemed enough when evaluating experience. Spiel et al. (2017) critiqued pragmatism approaches relying solely on empathy, such as Wright and McCarthy (2008), especially in cases where designers' daily lives and experiences differ greatly from those of the people they design with/for.

As the focus in HCI shifted from the workplace to the home and interfaces found their ways into our everyday lives and culture through physical, tangible artefacts, technology became a support for situated action in the world seen from a phenomenological point of view. In this view, all action is embodied. The phenomenological and situated nature of third-wave HCI centralised human experience in research and practice and focused on understanding emotions, movement, and embodied experiences. Coining embodied interaction, Dourish (2001) elaborated on embodiment as a central part of our interaction with technology. His definition was based on the phenomenological perspectives of philosophers such as Heidegger (2005) and Merleau-Ponty (2012) who challenged the Cartesian body-mind dualism and advocated that human perception and experience were grounded in our bodily interactions with the environment. In embodied interaction, direct action and shared meaning in the interaction with technology are inseparable (Dourish, 2001). Following this paradigm, HCI researchers explored people's subjective embodied experiences as situated agents in their interaction with technological artefacts through phenomenology (Svanæs, 2013; Hummels and van Dijk, 2015; Stienstra, 2015; van Dijk, 2018; Prpa et al., 2020) and soma design (Höök, 2018; Höök et al., 2019). Looking at the intricacies of subjective experiences resulted in a more nuanced understanding of interactions with technological artefacts.

As HCI moved towards tangible computing (Hornecker and Buur, 2006) and UbiComp, computing became entangled with the concept of space (Ciolfi, 2004b; O'Neill et al., 2006; Takeuchi, 2014). Further advancements in media architecture (e.g., Moere and Wouters, 2012; Fischer et al., 2018), interactive environments (e.g., Ciolfi, 2004a; Biloria, 2007), urban interaction design (UIxD) (e.g., Houghton, 2014), and HBI (e.g., Nembrini and Lalanne, 2017) denote that computation is now defined spatially too and that spatiality is integral to the experience of a computing system (Hornecker and Buur, 2006). However, few HCI scholars research spatial experience (e.g., Saarinen et al., 2012; Paananen et al., 2021; Hirsch et al., 2022) or space-related aspects of experience in HCI (e.g., Dalsgaard, 2008; Wiltse and Stolterman, 2010; Benyon, 2014). The majority of these works look at urban and outdoor spatial experiences (e.g., Freeman et al., 2019; Stals et al., 2019; Paananen et al., 2021; Hirsch et al., 2022), while the ones who do address experience in architecture and indoor spaces (e.g., Dalsgaard, 2008; Wiltse and Stolterman, 2010; Saarinen et al., 2012) do not provide a holistic framework on how to assess spatial experience with a physical artefact or its relation to other dimensions of human experience, such as embodiment.

Following this brief review of the notion of experience in HCI, in the next subsection we explore how the field of architecture approaches lived experience both in theory and in practice.

2.3 Experience in architecture

Architecture in itself is a hybrid entity that serves two masters; nature and culture, art and science. Therefore, theories, methodologies, and methods from other disciplines weave their way into the architecture field. The notion of experience in architecture is no exception to this rule; experience has been defined and explored under diverse epistemologies and by borrowing theories from other fields. In this section, we outline theory and works on experience in architecture from various schools of thought and examine the role of experience in architecture.

Not far from third-wave HCI theories on experience, architecture scholars, such as Rasmussen (1959) and Norberg-Schulz (1980) and more recently (Vesely, 2005; Holl et al., 2006; Pallasmaa, 2012; Pérez Gómez, 2016; Seamon, 2023), have argued that architecture is experienced through our senses and can elicit emotional and psychological reactions. These theorists drew from Heidegger's (1971) seminal writings on phenomenology, ‘Building, dwelling, thinking', in which he theorised that architecture and the built environment play a crucial role in shaping human experience, and to a lesser extent from Husserl (1989) and Merleau-Ponty (2012). Scholars who follow this phenomenological school of thought, refer to these experiences as ‘sensory.' Early on, Rasmussen in his book Experiencing Architecture defined experience as the way in which architecture engages and affects our senses, emotions, and intellect (Rasmussen, 1959). Rasmussen proposed that architecture should be designed not only for functionality purposes or to be seen, but to be experienced in a holistic way that engages all of our senses. In his own words, ‘it is not enough to see architecture; you must experience it. You must observe how it was designed for a special purpose and how it was attuned to the entire concept [] You must dwell in the rooms, feel how they close about you' (Rasmussen, 1959, p. 33). Advocating for a human-centred approach to design, Rasmussen prioritised the needs and experiences of people over purely aesthetic or functional considerations. However, Norberg-Schulz (1980), an esteemed architecture theorist, critiqued Rasmussen's theory (Rasmussen, 1959) on the premise that it failed to acknowledge that, aside from individual or collective memories and ‘multi-sensory feelings experienced when being-in-the-place' (Lewicka, 2011, p. 221) the meaning and identity of physical spaces derive from the symbolic (e.g., religious or national) content of architecture. His influential work, Genius loci — the spirit of place — emphasises on the importance of place and the sensory and emotional aspects of experiencing architecture in contrast to Euclidean-defined space (Norberg-Schulz, 1980).

Focusing on sensory stimuli as well, the world-renowned architect Zumthor (2006) begins his architectural manifesto, Atmospheres, by describing an early memory of his aunt's house saturated with touch, smell, and sound experiences-stimuli exemplifying how the materiality and sensual qualities of the surroundings can evoke experience through memory (Sharr, 2007). Similarly, philosopher Gaston Bachelard stressed the importance of imagination and the senses in experiencing architecture, arguing that buildings and spaces have a poetic quality as they can be imbued with personal and cultural significance (Bachelard, 1994). This poetic quality can evoke feelings of nostalgia, longing, and intimacy and thus impact human's emotional and psychological well-being (Bachelard, 1994). Echoing Zumthor (2006) and influenced by Heidegger's (1971), Norberg-Schulz (1980), Bachelard (1994), and Merleau-Ponty (2012), Pallasmaa linked emotion, imagination, memory, and perception in architectural experience as embodied, interconnected acts (Pallasmaa, 2014). Building on Dewey's (1934) ‘immediate, embodied, emotive, and subconscious essence of experience' (Pallasmaa, 2014, p. 230), Pallasmaa (2014, p. 230) emphasises the immediate, complex, emotive, and multi-sensory nature that characterises the experience of an architectural space which is ‘grasped as an overall atmosphere, ambience, feeling or mood.' Like Rasmussen (1959), he stresses the importance of embodied and especially tactile experiences in architecture, by critiquing the hegemony of designing for visual experience that characterises architectural practice (Pallasmaa, 2012).

Although very prominent, the operationalisation of these phenomenological frameworks in the form of case-studies and empirical investigations occurred only in recent years within the field of architecture. Some examples involve investigating the authenticity of place in relation to the urban built environment (Wesener, 2016), people's spatial experiences of ambient atmospheres (Jouan et al., 2021; Pritzen et al., 2023) and healthcare settings (Annemans et al., 2018), people's everyday bodily experience of the urban environments (Peri Bader, 2015; Bader and Peri Bader, 2016), and architects' lived experiences during COVID-19 lockdowns (Marco et al., 2022), among others. Albeit limited in numbers, these case studies contribute to a deeper understanding of the lived experiences within architectural contexts. First and second-wave HCI topics found their way in the architecture domain as well. For instance, similar to the first wave HCI Eilouti (2023), introduced an architecture ergonomics framework. In addition, scholars wrote on incorporating usability in the built environment (Andersson, 2004) in combination with UX principles (Andersson, 2004; Juliá Nehme et al., 2020; Attaianese and Sarmento, 2022; Vilar et al., 2022).

Other approaches in architecture look at the social experience of buildings, such as the applied theory of space syntax which was conceived by professor of architectural and urban morphology Bill Hillier, together with Julienne Hanson, and other colleagues from the Bartlett (UCL, London) (Hillier and Hanson, 1984). Space syntax explores the mutually constructive relation between society and space in built environments, which supports that space is an active component that shapes social interactions and human behaviour. Space syntax investigates the influence of spatial arrangement and connectivity in social interactions and movement. Hillier (1996) asserts that a building's multi-functionality, aside from producing and protecting space, encompasses three interconnected core aspects: physical, social, and spatial. The physical dimension refers to the building's construct (walls, floors, ceilings, external envelope) whereas the spatial refers to the space in which human activity occurs (Hillier, 1996). The social value of a building is carried by both its physical and spatial dimensions: the socio-physical by shaping and decoration of elements that hold functional or cultural significance with functional or cultural significance, and the socio-spatial pertains to the provision of spatial arrangements that dictate patterns of activities and relationships. According to Hillier (1996), space syntax sets off from the idea that the evidence for spatiality must be the ways in which human beings organise and arrange real space.

Research aside, experience is prioritised in architecture practice too, however, not in the same way interaction design practice centralised experience. For example, the term UX is not popularised in architecture and there is no UX Architecture the way there is UX design. As a central part of architectural design, experience encompasses aspects such as sensory engagement, emotional resonance, and spatial narratives. Architects strive to craft environments that evoke particular feelings and reactions, stimulate the senses, and facilitate meaningful interactions with and through the built environment (Lawson, 2003). To achieve this architects rely on strategies such as designing based on human proportions, resonating sensory qualities, attending to social dimensions, planning volumetric space and building circulation. The notion of experience in architecture is closely linked to the concept of place-making (Tuan, 1975), where architects create buildings that align with the specific cultural, historical, and contextual aspects of the place. By understanding and integrating the unique characteristics of a place, architects can design a building harmonious with its surroundings and, thus, contribute to the sense of identity and connection.

Nevertheless, in conventional architecture projects aside from the main client, there is a lack of involving people who will inhabit the end product of the design process. Actions like taking into account their needs, dreams, and aspirations before setting on a design or assessing the building occupant's satisfaction and frustrations with the resulting design, are generally absent. Even in the case of post-occupancy evaluations¹ (POE) (Preiser et al., 2015; Hay et al., 2018) the findings are rarely conveyed to the architects as a form of feedback and recommendations for future projects. Perhaps one of the reasons for this lack of probing occupants' experience in architecture is the long-standing tradition of the field. As Achten and Kopriva (2010) propose, architects take for granted how people act around certain configurations of the built environment due to experience and tacit knowledge passed on through architecture education and practice.

2.4 Juxtaposing experience in HCI and architecture: commonalities and differences

In his article, ‘Do Architects and Designers Think about Interactivity Differently?' (Kirsh, 2019), a cognitive science professor who drives initiatives in neuroscience and architecture, argues that architecture takes on a more embodied and social approach than HCI. He suggests that meaning and narrative are aspects that propel architectural design as opposed to efficiency and functionality that frequently drive the HCI field (Ingram, 2009; Kirsh, 2019). This position only holds true for the two first waves of HCI, whereas in its third wave, HCI has witnessed a turn towards approaches that centralise embodied and social aspects of experience. We find that both fields have similar views on the importance of embodied and social aspects of experience.

A main difference in the conception of experience in HCI and architecture is that experience in architecture is mainly concerned with the design of architectural form. The research examples that investigate the lived subjective experience in architecture do so on existing building structures. To the best of our knowledge, research in the architecture domain on the design, implementation, and assessment of lived experience has only become popular in recent years and not to the same extent as research on lived experience in HCI. Within the practice of architecture, the lived experience is imagined and simulated before construction, however, after the structures are built and inhabited, experience assessment is most frequently performed through quantitative tools (POE) measuring building performance rather than collecting qualitative personal accounts of the users. Another difference worth mentioning is while experience in HCI is a broadly-researched aspect of taking the embodied and social dimensions into account, the spatial element of experience is under-explored. A limited line of works that stem from the areas of UbiComp and interactive environments address the spatial experience of computing systems or the space-related aspect of experience, such as McCullough (2005); O'Neill et al. (2006); Lentini and Decortis (2010); Saarinen et al. (2012); Paananen et al. (2021), but not its relation to the other two dimensions of experience (social and embodied aspects).

Addressing the lack of a comprehensive approach to the lived experience in HBI and the aforementioned gaps, we introduce an initial framework aiming to provide a holistic view of how these three dimensions (embodied, social, spatial) interrelate in the context of HBI artefacts. In the following section, we present a case study which informed the development of our framework.

3 Designing for the embodied lived experience in HBI: a case study

The case study presented here is part of a larger project, Hybrid Interactions in Vibrant Environments (HIVE), dedicated to the scientific and novel improvements of redesigning early childhood education and care facilities. HIVE is a three-year project that combines architecture, environmental psychology, and digital technologies by bringing together researchers from the HCI Division, Department of Artificial Intelligence and Human Interfaces of the University of Salzburg with backgrounds in HCI, architecture, and early childhood education, and the Ecomedicine department of the Paracelsus Medical University. In what follows we present our case study of designing, constructing, deploying, and evaluating an HBI artefact within the broader scope of this interdisciplinary project.

3.1 Case study context and approach

Within the scope of the HIVE project, our team of HCI and design researchers collaborated with a team of environmental psychology scholars and architects and employed a PD approach with the community of a selected public kindergarten in Salzburg, Austria. The kindergarten community comprised of young children, teachers, and the headmistress, during the design and planning phase of their kindergarten building's renovation. The participating kindergarten was recruited by the partnering company, Salzburg Wohnbau, which is the municipally-selected service provider responsible for the planning and construction of the kindergarten building's renovation. The project explored how new living and learning spaces can be designed to foster a symbiotic relationship between children, their teachers, interactive technologies, and architecture. Based on the needs, wishes, and requirements expressed by children and teachers of the selected kindergarten, we designed and deployed two technological interfaces that support their everyday activities. The project has received trust and support from its corporate project partners: the municipal construction service provider Salzburg Wohnbau GmbH, the partnering construction company, STRABAG, the region's electricity provider, Salzburg AG, and the architects in charge of the upcoming building renovation of the selected kindergarten.

In a traditional PD approach within the context of a building project (Rigolon, 2011), design activities are preceded by an analysis of the status quo in the given context and the users' needs through methods such as interviews and questionnaires, or drawings and photographs when involving children. This needs analysis provides starting points for debating the design phase that follows. In the design phase, the community is invited to contribute ideas through creative activities such as design charrettes (Sutton and Kemp, 2006), model making, and drawings. The construction phase comes after the refinement of a selected idea. When the construction is complete, a final evaluation occurs where the community is asked to reflect on the developed designs and contrast them with their initial aspirations. As a last step, Rigolon (2011) suggests leaving space for ongoing modifications so that spaces can be adapted to accommodate users' changing needs.

In our participatory design approach we deployed an adapted version of cultural probes (Gaver et al., 1999) for the children and a contextual inquiry in the form of interviews, surveys, and a building tour performed with the teachers and the headmistress of the kindergarten. The consolidated findings from our probes returns and the contextual inquiry, which we wrote about in another publication (Economidou et al., 2023), revealed the children's need for controlling their environment through place-making and for noise mitigating strategies. These findings formed the inspiration for performing a set of PD workshops which resulted in the idea of an artefact-based HBI prototype that would support children's place-making actions and combat noise in their kindergarten environment. As our focus in this article is the prototype's assessment, we briefly present the design concept and the final design without referring to the design process and final fabrication in detail. However, it is worth mentioning that the findings from our interviews with the teachers indicate that our artefact lowered noise in the room and evidently from the children-made architectures, the children were able to create place-making structures.

3.2 Design: concept, development, and implementation

As the aim of our prototype was to support place-making, we find it important to frame space, place, and place-making in relation to our study's research objective.

One cannot define place-making without referring to ‘space' and ‘place', the two conceptually-loaded concepts which carry diverse meaning (Ciolfi, 2013). According to human geographer, Tuan (2011), ‘space' is generally perceived as a specific geographic locale that can be defined through coordinates, while ‘place' is a ‘space' endowed with value, meaning, and purpose to which an individual forms connection (Cresswell, 2004). Inspired by the two notions, HCI researchers further support that ‘place' holds social meaning and cultural understandings and is governed by notions of behavioural appropriateness like the concept of ‘home' (Harrison and Dourish, 1996). Place-making is the process of creating meaning through the creation of place which triggers emotional and intellectual responses (Rigolon, 2011). Based on Harrison and Dourish (1996), designers can only provide the tools for a space to become a place, the rest is up to the occupants and community. Therefore, one of the aims of our HBI interface was to support children's place-making actions, which can be either individual or collective (Falk et al., 2023), by providing them with the tools for creating retreat areas and other place-related structures.

In HBI, the technological interface is an environment, which according to Benedikt (1979) is open ended and immobile. However, we argue that HBI artefacts may not be embedded in the building structure as they can be mobile, such as the actuating wall example by Nguyen et al. (2022). Moreover, HBI artefacts can be smaller tangible interactive interfaces or TUIs (e.g., Moradi et al., 2018; Zhong et al., 2020; Economidou et al., 2021; Nabil and Kirk, 2021). Our design concept consisted of an artefact-based HBI interface comprised of a set of interlocking panels and projections with interactive capabilities. These artefacts facilitated place-making by creating (sub)spaces within a given environment (e.g., group-room) and floor-based projections acting as backdrops to children's activity that could be manipulated using a set of tangible controls. Other interactive artefacts using partitions (e.g., Peiris et al., 2011; Hyelip Lee et al., 2013; Peiris et al., 2013; Onishi et al., 2021; Nguyen et al., 2022) and floor projections (e.g., Mora-Guiard and Pares, 2014; Mora-Guiard et al., 2016; Takahashi et al., 2018; Kender et al., 2020) exist in HCI research; however, in their majority, these examples are not classified as HBI artefacts and none of these works investigates the role of the artefacts in facilitating place-making actions by these artefacts.

We explored our design concept through a series of sketches, scenarios, mock-ups, 3D CAD models, and 1:1 cardboard models before moving on to the final physical implementation (see Figures 1A–D). We opted for keeping the design as open-ended and flexible as possible to avoid restricting children in their activities and encourage exploration and appropriation. For example, we allowed freedom in constructing spaces of any arrangement and did not prescribe what activities they could do with the provided artefact. The flexibility and adaptability of the design renders the artefact capable to be spatially reconfigured as required, accommodating changing needs over time.

As a final step in the design process, a high-quality final prototype was designed and fabricated. The prototype was deployed and evaluated in the kindergarten context (see Figures 2A, 3A). It was further refined a final time with the intention to be deployed in the kindergarten as long as it is deemed usable and needed. As of December 2022, the prototype remains deployed in the kindergarten for an indefinite amount of time.

The final design concept (see Figures 1C, D, 2A) targets place-making and space division actions according to spatial needs by projecting different circular play areas on the floor and through 12 interconnecting space divider elements. The children can browse through the projection themes (a farm, a race track, water, a village, etc.) and modify them in size based on their different play activities. The projections can be made larger (maximum diameter 2 m) and smaller (minimum diameter <5 cm) and switched through sequentially. Floor projections were chosen for their flexibility, adaptability, and versatility as they are easily deployed in various locations and can be adapted to different spaces by means of size, placement, and content. They are also suitable for indicating spatial areas and fostering group participation and shared experience. Moreover, their implementation is straightforward as opposed to other artefact-based HBI interfaces such as shape-changing interfaces (e.g., Suzuki et al., 2020; Economidou and Hengeveld, 2021). We opted for circular projections as those would fit better within the triangular-shaped areas of our projector mounts rather than the regular rectangular projections (see Figures 1D, 3D). The divider elements are made out of wood and are self-standing. These elements can contribute towards both place-making and lowering noise levels as they are covered in passive noise-absorbing hive-shaped foam. The 12 wooden panels come in three different shapes and can be connected to one another to form structures like walls using connecting pieces that ‘lock' the panels together sequentially (see Figure 2D).

3.2.1 System description

Our design features a ‘tower' element holding in place three projectors facing the floor for surface projections. Each of the three floor-projected areas around this tower structure has a circular shape which can be controlled via two building blocks placed on the floor and a button switch embedded in each surface of the structure holding the projectors, at a reachable height (three switches in total). Both the building blocks and the button switches are designed to be used and manipulated by the children. The technology is implemented in Python on a Raspberry Pi 3 microcontroller with three connected cameras facing the floor. On the surface of each block there is an ArUco marker sticker that is detectable by the camera and recognised by the microcontroller using computer vision technology (see Figure 2B). By moving the two building blocks apart from each other on the floor surface the projected circular area becomes larger, while moving the building blocks closer together makes the projection smaller (see Figure 2C). There are six building blocks in total, two for interacting with each projection. Each projection's theme can be changed using the corresponding button switch.

3.3 Method

Our study's objective was to investigate young children's experience of interacting with our HBI artefact. Since, as mentioned before (see Section 3.2), place holds social meaning and behavioural appropriateness (Harrison and Dourish, 1996) we chose to focus on children's social interactions. As we wanted to discover if and how children would appropriate our designs, we inquired on the changing role of the artefact. Additionally, as ‘spatiality'—the everyday space encounters and opportunities for action the space affords—is a product of lived experience and embodied action (Dourish, 2006), we chose to include embodied actions. Finally, we chose to look at moments of conflict as conflict among young children may reveal feelings, intentions, and social rules (Shantz, 1987) and points raised through conflict are usually the most informative (Sengers and Gaver, 2006), therefore, they could reveal nuance on children's experience with the HBI artefacts. We pose a research question for each of these dimensions as we would like in order to investigate how they interrelate with one another. Consequently, the study analysis was guided by the following research questions:

RQ 1: What are young children's lived experiences of our HBI artefact?

RQ 1A: What social interaction is observed when children interact with each other using the artefact?

RQ 1B: What is the role the artefact assumes? What kind of places do children create?

RQ 1C: How do children bodily interact with the artefact's elements?

RQ 1D: For what and when does conflict occur?

3.3.1 Participants

For our four-day evaluation study, we recruited two of the four groups of children attending the kindergarten. Each of the two groups has their own dedicated and colour-coded group room and consists of up to 25 children of ages three to six years old. Each group used the prototype for 2 days. Most children hailed from the same region, apart from a second-generation immigrant child who displayed no issues in interacting and communicating with the rest of them. The children were familiar and at ease with each other. The first group consisted of 16 children (10 girls and 6 boys) on the first day and 15 children (9 girls and 7 boys) on the second day. The second group consisted of 17 children (12 girls and 5 boys) on the first day and 19 children (13 girls and 6 boys) on the second day. A total number of 36 children interacted with our artefact during the entire duration of the study.

3.3.2 Study design and procedure

The study design consisted of one time slot for data collection (data was collected through video material, interviews, handwritten notes, and photographic material). The collected data was transcribed and underwent two rounds of data analysis. Prior to the deployment of our prototype, we established rapport with the children in the form of postcards that informed them of our arrival and the purpose of our visit. Some children already knew three of the members of our team from the previous workshops we performed with them during April and May 2022. As a result, the children were told about our arrival by the teachers and anticipated our visit. All participating teachers and children's parents or guardians signed an informed consent form which was approved by Salzburg University's ethics board.

Over the span of four days in November 2022², our team of four researchers with backgrounds in HCI, architecture, exhibition design, and early childhood education conducted a study deploying the HBI prototype in two of the four group rooms of the kindergarten. The prototype was placed in each group room the morning of the first day of the study, before the children arrived at the kindergarten. The prototype remained in each of the two group room for two consecutive days for children to explore and make use of it in their environment. Two members of our team were present during the study, one researcher introduced the artefact to the children, answered questions, conducted contextual interviews and made sure there were no technical issues, while the other research member took notes on the ongoing interactions.

3.3.3 Data collection

Although traditional observation methods can provide a spectrum of insightful information on experience, it could be a challenging to capture details in a live setting, especially when complex interactions occur simultaneously among participants. We chose to capture interactions through video recording as it would allow us to synthesise and interpret data and verify them against children's views, values, and emotions obtained from short contextual interviews with them and their teachers. In addition to video and short contextual interviews on the observed interactions we opted for taking handwritten notes and photographic material on surprising or unexpected aspects that caught our attention.

3.3.4 Data collection instruments

To capture video material, two partially hidden wide-angle video-cameras were placed on opposite sides of the room facing the artefact. Contextual interviews were recorded using a specialised audio recorder. All sensitive data was handled following the European GDPR rules.

3.3.5 Data analysis

We conducted eight interviews (four individual interviews with teachers and four group interviews with nine children) which were transcribed verbatim and anonymized (interview transcripts translated to English can be found in Supplementary materials S1, S2). The video material was organised by day and prepared for analysis. We used ELAN (RRID:SCR-021705), a computer software for multi-modal data analysis, to analyse our videos. For the video analysis, two research members went through all the video material from both cameras and identified moments where the artefact changed role or was used differently than designed, moments of conflict, embodied interactions, and social actions by assigning a tier for each of these categories and marking these moments with descriptive annotations (see Supplementary material S3). When multiple interactions happened at the same time, these were coded by adding additional tiers in the same category. These annotations were then exported as a video transcript. For the second part of the analysis, we analysed both the video and interview transcripts using reflexive thematic analysis as introduced by Braun and Clarke (2006) due to its versatility. The main author coded the dataset in two iterations using MAXQDA, a qualitative data analysis software. As a first step, the coder immersed herself in the data by carefully reading and re-reading through the transcripts. We assigned initial code segments to each piece of data in the dataset inductively, split into answering each research question; an action which produced 183 unique codes (see exported codes from MAXQDA in Supplementary material S4). The coder used memos to keep track of code changes and merges, clustered the codes together and, finally, constructed themes relevant to the research question and aims (see Supplementary material S5).

3.4 Results

Regarding our research questions, the second round of thematic analysis led us to themes that were prominent throughout the entire dataset, as seen in Figure 4. Notably, these themes which were related to the research questions are interconnected and cannot be interpreted in isolation. We identified six themes related to social interactions (RQ 1A), seven themes regarding artefact appropriation (RQ 1B), three themes relevant to embodied actions (RQ 1C) dimension, and three themes concerning conflicts (RQ 1D). In the following section, we describe and reflect on these themes in more detail.

3.4.1 Social interaction

Six broad themes were constructed in relation to children's social interactions with the artefacts: place-making actions, inclusion, exclusion, playfulness, expression of values and affection, and calling for attention.

The largest theme, place-making actions was further divided into two sub-themes: creating areas of retreat and sense of ownership. Children created areas of retreat within their group rooms (see Figures 3A–C, 5) using the divider elements where they nested, shielded, or hid from other children and adults. A strong sense of ownership was observed when children ‘stole' artefact elements from one group to another by removing them from where they were initially placed and placing them in another area that was owned by ‘another' group of children. This held true for both parts of the artefact, the projection markers and the panel connectors. The children were more interested in ‘owning' the projection markers than control the projections with them. Similar behaviour was exhibited regarding the projection switches, where children would have conflicts over who would press the button or agree and proceed to press it one after the other. Regarding the divider elements, this theme was quite evident in one of the kindergarten group rooms, as the children created two areas which were divided by gender (see Figures 5D, E). When children belonging to one group attempted to enter this created area they were met with hostility, with children defending their fortified area by pushing other children's hands and bodies away. This behaviour was categorised as border protection. In a final occurrence where the sense of ownership was observed, children attempted to take an object from another child, where the object was either a projection marker or a panel connector.

The second largest theme we noted was inclusion which was evident when children collaborated with each other towards a common goal. For example, some children exhibited collaboration in making the projected area larger and/or changing its theme, by taking turns or handling one of the two projection markers often attributing the changes to magic. At times, they also decided on the projected theme together. Additionally, children collaborated in cleaning up and organising their group room, formed social groups, and allowed entry to other children in their created area.

In contrast, exclusion was the third theme we took note of. We noticed children preventing or obstructing interaction by placing their body in front of the projection switch, pushing other children's hands away or taking and throwing the projection markers and connectors away from another child. With regard to the divider elements, children prevented entry in and/or exit from the created structures, with arguments that escalated into physical conflict, such as shaking the divider elements as one would shake a fence, and children being pulled into or pushed out of the created area by another child. In some cases, children proceeded to ‘imprison' a child in an enclosed area by blocking the exit and not letting them get out.

Children expressed playfulness through various behaviours. They engaged in mimicking other children's actions, through role-play (i.e., playing house), playing on the floor within the floor projections with or without toys and artefacts, running around in the room chasing other children, and teasing other children using the projection controls (e.g., by placing an object such as a projection marker on another child's forehead). Moreover, we observed a strong hierarchy in children's pretend play activities, with older children taking the lead and dictating other children's course of action.

A compelling theme we noticed was the expression of values and affection among children. Curiosity was a prominent value with children gathering around others performing an action (e.g., connecting boards together or modifying the projections on the floor) and observing other children interacting together using all the elements of the artefact. A surprising category was the strong expression of designated use where children's behaviour indicated that some children believed there was a ‘right way' to handle artefacts and therefore corrected or undid other children's actions (e.g., by removing placed or stacked connectors, gathering markers together in a pile, and placing the divider elements in the spot they placed them initially after other children had moved them). Generosity and cooperation were strongly expressed values with children exchanging or giving the objects to one another (e.g., toys or the smaller parts of the artefact, projection markers and connectors). In sporadic cases, the children expressed affection for the divider elements of the artefact by placing each of their hands on either side of the board and leaned forward in an embrace, as if they were hugging them.

The final theme with the lowest observed occurrence was the calling for attention theme. Children expressed their need for attention by showing other children or the teachers their creation using the artefacts (e.g., holding close to another person the stacked connectors or verbally attracting attention by calling someone's name). They also pointed to artefacts, such as their toys in the projected images, or waved at another child on the other side of the created area.

3.4.2 Appropriation

Depending on the scale, form and function of the artefacts, the children inscribed to the artefact various roles. We merged these observed roles into seven themes; place-making structure, tool, toy, scaffolding for play, switch, valuable possession, and personified or animate object.

By far, the largest theme we observed was the place-making structure role for both artefact elements where the artefacts were put to use by the children to indicate, create, or expand areas and sub-divisions within the kindergarten group room (see Figure 5 for created structures). The artefact with the projectors served as a centrepiece with children running around it and hiding behind its structure. Most prominently, the room divider was handled as an area-indication element, that periodically portrayed wall or space divider, a door (see Figure 5B), a threshold, or an obstacle between the created area and the ‘outside' (see Figures 5D, E). As shared with us in the children's and teachers' interviews, the created areas were perceived as houses, prisons, caves, castles, police stations, or hide-out nooks. Other place-making structure roles the divider elements assumed were for creating a hideout, a nest, and pieces of furniture that children would lean or sit on, place fabrics to create a roof over them (see Figures 3B and 5A) or decorate them by placing other elements on the foam surfaces.

In the second most prominent theme, the artefacts were seen as tools, with the smaller parts of the artefacts used as fabric weights, fastener supports, and reinforcement tools that connect the artefact pieces together. For instance, the connector pieces were used to fasten one divider element to another by placing the protruding elements of the connector onto the holes on the side of each divider element (see Figure 6A). To reinforce this connection, the children stacked the connectors on top of each other. The same pieces were alternatively used as weights placed around the perimeter of the textile ‘roofs' to secure them on the divider elements (see Figure 6B). The divider elements were perceived as support tools where children could lean on, rest their heads on, or hold on to.

The smaller parts of the artefact, the connectors and the projections markers, were handled as toys in the third theme as they were stacked one on top of the other like one would make a tower using building blocks. These stacks or towers were then demonstrated to other children or adults in the room, left as towers in the middle of a projection, or towering connectors of two divider panels and later on dismantled or demolished.

An unexpected discovery was the fourth largest theme where all elements of the artefacts served as scaffolding for play. We witnessed children letting their creativity roam free and imagining the artefacts as scaffolding or backdrop to their play activities, by bringing other toys into the projections or engaging in pretend-play and other activities (making puzzles, drawing) in the created areas. For example, one child placed dinosaurs in the projected area and switched the pattern to ‘desert' and left them there (see Figure 2C), while another used the angled surface of the top part of the divider panel as a slope for their car toy. A frequent sight was children using the race-track-themed projection for their toy cars or walked with one foot after the other in the projected ‘lanes'.

As intended by design, children perceived the buttons on the projector artefact as switches for changing the projections as this change had an immediate effect on the environment. Children were fast in linking their button press action with the immediate effect of changing in the projection theme and were enthusiastic about altering the projected circles, and pressed the switches are random intervals while passing by the structure.

In the second to last theme, children saw the smaller pieces of the artefacts as valuable possessions or collectables as they gathered them in piles or in baskets and placed them on the divider element foam pieces as decoration (see Figure 6C).

Surprisingly, in a pair of cases and the least observed theme we noticed the children were perceiving artefact elements as personified or animate objects, for example, children were hugging and kissing the divider elements.

3.4.3 Embodied (inter)action

We captured bodily-interaction-related behaviours split among the manipulation of an object, bodily motion in space and physical gestures.

Manipulating an object was the largest observed theme and it concerns a wide range of interactions with an object which we divided into the two large groups of gross motoric movements and fine motoric movements and a smaller group of interactions using the feet. The children performed gross motoric motions across the room when they mainly lifted, held, carried, and placed the divider elements in order to create sub-areas within the room or divide the room into smaller areas. Other motions included in this category were pushing and shaking the divider elements and throwing objects over or under the board, towards or away from other children. The fine motoric movements category involved movements mainly performed using the hands such as moving the building blocks with the ArUco markers across the floor to enlarge the projections, gathering, or stacking the building blocks or the connectors (see Figure 6D), connecting or disconnecting the divider elements, touching the artefacts with their hands to explore their shapes through pressing or finger- and hand-tracing the various elements. The smallest category observed in this theme had to do with using the feet to interact which involved kicking either the divider elements or the markers and connectors away from or towards other children.

The second largest theme was bodily motion in space which was divided into three categories, motion on the floor, motion around an artefact, and bending or reaching over an artefact. The first category, motion on the floor, involved children sitting or lying on the floor next to or on the projected area, crawling on the floor in the created areas, and rolling on the floor in the projected space. The second category, motion around an artefact, included reaching or bending over the divider elements, squeezing through between the divider elements and the wall surfaces, shielding from other children, observing others and jumping into, dancing or walking around the projected areas on the floor. The third category, bending or reaching over an artefact, involved children bending over the divider elements or reaching over them either due to conflict or out of curiosity.

The final theme involved physical gestures split into the categories of pointing to artefacts or pointing to grab attention, waving at the projection artefact, tapping the divider elements as if knocking on a door, hugging the divider elements, and finally, showing their creations to others.

3.4.4 Conflict

To capture conflict, we coded children's behaviour in moments of dispute, rivalry, and dissidence or clash. In general, the video analysis suggests that conflict occurred when feelings and behaviours related to the sense of ownership, exclusion, and designated use were present.

To a large extent, the sense of ownership was a prominent factor in conflicts, particularly related to the smaller artefact elements. Children exerted proprietary rights over the connector and projection markers by ‘stealing' them from other children or groups of children, and taking them by throwing them away from others and preventing others from touching them, reaching them or removing them from their perceived ‘designated' spot, the final conflict-igniting theme. The sense of ownership led children to treat another child's entry in an area enclosed by the divider elements as ‘trespassing'. The trespassing element was present in the two counter-perceptions of border breach and border protection, where children either breached a ‘foreign' area or protected the area they considered as theirs. For example, when a child attempted to enter an area, either indicated by the divider elements or by the projections, that another child felt belonged to them or their peer group, they were then considered as a trespasser, resulting in conflicts (e.g., see Figure 3C).

The trespassing category is closely linked to the theme of exclusion, with the offender, most often, physically confronted by being pushed or pulled out of the perceived territory, thus, preventing or obstructing the child's entry into the area. Ostracised children attempted at trespassing again or reacted by shaking the boards as one would shake a fence. Aside from entering, exiting the area was sometimes obstructed as well, with ‘imprisonment' or being held in an area against one's will being another cause of conflict. This let to children being excluded from certain areas and play activities.

In instances where we observed designated use children behaved in a way that indicated that the artefacts had to be used or handled in a certain way. Action that deviated from these unwritten rules was met with conflict which resulted in children correcting or undoing the wrong-doers actions and stealing from one group to another. Some of these unwritten rules were: ‘closing' the ‘door' whenever a child entered or exited a created area or playing only with specific toys in the projected themes (e.g., no dolphin toy was allowed in the desert theme projection).

3.5 Reflection on findings

Upon further scrutiny and reflection on our developed themes, we discovered a set of experience aspects through the interconnections among the four research questions. These aspects have the form of five descriptive word pairs: privacy—publicness, designed affordances—unplanned affordances, inclusion - exclusion, spatial arrangement—decoration, and bodily motion in space—direct manipulation. These aspects feed into our framework as elements of the interrelated embodied, social, and spatial dimensions of HBI experience.

3.5.1 Privacy and publicness

A lot of our identified themes revolved around the two concepts of public and private. We observed that conflict arose in relation to ownership, and the constructed artefact roles. Friction occured when there was a discrepancy between privately owned (by a group of children) or a publicly owned (by all the children in the room) artefact or area. Similarly, conflicts arose when children individually (privately) assigned a role for the artefact opposed to the commonly or publicly constructed role of the artefact (e.g., opening or closing the door panel of their construction). As supported by Altman (1975), privacy is the ability or expressed agency of an individual to regulate the level of social interaction whereas other scholars noted that privacy and publicness are inherent aspects of spatial experience (Kuliga et al., 2013).

3.5.2 Inclusion and exclusion

In close relation to the privacy and publicness dichotomy, inclusion and exclusion were pivoting aspects of the experience. Segregating the developed places between public and private led to either including children in the created area or excluding them. Spatial exclusion leads to marginalisation of people from a community. In our kindergarten case study, this led to conflict and the expression of negative feelings (e.g., shaking the divider elements in order to enter the created area or conflicts over ‘trespassing'). On the other hand, spatial inclusion fosters community and a sense of belongingness. As seen in our case study, children reacted to inclusion by collaborating and sharing a common understanding over the ‘play rules' and roles of the artefact.

3.5.3 Designed and unplanned affordances

Themes related to embodied action and the constructed/assigned role of the artefact reflect the designed affordances³ (in line with the intentions of the designer) (Norman, 2005) and unplanned affordances (unintended affordances that were not planned by the designer) of the artefact. Unplanned affordances lead to children appropriating the form, size or other attribute of the artefact for their own purpose through gross or fine motoric movements. For example, when stacking connectors on top of each other or creatively using the tops of the divider panels as slopes for their car toys.

3.5.4 Spatial arrangements and decoration

We took note of spatial arrangement actions as opposed to decoration actions among children's place-making activities. The creation of spatial arrangements within the room is an act of place-making that holds a socio-spatial value dictating patterns of activities and relationships, a finding also supported by Hillier and Hanson (1984). Arrangements in our case affected bodily movement in the room and created boundary places. Smaller elements of the artefact (ArUco marker building blocks and connectors) were used as decoration or ornamentation as another place-making action which holds a socio-spatial significance. By decorating a space, it becomes a meaningful place that can foster a sense of belonging with the place and community, as explored by Nabil et al. (2018).

3.5.5 Bodily motion in space and direct manipulation

Lastly, we noticed the prominence of bodily motion in space and direct manipulation of the HBI artefact through touch. Through bodily motion in space, children used large movements such as full body interaction and motion around the artefact and the created places. Through direct manipulation, they used touch and gross or fine motoric movements to manipulate objects in relation to the HBI artefacts. This distinction among spatial and tangible interaction whether the artefact is an object or an environment falls in line with findings by Hornecker and Buur (2006).

4 An experience framework for assessing lived experience in HBI

Underpinned by the literature discussed in Section 2 and selected theory in Section 4.1, we propose an HBI experience framework as depicted in Figure 7. The framework is grounded on the embodied, social, and spatial dimensions (Figure 7A) of HBI experience drawn from related literature and theory. An active network schema on the relations of the three dimensions can be seen in Figure 7B. The three dimensions are detailed through ten aspects of experience split into five descriptive word pairs (Figure 7C) which derived from our case-study findings (Section 3.5). This framework aims to serve as a post-design analytical tool to understand better the experience of HBI artefacts.

4.1 Framework foundation: dimensions of HBI lived experience

Our literature review on experience in both constituting fields (see Section 2) as well as related work (Alavi et al., 2019a,b) and relevant theory (affordances, ANT, and embodiment) discussed below provides the foundation of the three dimensions our framework builds upon: embodied, social, and spatial, as seen in Figure 7A. The dimensions are of equal importance and are presented below.

• The first dimension of experience refers to embodied action

The experience of HBI artefacts is fundamentally embodied as we, humans, are embodied entities. Our bodies interact with such interfaces through bodily movement and actions which are also impacted by the design and physical affordances of HBI artefacts. The term affordance, coined by Gibson (2015), pertains to the perceived action possibilities within an environment (Norman, 1988), regardless of whether the designers originally envisioned those possibilities or not. Through affordances we determine actions with the physical world, such as whether an object is suitable for lifting, throwing, turning, reaching, sitting on, and so on. Norman (1988) gives the infamous example of a door handle to illustrate the relevance of affordance. Depending on its physical form and positioning a door handle ‘affords' either pulling or pushing. If the door handle is not placed in accordance with the door's opening mechanism (a door handle with a pushing affordance placed on the side of the door that is meant to be pulled) that creates frustration for the people who use it. Affordances also depend on the person who perceives them, for example, a bench may afford adults to sit on but for young children it can function as a hide out or a place to lie under, a platform to climb on or to jump from.

The affordance theory can be applied to the experience of HBI artefacts, as their affordances communicate how to use and interact with them. HBI artefacts that are based on different modalities like sound might be more tricky to be perceived through their affordances than tangible artefacts. In our framework, affordances are part of the physicality and embodiment of an HBI artefact, defining possible action whether intentionally designed or unplanned. These embodied interactions influence our sense of agency, spatial understanding, and engagement with the built environment.

• The second dimension concerns social aspects of experience

While subjective individual experience holds importance, our process of creating meaning also occurs through an inter-subjective connection between oneself and others. Experiences are therefore not only an individualistic phenomenon, but they can also they can be shared and meaning can be created through them, such as when playing online chess with a friend. Based on ANT (Latour, 1996), the technology artefacts in the interaction equation are as active agents as the people using the artefacts. The theory on actor networks (AN) was introduced by sociologists challenging the nature-culture dichotomy. This theory shifts focus from the subject to the actors and networks involved, tracing the actions taken to understand the complex dynamics of relationships and the emergence of materials in specific contexts (Latour, 1996). Tracing action is a core principle of ANT as it reveals the nature of networks and the diverse ways in which relations unfold (Law, 2008). Some scholars associate ANT with entanglement studies which challenge the notion that individuals and objects are separate entities and instead focus on how meanings and materialities are enacted together in everyday practices (Barad, 2007). In relation to HBI, Baron and Gomez (2016) argue that technology is a social construct, whereby its attributes are predominantly, if not solely, determined by the interpretive frameworks and negotiations of the involved social groups; the users and designers of technological artefacts. Yaneva (2008) connects ANT to architecture, proposing that a building is a sophisticated mediator that redistributes agency between human and non-human actors contributing to the transformation of social meanings.

We adopt ANT, as we find it useful in providing us with a vocabulary and a theoretical lens for articulating and defining the different actors and the interrelations with one another in the experience of an HBI artefact. In conventional HCI studies that assess the experience of physical, tangible technological artefacts, the artefact or TUI is an object that is self-contained, mobile, and rather small sized in comparison to human bodies. As mentioned earlier in Section 3.2, HBI artefacts may have environment and/or object forms, including other large-scale interactive objects and tangible user interfaces (TUIs) that may not be an inherent part of the building, like our ArUco markers or other TUIs [for example Zhong et al. (2020) and Nabil and Kirk (2021)]. Therefore, we argue that HBI artefacts can be both manipulated through movement in space, direct manipulation, and manipulation using an interactive or non-interactive object. Hence, the HBI artefact, tangible object(s), and tangible interactive object(s) are all considered to be actors. In our framework, we leverage ANT's theoretical foundation and terminology to create a network of all the identified actors and their interrelations. As part of our framework, we provide a schema of such an active network in relation to HBI experience as seen in Figure 7B. The schema illustrates the constituting elements of an active network within this context which are the users, the surrounding digitally-augmented physical environment, tangible objects, and tangible objects with interactive capabilities. These elements are entangled and affect one another in an interrelated way.

• The third dimension refers to spatial aspects of experience

Existence is inherently place-bound, hence, places are unavoidably existence-bound expressions of specific human interaction with the world. Everything we do—actions, thoughts, emotions, conversations, interactions with technology—occurs somewhere in a particular locale (Merleau-Ponty, 2012). As established before in Section 3.2, place is always occupied and situated within a specific space, imbued with meaning, and can engender emotions and values (Rigolon, 2011; Tuan, 2011). As part of the spatial realm, HBI interfaces can alter atmospheres and afford place-making. As highlighted in the previous section on embodied action, humans are spatial beings, therefore our bodies play a crucial role in how we perceive the world. Spatial perception and bodily motion in space are deeply interconnected. We relate spatial qualities of a space such as size, depth, and proportions to our bodies. These physical-spatial relationships influence how we perceive the environment around us.

In Section 2.2 we wrote that based on phenomenological perspectives all action is embodied since we, humans, have an inescapable embodied nature. Merleau-Ponty (2012) along with Heidegger (2005) theorised that knowledge originates through embodied experiences which precedes intellectual classification. Merleau-Ponty (2012) introduced a new idea of awareness, stressing the significance of bodily intentionality as the primary way of understanding a situation and enabling us to effectively engage with the ongoing stream of experiences. Inspired by these theories, HCI scholars linked the concepts of physical and tangible to social computing through embodiment, arguing that embodiment is the common way we, humans, encounter social and physical phenomena (Dourish, 2001). This real-world engagement requires enaction, a balance between body and environment (Varela et al., 2016). Enactment theory (Varela et al., 2016) and the broader framework of 4E cognition (Burnett and Gallagher, 2020) suggest that our cognition and perception are not confined to our brains alone. We share the view that our cognition and perception extend to our bodies, our surroundings, and our social interactions, ultimately shaping how we perceive and experience the world. As a result, our framework is governed by the notion that there is an inseparable relationship between our physical bodies, social actions, and interactions with the environment (see Figure 7A).

4.2 Framework aspects for assessing the experience of HBI artefacts

The experience framework serves as a translation of our findings from our case study (see Sections 3.4 and 3.5) into an analytical tool for exploring the experience of HBI artefacts and environments. Based on our case study, we identified five aspect pairs of HBI lived experience (see Figure 7C). The pairs of privacy-publicness, inclusion-exclusion, and spatial arrangements-decoration refer to socio-spatial dimensions. The pair of designed affordances-unplanned affordances primarily refers to physical-spatial dimensions, while the last pair, bodily movement in space-direct manipulation, refers to spatial-embodied dimensions. The sequence of these aspects comes in no particular order. These aspects may be employed in the evaluation phase of a design process to assess the relations between them which construct the overall experience.

• Privacy and publicness

Privacy denotes aspects of the experience that are individualistic or personal to an individual or a selected group of few. Private experiences could refer to ownership (private possessions), the role of the artefact, and control over one's activities such as maintaining solitude and protecting personal space. Publicness denotes aspects of the experience that are mutual, shared, or communal as part of a group. Public experiences could refer to ownership (common or public property), the role of the artefact, and being accessible and open to everyone in relation to activities and space.

• Inclusion and exclusion

Inclusion within an HBI experience denotes going beyond mere access to actively promote equity in the participation, involvement and engagement of diverse individuals in community activities in space, encouraging collaboration and cooperation among them. Spatial inclusion recognises the importance of social interaction and encourages the design of spaces that foster social connections and a sense of community. Exclusion within an HBI experience denotes individuality or an exclusive group of people, by intentionally restricting access to someone or something, or a group of people whether that is an activity, a space, or a social group.

• Designed and unplanned affordances

Designed affordances denote affordances that are intended by the designer, are prominent in the design and meant to dictate the use of an HBI artefact. Unplanned affordances denote affordances that were not part of the design but were discovered by the users and occupants of the space the HBI artefact is placed in.

• Spatial arrangements and decoration

Spatial arrangements denote socio-spatial actions altering the organisation, grouping, and configuration of spaces within a building or a built environment. Spatial arrangements influence patterns of activities and relationships within a given space. Design aspects such as flexibility and adaptability can accommodate changing needs. Decoration denotes actions relevant to the enhancement and visual appeal of a place, by selecting and arranging decorative elements. These elements can foster a sense of identity, engagement, and emotional connection within a specific space.

• Bodily movement in space and direct manipulation

Bodily movement in space denotes all the possibilities for full-body interaction within a given space, requiring gross-motoric and expressive movements that can carry meaning in the interaction with the system like moving around it or inside it. Direct manipulation denotes tangible interaction with the artefact for a desired effect through feeling and moving an object around though gross or fine motoric movements.

4.3 Framework use

As seen in our active-network schema in Figure 7B, researchers or designers wishing to assess an HBI experience need to first identify the range of actors including human (like people using the HBI artefact) or non-human (the artefact or other tangible or intangible notions affecting the interaction) entities. The objective is to trace actions among these identified actors through the lens of embodied, social, and spatial inter-dependencies and associations. Researchers can gather data on these actions, interrelations, and spatial movement through observational methods (e.g., on-site observations or video recordings) and descriptions on elements of interest during the interaction. To detail these relations, researchers would then need to analyze their collected qualitative data and trace the actions and interrelations among the constituting actors. In identifying commonalities and variations researchers would then explore relations or differences between the suggested ten aspects on HBI experiences in order to capture the diverse range of human experience when interacting with a physical HBI artefact. For example, other researchers may use the suggested pairs as a post-analysis tool to explore and identify whether these aspects are present, partially present, or absent in their results. Over time and through other cases, new pairs or lenses may emerge, while existing ones may be modified. Moreover, further pair connections may be examined and found related to whether the HBI interface is environment based or object/artefact based. Apart from a post-analysis tool, our framework may be used to plan and conduct studies. The suggested pairs can guide further exploration, such as in observing interactions in space or by incorporating them into participant interviews.

5 Discussion

In this article, we address the lack of a comprehensive approach to lived experience in the HBI domain, by providing an initial experience framework. This falls in line with the identification of human experience as a possible research dimension of HBI (Becerik-Gerber et al., 2022). We are positive that this framework can serve as a guiding compass for designers and researchers of HBI artefacts in getting a grip on the embodied, social, and spatial dimensions of HBI experience. By identifying the actors at play in each situation, their interrelations, and relating empirical findings in relation to our suggested aspects, designers and researchers can assess the lived experience of an HBI artefact. This experience is entangled and complex, as the multiple actors are involved in interrelations with one another and constantly shape and re-shape the interaction and experience.

Through the latest technological advancements in artificial intelligence (AI), quantified and smart buildings, and the emerging concept of the Metaverse (e.g., Farzaneh et al., 2021; Panchalingam and Chan, 2021; Mazhar et al., 2022), it is evident that building-related commercially available technology is rapidly moving towards a more intangible and virtual state. Bodily movement, social actions, and the spatial qualities of physical objects cannot be fully replicated in the virtual realm. Therefore, it is important to recognise that amidst this digital revolution, there remains immense value in designing, implementing, and evaluating physical HBI artefacts. Our work serves as a proof of this statement, as we exemplify that physical HBI artefacts offer rich and nuanced experiences.

As the field of HBI continues to expand, we hope to see more qualitative approaches, case studies, and empirical findings that focus on physical HBI artefacts and their experiential aspects. While assessing indoor environmental quality (e.g., Zhong et al., 2021) and building energy performance (e.g., Chen et al., 2021), are undoubtedly important, it is equally crucial to delve deeper into the subjective experiences evoked by physical HBI artefacts in real-life settings. By exploring the human experience through qualitative research methods, we can gain insights into how physical HBI artefacts impact place-making and bodily, social, and spatial interactions. Therefore, it is essential to encourage and support such research efforts within the field that go beyond investigating comfort or using traditional performance metrics.

Below we discuss strengths, challenges and limitations of the proposed HBI experience framework and future research directions.

5.1 Strengths, challenges, and limitations of the proposed HBI experience framework

To summarise, we drew from literature on experience from both constituting fields and theories on affordances (Norman, 1988), ANT (Latour, 1996), and embodiment (Dourish, 2001) to define HBI experiences as primarily spatial, socially constructed, and embodied. We exemplified the interrelations of the actors in the context of HBI experience through an active-network schema. Finally, we detailed aspects of these interrelations of HBI experience through ten aspects separated into word pairs derived from our case-study.

We consider our proposed framework an important first step towards advancing and incorporating lived experience as a fundamental concept in the HBI domain. As supported by the literature review (see Section 2), experience is a fundamental concept in both the research and practice of HCI and architecture. By grounding our framework's experience dimensions on literature review (Section 2) and on existing theories (Section 4.1) we establish a strong foundation for considering and understanding experience in HBI.

Through our theoretical grounding of the three dimensions of HBI experience (see Section 4.1), the boundaries blur as we merge HCI and architecture perspectives and theories. In the research context of HBI artefacts these theories may gain from yet unknown perspectives. For example, HCI-related theory could expand on the definition of affordances in HCI artefacts through the under-explored notion of unplanned, emergent, or unintended affordances that lead to appropriation. Additionally, HBI research that makes use of affordance theory (e.g., Nguyen et al., 2021) could further explore unplanned affordances in the context of HBI artefacts.

One of the main limitations of our framework is that it is partially underpinned by findings derived from our case-study in which we assessed the experience of an HBI artefact by young children. Although the experience dimensions of our framework are validated through related literature and theory, the experience aspects are dependent on our particular design artefact and empirical findings from our evaluation study in a particular context. In our analysis, we acknowledge the potential impact of participants' perceptions and cultural backgrounds may have influenced the identified themes. Although the participating groups were culturally homogeneous, we recognise the presence of diversity in terms of gender and age, which may have affected some of our findings. Considering this mix of young participants, we carefully evaluate how these factors may have influenced presented the themes. The transferability of our findings may be limited, as observed in other case-based technological research (Wulf et al., 2011). Knowledge transfer to other similar cases might prove to be a challenge but also puts forward a research agenda of looking into further contexts, users, and artefacts to expand, iterate and challenge the current state of the framework. We can only speculate whether these dimensions and aspects would also apply in other kindergarten contexts or in the case of older children or adults experiencing the same artefact. To verify the applicability and validity of our framework, further research and empirical findings on deployed case-studies are required in other contexts using different HBI designs.

Apart from transferability challenges, our framework is limited in the way it can be applied as it only addresses physical and tangible HBI artefacts. A short line of research within HBI refers to interactions and experiences within virtual (Bjorn et al., 2021; Chokwitthaya et al., 2023), augmented (Malkawi and Srinivasan, 2005), or cross-reality (Nguyen et al., 2021) environments. Thus, we cannot predict how our suggested dimensions and experience aspects would apply in VR, AR and XR-related research, if at all.

5.2 Future directions

Our framework provides an initial and non-exhaustive set of dimensions and aspects for assessing HBI experience. It is unclear how this initial conception of experience will influence the way we design and implement HBI artefacts. Therefore, it opens up the space for questions like: What are other dimensions and aspects of HBI experience? How to design HBI artefacts with attributes that target a specific HBI experience aspect? How do we involve architects and HCI researchers in the design and assessment of experience in the context of HBI? Additional case studies and evaluation findings on the experience of HBI artefacts could reveal further dimensions or extensions to this framework. For instance, one dimension that could be explored to further expand on our framework could be time. Our framework at its current state is based on a short-lived study in relation to interactions and experiences of environments. To further explore HBI experiences, longitudinal studies are necessary to understand the adoption and appropriation of the artefact over time. Temporal aspect pairs could emerge, by analysing the actor-networks development and overall experience with an HBI artefact over time and how they differ from or align to the initial findings. Pairs related to a temporal dimension could be ephemeral-permanent, instant-delayed, rapid-gradual, and could either be present in conjunction or in absence of our suggested aspects. Likewise, in exploring the emotional dimension of HBI experiences these pairs could be enthusiasm-indifference, empathy-apathy, and satisfaction-frustration. In addition, future studies may wish to investigate how these experience dimensions and aspects could be intentionally designed as in the case of Döring et al. (2013a,b), thus opening up expanding the design space of HBI environments and artefacts.

Furthermore, we strongly believe that spatiality is an integral part of every human interaction, whether that involves computational systems or not. The HCI field needs to realise the importance of space and spatiality when it comes to HCI artefacts. Our framework contributes towards expanding the understanding of experience by incorporating the spatial dimension of experience in the design and evaluation of HCI artefacts. Correspondingly, we hope to stir the architecture field into looking at experience more holistically and assessing experience of architectural designs using qualitative measures.

6 Conclusion

HBI is an emerging research domain still at its infancy in relation to defining, designing for, and assessing people's experiences—particularly, when it comes to lived experience. The limited studies on subjective, contextual, and embodied experiences of people with HBI artefacts and the relative absence of lived experience as a concept in HBI studies urged us to propose a model that links together embodied, social, and spatial aspects of people's interactions with a technology-augmented built environment. In this research, we reviewed literature on experience from both HCI and architecture research domains, and identified differences and gaps. Addressing the gaps in research, we present our framework underpinned by three dimensions of looking at HBI experience derived from theory and our case study findings from deploying an HBI artefact and the resulting lived experiences. All conclusions were consolidated into a single framework elaborating on the interrelations between the embodied, social, and spatial dimensions and aspect pairs in relation to the experience of an HBI artefact. While we cannot claim that the developed framework fills all the gaps related to the experience research in HBI, however, we believe that it is a first step towards introducing lived experience as a concept in HBI that will guide the design and evaluation methods for the future generation of HBI artefacts. Furthermore, the HCI field cannot keep on ignoring the spatial aspect that characterises all experiences. Spatiality is an integral part of every human interaction, with or without technology. After all, paraphrasing (McCarthy and Wright, 2004), we do not only use technology, we live in it.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by the University of Salzburg (Ethikkommission der Universität Salzburg). Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author contributions

AI acquired project funding and provided guidance in writing the manuscript. CF supervised the study conduction and administrated the project. EE and CF conceptualised the research and contributed to study design, methodology, and findings consolidation. EE contributed to the data collection, performed data curation and data analysis, integrated, visualised the findings, and wrote the original draft of the manuscript. All authors contributed to manuscript submission, read, and approved the submitted version.

Funding

The project reported in this work was supported by the Federal State Government of Salzburg, Austria, grant number PFL142230_01 and WISS2025.

Acknowledgments

We thank the children, teachers, and kindergarten headmistress who participated in our research and permitted us to conduct this research under their roof. A special thank you to our colleagues, Moritz Kubesch, Nico Etschberger, and Lisa Hofer, for supporting the implementation of the study reported in this manuscript. We also thank Alicia Julia Fradera for the transcription of the children's and teachers' interviews in English and Leyli Bunyadzade for their participation and contributions in the first data analysis phase. We appreciate the trust and support of our corporate project partners, the municipal construction service provider Salzburg Wohnbau GmbH, the STRABAG construction company, the Schlotterer sun-protection company, the region's electricity provider Salzburg AG, and architects responsible for the kindergarten's building renovation. We are very grateful for the financial support by the Federal Government of Salzburg, Austria. Our heartfelt gratitude to Thijs Hesby Roeleven for providing invaluable proofreading assistance on earlier versions of this document.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomp.2023.1233904/full#supplementary-material

Supplementary Data Sheet S1. Children's interview transcription (English translation).

Supplementary Data Sheet S2. Teacher's interview transcription (English translation).

Supplementary Data Sheet S3. ELAN video analysis excerpt.

Supplementary Data Sheet S4. Codebook of thematic analysis.

Supplementary Data Sheet S5. Thematic analysis' theme outcomes.

Footnotes

References