Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgo... more Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgoals is computationally costly. What algorithms might enable us to reap the benefits of planning using subgoals while minimizing the computational overhead of selecting them? We propose visual scoping, a strategy that interleaves planning and acting by alternately defining a spatial region as the next subgoal and selecting actions to achieve it. We evaluated our visual scoping algorithm on a variety of physical assembly problems against two baselines: planning all subgoals in advance and planning without subgoals. We found that visual scoping achieves comparable task performance to the subgoal planner while requiring only a fraction of the total computational cost.Together, these results contribute to our understanding of how humans might make efficient use of cognitive resources to solve complex planning problems.
To study the cognitive role that tangible objects play in design thinking, we gave 17 architects ... more To study the cognitive role that tangible objects play in design thinking, we gave 17 architects and novice students a set of blocks and asked them to design their dream house. Although the blocks seem simple they are filled with perceptual surprises. We regard manipulating blocks as a form of physical thinking because through interaction designers increase the dimensionality of their design space. This happens because a) perceptual ambiguity leads to multiple semantics multiple ways of identifying what shapes are out there, and b) kinesthetic and other forms of non-visual interaction enables designers to feel inertia, mass, force and gravity and thereby encounter blocks and their relations in additional ways. The effects of tangibility and enactive forms of perception is that the design space expands, often leading architects to more divergent thinking. Physical interaction broadens the basis of creativity.
When people make sense of situations, illustrations, instructions and problems they do more than ... more When people make sense of situations, illustrations, instructions and problems they do more than just think with their heads. They gesture, talk, point, annotate, make notes and so on. What extra do they get from interacting with their environment in this way? To study this fundamental problem, I looked at how people project structure onto geometric drawings, visual proofs, and games like tic tac toe. Two experiments were run to learn more about projection. Projection is a special capacity, similar to perception, but less tied to what is in the environment. Projection, unlike pure imagery, requires external structure to anchor it, but it adds ‘mental’ structure to the external scene much like an augmented reality system adds structure to an outside scene. A person projects when they look at a chessboard and can see where a knight may be moved. Because of the cognitive costs of sustaining and extending projection, humans make some of their projections real. They create structure exte...
Our ability to plan and build a wide array of physical structures, from sand castles to skyscrape... more Our ability to plan and build a wide array of physical structures, from sand castles to skyscrapers, is a defining feature of modern human intelligence. What cognitive tools enable us to create such complex and varied structures? Here we investigate how practice “reverse-engineering” a set of physical structures impacts the procedures that people subsequently use to build those structures, as well as how well they build them over time. Participants (N=105) viewed 2D silhouettes of 8 unique block towers in a virtual environment simulating rigid-body physics, and aimed to reconstruct each one in less than 60 seconds. We found that people learn to build each tower more accurately and quickly across repeated attempts, and that these gains reflect both group-level convergence upon a smaller set of viable policies, as well as error-dependent updating of each individual’s strategies. Taken together, our study provides novel insight into how humans learn from prior experience to discover be...
ACM Transactions on Computer-Human Interaction, 2019
This essay has three parts. In Part 1, I review six biases that frame the way architects and huma... more This essay has three parts. In Part 1, I review six biases that frame the way architects and human–computer interaction (HCI) practitioners think about their design problems. These arise from differences between working on procedurally complex tasks in peripersonal space like writing or sketching and being immersed in larger physical spaces where we dwell and engage in body-sized activity like sitting, chatting, and moving about. In Part 2, I explore three types of interface: classical HCI, network interfaces such as context-aware systems, and socio-ecological interfaces. An interface for an architect is a niche that includes the very people who interact with it. In HCI, people are still distinct from the interface. Because of this difference, architectural conceptions may be a fertile playground for HCI. The same holds for interactivity. In Part 3, I discuss why interactivity in HCI is symmetric and transitive. Only in ecological and social interaction is it also reflexive. In ecol...
To study the value of interactivity and chance in creative cognition we looked at the creative pr... more To study the value of interactivity and chance in creative cognition we looked at the creative process in architecture, choreography and word discovery. Seventeen architects and novice students were given a set of blocks and asked to design their dream house. Although the blocks seem simple they are filled with perceptual surprises. Manipulation led to seeing new things and these in turn led to thinking up new structural forms. In choreography we studied the creative method of a noted choreographer and observed how random objects in the environment often figure in tasks he assigned his dancers. The dancers would look for interesting attributes in the objects that they then played off of in interesting ways. In a word discovery task we gave subjects a string of 7 letters and asked them to call out all the words of three or more letters they could make. They were tested in three conditions: static letters are fixed, interactiveletters can be moved, and shuffle spacebar randomly reorde...
ACM Transactions on Computer-Human Interaction, 2013
The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about... more The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about interaction and new principles for better designs. I support this claim with four ideas about cognition: (1) interacting with tools changes the way we think and perceive -- tools, when manipulated, are soon absorbed into the body schema, and this absorption leads to fundamental changes in the way we perceive and conceive of our environments; (2) we think with our bodies not just with our brains; (3) we know more by doing than by seeing -- there are times when physically performing an activity is better than watching someone else perform the activity, even though our motor resonance system fires strongly during other person observation; (4) there are times when we literallythinkwith things. These four ideas have major implications for interaction design, especially the design of tangible, physical, context aware, and telepresence systems.
Why do people create extra representations to help them make sense of situations, diagrams, illus... more Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation – external representations save internal memory and computation – is only part of the story. I discuss eight ways external representations enhance cognitive power: they provide a structure that can serve as a shareable object of thought; they create persistent referents; they change the cost structure of the inferential landscape; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought – they help coordinate thought.
... Communication and Cooperation at Smith College Emilee Mooney EVS 300 May 4, 2005 Page 2. 2 ..... more ... Communication and Cooperation at Smith College Emilee Mooney EVS 300 May 4, 2005 Page 2. 2 ... Those peer institutions that have impressive sustainability programs, like Colby and Middlebury, also have sustainability as an explicit part of their overall mission (Baumer, pers. ...
Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the h... more Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the head. An alternative, more interactionist view is presented here, where working with objects, media and other external things is seen as a fundamental component of creative thought. The value of chance interaction and chance cueing — practices widely used in the creative arts — is explored briefly in an account of the creative method of choreographer Wayne McGregor and then more narrowly in an experimental study that compared performance on a Scrabble-like word discovery problem. Subjects were presented with seven letters and given two minutes to call out three-to-seven-letter English words. There were three conditions: The tiles were fixed in place, subjects were free to move the tiles manually or the tiles could be randomly shuffled. Results showed that random shuffling was best, with manual movement second. Three reasons are provided: Shuffling is faster and cheaper than mentally think...
The use of a double check by 2 nurses has been advocated as a key error-prevention strategy. This... more The use of a double check by 2 nurses has been advocated as a key error-prevention strategy. This study aims to determine how often a double check is used for high-alert medications and whether it increases error detection. Emergency department and ICU nurses worked in pairs to care for a simulated patient. Nurses were randomized into single- and double-check groups. Errors intentionally introduced into the simulation included weight-based dosage errors and wrong medication vial errors. The evaluator recorded whether a double check was used, whether errors were detected, and observational data about nurse behavior during the simulation. Forty-three pairs of nurses consented to enroll in the study. All nurses randomized to the double-check group used a double check. In the single-check group, 9% of nurses detected the weight-based dosage error compared with 33% of nurses in the double-check group (odds ratio 5.0; 95% confidence interval 0.90 to 27.74). Fifty-four percent of nurses in...
To study how designers explore ideas when making physical models we ran an experiment in which ar... more To study how designers explore ideas when making physical models we ran an experiment in which architects and undergraduate students constructed a dream house made of blocks. We coded their interactions in terms of robotic pick and place actions: adding, subtracting, modifying and relocating blocks. Architects differed from students along three dimensions. First, architects were more controlled with the blocks; they used fewer blocks overall and fewer variations. Second, architects appear to think less about house features and more about spatial relationships and material constraints. Lastly, architects experiment with multiple block positions within the model more frequently, repeatedly testing block placements. Together these findings suggest that architects physically explore the design space more effectively than students by exploiting material interactions. This embodied know-how is something next generation robots will need to support. Implications for material-based robotic interaction are discussed.
Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgo... more Planning is hard. The use of subgoals can make planning more tractable, but selecting these subgoals is computationally costly. What algorithms might enable us to reap the benefits of planning using subgoals while minimizing the computational overhead of selecting them? We propose visual scoping, a strategy that interleaves planning and acting by alternately defining a spatial region as the next subgoal and selecting actions to achieve it. We evaluated our visual scoping algorithm on a variety of physical assembly problems against two baselines: planning all subgoals in advance and planning without subgoals. We found that visual scoping achieves comparable task performance to the subgoal planner while requiring only a fraction of the total computational cost.Together, these results contribute to our understanding of how humans might make efficient use of cognitive resources to solve complex planning problems.
To study the cognitive role that tangible objects play in design thinking, we gave 17 architects ... more To study the cognitive role that tangible objects play in design thinking, we gave 17 architects and novice students a set of blocks and asked them to design their dream house. Although the blocks seem simple they are filled with perceptual surprises. We regard manipulating blocks as a form of physical thinking because through interaction designers increase the dimensionality of their design space. This happens because a) perceptual ambiguity leads to multiple semantics multiple ways of identifying what shapes are out there, and b) kinesthetic and other forms of non-visual interaction enables designers to feel inertia, mass, force and gravity and thereby encounter blocks and their relations in additional ways. The effects of tangibility and enactive forms of perception is that the design space expands, often leading architects to more divergent thinking. Physical interaction broadens the basis of creativity.
When people make sense of situations, illustrations, instructions and problems they do more than ... more When people make sense of situations, illustrations, instructions and problems they do more than just think with their heads. They gesture, talk, point, annotate, make notes and so on. What extra do they get from interacting with their environment in this way? To study this fundamental problem, I looked at how people project structure onto geometric drawings, visual proofs, and games like tic tac toe. Two experiments were run to learn more about projection. Projection is a special capacity, similar to perception, but less tied to what is in the environment. Projection, unlike pure imagery, requires external structure to anchor it, but it adds ‘mental’ structure to the external scene much like an augmented reality system adds structure to an outside scene. A person projects when they look at a chessboard and can see where a knight may be moved. Because of the cognitive costs of sustaining and extending projection, humans make some of their projections real. They create structure exte...
Our ability to plan and build a wide array of physical structures, from sand castles to skyscrape... more Our ability to plan and build a wide array of physical structures, from sand castles to skyscrapers, is a defining feature of modern human intelligence. What cognitive tools enable us to create such complex and varied structures? Here we investigate how practice “reverse-engineering” a set of physical structures impacts the procedures that people subsequently use to build those structures, as well as how well they build them over time. Participants (N=105) viewed 2D silhouettes of 8 unique block towers in a virtual environment simulating rigid-body physics, and aimed to reconstruct each one in less than 60 seconds. We found that people learn to build each tower more accurately and quickly across repeated attempts, and that these gains reflect both group-level convergence upon a smaller set of viable policies, as well as error-dependent updating of each individual’s strategies. Taken together, our study provides novel insight into how humans learn from prior experience to discover be...
ACM Transactions on Computer-Human Interaction, 2019
This essay has three parts. In Part 1, I review six biases that frame the way architects and huma... more This essay has three parts. In Part 1, I review six biases that frame the way architects and human–computer interaction (HCI) practitioners think about their design problems. These arise from differences between working on procedurally complex tasks in peripersonal space like writing or sketching and being immersed in larger physical spaces where we dwell and engage in body-sized activity like sitting, chatting, and moving about. In Part 2, I explore three types of interface: classical HCI, network interfaces such as context-aware systems, and socio-ecological interfaces. An interface for an architect is a niche that includes the very people who interact with it. In HCI, people are still distinct from the interface. Because of this difference, architectural conceptions may be a fertile playground for HCI. The same holds for interactivity. In Part 3, I discuss why interactivity in HCI is symmetric and transitive. Only in ecological and social interaction is it also reflexive. In ecol...
To study the value of interactivity and chance in creative cognition we looked at the creative pr... more To study the value of interactivity and chance in creative cognition we looked at the creative process in architecture, choreography and word discovery. Seventeen architects and novice students were given a set of blocks and asked to design their dream house. Although the blocks seem simple they are filled with perceptual surprises. Manipulation led to seeing new things and these in turn led to thinking up new structural forms. In choreography we studied the creative method of a noted choreographer and observed how random objects in the environment often figure in tasks he assigned his dancers. The dancers would look for interesting attributes in the objects that they then played off of in interesting ways. In a word discovery task we gave subjects a string of 7 letters and asked them to call out all the words of three or more letters they could make. They were tested in three conditions: static letters are fixed, interactiveletters can be moved, and shuffle spacebar randomly reorde...
ACM Transactions on Computer-Human Interaction, 2013
The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about... more The theory of embodied cognition can provide HCI practitioners and theorists with new ideas about interaction and new principles for better designs. I support this claim with four ideas about cognition: (1) interacting with tools changes the way we think and perceive -- tools, when manipulated, are soon absorbed into the body schema, and this absorption leads to fundamental changes in the way we perceive and conceive of our environments; (2) we think with our bodies not just with our brains; (3) we know more by doing than by seeing -- there are times when physically performing an activity is better than watching someone else perform the activity, even though our motor resonance system fires strongly during other person observation; (4) there are times when we literallythinkwith things. These four ideas have major implications for interaction design, especially the design of tangible, physical, context aware, and telepresence systems.
Why do people create extra representations to help them make sense of situations, diagrams, illus... more Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation – external representations save internal memory and computation – is only part of the story. I discuss eight ways external representations enhance cognitive power: they provide a structure that can serve as a shareable object of thought; they create persistent referents; they change the cost structure of the inferential landscape; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought – they help coordinate thought.
... Communication and Cooperation at Smith College Emilee Mooney EVS 300 May 4, 2005 Page 2. 2 ..... more ... Communication and Cooperation at Smith College Emilee Mooney EVS 300 May 4, 2005 Page 2. 2 ... Those peer institutions that have impressive sustainability programs, like Colby and Middlebury, also have sustainability as an explicit part of their overall mission (Baumer, pers. ...
Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the h... more Individual creativity is standardly treated as an ‘internalist’ process occurring solely in the head. An alternative, more interactionist view is presented here, where working with objects, media and other external things is seen as a fundamental component of creative thought. The value of chance interaction and chance cueing — practices widely used in the creative arts — is explored briefly in an account of the creative method of choreographer Wayne McGregor and then more narrowly in an experimental study that compared performance on a Scrabble-like word discovery problem. Subjects were presented with seven letters and given two minutes to call out three-to-seven-letter English words. There were three conditions: The tiles were fixed in place, subjects were free to move the tiles manually or the tiles could be randomly shuffled. Results showed that random shuffling was best, with manual movement second. Three reasons are provided: Shuffling is faster and cheaper than mentally think...
The use of a double check by 2 nurses has been advocated as a key error-prevention strategy. This... more The use of a double check by 2 nurses has been advocated as a key error-prevention strategy. This study aims to determine how often a double check is used for high-alert medications and whether it increases error detection. Emergency department and ICU nurses worked in pairs to care for a simulated patient. Nurses were randomized into single- and double-check groups. Errors intentionally introduced into the simulation included weight-based dosage errors and wrong medication vial errors. The evaluator recorded whether a double check was used, whether errors were detected, and observational data about nurse behavior during the simulation. Forty-three pairs of nurses consented to enroll in the study. All nurses randomized to the double-check group used a double check. In the single-check group, 9% of nurses detected the weight-based dosage error compared with 33% of nurses in the double-check group (odds ratio 5.0; 95% confidence interval 0.90 to 27.74). Fifty-four percent of nurses in...
To study how designers explore ideas when making physical models we ran an experiment in which ar... more To study how designers explore ideas when making physical models we ran an experiment in which architects and undergraduate students constructed a dream house made of blocks. We coded their interactions in terms of robotic pick and place actions: adding, subtracting, modifying and relocating blocks. Architects differed from students along three dimensions. First, architects were more controlled with the blocks; they used fewer blocks overall and fewer variations. Second, architects appear to think less about house features and more about spatial relationships and material constraints. Lastly, architects experiment with multiple block positions within the model more frequently, repeatedly testing block placements. Together these findings suggest that architects physically explore the design space more effectively than students by exploiting material interactions. This embodied know-how is something next generation robots will need to support. Implications for material-based robotic interaction are discussed.
Uploads
Papers by David Kirsh