Cognitive Grammar and Gesture Points of Convergence Advances and Challenges
Cognitive Grammar and Gesture Points of Convergence Advances and Challenges
Cognitive Grammar and Gesture Points of Convergence Advances and Challenges
Unauthenticated
Download Date | 7/20/19 4:23 PM
68 Kasper I. Kok and Alan Cienki
1 Introduction
According to Cognitive Grammar’s (henceforth CG; Langacker 1987, 1991, 2008a)
usage-based perspective on language structure, the realm of the linguistic need
not be limited to verbal expression. Several scholars have discussed the poten-
tial of speakers’ gestures (in the sense of Kendon 1980, 2004) to attain gramma-
tical status. Langacker, himself, already alluded to the potential of certain
gestures to be part of a linguistic system, claiming that any type of expressive
behavior can in principle become entrenched as a symbolic structure (Langacker
1987). This potential received further attention in Langacker’s (2001, 2008a)
writings, and was evaluated in more detail by Cienki (2012, 2014, in press).
With respect to signed languages, Wilcox (2004), Wilcox and Xavier 2013), and
Liddell (2003), among others, have shown that CG provides useful analytical
constructs for detailing the relation between (ASL) grammar and iconicity.
Others have adopted CG-terminology in their discussions of sequential speech-
gesture compositions (Enfield 2004; Ladewig 2011b) and the phenomenon of
gestural iteration (Bressem 2012).
Despite the many connections between CG and gesture research made by
these scholars, CG has thus far not developed so as to adopt multimodality in its
basic design, nor has it become clear what it entails to incorporate speakers’
gestures in actual cognitive grammatical analysis. The aim of this article is to
provide a more comprehensive overview of the points of convergence between
CG and gesture than is currently available, and to advance the incorporation of
gesture in CG as appropriate. After reviewing current literature on this topic, we
show how CG’s analytical tools can be applied to language as multimodal
(spoken-gestured). We conclude with a discussion of theoretical and operational
challenges.
2 Points of convergence
Whereas the study of speakers’ gestures from a linguistic perspective is still in
its infancy (Müller et al. 2013), several scholars have already explored the
potential of CG as a framework for understanding the grammar-gesture rela-
tionship. At least three aspects of the theory are of particular interest in this
respect. First of all, CG views grammar as emergent from actual communica-
tion, without posing restrictions on the kind of behaviors that may constitute a
linguistic system. Second, CG holds that grammatical meaning resides in
conceptualization – a cognitive process that has often been hypothesized to
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 69
Unauthenticated
Download Date | 7/20/19 4:23 PM
70 Kasper I. Kok and Alan Cienki
What follows from this view is that whether or not elements of expression
qualify as linguistic does not depend on the modality through which they are
expressed. Rather, the grammatical potential of co-verbal behaviors is to be
assessed according to their degree of entrenchment as symbolic structures in
an individual’s mind and the degree of conventionalization of those symbolic
structures within a given community (Langacker 1987; Zima 2014). Taking this
continuous view as a starting point, the picture arises that some gestural
behaviors are clearly candidates to be part of the linguistic system (as a general
category, or of a given language; Cienki 2012), whereas others are more on the
periphery of that system.
Co-verbal manual behaviors come in many varieties. Kendon (1980) first ana-
lyzed different types of gesture as forming a continuum from fully autonomous
to more idiosyncratic – later coined ‘Kendon’s continuum’ by McNeill (1992).
Kendon’s classification mirrors CG’s tenet that entrenchment is a matter of
degree rather than a dichotomy.
The entrenched status of so-called emblems (Efron 1941 [1972]; Kendon
2004: Ch. 16) is rather clear-cut. These are gestures with a stable form, that
can be attributed a meaning within a given culture in the absence of verbal
context (e.g., forming a ring with the thumb and index finger to say ‘OK’ in
North American and some other cultures, or extending the index finger and
middle finger in a V-shape, with the palm of the hand facing outward, to say
‘peace’). The use of emblems is to some extent language-specific, with different
repertoires being employed by different cultural communities (Efron 1941 [1972];
Payrató 1993). Hence, this category of gestures can be assumed to consist of
symbolic units that are both entrenched and conventionalized.
For most other categories of gestures, the grammatical status involved is only
evident on a higher level of abstraction. Even for gestural behaviors that may
seem rather systematic, such as manual pointing gestures, substantial variation in
form and meaning exists. Whereas pointing gestures often involve a stretched
index finger (in most Western cultures), there appear to be no strict constraints on
the shape of the hand and the degree of tension in the fingers. Likewise, in terms
of their meaning, pointing gestures may serve a variety of functions: they may be
used to draw the interlocutor’s attention to some contextual or abstract referent; to
resolve referential ambiguity, or to give or request the turn in a conversation
(Bavelas et al. 1992; Kita 2003). From a CG perspective, this variability in form and
meaning across different contexts “correlates with what cognitive grammarians
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 71
1 Also note that this term does not imply that the speaker has the intention of being creative.
Unauthenticated
Download Date | 7/20/19 4:23 PM
72 Kasper I. Kok and Alan Cienki
It may be that there are conventions which allow interpreters to recognize that a person is
doing an illustrative tracing gesture, based presumably on formal distinctions in types of
hand movement in combination with attention-directing eye gaze toward the gesture space.
While the exact form of a tracing gesture cannot be pre-specified, its general manner of
execution may be sufficient to signal that it is a tracing gesture. (Enfield 2013: 701)
Form Largely More or less fixed Common ‘formational Flexible, but not
fixed within a given core’, but strong unrestricted – bound to
within a culture, with some variation within high-level norms and
given degrees of freedom. categories. constrained by the
culture. grammar of the verbal
channel.
Meaning Largely Variable, but Only characterizable in Mostly context-
fixed with clear abstract terms and/or dependent; generally
within a commonalities – with constructs such as associated with some
given typically related to image schemas or ICMs. modification of the
culture. attention allocation. verbal channel or
emphasis of some
aspect of it.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 73
This continuum aligns well with the CG perspective that verbal structures such as
words, morphemes and syntactic constructions can be placed along a continuum of
schematicity. As follows from the discussion above, different types of co-speech
gestures can be characterized in a similar fashion. They are not principally different
from one another or from verbal structures, but rather analyzable as a set of structures
that range from fixed to more variable and schematic in their phonology and seman-
tics. A view of this kind may help to answer the question whether gestures are to be
seen as subject matter for linguistic theory. To echo Langacker’s (2008b) view on this
matter: some gestures are clearly part of a grammatical system, others less so.
2.3.1 Specificity
Unauthenticated
Download Date | 7/20/19 4:23 PM
74 Kasper I. Kok and Alan Cienki
telephone, perhaps even just the handset of a landline phone).2 Gestures with a
grip handshape, whereby all five fingers are bent as if holding something in the
fist, are much less specific: they most likely (but not exclusively) refer to small-
sized, round or cylindrical objects. A type of gesture on the most schematic end
is the palm-up-open-hand gesture (Kendon 2004; Müller 2004). Palm-up-open-
hands may, via metonymy, refer to any type of object that might be held on the
hand, either concrete (e.g., the physical referent of the co-expressed noun
phrase) or abstract (e.g., a position in a debate) (see further discussion in
Mittelberg and Waugh 2009).
2 The intended meaning of a phone handshape may of course go further than mere reference to
a phone, e. g., when used as a full pragmatic move, as to say ‘I’ll call you’. The current
argument is concerned only with a simple, referential use of this gesture (e.g., as performed
while telling a story that involves the action of picking up a phone).
3 Note that neither of these gestures has an absolute reference frame: both draw upon the
spatial configuration of the hand of the speaker relative to the rest of the body. Where they
differ, however, is in whether the real-space physical position of the body matters to the
interpretation of the utterance.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 75
situation as objective – e. g., when tracing the path of the car by moving the
hand through space along a certain trajectory. A more subjective construal of the
same scene can be achieved when a character perspective is adopted, e. g., when
one mimics the action of driving in a car by impersonating hand movements of
the driver holding the steering wheel.
Figure 2: The profile of the verbal expression may serve as (part of) the conceptual base with
respect to the gesture.
Unauthenticated
Download Date | 7/20/19 4:23 PM
76 Kasper I. Kok and Alan Cienki
At the same time, gestures may further specify the verbally presented content.
The different representation techniques people use for making iconic reference –
molding, drawing, holding or enacting (Müller 1998) – profile different aspects of
the referent. While referring to a rim, a gesture whereby the hands draw its
contours in the air profiles its shape and outline; when using the hands as if
interacting with the rim, one profiles its physical affordances.
The trajector-landmark distinction that holds between primary and second-
ary participants within the focus domain can also have a gestural correlate, as
demonstrated in Enfield’s (2004) description of the symmetry-dominance con-
struction in Lao. Enfield describes a gesture sequence whereby the speaker first
depicts a fish trap by means of a two-handed, symmetrical gesture, and then,
while one hand is held in place, makes a gesture with the other hand that
represents the fish moving into the trap. In CG terms, the moving hand is the
Trajector with respect to the non-dominant hand, which takes the role of
Landmark. Note that in this example, the focus-background relation is not
only mapped onto an asymmetry in perceptual prominence (moving vs. non-
moving), but also on the temporal ordering of the two subsequent gestures.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 77
(Ladewig 2011b). All gestures that perform the reification of some conceptual
content (thereby construing some referent as a thing) evoke a conceptual domain
that overlaps with that of nouns. Gestures whereby the hands of the speaker (in
addition) depict some static property of the referent bear correspondence to the
semantic domain of adjectives: they profile a non-processual relationship
between the referenced entity and some other conceptual structure. In cases
where the hand moves through space to represent the motion of some entity, the
gesture can be said to have a verb-like character, as it designates a processual
relationship (Figure 3).
Figure 3: A CG representation of the semantic poles (in their most schematic form; adopted
from Langacker 1987) of three major grammatical classes and the features of gestures that
relate to them. Things in CG are represented by circles, whereas squares stand for entities
(a broader category, which may encompass any kind of conceptual structure).
Two possible caveats of the proposed parallel can be noted at this moment. First:
since gestural expression is not linearized in the same way as speech, it is very
well possible that multiple conceptual structures are simultaneously evoked by a
Unauthenticated
Download Date | 7/20/19 4:23 PM
78 Kasper I. Kok and Alan Cienki
single gesture. In fact, gestures that depict the shape of an object are likely to be
interpreted as simultaneously performing reification (cf. Kok et al. in press).
Second, it should be noted that movement of the hand(s) does not necessarily
evoke a representation of a process. As Ladewig (2011b) notes, the movement of
the hands may instead be part of the act of reference, e. g., when referring to a
physical object by tracing its contours in the air. The gesture in this case can still
be seen as an object-process synthesis, where the hand is some drawing utensil
and the movement represents the act of drawing, but it serves a meta-referential
function: not the act of drawing but the drawn object is part of the situation
communicated.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 79
verbal context, a cyclic gesture may, for instance, represent a cyclic movement
of some object, but may also indicate that the speaker is searching for the right
words (Ladewig 2011a).
It is worth emphasizing that autonomy-dependence is not always a strictly
asymmetrical relationship. In the case of multimodal demonstrative utterances
like the man over there [ + pointing gesture], speech and gesture are mutually
dependent: the verbal component does not only presuppose the performance of
a deictic gesture, but the meaning of the gesture is at the same time elaborated
by the speech (the gesture could perhaps have pointed to the woman standing
next to the man). This symbiotic character of speech-gesture compositions is in
accordance with Langacker’s characterization of the autonomy-dependence rela-
tionship as variably asymmetrical:
Canonically the structures in a valence relation manifest substantial asymmetry, with one
of them (on balance) clearly dependent, and the other autonomous. As always, though,
recognition of the prototype must not be allowed to obscure the existence of other
possibilities. Nothing in the definition precludes a relation of mutual dependence between
two structures, or guarantees that there will always be a significant relation of dependence
in one direction or the other. (Langacker 1987: 300)
Unauthenticated
Download Date | 7/20/19 4:23 PM
80 Kasper I. Kok and Alan Cienki
demonstrated that CG’s notation conventions can be useful for analyzing the
phonological and semantic structure of ASL signs.4 The benefit of these dia-
grams over text-only notations is most obvious when it comes to displaying
iconic mappings between phonological space and semantic space, e.g., when
the position or movement of the hands is isomorphic to the conceptualized
position or movement of some object. This potential is demonstrated Section 3
with respect to data from a video corpus.
Like verbal expressions, gestures are complex structures that combine different
types of semiotic signs. On the one hand, as we have seen in Section 2.2,
gestures can be recognized as belonging to a particular class or category (e. g.,
cyclic gesture, thumbs-up emblem, tracing gesture). On the other hand, they
typically evoke form-meaning mappings that are specific to the particular con-
text in which they are used. As Mittelberg (2014: 1714) notes, “when producing
[creative] gestures, speakers-gesturers do not select from a given form inventory
of a system, […] but they create semiotic material each time anew.” Especially
iconic and indexical components of gestures, indeed, rely heavily on mappings
between form and conceptualization that are created on the fly and tailored to
the context of the utterance as a whole. Thus, as noted by Enfield (2009, 2013),
many gestures combine conventional and non-conventional (ad hoc) signs.
4 Liddell’s analyses, which extend CG notations with constructs from conceptual blending
theory, differs from the ones presented in this paper, which draw on CG notions only.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 81
This dual structuring is perhaps most clearly illustrated with respect to pointing
gestures: the pointing handshape itself is categorically linked to an expectation
of deictic reference or placement, but the location and direction of the pointing
gesture, which elaborate that expectation, are ad hoc and analog in nature
(cf. Liddell’s 2003 discussion of pronouns in ASL). Because this component of
the gesture does not (directly) rely on entrenchment in long-term memory or
convention, its symbolic status is of a different nature than that of prototypical
grammatical units.5 As Langacker (1987: 91) proposes, an iconic or indexical
structure is interpretable as a result of being “put in correspondence with itself”.
In other words, as a result of their inherent conceptual value, manually
expressed phonological structures have the potential to be self-symbolizing
(cf. Wilcox’s 2004 notion of cognitive iconicity).
At the risk of considerable oversimplification (for more detailed accounts,
see Mittelberg 2014; Mittelberg and Waugh 2014; Taub 2001), the analyses
below pursue the view here that instances of creative gesticulation can be
decomposed into a conventional component and an ad hoc component. An
example analysis of the internal structure of an instance of creative gesticulation
is given in Figure 5. The analysis concerns a fragment of the Speech and Gesture
Alignment (SaGA) corpus (Lücking et al. 2013), which contains video recordings
of German direction giving discourse. As seen in Figure 4, speech and gesture
are employed in tandem to describe a physical landmark that is relevant for the
route description. While the speaker says auf den Seiten sind zwei blaue
Wendeltreppen ‘on the sides there are two blue spiral staircases’, he simulta-
neously makes an upward spiral-shaped tracing gesture with the index fingers of
both hands. The time course of the gesture relative to speech is represented
below the transcript, following the conventions of Kendon (2004).6
A B C
(1)
auf den Seiten sind zwei blaue Wendeltreppen
|~~~~~|*************| .-.-.-.-.-.-.-.|
Prep Stroke |Recovery
‘on the sides there are two blue spiral staircases’
5 This is not to deny that many conventionalized units have iconic/analog features or that
analog representations may be susceptible to conventionalization. The point at stake here
concerns only those aspects of gestures that generated ‘on the fly’ and meaningful through
the construction of an ad hoc iconic mapping.
6 ‘Prep’ stands for the preparatory phase of the gesture; the ‘Stoke’ is the most energetic and
salient phase of the gesture; the ‘Recovery’ is the phase where the hands are retracted to rest
position.
Unauthenticated
Download Date | 7/20/19 4:23 PM
82 Kasper I. Kok and Alan Cienki
Figure 4: Video stills of the fragment of the SaGA corpus analyzed in example (1).
Figure 5 shows how the internal structure of the tracing gesture can be analyzed,
building on the assumption that the conventional (long-term memory) and the
ad hoc (self-symbolized) components of the gesture can be thought of as separ-
able symbolic structures.
The diagram on the bottom-left part of the figure represents the assumption
that the handshape instantiates a category of tracing gestures. This category
is analyzed as akin to a schematic grammatical class that subsumes adjectives
as well as adverbs: it profiles a relationship between some physical characteristic
(a region of shape, contour or trajectory space; analogous to Langacker 2008a: 102)
and some THING. In line with this interpretation, the diagram on the left contains
two elaboration sites. The site on the right corresponds to the shape or contour
that is drawn by the hand; the one on the left corresponds to the entity to which
this shape or contour is to be attributed (presumably elaborated in the verbal
channel). Thus, the bottom-left diagram in Figure 4 indicates that the conven-
tional component of the gesture conveys as much as there is some entity that has
some spatial property – presumably a path, motion, or contour.
The ad hoc component of the gesture signifies, through self-symbolization, a
shape or path that is homologous to the tracing movement that is performed.7
7 The iconicity that governs the self-symbolizing tracing is not as straightforward as suggested
by the diagram in Figure 4. Because iconicity rarely involves full overlap between form and
meaning, some degree of arbitrariness remains as to what aspects of the referent are profiled by
the gesture. With the rare exception of cases where a speaker’s hand actually represents her
own hand at the moment of speaking, iconic reference involves some degree of schematization.
The challenge of capturing the systematicity that exists on this ‘iconicity-internal’ level in a CG-
based analysis remains outside the scope of this article.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 83
Figure 5: A CG analysis of the internal structure of a spiral tracing gestures as being composed
of a conventional part and a self-symbolized, analog part.
This is represented on the bottom-right part of the figure, where the label ‘self-
symbolized’ is added to emphasize that this dimension of the gestural structure
is not a direct manifestation of an entrenched/conventionalized mental struc-
ture. As seen in the upper part of the diagram, the unification of the two
symbolic structures simply entails recognition that the ad hoc component of
the gesture (the trace) elaborates one of the elaboration sites invoked by the
handshape: it restricts the region of shape space that is being attributed. To
emphasize the direction of the elaboration while keeping the diagrams manage-
able, the elaboration site is marked with a hatched area and the process of
Unauthenticated
Download Date | 7/20/19 4:23 PM
84 Kasper I. Kok and Alan Cienki
elaboration with a single dashed arrow (cf. the notation with two separate lines
for correspondence and elaboration used by Langacker 2008a: 198 and else-
where). What results from the integration of the component structures is a
construct that functionally expresses: ‘there is some entity that has a spatial
feature resembling the trace of the hand.’
The semantic pole of the gesture in Figure 4 is not only determined by the
tracing structure, however. An additional symbolic unit is established by the use
of two hands that are conspicuously positioned on the sides of the body, outside
central gesture space. The meaning of this aspect of the gesture is at least as
schematic as that of the tracing component. As diagrammed in Figure 6, it can
be assumed to signify a spatial relation between two unspecified THINGS along a
horizontal axis.8
Figure 6: An example CG analysis of the interaction between the symbolic structures evoked by
the tracing-component and the location-component of the gesture.
8 It remains an open question how this symbolic structure is most accurately characterized.
The use of two hands can also refer to a single object, for instance when the hands are being
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 85
As seen in the middle diagram in the lower part of the figure, the positioning
of the hands is analyzed as a symbolic structure that evokes two THINGS, with the
conception of a horizontal axis as a salient part of the conceptual base. Like the
tracing component, this element of the gesture involves self-symbolization: the
physical distance between the hands maps onto the conceptualized spatial
relation (although on a different scale). The two THINGS that are schematically
referred to by the hand-positioning component of the gesture correspond to
those evoked by the tracing structure (Figure 5). This correspondence is perhaps
so obvious that it may give the impression of redundancy, but it nevertheless
demonstrates how CG can be employed to disentangle the elementary compo-
nents of gestures and their semantic qualities. The simplified, integrated repre-
sentation on the top part of the figure will serve as the basis for analyzing the
interaction with the verbal component of the utterance in the subsequent sec-
tion. Before doing so, it is important to note that this analysis, although it
exceeds the level of detail usually found in the gesture literature, is still likely
to be incomplete. Formal aspects such as the vertical position of the hands, their
position relative to gestures that have been made previously, the degree of
tension in the fingers may also carry some semantic import. Understanding
how these are best described in terms of symbolic structures requires more
empirical and analytical work, and is left out of consideration here (see
Section 4.1 for further discussion).
held with the palms towards each other, as if holding an object. In this case, however, this
interpretation is precluded by the handshapes and by the respective orientation of the hands,
which do not give the appearance of holding something.
Unauthenticated
Download Date | 7/20/19 4:23 PM
86 Kasper I. Kok and Alan Cienki
Section 3.3 for a more incremental analysis, where the gesture interacts with
elements of the verbal channel before the meaning of the utterance as a whole is
established).
The analysis of the gestural channel is already given in the previous section:
each of the hands schematically refers to a THING and attributes a self-symbolized
spatial characteristic to it. In combination with the noun phrase zwei blaue
Wendeltreppen ‘two blue spiral staircases’, the most obvious interpretation is that
each of the elaboration sites evoked by the hands corresponds to one of the
conceptualized objects (the spiral staircases) that are profiled by the verbal compo-
nent of the utterance. That is, the schematic structures evoked by the hands become
elaborated by the more specific structures evoked by the noun phrase zwei blaue
Wendeltreppen. The integrated meaning can be represented as in the upper part of
the diagram: the full multimodal utterance profiles the existence of two spiral
staircases (where the quantity is expressed in speech as well as in gesture) as
well as their color (only evoked through speech) and their shape and orientation
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 87
The interaction between speech and gesture can take higher degrees of complex-
ity than in the example outlined above. Here, we provide an analysis of the
intersection between speech and gesture in the case of a multimodal deictic
utterance. It should be noted that our goal is not to take any particular stance in
deixis theory, nor do we hold that our analysis is the only one possible; the main
aim instead is to illustrate CG’s potential in analyzing the interaction between
verbal and gestural forms of spatial expression. In the example displayed in
Figure 8 and example 2, again taken from the SaGA corpus, the speaker per-
forms two pointing gestures while referring to a physical landmark (a spiral
staircase again, coincidentally). First, while she says sieht ein bisschen aus wie
die Wendeltreppe ‘looks a bit like the spiral staircase’, she points sideways,
ostensibly in the direction of the actual hallway in the university building
where she is located. Shortly thereafter she points again, roughly in the same
direction, while she says hier in der Halle ‘here in the hallway’. A more precise
representation of the timing of the pointing gestures relative to the speech is
given in Figure 8.
(2) A B C D E F
sieht ein bisschen aus wie die Wendeltreppe hier in der in der Halle
|~~~~~~ |*******/*******|.-.-.-.-.-| |~~~~~|************
Prep Stroke Hold Recovery Prep Stroke
‘looks a bit like the spiral staircase here in the in the hallway’
We see that the first of the two consecutive pointing gesture is performed
right before the noun phrase die Wendeltreppe ‘the spiral staircase’ is voca-
lized. The combination of these two components of the utterance is analyzed in
Figure 9.
9 The meaning of the noun Wendeltreppe ‘spiral staircase’ may already involve experiential
knowledge regarding its typical shape and orientation. The gesture makes these characteristics
of the referent more specific and marks them as salient.
Unauthenticated
Download Date | 7/20/19 4:23 PM
88 Kasper I. Kok and Alan Cienki
Figure 8: Video stills of the fragment of the SaGA corpus analyzed in example (2).
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 89
Figure 9: CG-analysis of the first pointing gesture in combination with the definite noun phrase
die Wendeltreppe ‘the spiral staircase’.
Unauthenticated
Download Date | 7/20/19 4:23 PM
90 Kasper I. Kok and Alan Cienki
definite article. This suggests that the referent of the noun phrase is contextually
salient. Langacker (2008a: 285) represents the most basic meaning of the definite
article roughly as in the bottom-right diagram. The rounded rectangles represent
the attentional frames that are postulated in CG to analyze the linear-sequential
dimension of discourse organization (see Langacker 2001). As shown in the
diagram, the definiteness of the article suggests that the referent of the
(projected) noun phrase has been attended to in some previous attentional
frame.10 Because of the close temporal succession between the gesture and the
noun phrase, we can assume that the attentional frame that is schematically
referred to by the definite article aligns with the one in which the gesture was
performed. Thus, the referential content of the gesture elaborates one of the
elaboration sites set up by the definite article die ‘the’; a correspondence is
established between the referent of the gesture and that of the (projected) noun
phrase. The subsequently articulated noun Wendeltreppe ‘spiral staircase’
further specifies the type of entity that is being referred to. Here, the relatively
schematic referential content of the article-gesture combination becomes elabo-
rated by the more specific meaning of the noun, as shown in the middle and
upper segments of the figure.11 Overall, the verbal and gestural components of
the utterance segment are mutually informative: the noun phrase elaborates the
elaboration site that is created by the gesture (it classifies the referent that is
being singled out) and conversely, the gesture contributes to grounding the
utterance in the immediate physical context; it forms part of the conceptual
base for the interpretation of the noun phrase.
The analysis provided in Figures 10, 11 and 12 concern the part of the
utterance that follows, comprising the phrase hier in der Halle ‘here in the
hallway’ and the concurrent pointing gesture (the second of the two pointing
gestures). In all, this part of the multimodal utterance contains three deictic
elements: the spatial adverb hier ‘here’, the prepositional phrase in der Halle ‘in
the hallway’ and the pointing gesture. Figure 10 presents separate analyses of
each of these elements.
The locative adverb hier ‘here’ profiles a region of space in that is proximal
to the location of the speaker and hearer (Langacker 1991: 222). Its construal is
strongly subjective; the designated region is conceptualized relative to location
10 It may also be contextually salient for another reason. The current analysis however is
limited to the ‘anaphoric’ use of the definite article.
11 In fact, the semantics of the gestural component, in the preceding discourse frame, also gets
elaborated by the noun; to maintain clarity in the diagrams, these correspondences are not
depicted.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 91
Figure 10: CG-analyses of the three deictic terms in the second part of the multimodal
utterance in (2).
of the speaker and hearer (denoted with S and H), which serves as a reference
point. The prepositional phrase in der Halle ‘in the hallway’, in contrast, desig-
nates a region of space in a more objective manner. It profiles a landmark Halle
‘hallway’ and a relation of inclusion with an unspecified trajector. The pointing
gesture, finally, has a subjectively construed semantic pole: the location of its
referent is defined relative to the location and orientation of the hand of the
speaker, as discussed before. In the analysis, it is furthermore assumed that the
gesture profiles an entity that is associated with a point on the projected vector,
namely the spiral staircase. This assumption is motivated by the fact that the
spiral staircase was brought to focus in the first part of the utterance and was the
referent of the first pointing gesture, as discussed above. Thus, the representa-
tion of the pointing gesture in Figure 10 and beyond includes its anchoring to
the preceding noun phrase die Wendeltreppe ‘the spiral staircase’, as well as the
relative frame of spatial reference that specifies one dimension of its location
(note that we here use a strongly simplified notation relative to analysis shown
in Figure 9). Figures 11 and 12 represent the sequential unification of the point-
ing gesture with the two other spatial elements in the utterance. The analysis
assumes that the gesture first forms a unit with the adverb hier ‘here’, which
subsequently elaborates the trajector schematically referred to by in der Halle ‘in
the hallway’. Given the immediate discourse context described above, this
interpretation is deemed more plausible than alternative analyses (e. g., where
the gesture directly grounds the referential content of in der Halle as to dis-
ambiguate which hallway is being referred to exactly).
The combined meaning of hier ‘here’ + [gesture] derives from the overlap of the
two spatial frames evoked. Crucially, hier profiles a region of space that includes the
Unauthenticated
Download Date | 7/20/19 4:23 PM
92 Kasper I. Kok and Alan Cienki
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 93
The semantic representation of hier in der Halle ‘here in the hallway’ + [pointing
gesture] entails the integration of the structure depicted in Figure 10(a) and the upper
diagram in Figure 11. As diagrammed in Figure 12, the trajector schematically referred
to by the prepositional phrase in der Halle ‘in the hallway’ is elaborated by the
referential content of the hier + [pointing gesture] construct: the spiral staircase.
The resulting semantic representation involves an intersection between three spatial
frames. Two of these are defined relative to the location of the speakers – one through
a proximity relation and one through a position on the vector line that extends from
the stretched finger. The third is specified relative to the landmark Halle ‘hallway’,
which is contained in the scope of hier ‘here’.
Notwithstanding that there may be alternative interpretations of the semantics
of pointing gestures (see, e. g., Fricke 2007 for a comprehensive discussion), the
current analysis demonstrates CG’s potential utility in detailing the interaction
Unauthenticated
Download Date | 7/20/19 4:23 PM
94 Kasper I. Kok and Alan Cienki
4 Further challenges
In order to further develop the application of CG to language as a multimodal
activity, various challenges remain to be addressed. We here discuss three
potential hurdles as they apply to the analyses presented above. Although
these might not necessarily be specific to CG – they are relevant for other
approaches to multimodal grammar as well (e. g., Kok in press) – a brief
discussion of these issues is important at this point because they will be
encountered by anyone pursuing a CG-analysis of spoken-gestured data.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 95
Unauthenticated
Download Date | 7/20/19 4:23 PM
96 Kasper I. Kok and Alan Cienki
5 Conclusion
In view of the incorporation of speakers’ gesture in grammatical theory, CG
has various theoretical and operational strengths. Due to its non-restrictive,
usage-based nature, it does not require fundamental amendments or a supple-
mentary ‘gesture component’. Because CG conceives of grammar from the
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 97
Funding: We are also grateful for research support from the Netherlands
Scientific Organization (NWO; grant PGW-12-39) to the first author and from
the Russian Science Foundation (grant #14-48-00067) to the second author.
References
Bavelas, Janet Beavin, Nicole Chovil, Douglas A. Lawrie & Allan Wade. 1992. Interactive
gestures. Discourse Processes 15(4). 469–489.
Bressem, Jana. 2012. Repetitions in gesture: Structures, functions, and cognitive aspects.
Frankfurt (Oder): European University Viadrina thesis.
Cienki, Alan. 2005. Image schemas and gesture. In Beate Hampe & Joseph E. Grady (eds.),
From perception to meaning: Image schemas in cognitive linguistics, 421–441.
Berlin: Mouton de Gruyter.
Unauthenticated
Download Date | 7/20/19 4:23 PM
98 Kasper I. Kok and Alan Cienki
Cienki, Alan. 2012. Usage events of spoken language and the symbolic units we (may) abstract
from them. In Krzysztof Kosecki & Janusz Badio (eds.), Cognitive processes in language,
149–158. Frankfurt am Main: Peter Lang.
Cienki, Alan. 2013. Cognitive linguistics: Spoken language and gesture as expressions of
conceptualization. In C. Müller, A. Cienki, S. Ladewig, D. McNeill & S. Teßendorf (eds.),
Body – language – communication: An international handbook on multimodality in human
interaction, 1, 182–201. Berlin: Mouton de Gruyter
Cienki, Alan. 2014. Grammaticheskie teorii v kognitivnoi lingvistike i polimodal’nost’
kommunikacii. [Grammatical theories in cognitive linguistics and the multimodality of
communication.]. In O. V. Fedorova & A. A. Kibrik (eds.), Mul’timodal’naja kommunikacija:
Teoreticheskie i èmpiricheskie issledovanija [Multimodal communication: Theoretical and
empirical research], 86–98. Moscow: Buki Vedi.
Cienki, Alan. 2015. Spoken language usage events. Language and Cognition 7(4). 499–514.
Clark, Andy. 2013. Gesture as thought. In Zdravko Radman (ed.), The hand, an organ of the
mind: What the manual tells the mental, 255–268. Cambridge, MA: MIT Press.
De Ruiter, Jan Peter. 2007. Postcards from the mind: The relationship between speech, ima-
gistic gesture, and thought. Gesture 7(1). 21–38.
Efron, David. 1941 [1972]. Gesture, race and culture. The Hague: Mouton de Gruyter.
Enfield, Nick. 2004. On linear segmentation and combinatorics in co-speech gesture: A sym-
metry-dominance construction in Lao fish trap descriptions. Semiotica 149. 57–124.
Enfield, Nick. 2009. The anatomy of meaning: Speech, gesture, and composite utterances.
Cambridge: Cambridge University Press.
Enfield, Nick. 2013. A “Composite Utterances” approach to meaning. In C. Müller, A. Cienki, E.
Fricke, S. H. Ladewig, D. McNeill & J. Bressem (eds.), Body – language – communication:
An international handbook on multimodality in human interaction, 2, 689–707.
Berlin & Boston: De Gruyter Mouton.
Fricke, Ellen. 2007. Origo, Geste und Raum. Lokaldeixis im Deutschen. Berlin: Walter de Gruyter.
Fricke, Ellen 2012. Grammatik multimodal: Wie Wörter und Gesten zusammenwirken. Berlin:
Mouton de Gruyter.
Fricke, Ellen 2013. Towards a unified grammar of gesture and speech: A multimodal approach.
In C. Müller, A. Cienki, S. Ladewig, D. McNeill & S. Teßendorf (eds.), Body – language –
communication: An international handbook on multimodality in human interaction,
1, 733–754. Berlin: Mouton de Gruyter.
Hassemer, Julius. 2015. Towards a theory of gesture form analysis: Principles of gesture
conceptualisation, with empirical support from motion-capture data. Aachen: RWTH
Aachen University dissertation.
Hostetter, Autumn B. & Martha W. Alibali. 2008. Visible embodiment: Gestures as simulated
action. Psychonomic Bulletin & Review 15(3). 495–514.
Kendon, Adam. 1970. Movement coordination in social interaction: Some examples described.
Acta psychologica 32. 101–125.
Kendon, Adam. 1980. Gesticulation and speech: Two aspects of the process of utterance.
In M. R. Key (ed.), The relationship of verbal and nonverbal communication, 207–227.
The Hague: Mouton de Gruyter.
Kendon, Adam. 1988. How gestures can become like words. In F. Poyatos (ed.), Cross-cultural
perspectives in nonverbal communication, 131–141. Toronto: CJ Hogrefe.
Kendon, Adam. 2004. Gesture: Visible action as utterance. Cambridge: Cambridge
University Press.
Unauthenticated
Download Date | 7/20/19 4:23 PM
Cognitive Grammar and gesture 99
Kita, Sotaro (ed.). 2003. Pointing: Where language, culture, and cognition meet. Mahwah,
NJ: Lawrence Erlbaum Associates.
Kok, Kasper I. in press. The grammatical potential of co-speech gesture: A functional discourse
grammar perspective. Functions of Language 23.
Kok, Kasper I., Kisten Bergmann, Alan Cienki & Stefan Kopp. in press. Mapping out the
multifunctionality of speakers’ gestures. Gesture 15.
Ladewig, Silva. 2011a. Putting the cyclic gesture on a cognitive basis. CogniTextes 6,
http://cognitextes.revues.org/406.
Ladewig, Silva. 2011b. Syntactic and semantic integration of gestures into speech: Structural,
cognitive, and conceptual aspects. Frankfurt (Oder): European University Viadrina thesis.
Lakoff, George. 1987. Women, fire, and dangerous things: What categories reveal about the
mind. Chicago: University of Chicago Press.
Langacker, Ronald W. 1987. Foundations of cognitive grammar, Vol. I: Theoretical prerequisites.
Stanford: Stanford University Press.
Langacker, Ronald W. 1991. Foundations of cognitive grammar, Vol. II: Descriptive application.
Stanford: Stanford University Press.
Langacker, Ronald W. 2001. Discourse in cognitive grammar. Cognitive Linguistics 12(2). 143–188.
Langacker, Ronald W. 2008a. Cognitive grammar: A basic introduction. Oxford: Oxford
University Press.
Langacker, Ronald W. 2008b. Metaphoric gesture and cognitive linguistics. In Alan
Cienki & Cornelia Müller (eds.), Metaphor and gesture, 249–251. Amsterdam:
John Benjamins.
Langacker, Ronald W. 2012. Interactive cognition: Toward a unified account of structure,
processing, and discourse. International Journal of Cognitive Linguistics 3(2). 95–125.
Lapaire, Jean-Rémi. 2011. Grammar, gesture and cognition: Insights from multimodal
utterances and applications for gesture analysis. Visnyk of Lviv University. Series
Philology 52. 87–107.
Liddell, Scott K. 2003. Grammar, gesture, and meaning in American Sign Language.
Cambridge: Cambridge University Press.
Lücking, Andy, Kirsten Bergman, Florian Hahn, Stefan Kopp & Hannes Rieser. 2013.
Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture
Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces
7(1–2). 5–18.
McNeill, David. 1992. Hand and mind. What gestures reveal about thought. Chicago: University
of Chicago Press.
McNeill, David & Susan D. Duncan. 2000. Growth points in thinking-for-speaking. In David
McNeill (ed.), Language and gesture, 141–161. Cambridge: Cambridge University Press.
Mittelberg, Irene. 2014. Gestures and iconicity. In C. Müller, J. Bressem, A. Cienki, E. Fricke,
S. H. Ladewig, D. McNeill & J. Bressem (eds.). Body – language – communication:
An international handbook on multimodality in human interaction, 2, 1712–1732.
Berlin/Boston: De Gruyter Mouton.
Mittelberg, Irene & Linda R. Waugh. 2009. Metonymy first, metaphor second: A cognitive-
semiotic approach to multimodal figures of thought in co-speech gesture. In Charles
Forceville & Eduardo Urios-Aparisi (eds.), Multimodal metaphor, 329–356. Berlin:
Mouton de Gruyter.
Mittelberg, Irene & Linda R. Waugh. 2014. Gestures and metonymy. In C. Müller, A. Cienki,
E. Fricke, S. H. Ladewig, D. McNeill & J. Bressem (eds.), Body – language – communication.
Unauthenticated
Download Date | 7/20/19 4:23 PM
100 Kasper I. Kok and Alan Cienki
Unauthenticated
Download Date | 7/20/19 4:23 PM