2010 International Conference on Computer and Computational Intelligence (ICCCI 2010)

Mathematical Notation, Representation, and Visualization of Musical Rhythm: A

Comparative Perspective

Yang Liu Godfried T. Toussaint

School of the Museum of Fine Arts Boston Department of Music
Boston, MA, USA Harvard University
E-mail: yangliu1971@gmail.com Cambridge, MA, USA
E-mail: godfried@cs.mcgill.ca

Abstract—Several methods for the mathematical notation, [8]. Nevertheless, due to space limitations we focus only on
representation, and visualization of musical rhythm at the mathematical, and in particular geometric, methods for the
symbolic level are illustrated and compared in terms of their symbolic notation of musical rhythm, and we provide a small
advantages and drawbacks, as well as their suitability for sample of examples.
particular applications.
II. MNEMONIC NOTATION
Keywords: music, rhythm, music notation, color painting
notation, music representation, acoustic-mnemonic systems,
It is well known that ordinary speech in any language
binary sequences, box notation, Schillinger notation, ancient possesses rhythm caused by the patterns of accents or
Persian notation, necklace notation, convex polygon notation, stresses [39], [40]. Indeed, the employment of similar
interval-content representation, visualization. methods that use acoustic phonetic features of vowels and
consonants appears independently in geographically distant
cultures [12], [49], [50]. These systems are particularly
I. INTRODUCTION useful for teaching rhythm, and have been an invaluable tool
To the uninitiated, music and mathematics may appear as for transmitting rhythms in cultures based on oral traditions.
antithetical activities, one expressing the emotions of the For instance, the mnemonic system of syllables described in
heart in a phenomenological world, and the other exploring the Persian 13th Century book kit!b al-Adw!r [43] uses two
the precise and rigorous structures of the Platonic universe. syllables for the strong primary beats: ta and tan, for short
However, their intertwining relationship has a long history and long beats, respectively. Similarly two additional
that in Europe goes back to Pythagoras of Samos, who in the syllables are used for secondary beats: na and nan, for short
6th Century B.C. developed a musical scale based on the and long beats, respectively. Here the long beats last twice as
numerical integer ratio 3:2 [29]. The origin of using a scale long as the short beats. Using this system the clave rumba
consisting of twelve fundamental tones, however, appears to rhythm of Cuba is notated as: tanan tananan tanan tan
originate two millennia earlier in China with Huang-Ti, the tananan. Here the five beats of the clave rumba correspond
Yellow Emperor, circa 2700 B. C. [38]. The advent of music to the five ta sounds at the beginnings of the words.
notation on the other hand, whether possessing greater or
lesser mathematical structure, is more recent. Wulstan [35] III. MATHEMATICAL NOTATION
traces music notation back to the Babylonians in 1300 B.C., A variety of mathematical methods exist that are used to
and according to West [34] it goes back to at least 1800 B.C. notate a rhythmic or melodic sequence of tones. The simplest
In recent history there has been a renewed and energetic such method for rhythms is the binary sequence, in which a
surge in the mathematical and computational aspects of “1” is used to denote a sound, and a “0” to denote a silence
music [14], [16], [19], [23]-[33]. or rest [1]. Here both symbols represent one unit of time.
Music notation exists at many levels of abstraction Thus the clave rumba is notated as [1001000100101000].
ranging from the most concrete continuous acoustic signal Such a representation has obvious advantages for processing
(usually a waveform) through the most abstract discrete rhythms by computer, but its iconic value is minimal. An
symbolic notation [7], [9], [22], to the notation of emotion by improvement is the box-notation system, widely employed
means of facial expressions [20]. Some notation systems, by ethnomusicologists, in which the two symbols used to
such as Gongche notation, popular in ancient China, mark denote sound and silence are highly dissimilar [17], [18].
only the pitch of the notes, and not their duration [11]. Other One method employs “x” for sound and “-” for silence. The
notation systems use characters to indicate the finger result is [x - - x - - - x - - x – x - - -] for the clave rumba. A
positions for specific instruments, such as the Okinawan second more graphical rendering of box notation actually
notation system for the samisen [37], and the guitar tablature uses boxes that are either empty (silence) or contain a
notation in the West [6]. Composers have invented a variety symbol (sound). Different symbols may denote dissimilar
of notation systems for specific purposes when they felt that sounds. For instance, Fig. 1 shows the acoustic phonetic
traditional western notation did not serve their needs. For mnemonic for the clave rumba embedded in box notation,
instance, the painter and composer Michael Poast made where a black filled circle denotes a primary (strong) beat, a
extensive use of color in his paintings that serve as scores for grey filled circle a secondary (weak) beat, and an empty box
his compositions, and that stimulate musical expressiveness

978-1-4244-8950-3/10/$26.00 ©2010 IEEE V1-28

 2010 International Conference on Computer and Computational Intelligence (ICCCI 2010)

a silence. The actual clave rumba pattern consists of only the occurrences of the note onsets. For example, the regular 4-
five strong beats. beat, 16-pulse rhythm with intervals [4444], and the clave
son with intervals [33424] are described by the two curves
shown in Fig. 4 (top and bottom, respectively).

Figure 4. The clave son (bottom) in Schillinger rhythm notation.

Figure 1. Acoustic phonetic mnemonic and box-notation.
One approach to geometrically visualizing the interval
The box-notation system has been generalized to handle vector of a rhythm is by means of spectral notation [3].
several rhythms played simultaneously on different Consider the clave rumba with interval vector [34324]. In
instruments as shown in Fig. 2. Here the white circle spectral notation this vector is converted to a graph in which
indicates that the hi-hat is played in the open position. This the vertical axis marks the durations of the inter-onset
kind of notation is called drum tablature notation [1]. intervals, and the horizontal axis marks the onset number
(index), as illustrated in Fig. 5. The upper envelope of this
graph clearly highlights the pattern of variation among the
inter-onset intervals as the rhythm unfolds in time.

Figure 2. Drum tablature notation.

A more compact numerical notation system that favors

certain algebraic approaches to analysis and composition,
codifies the inter-onset intervals themselves using numbers
to obtain an interval vector [2]. In this notation the clave
rumba rhythm of Fig. 1 is coded as the 5-dimensional vector
[3,4,3,2,4]. One drawback of this scheme however, is that
rhythms containing different numbers of onsets yield vectors
in spaces of different dimensionalities that complicate certain Figure 5. The clave rumba in spectral notation.
kinds of analyses.
In spectral notation, at every onset one may readily
IV. GEOMETRIC NOTATION observe whether the subsequent inter-onset interval is greater,
In the early Twentieth Century in New York city, Joseph smaller, or remains equal. More careful observation reveals
Schillinger became famous for developing a detailed and the exact magnitude of these changes. However, from the
comprehensive mathematical methodology for analyzing, psychological perceptual perspective, humans have more
teaching, and composing music [4], [5], [41]. In his work he difficulty perceiving the quantitative aspects of these changes
made extensive use of geometric methods. His approach to than their qualitative counterparts. For practical purposes the
the notation of melodies in music is illustrated in Fig. 3, qualitative changes suffice. These qualitative changes are
which shows a fragment of Haydn’s Symphony No. 47 in G. referred to as the rhythmic contour in the music theory
In this diagram the width of each column corresponds to the literature [46], [47], [48]. Rhythmic contours are usually
duration of the shortest note employed, and the height notated with the three symbols “+”, “-“, and “0”, which for a
indicates the pitch in semitones. given inter-onset interval denote, respectively, that the
interval in question is greater, smaller, or has the same
duration as the previous interval. Thus in contour notation
the clave rumba is given by the sequence [+ - - + -], whereas
the clave son yields the contour [0 + - + -].
One drawback of spectral notation is that the time
information along the horizontal axis is lost. To make up for
Figure 3. A Haydn piece in Schillinger music notation. this deficiency, in 1988 Gustafson introduced what he called
TEDAS notation, an acronym that stands for Temporal
Schillinger also employed a geometric notation restricted Elements Displayed As Squares [3]. In this simple but
to pure rhythm without pitch information. For this purpose original and effective system the durations of the inter-onset
he used a rectilinear curve as a function of time, the height of intervals are, as in spectral notation, displayed along the
which alternates between two levels from the upper level to vertical axis, but they are also displayed along the horizontal
the lower level (and vice versa) at the locations of the

V1-29
 2010 International Conference on Computer and Computational Intelligence (ICCCI 2010)

axis. Therefore each inter-onset interval becomes a square, the inter-onset intervals marked are [34324]. The rumba
and the rhythm is displayed as a sequence of squares. For a clave consists of the five beats occurring at the first pulses of
concrete example consider the Manchu rhythm with interval each of these intervals.
vector [443122]. In TEDAS notation this rhythm becomes
the graph shown in Fig. 6. This notation not only highlights
the rhythmic contour of the rhythm, but it maintains its
temporal accuracy, and also affords a natural way to measure
rhythmic similarity [51].

Figure 8. The clave rumba in ancient Persian notation.

Variants of circular notation have been rediscovered

several times by different researchers. The typical
contemporary employment of circular notation uses a
Figure 6. A 6-onset Manchu rhythm in TEDAS notation.
“clock” diagram (also referred to as necklace notation) in
which all the equally spaced pulses in the cycle are marked
In 2002 Hoffman-Engl independently proposed a and numbered, the onsets of the rhythm are highlighted, and
notation for rhythms that is almost identical to Gustsfson’s time flows in the clockwise direction [1]. In another variant
TEDAS notation [10]. His chronotonic notation is illustrated (polygon notation) the onsets are connected with line
in Fig. 7 with the clave rumba rhythm. Recall that the clave segments to create a convex polygon [17], [18], [52]. Three
rumba has interval vector [34324]. In chronotonic notation examples of Manchu rhythms [36] in polygon notation are
the rhythm is displayed as a curve that connects a set of illustrated in Fig. 9, where black circles indicate main (strong)
points with line segments. One point occurs at every pulse beats and grey circles indicate secondary (weak) beats. More
position in time, and its height is equal to the duration of the recently, Benadon [45] has extended this notation by making
inter-onset interval to which it corresponds. Thus there are 3 the radial distance of a beat proportional to its duration, and
points at height 3, followed by 4 points at height 4, followed has found it useful for the study of expressive timing.
by 3 points at height 3, and so on. Hoffman-Engl defined a
rhythmic similarity measure based on this notation, and
reported experimental results that showed the measure was
correlated with human judgments of rhythm similarity.

Figure 9. Examples of Manchu rhythms [36] in polygon notation.

Symmetry is an important feature of music in general and

rhythm in particular [15], [21]. Polygon notation provides an
Figure 7. The clave rumba in chronotonic notation. effective means of visualizing the various symmetries that
may be present in cyclic rhythms. Consider for example the
A natural way to notate cyclic rhythms that repeat over three Manchu rhythms pictured in Fig. 9. The rhythm on the
and over during a piece of music is by means of a circular left has mirror symmetry about the line through pulses 4 and
diagram (also called a clock diagram). One of the earliest 12, and the center rhythm about the line through pulses 2 and
such methods was used by Safi al-Din in Thirteenth Century 10. The three strong beats in the center rhythm have mirror
Bagdad in his book kit!b al-Adw!r [42], [43]. An example symmetry about the line through pulses 0 and 8. The rhythm
of the clave rumba expressed in his notation is illustrated in on the right on the other hand possesses no mirror symmetry.
Fig. 8. The outer circle marks off the sixteen pulses in the
cycle with equally spaced small black dots. The inner circle V. REPRESENTATION
is reserved for identifying the rhythm (in this case rumba). The word notation suggests that it facilitates the reading
The sixteen pulses are separated into groups (feet) by small of music by a performer. However, an aspect of music may
white circles connected with line segments to the interior be displayed in ways that usefully illustrate some important
circle that help to visualize the temporal structure of the property of music, and that may in fact impede its readability
rhythm. An arrow marks the start of the rhythm as well as during performance. Such renderings of musical properties
the direction of the flow of time (counter-clockwise). Thus are perhaps better described by the word representation. In

V1-30
 2010 International Conference on Computer and Computational Intelligence (ICCCI 2010)

any case, the interdependence of notation and representation of the simplest and most elegant elementary proofs of this
systems, and the musical information they provide has been theorem is an induction proof due to Iglesias [54].
well documented. Cohen and Katz are careful to emphasize
that “no system of notation nor any kind of preservation of
musical information, be it the most highly developed, is truly
comprehensive” [13]. One of the most well-known and
studied representations of rhythms displays a rhythm’s full
interval content in the form of a histogram. It is common to
find music information retrieval systems that calculate global
features of this histogram to characterize and classify Figure 11. Two non-congruent homometric rhythms.
rhythms. Consider the six-onset Manchu rhythm shown in
Fig. 10 (left). The six adjacent inter-onset intervals are ACKNOWLEDGMENT
[443122]. In addition to these there are nine other non-
adjacent inter-onset intervals indicated by line segments. The This research was financially supported by the National
histogram of all fifteen intervals is shown on the right. Sciences and Engineering Research Council of Canada
(NSERC) administered through McGill University, Montreal,
and by the Radcliffe Institute for Advanced Study at Harvard
University, Cambridge, MA, where the second author was
the Emeline Bigelow Conland Fellow during the 2009-2910
academic year.
REFERENCES
[1] W. Sethares, Rhythm and Transforms, London: Springer-
Verlag, 2007.
[2] J. L. Hook, “Rhythm in the music of Messiaen: An algebraic
study and an application in the Turangalila Symphony,”
Figure 10. All the intervals of a Manchu rhythm and its histogram. Music Theory Spectrum, vol. 20, no. 1, April 1998, pp. 97-
120.
In a cyclic rhythm that lasts approximately a couple of [3] K. Gustafson, “The Graphical Representation of Rhythm,”
Technical report, Oxford University, 1988.
seconds, all the pair-wise inter-onset intervals are perceived
by the human mind. Therefore the interval content histogram [4] J. Schillinger, The Mathematical Basis of the Arts,
Philosophical Library, 1948.
provides instant information concerning the number and
[5] J. Schillinger, Encyclopedia of Rhythms, Da Capo Press,
durations of these intervals. For instance, examination of the 1976.
histogram in Fig. 10 reveals that apart from the four intervals [6] M. Gaare, “Alternatives to traditional notation,” Music
of duration 4, the rest of the histogram is quite flat, indicating Educators Journal, vol. 83, no. 5, March 1997, pp. 17-23.
that the irregular aspects of this Manchu rhythm are [7] R. G. Cromleigh, “Neumes, notes, and numbers: The many
embedded on a regular rhythmic framework consisting of methods of music notation,” Music Educators Journal, vol. 64,
beats at pulses 0, 4, 8, and 12. This representation has a no. 4, December 1977, pp. 30-39.
severe drawback however. Consider the two 6-onset rhythms [8] M. Poast, “Visual color notation for musical expression,”
in 12-pulse cycles pictured in Fig. 11 (left and center). The Leonardo, vol. 33, no. 3, 2000, pp. 215-221.
two rhythms are different in the most significant way [9] R. B. Dannenberg, “Music representation issues, techniques
possible: one is neither a rotation nor a mirror image of a and systems,” Computer Music Journal, vol. 17, no. 3,
Autumn. 1993, pp. 20-30.
rotation of the other, i.e., geometrically the two polygons are
[10] L. Hoffman-Engl, “Rhythmic similarity: A theoretical and
not congruent. This is clear from the fact that in the rhythm empirical approach,” Proc. Seventh International Conference
on the left the two intervals of duration 1 are separated by on Music Perception and Cognition, Sydney, Australia, 2002,
two intervals of duration 2, whereas in the rhythm in the pp. 564-567.
center they are separated by one interval of duration 2. Yet, [11] H. Xiangpeng and J. S. C. Lam, “Ancient tunes hidden in
surprisingly, both rhythms have exactly the same inter-onset modern Gongchee notation,” Yearbook for Traditional Music,
interval histogram shown on the right. Technically these two vol. 24, 1992, pp. 8-13.
rhythms are termed homometric. Therefore any features [12] D. W. Hughes, “No nonsense: The logic and power of
acoustic-iconic mnemonic systems,” British Journal of
calculated on the histogram of these two rhythms will fail to Ethnomusicology, vol. 9, no.2, 2000, pp. 93-120.
distinguish between them. One might hope that this [13] D. Cohen and R. Katz, “The interdependence of notation
disconcerting example is a rare anomaly that one can safely systems and musical information,” Yearbook of the
ignore in most real-life applications. Unfortunately this is not International Folk Music Council, vol.11, 1979, pp. 100-113
the case. Any two non-congruent complementary rhythms [14] T. Ellingson, “The mathematics of Tibetan Rol Mo,”
with k onsets and n pulses such that k = n/2 are homometric. Ethnomusicology, vol. 23, no.2, May 1979, pp. 225-243.
This result is known as the hexachordal theorem [53]. [15] P. Liebermann and R. Liebermann, “Symmetry in question
Surprisingly, this theorem has surfaced independently in both and answer sequences in music,” Computers and Mathematics
the music theory and crystallography literatures, where with Applications, vol. 19, no. 7, 1990, pp. 59-66.
several proofs of the theorem have been published [44]. One

V1-31
 2010 International Conference on Computer and Computational Intelligence (ICCCI 2010)

[16] F. Sobieczky, “Results in mathematics and music: and musical ideas,” British Journal of Ethnomusicology, vol.
Visualization of roughness in musical consonance,” Seventh 2, 1993, pp. 99-115.
IEEE Visualization, San Francisco, CA, October 27- [37] J. La Rue, “The Okinawan notation system,” Journal of the
November 1, 1996. American Musicological Society, vol. 4, no. 1, Spring 1951,
[17] G. T. Toussaint, “The geometry of musical rhythm,” In J. pp. 27-35.
Akiyama, M. Kano, and X. Tan, editors, Proc. Japan [38] K. Nakaseko, “Symbolism in ancient Chinese music theory,”
Conference on Discrete and Computational Geometry, Vol. Journal of Music Theory, vol. 1, no. 2, November 1957, pp.
3742, Lecture Notes in Computer Science, Springer, 147-180.
Berlin/Heidelberg, 2005, pp. 198-212.
[39] M. Liberman and A. Prince, “On stress and linguistic
[18] G. T. Toussaint, “Computational geometric aspects of musical rhythm,” Linguistic Inquiry, vol. 8, no. 2, Spring 1977, pp.
rhythm,” Abstracts 14th Fall Workshop on Computational 249-336.
Geometry, Massachusetts Institute of Technology, November
19-20, 2004, pp. 47-48. [40] B. Hayes, “The phonology of rhythm in English,” vol. 15, no.
1, Winter 1984, pp. 33-74.
[19] D. Tymoczko, A Geometry of Music: Harmony and
Counterpoint in the Extended Common Practice, Oxford [41] J. Schillinger, “General theory of rhythm as applied to
University Press, Oxford, UK, 2010. music,” Bulletin of the American Musicological Society, no.
2, June 1937, pp. 8-9.
[20] T. Nakanishi and T. Kitagawa, “Visualization of music
impression in facial expression to represent emotion,” Proc. [42] O. Wright, The Modal System of Arab and Persian Music
Third Asia-Pacific Conference on Conceptual Modelling, A.D. 1250-1300, London Oriental Series – vol. 28, Oxford
Hobart, Australia, 2006. University Press, Oxford, 1978.
[21] R. Donnini, “The visualization of music: Symmetry and [43] O. Wright, “A preliminary version of the kit!b al-Adw!r,”
asymmetry,” Computers and Mathematics with Applications, Bulletin of the School of Oriental and African Studies,
vol. 12B, nos. 1/2, 1986, pp. 435-463. University of London, vol. 58, no. 3, 1995, pp. 455-478.
[22] M. Naranjo and A. Koffi, “Geometric image modelling of the [44] G. T. Toussaint, "Elementary proofs of the hexachordal
musical object,” Leonardo, Supplemental Issue on Electronic theorem," Abstracts of the Special Session on Mathematical
Art, vol. 1, 1988, pp. 69-72. Techniques in Music Analysis - I, 113th Annual Meeting of
the American Mathematical Society, New Orleans, U.S.A.,
[23] D. Wright, Mathematics and Music, American Mathematical January 6, 2007.
Society, 2009.
[45] F. Benadon, “A circular plot for rhythm visualization and
[24] D. Temperley, Music and Probability, The MIT Press, 2007. analysis,” Music Theory Online, Vol. 13, No. 3, September
[25] G. Loy, Musimathics, Vol. 2: The Mathematical Foundations 2007.
of Music, The MIT Press, 2007. [46] I. Quinn, “Fuzzy extensions to the theory of contour,” Music
[26] G. Loy, Musimathics, Vol. 1: The Mathematical Foundations Theory Spectrum, vol. 19, no. 2, Autumn 1997, pp. 232-263.
of Music, The MIT Press, 2006. [47] I. Quinn, “The combinatorial model of pitch contour,” Music
[27] L. Harkleroad, The Math Behind the Music, Cambridge Perception, vol. 16, no. 4, Summer 1999, pp. 439-456.
University Press, 2006. [48] R. D. Morris, “New directions in the theory and analysis of
[28] T. H. Garland and C. V. Kahn, Math and Music: Harmonious musical contour,” Music Theory Spectrum, vol. 15, no. 2,
Connections, Dale Seymour Publications, 1995. Autumn 1993, pp. 205-228.
[29] J. Fauvel, R. Flood, and R. Wilson, (editors), Music and [49] B. Colley, “A comparison of syllabic methods for improving
Mathematics: From Pythagoras to Fractals, Oxford University rhythm literacy,” Journal of Research in Music Education, vol.
Press, USA, 2006. 35, no. 4, Winter 1987, pp. 221-235.
[30] D. Benson, Music: A Mathematical Offering, Cambridge [50] D. P. Ester, J. W. Scheib, and K. J. Inks, “A rhythm system
University Press, 2006. for all ages,” Music Educators Journal, vol. 93, no. 2,
[31] E. Chew, A. Childs, and C.-H. Chuan, eds., Mathematics and November 2006, pp. 60-65.
Computation in Music, (Proceedings of the Second [51] G. T. Toussaint, “A comparison of rhythmic similarity
International Conference), Springer-Verlag, Berlin, 2009. measures,” Proceedings of ISMIR 2004: 5th International
[32] T. Klouche and T. Noll, eds., Mathematics and Computation Conference on Music Information Retrieval, Universitat
in Music, (Proceedings of the First International Conference), Pompeu Fabra, Barcelona, Spain, October 10-14, 2004, pp.
Springer-Verlag, Berlin, 2009. 242-245.
[33] M. Keith, From Polychords to Pólya: Adventures in Musical [52] G. T. Toussaint, “Classification and phylogenetic analysis of
Combinatorics, Vinculum Press, Princeton, New Jersey, 1991. African ternary rhythm timelines,” Proceedings of BRIDGES:
Mathematical Connections in Art, Music, and Science,
[34] M. L. West, “The Babylonian musical notation and the University of Granada, Granada, Spain July 23-27, 2003, pp.
Hurrian melodic texts,” Music & Letters, vol. 75, no. 2., May, 25-36.
1994, pp. 161-179.
[53] M. Senechal, “A point set puzzle revisited,” European Journal
[35] D. Wulstan, “The earliest musical notation,” Music & Letters, of Combinatorics, vol. 29, 2008, pp. 1933-1944.
vol. 52, no. 4, October 1971, pp. 365-382.
[54] J. E. Iglesias, “On Patterson’s cyclotomic sets and how to
[36] L. Li, “Mystical numbers and Manchu traditional music: A count them,” Zeitschrift f¨ur Kristallographie, Vol. 156, 1981,
consideration of the relationship between shamanic thought pp. 187–196.

V1-32