Abstract
In this article, we describe a new computational model of switching between path-planning and cue-guided navigation strategies. It is based on three main assumptions: (i) the strategies are mediated by separate memory systems that learn independently and in parallel; (ii) the learning algorithms are different in the two memory systems—the cue-guided strategy uses a temporal-difference (TD) learning rule to approach a visible goal, whereas the path-planning strategy relies on a place-cell-based graph-search algorithm to learn the location of a hidden goal; (iii) a strategy selection mechanism uses TD-learning rule to choose the most successful strategy based on past experience. We propose a novel criterion for strategy selection based on the directions of goal-oriented movements suggested by the different strategies. We show that the selection criterion based on this “common currency” is capable of choosing the best among TD-learning and planning strategies and can be used to solve navigational tasks in continuous state and action spaces. The model has been successfully applied to reproduce rat behavior in two water-maze tasks in which the two strategies were shown to interact. The model was used to analyze competitive and cooperative interactions between different strategies during these tasks as well as relative influence of different types of sensory cues.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Arleo A, Gerstner W (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol Cybern 83(3): 287–299
Arleo A, Rondi-Reig L (2007) Multimodal sensory integration and concurrent navigation strategies for spatial cognition in real and artificial organisms. J Integr Neurosci 6(3): 327–366
Barrera A, Weitzenfeld A (2007) Bio-inspired model of robot spatial cognition: topological place recognition and target learning. In: CIRA, pp 61–66
Blaisdell A (2009) The role of associative processes in spatial, temporal, and causal cognition. In: Watanabe SB, Blaisdell AP, Huber L, Young A (eds) Rational animals, irrational humans. Keio University Press, Tokyo, pp 153–172
Brown M, Sharp P (1995) Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal formation and nucleus accumbens. Hippocampus 5(3): 171–188
Burgess N (2008) Spatial cognition and the brain. Ann N Y Acad Sci 1124: 77–97
Burnod Y (1991) Organizational levels of the cerebral cortex: an integrated model. Acta Biotheor 39(3–4): 351–361
Canal C, Stutz S, Gold P (2005) Glucose injections into the dorsal hippocampus or dorsolateral striatum of rats prior to T-maze training: modulation of learning rates and strategy selection. Learn Mem 12(4): 367–374
Chamizo V (2003) Acquisition of knowledge about spatial location: assessing the generality of the mechanism of learning. Q J Exp Psychol 56(1): 102–113
Chang Q, Gold PE (2003) Switching memory systems during learning: changes in patterns of brain acetylcholine release in the hippocampus and striatum in rats. J Neurosci 23(7): 3001
Chang Q, Gold PE (2004) Inactivation of dorsolateral striatum impairs acquisition of response learning in cue-deficient, but not cue-available, conditions. Behav Neurosci 118(2): 383–388
Chavarriaga R, Strösslin T, Sheynikhovich D, Gerstner W (2005) A computational model of parallel navigation systems in rodents. Neuroinformatics 3(3): 223–242
Conde F, Maire-Lepoivre E, Audinat E, Crepel F (1995) Afferent connections of the medial frontal cortex of the rat. II. Cortical and subcortical afferents. J Comp Neurol 352(4): 567–593
Cuperlier N, Quoy M, Gaussier P (2007) Neurobiologically inspired mobile robot navigation and planning. Front Neurorobotics 1: 1–15
Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12): 1704–1711
Descarries L, Lemay B, Doucet G, Berger B (1987) Regional and laminar density of the dopamine innervation in adult rat cerebral cortex. Neuroscience 21(3): 807–824
Devan B, White N (1999) Parallel information processing in the dorsal striatum: relation to hippocampal function. J Neurosci 19(7): 2789–2798
Devan B, McDonald R, White N (1999) Effects of medial and lateral caudate-putamen lesions on place-and cue-guided behaviors in the water maze: relation to thigmotaxis. Behav Brain Res 100(1–2): 5–14
Dijkstra E (1959) A note on two problems in connection with graphs. Numer Math 1(269–270): 269–271
Doeller CF, Burgess N (2008) Distinct error-correcting and incidental learning of location relative to landmarks and boundaries. Proc Natl Acad Sci USA 105(15): 5909–5914
Doeller CF, King JA, Burgess N (2008) Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory. Proc Natl Acad Sci USA 105(15): 5915–5920
Dolle L, Khamassi M, Girard B, Guillot A, Chavarriaga R (2008) Analyzing interactions between navigation strategies using a computational model of action selection. LNAI 5248: 71–86
Foster DJ, Morris RG, Dayan P (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10(1): 1–16
Franz MO, Mallot HA (2000) Biomimetic robot navigation. Rob Auton Syst 30(1): 133–153
Gibson B, Shettleworth S (2003) Competition among spatial cues in a naturalistic food-carrying task. Learn Behav 31(2): 143–159
Gibson B, Shettleworth S (2005) Place versus response learning revisited: tests of blocking on the radial maze. Behav Neurosci 119(2): 567–586
Girard B, Filliat D, Meyer J, Berthoz A, Guillot A (2005) Integration of navigation and action selection functionalities in a computational model of cortico-basal-thalamo-cortical loops. Adapt Behav 13(2): 115–130
Gold P (2004) Coordination of multiple memory systems. Neurobiol Learn Mem 82(3): 230–242
Grahn J, Parkinson J, Owen A (2008) The cognitive functions of the caudate nucleus. Prog Neurobiol 86(3): 141–155
Granon S, Poucet B (1995) Medial prefrontal lesions in the rat and spatial navigation: evidence for impaired planning. Behav Neurosci 109(3): 474–484
Groenewegen H, Berendse H, Meredith G, Haber S, Voorn P, Wolters J, Lohman A (1991) The mesolimbic dopamine system: from motivation to action. In: Willner P, Scheel-Kriiger J (eds) Functional anatomy of the ventral, limbic system-innervated striatum. Wiley, Chichester, pp 19–59
Guazzelli A, Corbacho F, Bota M, Arbib M (1998) Affordances, motivation, and the world graph theory. Adapt Behav 6(3): 435– 471
Hamilton D, Rosenfelt C, Whishaw I (2004) Sequential control of navigation by locale and taxon cues in the morris water task. Behav Brain Res 154(2): 385–397
Hartley T, Burgess N (2005) Complementary memory systems: competition, cooperation and compensation. Trends Neurosci 28(4): 169–170
Hasselmo ME (2005) A model of prefrontal cortical mechanisms for goal-directed behavior. J Cogn Neurosci 17(7): 1115–1129
Jankowski J, Scheef L, Hüppe C, Boecker H (2009) Distinct striatal regions for planning and executing novel and automated movement sequences. Neuroimage 44(4): 1369–1379
Kelly D, Gibson B (2007) Spatial navigation: spatial learning in real and virtual environments. Comp Cogn Behav Rev 2: 111–124
Khamassi M (2007) Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies. PhD thesis, University Paris 6
Kim J, Baxter M (2001) Multiple brain-memory systems: the whole does not equal the sum of its parts. Trends Neurosci 24(6): 324–330
Leising K, Blaisdell A (2009) Associative basis of landmark learning and integration in vertebrates. Comp Cogn Behav Rev 4: 80–102
Martinet LE, Passot JB, Fouque B, Meyer JA, Arleo A (2008) Map-based spatial navigation: a cortical column model for action planning. LNAI 5248: 39–55
McDonald R, White N (1993) A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behav Neurosci 107(1): 3–22
McDonald R, White N (1994) Parallel information processing in the water maze: evidence for independent memory systems involving dorsal striatum and hippocampus. Behav Neural Biol 61(3): 260–270
McDonald R, Devan B, Hong N (2004) Multiple memory systems: the power of interactions. Neurobiol Learn Mem 82(3): 333–346
Mizumori S (2008) Hippocampal place fields. Oxford University Press, USA
O’Keefe J, Dostrovsky J (1971) The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34(1): 171–175
O’Keefe J, Nadel L (1978) The hippocampus as a cognitive map. Oxford University Press, Oxford
Packard M, McGaugh J (1992) Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: further evidence for multiple memory systems. Behav Neurosci 106(3): 439–446
Packard M, McGaugh J (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65(1): 65–72
Packard M, Hirsh R, White N (1989) Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems. J Neurosci 9: 1465–1472
Pearce J (2009) The 36th Sir Frederick Bartlett Lecture: an associative analysis of spatial learning. Q J Exp Psychol 62(9): 1665–1684
Pearce J, Roberts A, Good M (1998) Hippocampal lesions disrupt navigation based on cognitive maps but not heading vectors. Nature 396(6706): 75–77
Pouzet B, Zhang W, Feldon J, Rawlins J (2002) Hippocampal lesioned rats are able to learn a spatial position using non-spatial strategies. Behav Brain Res 133(2): 279–291
Ragozzino M, Detrick S, Kesner R (1999) Involvement of the prelimbic-infralimbic areas of the rodent prefontal cortex in behavioral flexibility for place and response learning. J Neurosci 19(11): 4585–4594
Redish A (1999) Beyond the cognitive map: from place cells to episodic memory. The MIT Press, Cambridge
Rescorla R, Wagner A (1972) A theory of pavlovian conditioning: the effectiveness of reinforcement and non-reinforcement. In: Black A, Prokasy W (eds) Classical conditioning II: current research and theory. Appleton-Century-Crofts, New York, pp 64–69
Rich E, Shapiro M (2007) Prelimbic/infralimbic inactivation impairs memory for multiple task switches, but not flexible selection of familiar tasks. J Neurosci 27(17): 4747
Rich E, Shapiro M (2009) Rat prefrontal cortical neurons selectively code strategy switches. J Neurosci 29(22): 7208–7219
Roberts A, Pearce J (1999) Blocking in the Morris swimming pool. J Exp Psychol Anim Behav Process 25(2): 225–235
Save E, Poucet B (2000) Involvement of the hippocampus and associative parietal cortex in the use of proximal and distal landmarks for navigation. Behav Brain Res 109(2): 195–206
Sheynikhovich D, Chavarriaga R, Strösslin T, Arleo A, Gerstner W (2009) Is there a geometric module for spatial orientation? Insights from a rodent navigation model. Psychol Rev 116(3): 540–566
Skaggs W, Knierim J, Kudrimoti H, McNaughton B (1995) A model of the neural basis of the rat’s sense of direction. Adv Neural Inf Process Syst 7: 173–182
Stahlman W, Blaisdell A (2009) Blocking of spatial control by landmarks in rats. Behav Processes 81(1): 114–118
Strösslin T, Sheynikhovich D, Chavarriaga R, Gerstner W (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Netw 18(9): 1125–1140
Sutton R, Barto A (1998) Reinforcement learning: an introduction. Bradford Book. The MIT Press, Cambridge
Taube JS, Muller RU, Ranck JB Jr (1990) Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. J Neurosci 10(2): 420
Touretzky D, Redish A (1996) Theory of rodent navigation based on interacting representations of space. Hippocampus 6(3): 247–270
Uchibe E, Doya K (2005) Reinforcement learning with multiple heterogeneous modules: a framework for developmental robot learning. In: The 4th international conference on development and learning. IEEE Computer Society Press, pp 87–92
Ujfalussy B, Eros P, Somogyvari Z, Kiss T (2008) Episodes in space: a modelling study of hippocampal place representation. LNAI 5040: 123–136
Voermans N, Petersson K, Daudey L, Weber B, Van Spaendonck K, Kremer H, Fernández G (2004) Interaction between the human hippocampus and the caudate nucleus during route recognition. Neuron 43(3): 427–435
White N (2004) The role of stimulus ambiguity and movement in spatial navigation: a multiple memory systems analysis of location discrimination. Neurobiol Learn Mem 82: 216–229
White N (2009) Some highlights of research on the effects of caudate nucleus lesions over the past 200 years. Behav Brain Res 199(1): 3–23
White N, McDonald R (2002) Multiple parallel memory systems in the brain of the rat. Neurobiol Learn Mem 77: 125–184
Yin H, Knowlton B (2004) Contributions of striatal subregions to place and response learning. Learn Mem 11(4): 459–463
Young J, Shapiro M (2009) Double dissociation and hierarchical organization of strategy switches and reversals in the rat PFC. Behav Neurosci 123(5): 1028–1035
Zhang K (1996) Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. J Neurosci 16(6): 2112
Author information
Authors and Affiliations
Corresponding author
Additional information
Laurent Dollé, Denis Sheynikhovich—First authorship shared.
Rights and permissions
About this article
Cite this article
Dollé, L., Sheynikhovich, D., Girard, B. et al. Path planning versus cue responding: a bio-inspired model of switching between navigation strategies. Biol Cybern 103, 299–317 (2010). https://doi.org/10.1007/s00422-010-0400-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-010-0400-z