ARTICLE IN PRESS Journal of Theoretical Biology 251 (2008) 210–226 www.elsevier.com/locate/yjtbi Evolution of dominance under frequency-dependent intraspecific competition Stephan Peischla, Reinhard Bürgera,b, a Department of Mathematics, University of Vienna, Nordbergstrasse 15, 1090 Wien, Austria Program for Evolutionary Dynamics, Harvard University, One Brattle Square, Cambridge, MA 02138, USA b Received 19 June 2007; received in revised form 15 October 2007; accepted 14 November 2007 Available online 19 November 2007 Abstract A population-genetic analysis is performed of a two-locus two-allele model, in which the primary locus has a major effect on a quantitative trait that is under frequency-dependent disruptive selection caused by intraspecific competition for a continuum of resources. The modifier locus determines the degree of dominance at the trait level. We establish the conditions when a modifier allele can invade and when it becomes fixed if sufficiently frequent. In general, these are not equivalent because an unstable internal equilibrium may exist and the condition for successful invasion of the modifier is more restrictive than that for eventual fixation from already high frequency. However, successful invasion implies global fixation, i.e., fixation from any initial condition. Modifiers of large effect can become fixed, and also invade, in a wider parameter range than modifiers of small effect. We also study modifiers with a direct, frequency-independent deleterious fitness effect. We show that they can invade if they induce a sufficiently high level of dominance and if disruptive selection on the ecological trait is strong enough. For deleterious modifiers, successful invasion no longer implies global fixation because they can become stuck at an intermediate frequency due to a stable internal equilibrium. Although the conditions for invasion and for fixation if sufficiently frequent are independent of the linkage relation between the two loci, the rate of spread depends strongly on it. The present study provides further support to the view that evolution of dominance may be an efficient mechanism to remove unfit heterozygotes that are maintained by balancing selection. It also demonstrates that an invasion analysis of mutants of very small effect is insufficient to obtain a full understanding of the evolutionary dynamics under frequency-dependent selection. r 2007 Elsevier Ltd. All rights reserved. Keywords: Frequency dependence; Disruptive selection; Intraspecific competition; Balancing selection; Stable underdominance; Evolution of genetic architecture; Dominance modifier 1. Introduction Frequency-dependent intraspecific competition has been invoked in the explanation of a number of important evolutionary phenomena. These include the maintenance of high levels of genetic variation (Cockerham et al., 1972; Clarke, 1979; Asmussen and Basnayake, 1990; Gavrilets and Hastings, 1995), in particular, in quantitative traits (Bulmer, 1974; Slatkin, 1979; Christiansen and Loeschcke, 1980; Loeschcke and Christiansen, 1984; Bürger Corresponding author at: Department of Mathematics, University of Vienna, Nordbergstrasse 15, 1090 Wien, Austria. E-mail addresses: stephan.peischl@univie.ac.at (S. Peischl), reinhard.buerger@univie.ac.at (R. Bürger). 0022-5193/$ - see front matter r 2007 Elsevier Ltd. All rights reserved. doi:10.1016/j.jtbi.2007.11.014 2002a, b, 2005; Bürger and Gimelfarb, 2004; Schneider, 2006), the evolution of sexual dimorphism (Slatkin, 1984; Bolnick and Doebeli, 2003; Van Dooren et al., 2004), the evolutionary splitting of assortatively mating populations (Drossel and McKane, 2000; Bolnick, 2004; Kirkpatrick and Nuismer, 2004; Bürger and Schneider, 2006; Bürger et al., 2006; Schneider and Bürger, 2006), the evolution of assortative mating and sympatric speciation (Maynard Smith, 1966; Udovic, 1980; Doebeli, 1996; Dieckmann and Doebeli, 1999; Dieckmann et al., 2004; Matessi et al., 2001; Kirkpatrick and Ravigné, 2002; Gavrilets, 2003, 2004; Polechová and Barton, 2005), and the evolution of genetic architecture (van Doorn and Dieckmann, 2006; Kopp and Hermisson, 2006; Schneider, 2007). The latter includes the evolution of dominance, the topic ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 to which this work is devoted. For a recent review of various evolutionary responses to disruptive selection, see Rueffler et al. (2006). Evolution of dominance has been a controversial issue in population genetics which has its origin in the famous dispute between Fisher and Wright beginning in 1928; for a review see Mayo and Bürger (1997). Theories explaining dominance and its modification within a biochemical framework are discussed in Bürger and Bagheri (2008). The idea common to models of dominance evolution is that if some heterozygous and homozygous genotypes differ in fitness, dominance should evolve such that the phenotype associated with the disadvantageous genotype resembles the superior phenotype. However, a dominance modifier can attain an evolutionarily significant selective advantage only if heterozygotes are sufficiently frequent. This may occur during a selective sweep (e.g., Haldane, 1956; Wagner and Bürger, 1985), if there is heterozygous advantage (Sheppard, 1958; Bürger, 1983; Otto and Bourguet, 1999), under disruptive selection (Clarke and Sheppard, 1960), under frequency-dependent selection (e.g., Clarke, 1964; O’Donald, 1968) or, more generally, if heterozygotes are maintained at high frequency by some form of balancing selection, such as migration-selection balance or spatially varying selection (Fisher, 1931; Van Dooren, 1999; Otto and Bourguet, 1999). As noted by Mayo and Bürger (1997), most well-documented empirical examples for the evolution of dominance originate from studies of genetic variation in a relatively well-understood ecological context. Wilson and Turelli (1986) found stable underdominance in a model of differential utilization of two resources and argued that evolution of dominance would be an efficient mechanism for removing unfit heterozygotes. Also strong intraspecific competition for a single, continuously distributed resource is known to cause stable polymorphism and disruptive selection (e.g., Bulmer, 1974; Christiansen and Loeschcke, 1980). In related contexts, evolution of dominance has been demonstrated theoretically in subdivided populations (Otto and Bourguet, 1999) and in a heterogeneous environment (Van Dooren, 1999). Evolution of dominance caused by pesticide resistance may be a case in point (Otto and Bourguet, 1999). In both models, frequency-dependent disruptive selection results from variation in the direction of selection among the patches. Thus, under appropriate conditions balancing selection maintains heterozygotes at high frequency. It is tempting to speculate about evolution of dominance caused by intraspecific competition for resources in the African finch Pyrenestes (Smith, 1993), where two morphs differ substantially in lower mandible width. Apparently, these morphs are randomly breeding with respect to these traits. Disruptive selection is most likely related to seed quality, because large morphs feed more efficiently on a hard-seeded species of sedge and small morphs on a softseeded species. The trait under putative disruptive selection shows a distinct bimodal distribution, and there is evidence 211 that the bill-size polymorphism is caused by a single autosomal diallelic locus with complete dominance for the large-bill morph. Although the seed distribution is strongly bimodal, a putative ancestral generalist (Rueffler et al., 2006) may have had highest fitness under low population density or weak competition. Another example, where negative frequency-dependent selection on a single gene maintains a polymorphism, is the foraging gene in larvae of Drosophila melanogaster (Fitzpatrick et al., 2007). Also in this case, heterozygous larvae are similar to one of the homozygotes (to the rovers). A further example may be the jaw asymmetry in the scale-eating cichlid Perissodus microlepis (Hori, 1993). In this paper, we perform a population-genetic analysis of a two-locus two-allele model, in which the primary locus has a major effect on a quantitative trait that is under frequency-dependent selection caused by intraspecific competition for a continuum of resources. The modifier locus determines the degree of dominance at the trait level. Our approach differs in important respects from related ones. Motivated by molecular-genetic considerations, Van Dooren (1999) assumed a single locus and considered two specific genotype–phenotype maps for which dominance interactions between alleles derive from their promoter affinities. His population inhabits a heterogeneous environment according to a Levene model with two demes and soft selection. His analysis is within the adaptive-dynamics framework, i.e., successive invasion is studied of single mutants that change the dominance relation, each by a small amount. Otto and Bourguet (1999) investigated a population-genetic model similar in spirit to the present one, but had two habitats connected by migration and alternative alleles selectively favored in the two habitats. We explore the conditions under which an allele modifying the dominance relations at the primary locus can invade and when it becomes fixed if already frequent. These conditions are often not equivalent because an internal equilibrium may exist. In the case of asymmetric selection, depending on the direction, the induced dominance effect may have to be sufficiently strong for a modifier to be able to invade. In contrast to most previous studies, we admit modifiers with a (frequency-independent) deleterious effect on fitness. We show that modifiers with a modest deleterious effect can invade and rise to fixation if their modifying effect is sufficiently large. 2. The model Our model follows closely that of Bürger (2005). We consider a sexually reproducing population of diploid organisms with discrete generations. Both sexes have the same genotype distribution and are treated as indistinguishable. The population size N may be density regulated, but is sufficiently large so that random genetic drift can be ignored. Natural selection acts through differential viabilities on a quantitative trait. Individual fitness depends on two components: frequency-independent stabilizing selec- ARTICLE IN PRESS 212 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 tion on this trait, and frequency- and density-dependent competition among individuals of similar phenotype. A single locus with two alleles contributes to the trait. Alleles on a second locus, the modifier locus, determine the dominance relations on the primary locus. 2.1. Ecological assumptions The first fitness component of the quantitative trait is frequency independent and may reflect some sort of direct selection on the trait, for example, by differential supply of a resource whose utilization efficiency is phenotype dependent. We ignore environmental variation and deal directly with the fitnesses of genotypic values. For simplicity, we use the words genotypic value and phenotype synonymously. Stabilizing selection is modeled by the quadratic function SðgÞ ¼ 1  sðg  yÞ2 , (1) where sX0 measures the strength of stabilizing selection and y is the position of the optimum. Of course, SðgÞ is assumed positive on the range of possible phenotypes, which is scaled to ½1; 1. Thus, we have the constraint 0pspð1 þ jyjÞ2 . In addition, we restrict attention to the case jyjo1, so there is always stabilizing selection. The second component of fitness is frequency dependent. We assume that competition between phenotypes g and h can be described by aðg; hÞ ¼ 1  cðg  hÞ2 , (2) where 0pcp14. This implies that competition between individuals of similar phenotype is stronger than between individuals of very different phenotype, as it will be the case if different phenotypes preferentially utilize different food resources. Large c implies a strong frequency-dependent effect of competition whereas in the limit c ! 0, frequency dependence vanishes. Let PðhÞ denote the relative frequency of individuals with phenotype h. Then the intraspecific competition function āðgÞ, which measures the strength of competition experienced by phenotype g if the population distribution is P, is given by X āðgÞ ¼ aðg; hÞPðhÞ h and calculated to āðgÞ ¼ 1  c½ðg  ḡÞ2 þ V . W ðgÞ ¼ 1  sðg  yÞ2 þ sf ½ðg  ḡÞ2 þ V , (4) (5) where the dependence of W ðgÞ on P is omitted (Bürger, 2005). There, it was shown that W ðgÞ is the weak-selection approximation of fitness (i.e., to first order in s) in most models of intraspecific competition for a continuum of resources, e.g., in those of Bulmer (1974), Slatkin (1979), Christiansen and Loeschcke (1980), Loeschcke and Christiansen (1984), Doebeli (1996), Dieckmann and Doebeli (1999), Bürger (2002a, b), Bolnick and Doebeli (2003), Bolnick (2004), Bürger and Gimelfarb (2004), Gourbiere (2004), Kirkpatrick and Nuismer (2004), Schneider (2006), Schneider and Bürger (2006). Therefore, the present results are representative for a large class of functional forms for fitness if selection is not too strong. Density dependence will be treated in Section 3.6. Because W ðgÞ is quadratic in g, selection is disruptive if W ðgÞ is convex and the minimum is within the range of phenotypic values, ½1; 1. For a given population distribution P, ḡ and V are constants and W ðgÞ is twice differentiable. By straightforward calculation, we obtain that W ðgÞ is convex if and only if f 41. The minimum lies in the interior of the phenotypic range if and only if f4 1þy 1 þ ḡ and f4 1y . 1  ḡ (6) If both conditions are satisfied, f 41 holds. Therefore, the conditions (6) are necessary and sufficient for selection to be disruptive for every population distribution with mean ḡ. Hence, if f 41, then selection is disruptive if ḡ is sufficiently close to y, otherwise it is directional. If ḡ ¼ y, then f 41 is necessary and sufficient for selection to be disruptive. Therefore, we call intraspecific competition strong if f 41, and weak if f o1. 2.2. Genetic assumptions and evolutionary dynamics We consider a simple model, in which the trait value, g, is determined by a single, diallelic locus, which we shall call the primary locus. We label the two alleles a and A. The alleles at the modifier locus are m and M, and they affect the dominance relations on the primary locus as follows: mm (3) Here, ḡ and V denote the mean and variance, respectively, of the distribution P of genotypic values. In the following, it will be convenient to measure the strength of frequencydependent competition relative to the strength of stabilizing selection. Therefore, we define f ¼ c=s. We shall treat f and s as independent parameters and assume f X0. First, we ignore density dependence and assume that relative fitness is given by mM MM aa 1 1 1 aA a AA 1 b g 1 1: (7) We assume 0pjajobpgp1, where a can be positive or negative. Thus, we assume that M induces a higher degree of dominance than m. If a ¼ 0, the case to which we pay most attention, the alleles a and A contribute additively to the trait if M is absent. For biological purposes, it may be ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 more natural to assume yX0 and to admit positive and negative values of b and g because dominance may be modified in either direction. In the Discussion we reformulate our main results for this scenario. Mathematically, it is equivalent to the one used here and below because gXb40 and yo0 is equivalent to gpbo0 and y40. Admitting y 2 ð1; 1Þ and restricting b and g to positive values allows a more efficient formulation and visualization of many results. Modifiers with a direct deleterious effect on fitness will be treated in Section 3.3. The frequencies of the gametes am, Am, aM, and AM are denoted by p1 , p2 , p3 , and p4 , respectively. Due to the assumption of random mating and because gamete frequencies are measured after reproduction and before selection, we can use Hardy–Weinberg proportions and it is sufficient to follow gamete frequencies. We denote the recombination rate between the primary locus and the modifier locus by r, and linkage disequilibrium by D ¼ p1 p4  p2 p3 . The marginal fitness of gamete i is denoted by W i and can be calculated from (7) and (5). For instance, we have W 1 ¼ W ð1Þp1 þ W ðaÞp2 þ W ð1Þp3 þ W ðbÞp4 . Mean fitness is X W¼ W i pi ¼ 1  sðḡ  yÞ2 þ 2sfV , (8) 213 15 randomly chosen initial conditions, subject to the constraint that the minimum Euclidean distance between any pair is 0.2. More precisely, for several combinations of s, b, and g a fine grid of values of y and f was chosen. The equilibria and their properties were recorded. In addition to these iterations, results were also checked by solving the equilibrium equations numerically and by computing the eigenvalues. 3.1. Equilibrium and stability structure for neutral modifiers We exclude the degenerate case f ¼ 1 in which a manifold of equilibria exists which, apparently, attracts all trajectories. For f a0, the complexity of the model precludes explicit calculation of all possible equilibria. We have been able to determine all equilibria on the boundary of the state space, which is the simplex S4 (Fig. 1). In addition, an internal, i.e., fully polymorphic, equilibrium may exist. Our numerical results suggest that at most one stable internal equilibrium can exist, which is always in linkage equilibrium. Table 1 lists the types of equilibria found. Their properties are summarized below. A full i where ḡ ¼ ðp2 þ p4 Þ2ðp1 þ p3 Þ2 þ 2ap1 p2 þ 2bðp1 p4 þp2 p3 Þ þ2gp3 p4 and V ¼ ðp2 þp4 Þ2 þðp1 þ p3 Þ2 þ2a2 p1 p2 þ 2b2 ðp1 p4 þp2 p3 Þ þ2g2 p3 p4  ḡ2 . The genetic dynamics is given by the well known system of recursion relations W p0i ¼ pi W i  ri W ðbÞD; i ¼ 1; 2; 3; 4, (9) where r1 ¼ r4 ¼ r and r2 ¼ r3 ¼ r. 3. Results Of prime interest are the conditions when a modifier of dominance (M) can invade and when it becomes fixed in the population. To this aim, we find the possible equilibria and establish their stability properties. First, we assume a neutral modifier and a population of constant size, for instance, at demographic equilibrium. After considering several special cases in Section 3.2, we relax the condition of neutrality in Section 3.3 and investigate modifiers with a direct deleterious effect. Whereas in Sections 3.1–3.3, absence of dominance ða ¼ 0Þ is assumed at the primary locus in the presence of mm, the case aa0 is the subject of Section 3.4. The effect of recombination and the rate of evolution are treated in Section 3.5. Finally, in Section 3.6, we briefly report results for a neutral modifier if densitydependent population growth is admitted. Throughout, we restrict our attention to positive recombination, r40. Because not all properties of interest could be proved analytically, we complement the analytical work by numerical results. These were obtained primarily by iterating the recursion relations (9) for a large number of combinations of the parameters s; b; g; y, and f, each from Fig. 1. Schematic drawing of all possible equilibria for r40. Table 1 The five types of equilibria (a) (b) (c) (d) (e) Fixation of allele a, every point on the edge p1 þ p3 ¼ 1, in particular, the monomorphic equilibria p1 ¼ 1 and p3 ¼ 1 Fixation of allele A, every point on the edge p2 þ p4 ¼ 1, in particular, the monomorphic equilibria p2 ¼ 1 and p4 ¼ 1 A single-locus polymorphism at the primary locus with modifier allele m fixed, i.e., p1 þ p2 ¼ 1 and 0op1 o1; denoted p^ ðmÞ A single-locus polymorphism at the primary locus with modifier allele M fixed, i.e., p3 þ p4 ¼ 1 and 0op3 o1; denoted p^ ðMÞ An internal equilibrium in linkage equilibrium, i.e., D ¼ 0, where both loci are polymorphic; denoted by p^ LE ARTICLE IN PRESS 214 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 account of the mathematical results together with proofs is given in the Appendix A. 3.1.1. The edge equilibria, (a) and (b) They exist always because if the primary locus is fixed for either A or a, modifier alleles have no effect and there is no mutation. In general, only the stability conditions for the monomorphic equilibria, i.e., pi ¼ 1, can be given explicitly (Appendix A.1). Those for the equilibria on the edges can be determined approximately by assuming weak selection, i.e., s5r. In this case and if f 41, there is one eigenvalue 41 (and one is always 1 because these are manifolds of equilibria), hence none can be stable if f 41 (Appendix A.2). Our numerical results suggest that this is true for every admissible s. Stability of any of these equilibria implies the possible loss of all genetic variation on the primary locus. If all edge equilibria are unstable, there is a protected polymorphism at the primary locus. This is always the case if f 41, the situation we are most interested in. 3.1.2. The equilibrium (c) ^ ðmÞ This equilibrium satisfies p^ ðmÞ 3 ¼p 4 ¼ 0, and the frequency of allele A is 1 y ^ ðmÞ , (10) p^ ðmÞ A ¼p 2 ¼ þ 2 1þf where we denote allele frequencies by the corresponding subscripts. It exists, i.e., satisfies 0op^ ðmÞ A o1, if and only if f 42jyj  1. (11) Hence, p^ ðmÞ exists whenever jyjo12 or f 41. We note that if f 41, then (11) is equivalent to underdominance, whereas it is equivalent to overdominance if f o1. Stability of this equilibrium determines if a modifier allele that induces dominance can invade, i.e., invasion occurs if this equilibrium is unstable. If the optimum is symmetric, i.e., if y ¼ 0, the condition for local stability is simply f o1. In general, we set f2 ¼ 2y  1. b (12) Then, p^ ðmÞ exists and is asymptotically stable if and only if either maxð0; 2jyj  1; f 2 Þof o1, (13a) which requires b4y, or 1of of 2 , (13b) which requires boy (see Appendix A.3). To put it otherwise, if f 41, then invasion of the modifier allele M occurs if and only if f 4f 2 . (14) Notably, the invasion condition is independent of the recombination rate. A simple calculation yields ḡ ¼ 2y=ð1 þ f Þ at p^ ðmÞ , and conditions (6) show that disruptive selection acts on the equilibrium distribution if and only if f 41. Thus, any modifier allele with b40 can invade if yp0, whereas sufficiently large b (or f) is required if y40. Rearrangement of (14) shows that a modifier invades if and only if either bo0 or b42y=ðf þ 1Þ. This condition is easily shown to be equivalent to W ðbÞ4W ð0Þ, (15) where fitnesses are evaluated at p^ ðmÞ . Thus, if y40, only modifiers can invade that either have negative b or a sufficiently large b, such that the fitness valley can be crossed and a fitness gain is achieved. For graphical illustrations, see Fig. 2. 3.1.3. The equilibrium (d) ðMÞ It satisfies p^ ðMÞ ¼ p^ ðMÞ ¼ 0. In general, p^ A ¼ p^ ðMÞ is a 1 2 4 complicated solution of a cubic which does not give much insight. In Appendix A.4.1 it is shown that p^ ðMÞ exists if and only if   2y  1  g g  1  2y f 4f 0 ¼ max 0; ; . (16) 1g 1þg We note that f 41 implies f 4f 0 because we assume jyjo1. If MM causes complete dominance, i.e., g ¼ 1, then p^ ðMÞ is of much simpler form and a local stability analysis can be performed; see Section 3.2.3. Fig. 2. Fitness at the equilibrium p^ ðmÞ as a function of the trait value g. The positions of the resident genotypes are indicated on the horizontal axis and thick lines correspond to values of b such that a modifier M can invade. In the left panel the modifier is neutral ðt ¼ 0Þ and in the right panel deleterious ðt ¼ 0:01Þ. The other parameters are s ¼ 0:1; y ¼ 0:5; f ¼ 1:5 and a ¼ 0. ARTICLE IN PRESS 215 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 Stability of equilibrium (d) determines if a modifier inducing dominance becomes fixed if sufficiently frequent. If it is asymptotically stable, then this is the case. Since, as already noted, p^ ðMÞ is of very complicated form, a general linear stability analysis seems impossible. However, if f ¼ f 1 , where f1 ¼ ð1 þ bgÞ2 ð2y  b  gÞ , gð2  b2  g2 Þ þ bð1  b2 g2 Þ (17) then one eigenvalue of p^ ðMÞ equals unity because another equilibrium, p^ LE , passes through it. If f ¼ f 1 , then p^ ðMÞ A ¼ ð1 þ bÞð1 þ gÞ 2ð1 þ bgÞ (20) f 0 of ominð1; f 1 Þ. This is confirmed by numerical results, cf. Fig. 3. They also suggest that p^ ðMÞ is globally stable if f 4maxð1; f 2 Þ, (14), and that stability is independent of r. If f 41, fixation of M from sufficiently high frequency occurs if f 4f 1 , which translates into a complicated condition for b. In terms of fitnesses, it is equivalent to (21) W ðgÞ4W ðbÞ, ^ ðMÞ where fitnesses are evaluated at p . (18) and a local stability analysis can be performed if f ¼ f 1 þ  and  and s are sufficiently small. It demonstrates that p^ ðMÞ changes stability as p^ LE passes through at  ¼ 0, and p^ ðMÞ exists and is asymptotically stable if either f 4maxð1; f 1 Þ or (19) 3.1.4. The equilibrium (e) In general, this fully polymorphic equilibrium cannot be determined explicitly. Numerical computations indicate that it is always in linkage equilibrium and that no other internal equilibrium exists. In the special case b ¼ g, when the modifier allele M is dominant, p^ LE can be calculated Fig. 3. Regions of existence and stability of the equilibria. In region Omo , only monomorphic equilibria and those on the edges p1 þ p3 ¼ 1 and p2 þ p4 ¼ 1 can be stable. In region O1 , p^ ðmÞ and an interval of fixed points on the edge p1 þ p3 are stable. In O2 , p^ ðMÞ and an interval of fixed points on the edge p2 þ p4 ¼ 1 are stable. In region Om , the equilibrium p^ ðmÞ is globally asymptotically stable, and in OM , p^ ðMÞ is globally asymptotically stable. In Om_M , both p^ ðmÞ and p^ ðMÞ are asymptotically stable, and the fully polymorphic equilibrium p^ LE exists and is unstable. Thus, depending on the initial conditions, either m or M becomes eventually fixed. In Om^M , both p^ ðmÞ and p^ ðMÞ are unstable, and p^ LE is globally attracting. Parameter values are b ¼ 0:1 and g ¼ 0:2 in the upper left panel, b ¼ 0:5 and g ¼ 0:6 in the upper right panel, b ¼ 0:5 and g ¼ 1 in the lower left panel, and b ¼ g ¼ 1 in the lower right panel. ARTICLE IN PRESS 216 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 explicitly and a more detailed analysis is performed in Section 3.2.2. A bifurcation analysis suggests the following. If bog, then p^ LE exists if and only if (Appendix A.5) f 1 of of 2 . (22) Hence, existence requires y4b=2. As a function of f, p^ LE enters the simplex through the equilibrium p^ ðMÞ if f ¼ f 1 and leaves it through p^ ðmÞ if f ¼ f 2 , where p^ ðmÞ A ¼ ð1 þ bÞ=2 if f ¼ f 2 . The case b ¼ g is degenerate and needs separate treatment (see below). Since, in general, p^ LE cannot be calculated explicitly, a general linear stability analysis seems impossible. Apparently, if p^ LE exists, then it is unstable if f 41, and asymptotically stable if f o1. It exchanges stability with p^ ðMÞ and with p^ ðmÞ when it passes through them. For various combinations of b and g, Fig. 3 displays the regions of existence and stability of all equilibria as a function of y and f. The boundary between the regions Om and Om_M is given by f ¼ f 1 , and the boundary between Om_M and OM is given by f ¼ f 2 . The critical role of f ¼ 1 is eminent. If f 41, condition (22) is equivalent to W ðbÞoW ð0Þ at p^ ðmÞ and W ðbÞoW ðgÞ at p^ ðMÞ . (23) Thus, the internal equilibrium exists if and only if double heterozygotes have a lower fitness than the corresponding heterozygotes at the equilibria p^ ðmÞ and p^ ðMÞ . Then, p^ LE appears to be unstable and either m or M goes to fixation, depending on the initial gamete distribution. Similarly, if f o1, then double heterozygote advantage at p^ ðmÞ and p^ ðMÞ is necessary for existence of p^ LE . In this case, p^ LE appears to be globally attracting and a modifier gets stuck at an intermediate frequency. 3.2. Special cases 3.2.1. Symmetric optimum, y ¼ 0 If y ¼ 0, all equilibria listed in Table 1, except the internal equilibrium (e), exist for all parameter combinations. The internal equilibrium never exists. The equilibrium p^ ðmÞ is asymptotically stable if and only if f o1. Because we have f 1 ; f 2 o0 in this case, f 41 is sufficient for successful invasion and fixation of the modifier. Thus, if y ¼ 0 and f 41, any modifier increasing dominance invades and rises to fixation. 3.2.2. M is dominant, b ¼ g In this case, the internal equilibrium is given by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1þg gð1 þ 2f Þ  g3 f  2y LE ^ ; p ¼ 1  ¼ p^ LE , pffiffiffiffiffipffiffiffiffiffiffiffiffiffiffiffiffiffi M A 2 gf 1  g2 D^ LE ¼ 0. ð24Þ It exists if and only if 2y  g 2y of o  1, 2 gð2  g Þ g (25) which is impossible if b ¼ g ¼ 1. Thus, y4 2g is a necessary condition for existence. If f 41, then y4g is a necessary condition. If f ¼ 2y=g  1, a simple calculation shows that 2yg ^ LE p^ LE coincides with p^ ðmÞ . If f ¼ gð2g 2 Þ, then p M ¼ 1 and, due to the uniqueness of the fixed point p^ ðMÞ on p3 þ p4 ¼ 1, p^ LE has to (and does) coincide with p^ ðMÞ . If bog, then by (22) f 1 is the lower bound of the interval for f in which the internal equilibrium exists. It is easy to 2yg see that limb!g f 1 o gð2g 2 Þ, hence f 1 increases discontinuously at b ¼ g. Thus, the treatment in Appendix A.5 does not extend to the present case, which is degenerate. In addition, if b ¼ g, p^ LE enters the simplex through 2yg p^ ðMÞ ‘rapidly’ because for f ¼ gð2g we have 2 Þ þ , pffi pffi 2  g ð2g Þ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi þ OðÞ. p^ LE M ¼1 2 ð1g Þð2ygÞ 3.2.3. MM causes complete dominance, g ¼ 1 In this case, the equilibrium (d) has the simple representation sffiffiffiffiffiffiffiffiffiffiffi f y ðMÞ ^ ðMÞ ; p^ ðMÞ p^ 3 ¼ ¼ 1  p^ 3ðMÞ . (26) A ¼ p 4 2f It exists if and only if f 4y. Local stability of p^ ðMÞ can be determined under the simplifying assumption of weak selection (small s). Then, p^ ðMÞ is asymptotically stable if and only if f 41 (Appendix A.4.2). Numerical results suggest this to be true for any s. If, in addition, we assume b ¼ g ¼ 1, a more detailed analysis can be performed. It follows from (24) and (25) that the equilibrium p^ LE does not exist in this case. The eigenvalues of p^ ðMÞ are 1, 1  r, and l1 given by (A.16a). The eigenvalue 1 results from the fact that M is dominant. Therefore, the equilibrium p^ ðMÞ is asymptotically stable if and only if f 41. In fact, it appears to be globally stable in this case. In particular, p^ ðmÞ is unstable if f 41. 3.3. Deleterious modifiers Most mutants have slightly deleterious fitness effects. Here, we study whether and when evolution of dominance can occur if the modifier is not neutral. We assume that the fitness of a two-locus genotype is reduced by t, where tX0 is small, if it carries one M allele, and by 2t if it carries two M alleles. We require 2t þ sð1 þ jyjÞ2 o1, so that all genotypes have positive fitnesses irrespective of their frequency, cf. (1), (5). If t40, except for the monomorphic equilibria, the edge equilibria (a) and (b) from Table 1 no longer exist because the frequency of M decreases along these edges. None of the monomorphic equilibria can be stable if f 41 (Appendix A.1). Clearly, the equilibria p^ ðmÞ (c) and p^ ðMÞ (d), which are of most biological interest, remain unchanged by admitting t40, however, their stability properties change. Also an internal equilibrium, corresponding to (e) may exist if t40. In general, it is not in linkage equilibrium, and it cannot be calculated explicitly. ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 217 In Appendix A.3 it is shown that p^ ðmÞ is unstable, i.e., the modifier M can invade, if is asymptotically stable, i.e., the modifier allele M becomes fixed if sufficiently frequent, if to12 sbðf  1Þðb  WÞð1  W2 Þ þ Oðs2 Þ, 2 ^ ðMÞ to2p^ ðMÞ A ð1  p A Þ½W ðgÞ  W ðbÞ þ Oðs Þ. (27) where W ¼ 2y=ð1 þ f Þ. If f 41, the existence of a positive t requires that b4W, which is equivalent to (14). If W ¼ y ¼ 0, then (27) simplifies to to12 sb2 ðf  1Þ þ Oðs2 Þ. to12sbðf (28) 2 The condition  1Þðb  WÞð1  W Þ, the leading order estimate in (27), is easily shown to be equivalent to ðmÞ to2p^ A ð1  p^ ðmÞ A Þ½W ðbÞ  W ð0Þ. (29) The latter is fulfilled if and only if the mean fitness of an invading population with genotype Mm at the modifier locus is larger than the mean fitness of a resident population with mm. If t ¼ 0, then (29) reduces to (15). For the equilibrium p^ ðMÞ , no complete stability analysis can be performed. A condition analogous to (27) is given in Appendix A.4.4, however, it involves the allele frequency p^ ðMÞ A , which is a complicated solution of a cubic. Equivalently, and in analogy to (29), we obtain that p^ ðMÞ (30) Such positive values of t exist provided (19) holds. Apparently, if (30) is satisfied but not (27), then an unstable internal equilibrium exists. By contrast, if (27) is satisfied but not (30), then a stable internal equilibrium exists and the modifier M can invade but converges to some intermediate frequency. This occurs if t is sufficiently small relative to b but too large relative to g  b. Thus, a (partially) dominant modifier M may get stuck at intermediate frequency. A deleterious modifier can never become fixed if g ¼ b, i.e., if it is completely dominant (this could be proved only for b ¼ g ¼ 1, see Appendix A.4.2). Numerical results support the above (approximate) analytical results and suggest that successful invasion of the modifier M or fixation from high frequency occur if (27) or (30) are satisfied, respectively. Fig. 4, which is analogous to Fig. 3, displays regions of existence and stability of equilibria for a deleterious modifier. Fig. 4. Regions of existence and stability of the equilibria for a deleterious modifier. We have s ¼ 0:1 and t ¼ 0:01, all other parameters, as well as notation, are as in Fig. 3. However, in Omo only p1 ¼ 1 or p2 ¼ 1 can be stable because the edge equilibria do not exist. The reader may note the different scales for f. ARTICLE IN PRESS 218 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 We note that (27)–(30) provide estimates for the strength of selection on a neutral modifier close to the respective equilibria. dominance than mm. A further case, with t40, is also displayed in Fig. 5. 3.5. Recombination and the rate of evolution 3.4. mm causes dominance We briefly discuss an extension of our model. For mathematical simplicity, we assumed so far that in the presence of mm there is no dominance at the primary locus, i.e., a ¼ 0. If aa0, the stability analysis of the equilibrium p^ ðmÞ becomes mathematically equivalent to that of p^ ðMÞ . Therefore, the invasion condition for a deleterious modifier is ðmÞ ^A Þ½W ðbÞ  W ðaÞ þ Oðs2 Þ. to2p^ ðmÞ A ð1  p (31) If t ¼ 0, y ¼ 0, and a40, then the modifier M can invade if either b4a or bobc for some bc o0 (see Fig. 5, lower left panel). Numerical results suggest that bc is always smaller than a. Apparently, this asymmetry is due to the position of the fitness minimum which for aa0 is in general not at 0. Thus, Mm has to induce a higher degree of Although the conditions for invasion and for fixation from high frequency are apparently independent of the recombination rate r (we have a mathematical proof only for the independence of the invasion condition), linkage between primary locus and modifier may have a strong influence on the rate of evolution. If, initially, one of the primary alleles is rare, tightly linked modifiers invade much more rapidly and, usually, become fixed more rapidly (Fig. 6, left panel). The initial steep increase occurs because first the allele A increases in frequency to approximate equilibrium proportions at the primary locus, i.e., close to p^ ðmÞ . Then the modifier starts to increase due to his own, indirect fitness advantage. If invasion of the modifier starts near the equilibrium p^ ðmÞ , i.e., the primary locus is close to equilibrium, then recombination has only a slight influence on the rate of evolution (Fig. 6, right panel). In any case, Fig. 5. Regions of invasion of modifier allele M as a function of y and b. In the top row, there is no dominance ða ¼ 0Þ in the absence of M; in the bottom row, A is slightly dominant ða ¼ 0:1Þ. In the left column, different shadings correspond to regions in which a modifier M with various deleterious effects t can invade. In the white region, no modifier can invade (even if neutral); light gray to dark gray shadings indicate regions in which modifiers with t ¼ 0, 0.004, 0.015, 0.03, 0.05 can invade. The other parameters are: s ¼ 0:1, r ¼ 0:5 and f ¼ 2. In the right column, different shadings correspond to regions in which a modifier with deleterious effect t ¼ 0:01 can invade if f assumes the values (from dark gray to light gray) 1.25, 1.5, 2.5, and 6. If f ¼ 6 is replaced by a larger values, the white region shrinks and, in the limit f ! 1, disappears. The other parameters are: s ¼ 0:1 and r ¼ 0:5. ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 219 Fig. 6. Rate of modifier evolution. The frequency of an initially rare modifier allele M is shown for s ¼ 0:1, t ¼ 0, and a ¼ 0. Black curves are for r ¼ 0, gray curves for r ¼ 0:5. In the left panel we have b ¼ 0:1, g ¼ 0:2, and trajectories start close to the equilibrium p1 ¼ 1, i.e., at ð0:99; 0:003; 0:003; 0:004Þ; solid lines are for f ¼ 5, y ¼ 0, dotted lines for f ¼ 2:5, y ¼ 0, and dashed lines for f ¼ 5, y ¼ 0:3. In the right panel we have b ¼ 0:5, g ¼ 0:6, and trajectories start close to the equilibrium p^ ðmÞ ; solid lines are for f ¼ 2:5, y ¼ 0, dotted lines for f ¼ 1:5, y ¼ 0, and dashed lines for f ¼ 2:5, y ¼ 0:6. stronger frequency dependence (larger f) or a stronger dominance effect (larger b, gÞ accelerate the sweep. 3.6. Population regulation If population regulation is taken into account, fitness is given by W ðgÞ ¼ F ðNÞ½1  sðg  yÞ2 þ sZððg  gÞ2 Þ þ V , (32) 0 where Z ¼ ZðNÞ ¼ fNF ðNÞ=F ðNÞ and F : ½0; 1Þ ! ½0; 1Þ is a strictly decreasing, differentiable function of N such that F ðNÞ ¼ 1 has a unique positive solution K, the carrying capacity. For more information, consult Bürger (2005) and Thieme (2003, Chapter 9). The genetic dynamics (9) has to be complimented by the demographic dynamics N 0 ¼ W N. (33) The inclusion of population regulation makes a detailed mathematical analysis prohibitively complicated. Comprehensive numerical iterations show that, in general, con^ occurs much vergence to demographic equilibrium, N, faster than convergence to genetic equilibrium. Concomitantly, during this first phase, ZðNÞ converges to Z^ . Then, with this value nearly fixed, convergence to genetic equilibrium occurs as studied above. Since the value of Z^ depends on the equilibrium reached, in principle, equilibrium configurations could occur which do not exist in the simplified model (cf. Bürger, 2005). However, we did not find such instances. Eq. (2.10) in Bürger (2005) shows that the population size in demographic equilibrium can be approximated by ! ^ þ ðḡ^  yÞ2 V N^ ¼ K þ sK 2f V^ þ þ Oðs2 C 2 Þ, KF 0 ðKÞ where C ¼ maxððḡ^  yÞ2 ; V^ Þ, a ^ signifies equilibrium values, and s is small. From this, an approximation for Z^ can be obtained (Bürger, 2005, Eq. (C.4)). For discrete logistic growth, one obtains to leading order in s (Bürger, 2005, Appendix C): N^  kðr  1Þ þ ksð2f V^ ðr  1Þ  V^  ðḡ^  yÞ2 Þ, Z^  f ðr  1Þ  sf rðV^ þ ðḡ^  yÞ2 Þ. With f ¼ Z^ , the local stability conditions derived above apply. Thus, the main features of the full model appear to be captured by the simplified model with constant population size. 4. Discussion Evolution of dominance under frequency-dependent selection is an old hat. Studies have been initiated already half a century ago in the context of evolution of mimicry (e.g., Sheppard, 1958; Clarke, 1964; O’Donald, 1968). In these models it is assumed that a mimic is best protected from predators if it has an intermediate, ‘focal’ frequency. In some of these models, rapid evolution of a modifier has been demonstrated by numerical examples, and selective advantages have been estimated to be considerable. This is in contrast to Fisher’s classical model (Mayo and Bürger, 1997). In this work, we are concerned with a different scenario, namely with the evolution of dominance if heterozygotes are underdominant and maintained by frequency-dependent disruptive selection. Compared with the evolution of assortative mating and speciation, evolution of dominance may be a genetically simple and evolutionary fast alternative to eliminate disadvantageous heterozygotes. It is known that evolution of dominance can occur on short time scales, sometimes considerably less than 100 generations. The most prominent case occurred during the spread of industrial melanism in the moth Biston betularia (e.g., Kettlewell, 1965). We start with a brief summary of our results and restrict attention to parameter values f 41, so that intraspecific competition is strong enough to induce disruptive selection. First, we deal with a neutral modifier, i.e., we assume t ¼ 0. In contrast to Section 3, here we consider the ecological ARTICLE IN PRESS 220 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 parameters s, f, and y as given (with 0pyo1Þ and ask for which values of b and/or g the modifier M can invade successfully and/or becomes fixed. Because heterozygotes can be modified in both directions, we admit values b and g such that either 0obpgp1 or 1pgpbo0. In the absence of the modifier allele M and if a ¼ 0 in (7), there is no dominance at the primary locus and a polymorphism, p^ ðmÞ , is maintained whenever f 41. If y ¼ 0, modifiers inducing any level of dominance will invade and become fixed. If ya0, this is not so because the minimum fitness is assumed at g ¼ y=ð1 þ f Þ, which differs from the genotypic value 0 of the Aa heterozygote. For ya0, negative b or sufficiently large b40 is necessary for successful invasion (see Fig. 2). Importantly, invasion of a neutral modifier also guarantees its eventual fixation. Indeed, condition (14) or, equivalently, (15) implies global fixation, i.e., from any initial condition. However, fixation of an already frequent modifier can occur in a wider parameter range than invasion, except when b ¼ g ¼ 1 (then invasion and fixation occur whenever f 41Þ. Fixation from sufficiently high frequency occurs if (19) or, equivalently, (21) holds. If (22) is valid, an internal unstable equilibrium exists and the modifier becomes fixed only if it is initially sufficiently frequent, a scenario which is biologically not unrealistic and may have occurred in some cases (Wagner and Bürger, 1985). Therefore, an invasion analysis is insufficient to predict the evolution of dominance. Because a large fraction of mutations is known to be slightly deleterious, we also investigated modifiers with a direct deleterious effect on fitness, independent of their modifying effect. Such modifiers can still invade if they induce sufficiently strong dominance, i.e., if (29) holds to leading order in s; for graphical representations, see Figs. 4 and 5. The condition for fixation of a modifier that is already sufficiently frequent is given by (30). If the modifier is deleterious, then an internal, stable or unstable, equilibrium may exist. If the deleterious effect t is sufficiently small so that M can invade, then M will converge to some intermediate frequency if g  b is too small to outweigh the deleterious effect 2t of the MM homozygous genotypes. In contrast, if b is relatively small and g  b is large, then invasion may be impossible but fixation of a frequent modifier can occur (see Fig. 4). Under the assumption of an infinitesimally small dominance effect b, deleterious modifiers can never invade. Also for an asymmetric optimum, modifiers of very small effect can invade only if they push the heterozygous effect b in the direction opposite to y. In only a few models of modification of dominance can modifiers with a frequency-independent deleterious effect invade and become fixed. One, for the evolution of mimicry, goes back to Sheppard (1958). It assumes that if a new mutant arises, heterozygotes have a fitness advantage over both homozygotes, and modifiers may be selected that increase the fitness of one of the homozygotes. It was proved that such a neutral modifier always invades and becomes fixed (Bürger, 1983). Thus, a bistable situation does not occur. In addition, also modifiers with a slightly deleterious fitness effect can invade and become fixed. The present model provides another example. As is shown explicitly by (27) and (28), condition (29) is frequency independent given the model parameters, although p^ ðmÞ A enters. The same applies to (30) and (31). We note that these conditions are similar in structure to those given by Otto and Bourguet (1999); e.g., their Eq. (8). Thus, their model constitutes a third example. In Otto and Bourguet’s model with soft selection, a polymorphism with stable underdominance is maintained. For small migration rates, the authors provided estimates for the (indirect) selection coefficient of an invading dominance modifier. For high recombination rates, these estimates are similar in structure to our condition (31) on the deleterious effect t which, along with (27)–(30), provides an estimate of the selection intensity on the modifier near the respective equilibrium. Interestingly, in Otto and Bourget’s model the selection coefficients depend strongly on the recombination rate. Thus, their invasion conditions depend on the linkage relation, whereas ours do not. The reason seems to be that one allele is favored in each deme and migration is weak, thus tight linkage facilitates invasion and quick fixation of the modifier. Although, Otto and Bourguet do not mention it, it can be shown that in their model, in which modifiers have no direct fitness effects, an internal equilibrium can exist, which may be stable or unstable; see Peischl (2006). Van Dooren’s (1999) model and results are difficult to compare with ours because his parameterization is very different and he studies the successive invasion of mutants, as is usual in adaptive dynamics. Still, his general conclusion, mainly based on numerical examples, that dominance relations evolve to a certain extent if a stable polymorphism is maintained, is qualitatively similar to ours. In addition, he found that for some parameter combinations alternative ESS with either no dominance or complete dominance coexist, a situation that may be related to our finding of an unstable internal equilibrium. Following Schneider (2007), a population-genetic study of long-term evolution, i.e., successive invasions of modifier alleles, could be performed for our model, because a, b, and g are parameters. In contrast to adaptivedynamics methodology, in population genetics we need not assume that the invading type differs only very slightly from the resident type. In fact, this seems to be an important aspect because in many empirical examples dominance relations have changed dramatically within a short time span, thus ‘modifiers’ seem to have large effects. In addition, in our model modifiers of arbitrarily small effect can invade only if they are neutral (or have a direct beneficial effect on fitness) and if they modify dominance in one direction (opposite to yÞ. Our population-genetic analysis reveals that modifiers can invade under much more general assumptions than a simple invasion analysis ARTICLE IN PRESS 221 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 considering mutations of infinitesimally small effects would suggest. Moreover, the conditions for invasion and fixation are not equivalent. If y ¼ 0, t ¼ 0 and a ¼ 0, then any modifier invades and becomes fixed that induces some degree of dominance (i.e., ba0Þ. If, for instance, a40, modifiers with positive b can invade if b4a. For negative b, it appears that bo  a is a necessary condition for successful invasion of a modifier. Thus, only modifiers inducing a higher degree of dominance can invade and we expect that in the long run, complete dominance of one of the alleles evolves. If yX0, t ¼ 0, and b ¼ g ¼ 1, then p^ ðMÞ is globally asymptotically stable and we expect that this is the only stable long-term equilibrium. For deleterious modifiers the situation becomes more complicated, especially as it cannot be expected that every new modifier allele has the same fitness effect. Evolution of dominance is one of several mechanisms by which inheritance systems and genetic architectures can evolve (Bagheri, 2006). Four recent theoretical studies have shown that the genetic architecture readily evolves under frequency-dependent disruptive selection. Matessi and Gimelfarb (2006) studied long-term evolution at a single multiallelic locus under frequency-dependent selection of essentially the same form as ours. In their model, homozygous and heterozygous effects evolve due to a sequence of new mutations, typically in such a way that at a long-term equilibrium (i.e., at an equilibrium that cannot be invaded by further mutations) only the two most extreme of all possible phenotypes are realized. Their genetic model is highly flexible, so that different alleles can code for the same phenotype and thus coexist, and for each individual the phenotype associated with a given genotype may be drawn from a distribution. van Doorn and Dieckmann (2006) investigated numerically a multilocus soft-selection Levene model, in which Gaussian stabilizing selection acts on a quantitative trait in each of two patches. If the fitness optima differ sufficiently much between patches, after evolutionary branching, one of the loci evolves larger and larger effects, whereas the others evolve increasingly small effects. After a sufficiently long time, one homozygote and the heterozygote become well adapted. Then, evolution of dominance starts and the phenotype of second homozygote evolves toward that of the heterozygote. Their study assumes additive genetics, i.e., no epistasis or dominance. For an ecological scenario as in the present paper, Kopp and Hermisson (2006) showed that in multilocus models with epistasis but no dominance, typically highly asymmetric genetic architectures evolve, in the sense that one or a few loci have large effects, whereas the other loci have very small effects. Finally, Schneider (2007) performed a complete analysis of longterm evolution of the same ecological model as used here and by Kopp and Hermisson. His genetic model is a multilocus–multiallele model with no dominance and epistasis, and he assumes linkage equilibrium. He classifies all possible long-term equilibria and finds that a highly asymmetric genetic architecture, as in van Doorn and Dieckmann (2006) and Kopp and Hermisson (2006), evolves only if the range of possible effects at some locus is sufficiently large to span the full range of ecologically favored phenotypes. The above findings provide further support for the suggestion of Wilson and Turelli (1986) that stable underdominance could mediate rapid evolution of dominance, although an analytical proof within their model seems impracticable. These findings show that evolution of dominance or, more generally, evolution of genetic architecture is a potent mechanism to remove unfit heterozygotes which naturally occur under disruptive selection. Whether and when evolution of dominance is indeed more efficient or rapid than evolution of assortative mating remains to be studied. Acknowledgment We thank J. Hermisson, K. Schneider, and T. van Dooren for useful comments on the manuscript. This work was supported by grant P16474-N04 of the Austrian Science Foundation (FWF) and by a grant of the Vienna Science and Technology Fund (WWTF). The Program for Evolutionary Dynamics is supported by Jeffrey Epstein. Appendix A. Existence and stability of equilibria We do not give complete algebraic proofs here, which would be very lengthy. However, all the formulas given can be easily checked with Mathematica or similar software. Unless otherwise stated, we admit aa0 and tX0. A.1. Monomorphic equilibria The eigenvalues of the four monomorphic equilibria are readily determined and are quite simple, even if initially there is some degree of dominance, i.e., aa0, and the modifier is deleterious, i.e., t40. Clearly, the two edges of equilibria do not exist if the modifier is deleterious. The eigenvalues of p1 ¼ 1 are l1 ¼ ð1  rÞf1  t þ s½ð1  bÞ2 f  ðb  yÞ2 g , 1  sð1 þ yÞ2 l2 ¼ 1  l3 ¼ t , 1  sð1 þ yÞ2 1 þ s½ð1 þ aÞ2 f  ða  yÞ2  . 1  sð1 þ yÞ2 (A.1a) (A.1b) (A.1c) Obviously, all eigenvalues are positive and l2 o1. Further, 2 1sð1þyÞ we have l1 o1 if and only if r4rc1 ¼ 1  1tþs½ð1bÞ . 2 f ðbyÞ2  ARTICLE IN PRESS 222 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 It can easily be seen that rc1 o0 if and only if f of r1 l3 ¼ t b  1  2y . þ ¼ 2 1þb sð1 þ bÞ (A.2) The eigenvalue l3 is smaller than 1 if and only if f of c1 ¼ a  1  2y . 1þa (A.3) Since f c1 is monotonically increasing in a, we have f r1 4f c1 . Therefore, p1 ¼ 1 is asymptotically stable if and only if f of c1 ; in particular, stability does not depend on r. The eigenvalues of p2 ¼ 1 are l1 ¼ ð1  rÞf1  t  s½ð1  bÞ2 f  ðb  yÞ2 g , 1  sð1  yÞ2 l2 ¼ 1  t , 1  sð1  yÞ2 (A.4a) (A.4c) 2 1sð1yÞ have l1 o1 if and only if r4rc2 ¼ 1  1ts½ðbyÞ . 2 ð1bÞ2 f  We have l1 o1 for every rX0 if and only if t 2y  1  b . þ 1b sð1  bÞ2 (A.5) In addition, l3 o1 if and only if 2y  1  a . 1a (A.6) to 2sðbaÞð1bÞð1yÞ . 1a It follows that f r2 of c2 if and only if Thus, for sufficiently small r and t, it is possible that l1 414l3 . Hence, stability may depend on r. A necessary condition for stability is f of c2 . The eigenvalues of p3 ¼ 1 are l1 ¼ ð1  rÞf1 þ t þ s½ð1 þ bÞ2 f  ðb  yÞ2 g , 1  2t  sð1 þ yÞ2 (A.7a) t , 1  2t  sð1 þ yÞ2 (A.7b) 1  2t  s½ðg  yÞ2  f ð1 þ gÞ2  , l3 ¼ 1  2t  sð1 þ yÞ2 (A.7c) l2 ¼ 1 þ and those of p4 ¼ 1 are l1 ¼ ð1  rÞf1  t  s½ð1 þ bÞ2 f  ðb  yÞ2 g , 1  2t  sð1  yÞ2 l2 ¼ 1 þ t , 1  2t  sð1  yÞ2 If t40, then l2 41 holds in both cases because fitnesses of all genotypes have to be nonnegative, i.e., 2t þ sð1 þ jyjÞ2 p1 must hold. Thus, neither p3 ¼ 1 nor p4 ¼ 1 can be locally stable. If t ¼ 0, one eigenvalue is always unity because of the two edges of equilibria. In this case, it can be shown that no monomorphic equilibrium can be stable if   a  1  2y 2y  1  a g  1  2y 2y  1  g ; ; ; f 4max . 1þa 1a 1þg 1g (A.9) As a consequence of our constraints on the parameters, monomorphic equilibria can never be stable if f 41, i.e., if there is disruptive selection. A.2. Edge equilibria Again, all eigenvalues are positive and l2 o1. Here, we f of c2 ¼ (A.8c) (A.4b) 1 þ s½ða  yÞ2  ð1 þ aÞ2 f  . l3 ¼ 1  sð1  yÞ2 f of r2 ¼ 1  2t  s½ðg  yÞ2  f ð1  gÞ2  . 1  2t  sð1  yÞ2 (A.8a) (A.8b) Here, we assume t ¼ 0 because otherwise no equilibria exist on the edges p1 þ p3 ¼ 1 and p2 þ p4 ¼ 1 except the monomorphic ones. Because the edges p1 þ p3 ¼ 1 and p2 þ p4 ¼ 1 consist of equilibria, one eigenvalue is always 1. The other two are the solutions of complicated quadratic polynomials. Under the assumption of weak selection, however, relatively simple expressions are obtained that allow simple conclusions. To leading order in s, the two nontrivial eigenvalues on p1 þ p3 ¼ 1 are given by 1  r þ OðsÞ and 1 þ sf2ð1 þ yÞ½1 þ að1  pM Þ2 þ 2bpM ð1  pM Þ þ gp2M  þ ðf  1Þ½1 þ aða þ 2Þð1  pM Þ2 þ 2bðb þ 2ÞpM ð1  pM Þ þ gðg þ 2Þp2M g þ Oðs2 Þ. The latter clearly is 41 whenever f 41. Hence, none of the equilibria can be stable. To leading order in s, the two nontrivial eigenvalues on p2 þ p4 ¼ 1 are given by 1  r þ OðsÞ and 1 þ sf2ð1  yÞ½1  að1  pM Þ2  2bpM ð1  pM Þ  gp2M  þ ðf  1Þ½1 þ aða  2Þð1  pM Þ2 þ 2bðb  2ÞpM ð1  pM Þ þ gðg  2Þp2M g þ Oðs2 Þ. Here, the latter is 41 whenever f 41 because 1 þ aða  2Þ X0 for a 2 ½1; 1. Hence, again, none of the equilibria can be stable if f 41. Numerical iterations suggest that if both monomorphic equilibria are stable/unstable, the same holds for all equilibria in between. If one is stable and the other is unstable, then stability changes exactly once on this edge. Examples are given and proved in Peischl (2006). ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 A.3. The equilibrium p^ ðmÞ A.3.1. The case a ¼ 0 This case is much simpler than aa0 because the equilibrium can be calculated explicitly if a ¼ 0. It is given by (10). The eigenvalue determining stability within the edge p1 þ p2 ¼ 1 is 2 l1 ¼ 1  223 If t ¼ 0 and under the assumption of weak selection (small s), asymptotic stability of p^ ðmÞ changes as f passes through f 2 ðaÞ, where f 2 ðaÞ ¼ ð1 þ abÞ2 ð2y  a  bÞ , að2  a2  b2 Þ þ bð1  b2 a2 Þ (A.12) cf. Sections A.4.3 and A.4.4 for more details. 2 sð1 þ f Þ½ð1 þ f Þ  4y  . 2ð1 þ f Þ2 þ s½ð1 þ f Þ2 ð2f  1Þ þ 2yð1  2f  f 2 Þ (A.10a) We have l1 o1 if and only if f 42jyj  1, i.e., whenever this equilibrium exists. The other two eigenvalues are the solutions of a complicated quadratic equation. To leading order in s, they are given by (A.13) o1 ¼ 2½1 þ f þ g2 ð3f  1Þ þ 2gð3f þ yÞ, ðA:10bÞ 2 þ 14st½ð2f  1Þð2  W2 Þ þ f 2 W2  þ Oðs2 Þ, hðxÞ:¼o0 þ o1 x þ o2 x2 þ o3 x3 ¼ 0, o0 ¼ ð1 þ gÞð1 þ f þ gðf  1Þ þ 2yÞ, þ rð1  2b2 þ 2bW  W2 Þ l3 ¼ 1  t þ 12sbðf  1Þðb  WÞð1  W Þ A.4.1. Existence On the edge p3 þ p4 ¼ 1, where M is fixed, an internal equilibrium is a solution p3 ¼ x 2 ð0; 1Þ of where l2 ¼ ð1  rÞð1  tÞ þ 12sðf  1Þ½bðb  WÞð1 þ W2 Þ þ 14stð1  rÞ½ð2f  1Þð2  W2 Þ þ f 2 W2  þ Oðs2 Þ, A.4. The equilibrium p^ ðMÞ o2 ¼ 12gf ð1 þ gÞ, o3 ¼ 8g2 f . We have ðA:10cÞ where W ¼ 2y=ð1 þ f Þ. Unless r is small relative to s, it follows that l3 determines stability, and (27) is obtained immediately. If t ¼ 0, then conditions (13a) are obtained. For general s, the characteristic polynomial evaluated at x ¼ 1 is given by s2 bðf  1Þ½ð1 þ f Þ2  4y2 ½bð1 þ f Þ  2yA p1 ¼ , ½ð1 þ f Þ2 ð2 þ sð2f  1Þ  2sy2 Þ þ 4sy2 3 where A ¼ sbðf 2  1Þ½bð1 þ f Þ  2y  rf2  2sðb  yÞ2 þ f ð4 þ s  2sb2 Þ þ 2f 2 ½1 þ sð1 þ b2  2by  y2 Þ þ f 3 sð1 þ 2b2 Þg. Because p^ ðmÞ exists only if ð1 þ f Þ2 44y2 , the stability properties of p^ ðmÞ can change only if f ¼ 1 or f ¼ f 2 or A ¼ 0. However, we have Ao0 whenever ðf  1Þ½bð1 þ f Þ  2yo0, which is satisfied if and only if one of conditions (13a) is fulfilled. Therefore, stability is independent of r and always given by (13a). A.3.2. The case aa0 The analysis of this case is mathematically equivalent to the analysis of the equilibrium p^ ðMÞ , but with g and a exchanged (see below). Therefore (16), p^ ðmÞ exists if and only if   2y  1  a a  1  2y ; . (A.11) f 4max 0; 1a 1þa Each of the following conditions is sufficient for existence: f 41, jyjo12, or 2y  1oao2y þ 1. hð0Þ ¼ o0 o0 if and only if f4 g  1  2y . 1þg (A.14) In addition, we get hð1Þ ¼ ð1  gÞð1 þ g þ f ð1  gÞ  2yÞ40 if and only if f4 2y  1  g . 1g (A.15) Thus, if f 40, then hð0Þ40 and hð1Þo0 cannot hold. If h has no or one critical point, then it is strictly monotonically increasing because it is a cubic and o3 40. Hence, a unique equilibrium exists if and only if hð0Þo0 and hð1Þ40. Otherwise, h has two critical points, y1 oy2 , with h0 ðy1 Þ ¼ h0 ðy2 Þ ¼ 0. Because h00 ðxÞ ¼ 24gf ð1 þ g  2gxÞ, we have h00 ðy1 Þo0, h00 ðy2 Þ40 and y2 41. Hence, y1 is a local maximum and y2 is a local minimum. Consequently, if hð0Þo0 and hð1Þ40, then there exists a unique solution of hðxÞ ¼ 0 in ð0; 1Þ. If hð0Þo0 and hð1Þo0, the following argument shows that no solution exists in ð0; 1Þ. Because hð0Þohð1Þ, we have hð0Þohð1Þo0 if and only if (A.15) is violated, i.e., if and only if b1 ¼ 12½f ð1  gÞ þ 1 þ goy. This requires f o1 and y40. In addition, a simple rearrangement shows that pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi! g2 þ 2f  1  2gy 1 1 pffiffiffiffiffi y1 ¼ 1þ  41 2 g 3f g ARTICLE IN PRESS 224 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 (assuming, of course, g2 þ 2f  1  2gyX0Þ is equivalent to b2 ¼ g2  1  f ð1  6g þ 3g2 Þ oy. 2g p^ ðMÞ A ðÞ ¼ Another simple calculation shows that b2 ob1 if f o1, thus hð0Þohð1Þo0 implies y1 41, and therefore no solution in ð0; 1Þ can exist. In the third and last possible case, hð0Þ40 and hð1Þ40, no solution in ð0; 1Þ exists because y2 41. Summarizing, a unique equilibrium exists if and only if hð0Þo0 and hð1Þ40, i.e., if both (A.14) and (A.15) hold. Simple calculations show that each of the following conditions is sufficient for existence: f 41, jyjo12, or g þ 142y4g  1. A.4.2. Stability if g ¼ 1 If g ¼ 1, we obtain hðxÞ ¼ 4ð1  xÞðf þ y  4fxþ 2fx2 Þ, and p^ ðMÞ is given by (26). Then, the eigenvalues can be calculated explicitly. The eigenvalue determining stability within the edge p3 þ p4 ¼ 1 is pffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffi 4sðf  yÞ3=2 ð 2f  f  yÞ . (A.16a) l1 ¼ 1  f ½1 þ sð2f  1  y2 Þ Recall that existence of p^ ðMÞ requires f 4y. It is readily shown that l1 40 and l1 o1 if and only if f 4jyj. The other two eigenvalues are the solutions of a complicated quadratic equation. To leading order in s and t, they are l2 ¼ ð1  rÞð1 þ tÞ þ OðsÞ þ Oðt2 Þ, (A.16b) pffiffiffiffiffiffiffiffiffiffiffi l3 ¼ 1 þ t  sð1  f 1 Þð1  b2 Þ f  y pffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffi ð 2f  f  yÞ þ Oðs2 Þ þ OðstÞ þ Oðt2 Þ. ðA:16cÞ If t is sufficiently much smaller than r, then stability is determined by l3 . Therefore, p^ ðMÞ is asymptotically stable if, approximately, tosð1  f 1 Þð1  b2 Þ pffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffi f  yð 2f  f  yÞ. (A.17) Because yp1, the right-hand side is positive if f 41. It increases nearly linearly in f. Therefore, p^ ðMÞ is locally asymptotically stable if t is sufficiently small. If t ¼ 0, then p^ ðMÞ is asymptotically stable whenever f 41. If b ¼ g ¼ 1, then the eigenvalues can be calculated explicitly, and we obtain l3 ¼ 1 þ t 1  2t  sð1  2f þ y2 Þ is given by through p^ ðMÞ . If f ¼ f 1 þ , p^ ðMÞ A (A.18) and l2 ¼ ð1  rÞl3 . Thus, M cannot become fixed if t40. ð1 þ bÞð1 þ gÞ þ a, 2ð1 þ bgÞ (A.19) where a is a complicated expression. The eigenvalue determining stability within the edge p3 þ p4 ¼ 1 is given by l1 ¼ 1  s ð1  b2 Þð1  g2 Þa1 2ð1 þ bgÞ2 ½gð2  g2  b2 Þ þ bð1  b2 g2 Þ þ Oðs2 Þ þ OðsÞ þ Oð2 Þ, where a1 ¼ g½1 þ 3g2  2g4 þ 3bgð1  g2 Þ þ 6bg þ b2 ð3 þ bg þ 3g2 þ bg3 Þ þ 2yð1  4g2 þ 2g4  3bg  3b2 g2  b3 g3 Þ. We have l1 o1 whenever f 1 41 because this implies yXb þ gX0. Since, a1 40 if y ¼ 0 and if y ¼ 1, it follows that a1 40 for every y 2 ½0; 1. The value of the characteristic polynomial for the other two eigenvalues at l ¼ 1 can be shown to be rsðg  bÞð1  b2 Þð1  g2 Þ  ½gð2  g2  b2 Þ þ bð1  b2 g2 Þ2 þ 4að1  g2 Þð1 þ bgÞ3 ð2y  b  gÞ 2ð1 þ bgÞ4 ½gð2  g2  b2 Þ þ bð1  b2 g2 Þ þOðs2 Þ þ Oð2 Þ. Therefore, the equilibrium changes stability at  ¼ 0 as  increases from below 0 to above 0, i.e., as p^ LE enters the simplex. In fact, numerical evaluation of the nominator shows that it is always positive (although a is negative). Because the derivative of the characteristic polynomial at 1 is r þ OðsÞ, it follows that p^ ðMÞ becomes stable as p^ LE moves through, i.e., if 40. A.4.4. Deleterious modifier M By comparing the mean fitness of a population which is in equilibrium with respect to the first locus and is homozygous for M with an analogous population which heterozygous at the modifier locus, we find that the condition for fixation of M should be given by (30). If fitnesses are calculated explicitly, this becomes ^ ðMÞ to2p^ ðMÞ A ð1  p A Þsðg  bÞ½ðb þ gÞðf  1Þ þ 2ðy þ f Þ ^ ðMÞ  4f p^ ðMÞ A ð1 þ gð1  p A ÞÞ, ðA:20Þ ðMÞ where p^ A is given as the unique solution of the cubic (A.13). The same stability result is obtained by calculating the (otherwise very complicated) eigenvalues to first order in s and in t (also omitting terms of order st). A.5. The equilibrium p^ LE A.4.3. Stability near f ¼ f 1 Here, we assume t ¼ 0. As f increases from below f 1 (17) to above f 1 , the equilibrium p^ LE traverses into the simplex Here, we assume a ¼ 0 and t ¼ 0. With Mathematica it is straightforward to check that if f ¼ f 2   ð40Þ and bo1, ARTICLE IN PRESS S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 then the following internal equilibrium exists: p^ LE A ¼ 1þb ðg  bÞgb2 þ þ Oð2 Þ, 2 2 2 4½ygðg  bÞ þ b ð1  b Þð2y  bÞ (A.21a) p^ LE M ¼ b3 þ Oð2 Þ, 2½ygðg  bÞ þ b2 ð1  b2 Þð2y  bÞ LE D^ ¼ Oð2 Þ. (A.21b) (A.21c) A similar analysis can be performed for f ¼ f 1 þ . Then, the internal equilibrium is given by p^ LE A ¼ ð1 þ bÞð1 þ gÞ þ a1 þ Oð2 Þ, 2ð1 þ bgÞ (A.22a) 2 p^ LE M ¼ 1  ðg  bÞa2 þ Oð Þ, (A.22b) LE D^ ¼ Oð2 Þ, (A.22c) where a1 and a2 are complicated, positive expressions in b, g and y. Eqs. (A.22) do not give a valid equilibrium in the case b ¼ g, which turns out to be degenerate, see Section 3.2.2. References Asmussen, M.A., Basnayake, E., 1990. Frequency-dependent selection: the high potential for permanent genetic variation in the diallelic, pairwise interaction model. Genetics 125, 215–230. Bagheri, H.C., 2006. Unresolved boundaries of evolutionary theory and the question of how inheritance systems evolve: 75 years of debate on the evolution of dominance. J. Exp. Zool. 36B, 1–31. Bolnick, D.I., 2004. Waiting for sympatric speciation. Evolution 58, 895–899. Bolnick, D.I., Doebeli, M., 2003. Sexual dimorphism and adaptive speciation: two sides of the same ecological coin. Evolution 57, 2433–2449. Bulmer, M.G., 1974. Density-dependent selection and character displacement. Am. Nat. 108, 45–58. Bürger, R., 1983. Nonlinear analysis of some models for the evolution of dominance. J. Math. Biol. 16, 269–280. Bürger, R., 2002a. Additive genetic variation under intraspecific competition and stabilizing selection: a two-locus study. Theor. Popul. Biol. 61, 197–213. Bürger, R., 2002b. On a genetic model of intraspecific competition and stabilizing selection. Am. Nat. 160, 661–682. Bürger, R., 2005. A multilocus analysis of intraspecific competition and stabilizing selection on a quantitative trait. J. Math. Biol. 50, 355–396. Bürger, R., Bagheri, H.C., 2008. Dominance and its evolution. In: Encyclopedia of Ecology. Elsevier, Amsterdam to appear. Bürger, R., Gimelfarb, A., 2004. The effects of intraspecific competition and stabilizing selection on a polygenic. Genetics 167, 1425–1443. Bürger, R., Schneider, K.A., 2006. Intraspecific competitive divergence and convergence under assortative mating. Am. Nat. 167, 190–205. Bürger, R., Schneider, K.A., Willensdorfer, M., 2006. The conditions for speciation through intraspecific competition. Evolution 60, 2185–2206. Christiansen, F.B., Loeschcke, V., 1980. Evolution and intraspecific exploitative competition. I. One-locus theory for small additive gene effects. Theor. Popul. Biol. 18, 297–313. Clarke, B., 1964. Frequency-dependent selection for the dominance of rare polymorphic genes. Evolution 18, 364–369. 225 Clarke, B.C., 1979. The evolution of genetic diversity. Proc. R. Soc. London B 205, 453–474. Clarke, C.A., Sheppard, P.M., 1960. The evolution of dominance under disruptive selection. Heredity 14, 73–87. Cockerham, C.C., Burrows, P.M., Young, S.S., Prout, T., 1972. Frequency-dependent selection in randomly mating populations. Am. Nat. 106, 493–515. Dieckmann, U., Doebeli, M., 1999. On the origin of species by sympatric speciation. Nature 400, 354–357. Dieckmann, U., Doebeli, M., Metz, J.A.J., Tautz, D., 2004. Adaptive Speciation. Cambridge University Press, Cambridge, UK. Doebeli, M., 1996. A quantitative genetic competition model for sympatric speciation. J. Evol. Biol. 9, 893–909. Drossel, B., McKane, A., 2000. Competitive speciation in quantitative genetics models. J. Theor. Biol. 204, 467–478. Fisher, R.A., 1931. The evolution of dominance. Biol. Rev. Cambridge Philos. Soc. 6, 345–368. Fitzpatrick, M.J., Feder, E., Rowe, L., Sokolowski, M.B., 2007. Maintaining a behavior polymorphism by frequency-dependent selection on a single gene. Nature 447, 210–213. Gavrilets, S., 2003. Models of speciation: what have we learned in 40 years? Evolution 57, 2197–2215. Gavrilets, S., 2004. Fitness Landscapes and the Origin of Species. Princeton University Press, Princeton, NJ. Gavrilets, S., Hastings, A., 1995. Intermittency and transient chaos from simple frequency-dependent selection. Proc. R. Soc. London B 261, 233–238. Gourbiere, S., 2004. How do natural and sexual selection contribute to sympatric speciation? J. Evol. Biol. 17, 1297–1309. Haldane, J.B.S., 1956. The theory of selection for melanism in Lepidoptera. Proc. R. Soc. London B 145, 303–306. Hori, M., 1993. Frequency-dependent natural selection in the handedness of scale-eating fish. Science 260, 216–219. Kettlewell, H.B.D., 1965. Insect survival and selection for pattern. Science 148, 1290–1296. Kirkpatrick, M., Nuismer, S.L., 2004. Sexual selection can constrain sympatric speciation. Proc. R. Soc. London B 271, 687–693. Kirkpatrick, M., Ravigné, V., 2002. Speciation by natural and sexual selection: models and experiments. Am. Nat. 159, S22–S35. Kopp, M., Hermisson, J., 2006. The evolution of genetic architecture under frequency-dependent disruptive selection. Evolution 60, 1537–1550. Loeschcke, V., Christiansen, F.B., 1984. Evolution and intraspecific exploitative competition. II. A two-locus model for additive gene effects. Theor. Popul. Biol. 26, 228–264. Matessi, C., Gimelfarb, A., 2006. Discrete polymorphism due to disruptive selection on a continuous trait—I: the one-locus case. Theor. Popul. Biol. 69, 283–295. Matessi, C., Gimelfarb, A., Gavrilets, S., 2001. Long-term buildup of reproductive isolation promoted by disruptive selection: how far does it go? Selection 2, 41–64. Mayo, O., Bürger, R., 1997. The evolution of dominance: a theory whose time has passed. Biol. Rev. 72, 97–110. Maynard Smith, J., 1966. Sympatric speciation. Am. Nat. 100, 637–650. O’Donald, P., 1968. Models of the evolution of dominance. Proc. R. Soc. B 171, 127–143. Otto, S.P., Bourguet, D., 1999. Balanced polymorphism and the evolution of dominance. Am. Nat. 153, 561–574. Peischl, S., 2006. Evolution of dominance modifiers under frequencydependent selection. Diploma thesis, University of Vienna. Polechová, J., Barton, N.H., 2005. Speciation through competition: a critical review. Evolution 59, 1194–1210. Rueffler, C., Vandooren, T.J.M., Leimar, O., Abrams, P.A., 2006. Disruptive selection and then what. Trends Ecol. Evol. 21, 238–245. Schneider, K.A., 2006. A multilocus-multiallele analysis of frequencydependent selection induced by intraspecific competition. J. Math. Biol. 52, 483–523. ARTICLE IN PRESS 226 S. Peischl, R. Bürger / Journal of Theoretical Biology 251 (2008) 210–226 Schneider, K.A., 2007. Long-term evolution of polygenic traits under frequency-dependent intraspecific competition. Theor. Popul. Biol. 71, 342–366. Schneider, K.A., Bürger, R., 2006. Does competitive divergence occur if assortative mating is costly. J. Evol. Biol. 19, 570–588. Sheppard, P.M., 1958. Natural Selection and Heredity. Hutchinson, London. Slatkin, M., 1979. Frequency- and density-dependent selection on a quantitative character. Genetics 93, 755–771. Slatkin, M., 1984. Ecological causes of sexual dimorphism. Evolution 38, 622–630. Smith, T.B., 1993. Disruptive selection and the genetic basis of bill size polymorphism in the African finch pyrenestes. Nature 363, 618–620. Thieme, H., 2003. Mathematics in Population Biology. Princeton University Press, Princeton. Udovic, D., 1980. Frequency-dependent selection disruptive selection and the evolution of reproductive isolation. Am. Nat. 116, 621–641. Van Dooren, T.J.M., 1999. The evolutionary ecology of dominancerecessivity. J. Theor. Biol. 198, 519–532. Van Dooren, T.J.M., Durinx, M., Demon, I., 2004. Sexual dimorphism or evolutionary branching. Evol. Ecol. Res. 6, 857–871. van Doorn, G.S., Dieckmann, U., 2006. The long-term evolution of multilocus traits under frequency-dependent disruptive selection. Evolution 60, 2226–2238. Wagner, G.P., Bürger, R., 1985. On the evolution of dominance modifiers II. A non-equilibrium approach to genetic systems. J. Theor. Biol. 113, 475–500. Wilson, D.S., Turelli, M., 1986. Stable underdominance and the evolutionary invasion of empty niches. Am. Nat. 127, 835–850.