At first glance, population geneticists typically assume that genotypes have constant fitness values, which are a measure of their reproductive success. In some case this is not a reasonable assumption. For example, the fitness of a genotype may depend on time, environment, or on the frequency-dependent selection. Positive frequency-dependent selection occurs when the fitness of a genotype increases as it becomes more common in a population. In contrast, negative frequency-dependent selection favors rare alleles by assigning the highest fitness to the rarest allele. Negative frequency-dependent selection can only give rise to balanced polymorphisms in a population because common genotypes will be selected against, limiting their chance of fixation. Examples of negative-frequency dependent selection included: self-compatibility alleles in plants, search images in predator-prey systems, mating preferences in Drosophila, and jaw handedness in cichlid fish.

Mating preferences in Drosophila melanogaster is an example of negative-frequency dependent selection. Photo credit: Steve A. Kay, The Scripps Research Institute, Courtesy of the National Science Foundation.

Consider a two allele system (A₁ and A₂) and let p and q represent the frequency of A₁ and A₂, respectively. Let w_ij represent the fitness of genotype A_iA_j (assume w_ij = w_ji for i ≠ j). Therefore, for this system w₁₁ is the fitness of allele A₁A₁, w₁₂ is the fitness of allele A₁A₂, and w₂₂ is the fitness of allele A₂A₂. We can express the new frequency of A₁ after one generation of selection using the rational function,

where is the mean fitness of the population. We are interested in finding equilibrium values of p, in other words values of p such that p' = p, indicating no change in allele frequency in the next generation. Setting p' = p gives,

Assuming p ≠ 0 we can cancel p on both sides of the above equation as,

Bringing all terms to the right-hand side of the equation and substituting q = 1 - p gives the polynomial,

Therefore, we have deduced that solutions to the above equations are equilibrium values of p (we are only concerned with biologically reasonable equilibria). Using fitness values:

where c > 0 is a constant, answer the following questions.