On an Objective Basis for the Maximum Entropy Principle

In [2], Jaynes does provide a principled basis for preferring the maximum entropy solution over alternative probability assignments. Specifically, let N be the number of repeated trials of an experiment with K possible outcomes {ω₁, ω₂, …, ω_K}, and with some constraint information, such as the mean die value measured based on these N repeated trials. Note that the outcomes of the individual trials are not known. Nor is it known the number of occurrences (N_k) of each distinct outcome, ω_k. However, suppose that we did know N_k, k = 1, …, K. For large N, by the weak law of large numbers, e.g., [3], we know that $\frac{N_k}{N}\to p_k$ with probability 1, where p_k, k = 1, …, K are the true probabilities. Thus, if (N₁, N₂, …, N_K) were known, a good choice for the probability assignments would be the frequency counts $\left(\frac{N_1}{N},\frac{N_2}{N},\dots ,\frac{N_K}{N}\right)$. Accordingly, estimating $\underset{¯}{p}=(p_1,p_2,\dots ,p_K)$ amounts to estimating (N₁, …, N_K). Let (x₁, x₂, …, x_N), x_i ∈ {ω₁, …, ω_K}, be a particular N-trial realization sequence (microstate) for the experiment, with associated macrostate (counts) $\left({N^{\prime }}_1,{N^{\prime }}_2,\dots ,{N^{\prime }}_K\right)$ that agrees with the given constraint information. Suppose all such microstates are a priori equally likely. Then the probability of macrostate (N₁, N₂, …, N_K) is: where the multinomial $\left(\stackrel{N}{N_1,\dots ,N_K}\right)$ is the number of distinct microstates that are consistent with the (constraint-achieving) macrostate. Since, for any given realization sequence, with macrostate (N₁, …, N_K), we would form the probability estimate as $\underset{¯}{p}=\left(\frac{N_1}{N},\dots ,\frac{N_K}{N}\right)$, P (N1, …, NK) is also the probability that we form the probability assignment $\left(\frac{N_1}{N},\dots ,\frac{N_K}{N}\right)$. Thus, if we choose (N₁, …, N_K) to maximize (2), we are determining the probability mass function $p_1,p_2,\dots ,p_K=\left(\frac{N_1}{N},\dots ,\frac{N_K}{N}\right)$ that we are most likely to produce as an estimate, given the specified constraint information and the number of die tosses. To maximize (2), one maximizes the multinomial coefficient $\left(\stackrel{N}{N_1,\dots ,N_K}\right)$. Based on Stirling’s approximation [4], $\left(\stackrel{N}{N_1,\dots ,N_K}\right)~e^{NH\left(\frac{N_1}{N},\dots ,\frac{N_K}{N}\right)}$, with H(·) Stirling’s entropy function. Accordingly, one can closely approximate maximizing (2) subject to, e.g., a mean value constraint ${\displaystyle \sum _{i=1}^{K}i{N}_i=\mu }$ by maximizing Shannon’s entropy function $H\left(\frac{N_1}{N},\dots ,\frac{N_K}{N}\right)$ subject to the constraint. Allowing unconstrained probabilities, rather than fractions of N, this amounts to solving: subject to ${\displaystyle \sum _{i=1}^{K}i{p}_i}=\mu$ and ${\displaystyle \sum _{i=1}^K{p}_i}=1$.

Note, too, that since (2) or its approximate is the probability of forming the estimate (p₁, …, p_K), we can use (4) to evaluate the relative likelihoods of producing different candidate probability assignments, $\frac{P(p_1,\dots ,p_K)}{P({p^{\prime }}_1,\dots ,{p^{\prime }}_K)}=e^{N(H(p_1,\dots ,p_K)-H({p^{\prime }}_1,\dots ,{p^{\prime }}_K))}$ Thus, the likelihood of the maximum entropy solution, relative to an alternative distribution, grows exponentially with the entropy difference.

In [1], they acknowledged this statistically-based justification for the maximum entropy distribution, as applied to the 6-sided Brandeis die problem. Moreover, they motivated the 3-sided die problem by stating that “suppose a friend later tossed the die many times…”. Thus, their 3-sided die problem genuinely does consider the scenario where the constraint information was accurately measured, based on many repeated die tosses. Accordingly, the above interpretation should be applicable to their 3-sided die problem, just as it is to the Brandeis die problem. For the 3-sided die problem, we have $\frac{P\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)}{P\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)}=2^{N({\log }_2(3)-1.5)}~2^{0.08N}$. For example, if N = 1000, the ME distribution is more than 10²⁴ times more likely to be produced as the estimate than the proposed distribution from [1]. The only real assumption in this analysis is that realization sequences (microstates) consistent with a given macrostate are equally likely.

In [1], they also stated that “since there is no reason to believe the die is fair, we could consider all probability distributions that satisfy the constraints in the problem equally probable. That is, we apply the principle of indifference to the probability values themselves”. While it is true that a priori there is no reason to believe the die is fair, there is also no evidence to support that all probability distributions are equally probable given that our only knowledge about the die is the mean value constraint. Applying a principle of indifference in this way, by [1], imposes a further (rather strong and unjustified) constraint which, as shown above, leads to a solution 10²⁴ times less probable than the solution built on only the available evidence. The “least presumptuous” strategy should only consider the mean value constraint and not impose further constraints. Nevertheless, a different distribution than the ME solution may be achieved if the principle of indifference is applied correctly; i.e., if it is supported by some evidence or by available knowledge about the distribution for a particular application.

Another related albeit somewhat different vantage point from which to evaluate the two candidate distributions $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ and $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ is with respect to the asymptotic equipartition principle (AEP) [4]. The AEP considers N independent trials of random draws from a given distribution (p₁, p₂, …, p_K). It defines the ϵ-typical set $A_{ϵ}^{(N)}$ as the set of sequences (x₁, …, x_N) such that 2^{−N(H(X)+ϵ)} ≤ P (x₁, x₂, …, x_N) ≤ 2^{−N(H(X)−ϵ)}. One can show the following [4]:

• The cardinality of this set is approximately 2^{NH(p1, p2, …, pK)}.
• Each sequence in this set is approximately equally likely.
• The typical set accounts for nearly all the probability, i.e., the joint pmf P (x₁, x₂, …, x_N) is very nearly approximated as a uniform distribution on all typical sequences, with a zero probability assignment on all non-typical sequences.

Note that not all sequences that are typical will precisely achieve the mean value constraint. However, their deviation from the constraint is made arbitrarily small as N gets large. From the perspective of the AEP, according to the $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ model, there are roughly 2^1.5N equally likely realizations (each with self-information very close to $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$. On the other hand, according to the ME distribution $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$, there are $2^{N{\log }_2(3)}$ equally likely realizations (typical sequences). Note that the sequences that are typical for $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ are not typical for $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$, and vice versa. Thus, in choosing between the distributions, one is either rejecting 2^1.5N realizations typical of $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ or $2^{N{\log }_2(3)}$ realizations typical of $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$. Again, for N = 1000, choosing $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ means rejecting more than 10²⁴ times the number of realizations than if one chooses $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$. In the absence of additional information, it seems one should reject the hypothesis that contains (far) fewer realizations that are (approximately) consistent with the measured constraints. Again, from this vantage, the ME solution is preferred.

The authors in [1] sought to independently validate the $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ distribution by considering a Bayesian learning setting, starting from an uninformative Dirichlet prior on the probabilities. Here, they imposed the mean value constraint, fixed N₁ = N₃, and assumed all (N₁, N₂) configurations consistent with satisfying the mean value constraint to be equally likely. They then let N → ∞ and evaluated the conditional probability of die value 2, which they found to be 0.5. While this analysis does appear to support the proposed $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ distribution, a concern with this analysis is the consistency between letting N → ∞ and the assumption that all (N₁, N₂) configurations meeting the constraint are equally likely. Specifically, by the weak law of large numbers, as $N\to \infty ,\frac{N_1}{N}\to P_1$ and $\frac{N_2}{N}\to P_2$ with probability 1, i.e., the assumption of equally likely macrostate pairs appears to be incompatible with letting N → ∞.

Objectively, it is of course possible that the true distribution (used to randomly generate N realizations and achieve a mean value of 2) is the $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ distribution. This would be revealed within the ME framework if we imposed one additional constraint. Specifically, if, in addition to the mean value constraint, we imposed the second moment constraint, ${\displaystyle \sum _{i=1}^3{i}^2{p}_i}=4.5$ the (unique) ME distribution is indeed the $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ distribution. Thus, ME is not incompatible with the $\left(\frac{1}{4},\frac{1}{2},\frac{1}{4}\right)$ distribution. It is simply producing the “best” distribution given whatever accurate constraint information is made available.