Mixture models: building a parameter space

Despite the flexibility and popularity of mixture models, their associated parameter spaces are often difficult to represent due to fundamental identification problems. This paper looks at a novel way of representing such a space for general mixtures of exponential families, where the parameters are identifiable, interpretable, and, due to a tractable geometric structure, the space allows fast computational algorithms to be constructed.

Authors and Affiliations

1. Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, ON, Canada

   Vahed Maroufy & Paul Marriott

Corresponding author

Correspondence to Vahed Maroufy.

Appendix

Following the methodology in Sect. 2.1 select suitably separated grid points \(\varvec{\mu }=(\mu _1,\ldots ,\mu _L)\), which are fixed throughout. Select initial proportions \(\varvec{\rho }^{(0)}=(\rho _1^{(0)},\ldots ,\rho _L^{(0)})\) and local mixture parameters \(\underline{\varvec{\lambda }}^{(0)} =(\varvec{\lambda }^{1,(0)},\ldots ,\varvec{\lambda }^{L,(0)})\). Suppose we have \(\varvec{\rho }^{(r)}\) and \(\underline{\varvec{\lambda }}^{(r)}\) at step r, where \(L_r\) is the number of non-zero proportions and \(L_r \le L\). To obtain the estimates at step \(r+1\) run the following steps.

1. 1.

   Calculate \(\rho _l^{(r+1)}=\frac{n_l}{n}\), where \(n_l= \sum \nolimits _{i=1}^{n}w^{(r+1)}_{il}\) and for \(x=1,\ldots ,n;\,\, l=1,\ldots ,L_r\)

   $$\begin{aligned} w^{(r+1)}_{il}=\frac{\rho _l^{(r)} g_{\mu _l}\left( x_i,\varvec{\lambda }^{l,(r)}\right) }{\sum \nolimits _{l=1}^{L_r} \rho _l^{(r)} g_{\mu _l}\left( x_i,\varvec{\lambda }^{l,(r)}\right) }. \end{aligned}$$
2. 2.

   Choose a positive value \(0 < \gamma < 1\), and check if there is any l such that \(\rho _l^{(r+1)} < \gamma \).

   1. (a)

      If yes: exclude the components corresponding to \(\rho _l^{(r+1)} < \gamma \), update \(L_r\rightarrow L_{r+1}\) and go back to step 1.

   2. (b)

      If no: go to step 3.

3. 3.

   Classify the data set into \(\varvec{x}^1,\ldots \varvec{x}^{L_{r+1}}\) by assigning each \(x_i\) to only one local mixture component. For each \(l=1,\ldots ,L_{r+1}\), update \(\varvec{\lambda }^{l,(r)}\) by

   $$\begin{aligned} \varvec{\lambda }^{l,(r+1)}= {\text {*}}{arg\,max}_{\varvec{\lambda } \in \Lambda _{\mu _l}} l_{\mu _l}(x^l,\varvec{\lambda }), \end{aligned}$$

   where \( l_{\mu _l}(x^l,\cdot )\) is the log-likelihood function for the component l as defined in Marriott (2002).

Remark 1

Step 2 restricts the number of required components for fitting a data set in a way that there is enough information necessary for running inference on each local mixture component. The value, \(\gamma \), has an influence on the final result of the algorithm in a similar way that an initial value affects the convergence of a general EM algorithm (Table 1).

Reprints and permissions