Abstract
We propose an exact global minimization framework for certain variational image segmentation models, such as the Chan–Vese model, involving four regions. A global solution is guaranteed if the data term satisfies a certain condition. We give a theoretical analysis of the condition for \(L^p\) type of data terms, such as in the Chan–Vese model and Mumford Shah model for \(p=2\). We show experimentally that the condition tends to hold in practice for \(p \ge 2\) and also holds in many cases for more advanced data terms. If the condition is violated, convex and submodular relaxations are proposed which are not guaranteed to produce exact solutions, but tend to do so in practice. We also build up simple convex relaxations for some other four region segmentation models, including Potts’ model. Algorithms are proposed which are very efficient compared to related work due to the simple and compact formulations.
Similar content being viewed by others
References
Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42, 577–685 (1989)
Chan, T., Vese, L.A.: Active contours without edges. IEEE Image Proc. 10, 266–277 (2001)
Vese, L.A., Chan, T.F.: A new multiphase level set framework for image segmentation via the Mumford and Shah model. Int. J. Comput. Vis. 50, 271–293 (2002)
Boykov, Yuri, Kolmogorov, Vladimir: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26, 359–374 (2001)
Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. J. R. Stat. Soc. Ser. B 51, 271–279 (1989)
Boykov, Yuri, Veksler, Olga, Zabih, Ramin: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1222–1239 (2001)
Bae, Egil, Yuan, Jing, Tai, Xue-Cheng: Global minimization for continuous multiphase partitioning problems using a dual approach. Int. J. Comput. Vis. 92(1), 112–129 (2011)
Brown, E.S., Chan, T.F., Bresson, X.: A convex relaxation method for a class of vector-valued minimization problems with applications to Mumford–Shah segmentation. UCLA, Applied Mathematics, CAM-report-10–43, July, 2010
Lellmann, J., Becker, F., Schnörr, C.: Convex optimization for multi-class image labeling with a novel family of total variation based regularizers. In: IEEE International Conference on Computer Vision (ICCV), pp. 646–653 (2009)
Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. In: Tai, X.-C., Mórken, K., Lysaker, M., Lie, K.-A. (eds.) Scale Space and Variational Methods in Computer Vision (SSVM 2009), volume 5567 of LNCS, pp. 150–162. Springer, New York (2009)
Pock, T., Chambolle, A., Bischof, H., Cremers, D.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Miami, Florida (2009)
Zach, C., Gallup, D., Frahm, J.-M., Niethammer, M.: Fast global labeling for real-time stereo using multiple plane sweeps. In: Vision, Modeling and Visualization Workshop (VMV) (2008)
Goldluecke, B., Cremers, D.: Convex relaxation for multilabel problems with product label spaces. In: Proceedings of the 11th European conference on Computer vision, ECCV’10, pp. 225–238. Springer, Berlin, Heidelberg (2010).
Strekalovskiy, E., Goldluecke, B., Cremers, D.: Tight convex relaxations for vector-valued labeling problems. In: IEEE International Conference on Computer Vision (ICCV) (2011)
Ishikawa, Hiroshi: Exact optimization for markov random fields with convex priors. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1333–1336 (2003)
Chan, T.F., Esedoḡlu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math., 66(5):1632–1648 (electronic), (2006)
Bae, E., Yuan, J., Tai, X.C., Boykov, Y.: A fast continuous max-flow approach to non-convex multilabeling problems. Technical report CAM10-62, UCLA, CAM (2010)
Yuan, J., Bae, E., Tai, X.C.: A study on continuous max-flow and min-cut approaches. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2217–2224. San Francisco (2010)
Yuan, Jing, Bae, Egil, Tai, Xue-Cheng, Boykov, Yuri: A spatially continuous max-flow and min-cut framework for binary labeling problems. Numerische Mathematik 126(3), 559–587 (2014)
Liu, Liman, Tao, Wenbing: Image segmentation by iteratively optimization of multiphase multiple piecewise constant model and four-color relabeling. Pattern Recognit. 44(12), 2819–2833 (2011)
Zhang, R., Bresson, X., Chan, T.F., Tai, X.-C.: Four color theorem and convex relaxation for image segmentation with any number of regions. UCLA, Applied Mathematics, CAM-report-13-16 (2013)
Hodneland, Erlend, Tai, Xue-Cheng, Gerdes, Hans-Hermann: Four-color theorem and level set methods for watershed segmentation. Int. J. Comput. Vis. 82(3), 264–283 (2009)
Bae, E., Tai, X.-C.: Efficient global minimization for the multiphase Chan–Vese model of image segmentation. In: Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition. LNCS, vol. 5681, pp. 28–41. Springer, New York (2009)
Bae, E.: Efficient global minimization methods for variational problems in imaging and vision. (Phd thesis, University of Bergen, https://bora.uib.no/handle/1956/5017), August 2011
Bae, Egil, Tai, Xue-Cheng: Graph cut optimization for the piecewise constant level set method applied to multiphase image segmentation. In: Tai, X.-C., Mórken, K., Lysaker, M., Lie, K.-A. (eds.) Scale Space and Variational Methods in Computer Vision, Volume 5567 of LNCS, pp. 1–13. Springer, New York (2009)
Brown, E.S., Chan, T.F., Bresson, X.: Completely convex formulation of the Chan-Vese image segmentation model. International Journal of Computer Vision. (2011) doi:10.1007/s11263-011-0499-y
Lie, J., Lysaker, M., Tai, X.C.: Piecewise constant level set methods and image segmentation. In: Scale Space and PDE Methods in Computer Vision, pp. 573–584. Springer, New York (2005)
Lie, J., Lysaker, M., Tai, X.C.: A binary level set model and some applications to Mumford–Shah image segmentation. IEEE Trans. Image Process. 15(5), 1171–1181 (2006)
Meyer, Y.: Oscillating patterns in image processing and nonlinear evolution equations, volume 22 of University Lecture Series. American Mathematical Society, Providence, RI, 2001. The fifteenth Dean Jacqueline B. Lewis memorial lectures
Strang, Gil: Maximal flow through a domain. Math. Program. 26, 123–143 (1983)
Pock, T., Schoenemann, T., Graber, G., Bischof, H., Cremers, D.: A convex formulation of continuous multi-label problems. In: ECCV ’08, pp. 792–805 (2008)
Bae, E., Lellmann, J., Tai, X.-C.: Convex relaxations for a generalized chan-vese model. In: 9th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 8081, pp. 223–236. Springer, New York (2013)
Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV ’03: Proceedings of the Ninth IEEE International Conference on Computer Vision, p. 26. IEEE Computer Society, Washington, DC (2003).
Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Readings in Uncertain Reasoning, pp. 452–472. Morgan Kaufmann Inc., San Francisco (1990)
Fulkerson, D., Ford, L.: Flows in Networks. Princeton University Press, Princeton (1962)
Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)
Kolmogorov, Vladimir, Rother, Carsten: Minimizing nonsubmodular functions with graph cuts: a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1274–1279 (2007)
Hammer, P.L., Hansen, P., Simeone, B.: Roof duality, complementation and persistency in quadratic 01 optimization. Math. Program. 28, 121–155 (1984)
Rother, C., Kumar, S., Kolmogorov, V., Blake, A.: Digital tapestry. In: IEEE Proceedings of the Computer Vision and Pattern Recognition (2005)
El-Zehiry, N.Y., Grady, L.: Combinatorial optimization of the discretized multiphase Mumford-Shah functional. Int. J. Comput. Vis. (2013) doi:10.1007/s11263-013-0617-0
Berkels, B.: An unconstrained multiphase thresholding approach for image segmentation. In: SSVM, pp. 26–37 (2009)
Hyman, J.M., Shashkov, M.J.: Natural discretizations for the divergence, gradient, and curl on logically rectangular grids. Comput. Math. Appl. 33(4), 81–104 (1997)
Ito, Kazufumi, Kunisch, Karl: Lagrange Multiplier Approach to Variational Problems and Applications. Society for Industrial and Applied Mathematics, Philadelphia (2008)
Ekeland, Ivar, Téman, Roger: Convex Analysis and Variational Problems. Society for Industrial and Applied Mathematics, Philadelphia (1999)
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
Chambolle, Antonin, Cremers, Daniel, Pock, Thomas: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 11131158 (2012)
Federer, H.: Geometric Measure Theory. Springer, New York (1969)
Vol’Pert, A.I.: Spaces BV and quasilinear equations. Mat. Sb. (N.S.) 73(115), 255302 (1967)
Chambolle, Antonin: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1), 89–97 (2004)
Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)
Boyle, J.P., Dykstra, R.: A method for finding projections onto the intersection of convex sets in hilbert spaces. In: Advances in Order Restricted Statistical Inference (Iowa City, Iowa, 1985). Lecture Notes in Statistics, vol. 37, p. 2847. Springer, Berlin, 1986 (1985)
Hodneland, E.: Segmentation of digital images. Cand. Scient Thesis, Department of Mathematics, University of Bergen. Available online at www.mi.uib.no/tai (2003)
Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the piecewise smooth Mumford-Shah functional. In: IEEE International Conference on Computer Vision (ICCV). Kyoto (2009)
Yuan, J., Bae, E., Tai, X.-C., Boykov, Yuri: A continuous max-flow approach to potts model. In: European Conference on Computer Vision. LNCS, vol. 6316, pp. 379–392 (2010)
Acknowledgments
We thank the anonymous reviewers for valuable feedback. This research has been supported by the Norwegian Research Council eVita project 214889 and eVita project 166075.
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix 1: A Proofs of Prop 2–5
1.1 Appendix 1.1: Proof of Prop 4
Proof
Let \(\frac{c_2+c_1}{2} \le I \le \frac{c_4+c_3}{2}\). Then
for any \(\beta \ge 1\). Therefore, adding these two inequalities
\(\square \)
1.2 Appendix 1.2: Proof of Prop 2
When \(\frac{c_1+c_2}{2} \le I \le \frac{c_4+c_3}{2}\), the result follows from Prop 4. Consider \(I < \frac{c_2+c_1}{2}\), then
Together with (64), this implies
Therefore, there exists \(\theta _1 \ge \theta _2 > 1\) such that
For \(\beta \ge {\beta _0}\)
where \(\theta _1^{\beta - {\beta _0}} \ge \theta _2^{\beta - {\beta _0}} > 1\), hence
where the last inequality follows from \(|I-c_1|^\beta \le |I-c_4|^\beta \). Exactly the same argument can be used to show Prop 2 when \(I > \frac{c_4+c_3}{2}\).
1.3 Appendix 1.3: Proof of Prop 3
Proof
Assume first \(I > c_3\), then
Therefore, there exists a \(\theta > 1\) s.t.
Pick \(\mathcal {C}_I^1 \in \mathbb {N}\) s.t.
Then
If \(I<c_2\), then
and thus the same argument can be used to show there exists \(\mathcal {C}_I^2 \in \mathbb {N}\) such that
In case \(c_2 \le I \le c_3\), the existence of such a \(\mathcal {C}\) was proved in Prop 4, e.g. \(\mathcal {C} = 1\).
Therefore the condition (64) is satisfied for any \(I \in [0,L]\) by choosing \(\beta \ge \mathcal {C} = \max _{I \in [0,L]} \max \{ \mathcal {C}^1_I,\mathcal {C}^2_I \}\). \(\square \)
1.4 Appendix 1.4 Proof of Prop 5
We will show the condition holds for \(\beta = 1\), which by Prop 2 implies it holds for all \(\beta \ge 1\). Observe that if \(c_1,c_2\) and \(c_3,c_4\) are equidistant it follows that \(c_1 + c_4 = c_2 + c_3\). For \(I < c_2\)
For \(I \ge c_3\)
Appendix 2: Relation Between Dual Problem (63) and Discrete Max-flow Problem
Recall that for each \(p \in \mathcal {P}\) the data term was represented by the six edges (31). The flow on each of these edges are constrained by the capacities defined in (35) and (34), i.e.
For notational convenience, we use capital letters \(P,Q\) to denote flow on discrete edges. The dual variables \(p_s^1, p_s^2, p_t^1, p_t^2, \tilde{p}^{12}, \tilde{p}^{21}\) over the continuous domain \({\varOmega }\) are inspired by these discrete flow functions. They are constrained by the capacities \(C_s^1(x), C_s^2(x), C_t^1(x), C_t^2(x), C^{12}(x)\) and \(C^{21}(x)\) respectively. Note that \(\tilde{p}^{12}\) and \(\tilde{p}^{21}\) should satisfy
for all \(x \in {\varOmega }\), but can be merged in \(p^{12} = \tilde{p}^{12} - \tilde{p}^{21}\). The above two constraints then transform into (51).
For each pair of neighboring pixels \((p,q) \in \mathcal {N}\), two edges were constructed in the discrete graph \(\mathcal {G}\): \((v_{p,1},v_{q,1})\) and \((v_{p_2},v_{q,2})\). Let \(Q^1\) denote the flow function on \((v_{p,1},v_{q,1})\) and \(Q^2\) the flow function on \((v_{p,2},v_{q,2})\). These flows are constrained by
for all \((p,q) \in \mathcal {N}\). In the same vein, the two spatial flow fields \(q^i \in (C^\infty ({\varOmega }))^N\) \(i=1,2\) are defined in the continuous setting, and should satisfy (52). Observe that the continuous counterpart allows to measure the magnitude of \(q^1,q^2\) with 2-norm, which in turn produces rotationally invariant results.
Flow conservation should hold at each \(v_{p,1}\) and \(v_{p,2}\), i.e.
\(\forall p \in \mathcal {P}\). Here \(N^+(v)\) is defined as all \(w \in \mathcal {V}\) such that \((v,w) \in \mathcal {E}\) and \(N^-(v)\) is defined as all \(w \in \mathcal {V}\) such that \((w,v) \in \mathcal {E}\). The final two constraints (61) and (62) are generalizations of discrete flow conservation to the continuous setting. The total amount of discrete flow in the graph is in this case
The objective functional (63) is a generalization of the above, and measures the total amount of flow that originates from the source.
Rights and permissions
About this article
Cite this article
Bae, E., Tai, XC. Efficient Global Minimization Methods for Image Segmentation Models with Four Regions. J Math Imaging Vis 51, 71–97 (2015). https://doi.org/10.1007/s10851-014-0507-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10851-014-0507-2