Nothing Special   »   [go: up one dir, main page]

Skip to main content
Log in

Efficient Global Minimization Methods for Image Segmentation Models with Four Regions

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

We propose an exact global minimization framework for certain variational image segmentation models, such as the Chan–Vese model, involving four regions. A global solution is guaranteed if the data term satisfies a certain condition. We give a theoretical analysis of the condition for \(L^p\) type of data terms, such as in the Chan–Vese model and Mumford Shah model for \(p=2\). We show experimentally that the condition tends to hold in practice for \(p \ge 2\) and also holds in many cases for more advanced data terms. If the condition is violated, convex and submodular relaxations are proposed which are not guaranteed to produce exact solutions, but tend to do so in practice. We also build up simple convex relaxations for some other four region segmentation models, including Potts’ model. Algorithms are proposed which are very efficient compared to related work due to the simple and compact formulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Mumford, D., Shah, J.: Optimal approximation by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42, 577–685 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  2. Chan, T., Vese, L.A.: Active contours without edges. IEEE Image Proc. 10, 266–277 (2001)

    Article  MATH  Google Scholar 

  3. Vese, L.A., Chan, T.F.: A new multiphase level set framework for image segmentation via the Mumford and Shah model. Int. J. Comput. Vis. 50, 271–293 (2002)

    Article  MATH  Google Scholar 

  4. Boykov, Yuri, Kolmogorov, Vladimir: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26, 359–374 (2001)

    Google Scholar 

  5. Greig, D.M., Porteous, B.T., Seheult, A.H.: Exact maximum a posteriori estimation for binary images. J. R. Stat. Soc. Ser. B 51, 271–279 (1989)

    Google Scholar 

  6. Boykov, Yuri, Veksler, Olga, Zabih, Ramin: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1222–1239 (2001)

    Article  Google Scholar 

  7. Bae, Egil, Yuan, Jing, Tai, Xue-Cheng: Global minimization for continuous multiphase partitioning problems using a dual approach. Int. J. Comput. Vis. 92(1), 112–129 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  8. Brown, E.S., Chan, T.F., Bresson, X.: A convex relaxation method for a class of vector-valued minimization problems with applications to Mumford–Shah segmentation. UCLA, Applied Mathematics, CAM-report-10–43, July, 2010

  9. Lellmann, J., Becker, F., Schnörr, C.: Convex optimization for multi-class image labeling with a novel family of total variation based regularizers. In: IEEE International Conference on Computer Vision (ICCV), pp. 646–653 (2009)

  10. Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. In: Tai, X.-C., Mórken, K., Lysaker, M., Lie, K.-A. (eds.) Scale Space and Variational Methods in Computer Vision (SSVM 2009), volume 5567 of LNCS, pp. 150–162. Springer, New York (2009)

    Google Scholar 

  11. Pock, T., Chambolle, A., Bischof, H., Cremers, D.: A convex relaxation approach for computing minimal partitions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Miami, Florida (2009)

  12. Zach, C., Gallup, D., Frahm, J.-M., Niethammer, M.: Fast global labeling for real-time stereo using multiple plane sweeps. In: Vision, Modeling and Visualization Workshop (VMV) (2008)

  13. Goldluecke, B., Cremers, D.: Convex relaxation for multilabel problems with product label spaces. In: Proceedings of the 11th European conference on Computer vision, ECCV’10, pp. 225–238. Springer, Berlin, Heidelberg (2010).

  14. Strekalovskiy, E., Goldluecke, B., Cremers, D.: Tight convex relaxations for vector-valued labeling problems. In: IEEE International Conference on Computer Vision (ICCV) (2011)

  15. Ishikawa, Hiroshi: Exact optimization for markov random fields with convex priors. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1333–1336 (2003)

    Article  Google Scholar 

  16. Chan, T.F., Esedoḡlu, S., Nikolova, M.: Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math., 66(5):1632–1648 (electronic), (2006)

    Google Scholar 

  17. Bae, E., Yuan, J., Tai, X.C., Boykov, Y.: A fast continuous max-flow approach to non-convex multilabeling problems. Technical report CAM10-62, UCLA, CAM (2010)

  18. Yuan, J., Bae, E., Tai, X.C.: A study on continuous max-flow and min-cut approaches. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2217–2224. San Francisco (2010)

  19. Yuan, Jing, Bae, Egil, Tai, Xue-Cheng, Boykov, Yuri: A spatially continuous max-flow and min-cut framework for binary labeling problems. Numerische Mathematik 126(3), 559–587 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  20. Liu, Liman, Tao, Wenbing: Image segmentation by iteratively optimization of multiphase multiple piecewise constant model and four-color relabeling. Pattern Recognit. 44(12), 2819–2833 (2011)

    Article  MATH  Google Scholar 

  21. Zhang, R., Bresson, X., Chan, T.F., Tai, X.-C.: Four color theorem and convex relaxation for image segmentation with any number of regions. UCLA, Applied Mathematics, CAM-report-13-16 (2013)

  22. Hodneland, Erlend, Tai, Xue-Cheng, Gerdes, Hans-Hermann: Four-color theorem and level set methods for watershed segmentation. Int. J. Comput. Vis. 82(3), 264–283 (2009)

    Article  Google Scholar 

  23. Bae, E., Tai, X.-C.: Efficient global minimization for the multiphase Chan–Vese model of image segmentation. In: Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition. LNCS, vol. 5681, pp. 28–41. Springer, New York (2009)

  24. Bae, E.: Efficient global minimization methods for variational problems in imaging and vision. (Phd thesis, University of Bergen, https://bora.uib.no/handle/1956/5017), August 2011

  25. Bae, Egil, Tai, Xue-Cheng: Graph cut optimization for the piecewise constant level set method applied to multiphase image segmentation. In: Tai, X.-C., Mórken, K., Lysaker, M., Lie, K.-A. (eds.) Scale Space and Variational Methods in Computer Vision, Volume 5567 of LNCS, pp. 1–13. Springer, New York (2009)

    Chapter  Google Scholar 

  26. Brown, E.S., Chan, T.F., Bresson, X.: Completely convex formulation of the Chan-Vese image segmentation model. International Journal of Computer Vision. (2011) doi:10.1007/s11263-011-0499-y

  27. Lie, J., Lysaker, M., Tai, X.C.: Piecewise constant level set methods and image segmentation. In: Scale Space and PDE Methods in Computer Vision, pp. 573–584. Springer, New York (2005)

  28. Lie, J., Lysaker, M., Tai, X.C.: A binary level set model and some applications to Mumford–Shah image segmentation. IEEE Trans. Image Process. 15(5), 1171–1181 (2006)

    Article  MATH  Google Scholar 

  29. Meyer, Y.: Oscillating patterns in image processing and nonlinear evolution equations, volume 22 of University Lecture Series. American Mathematical Society, Providence, RI, 2001. The fifteenth Dean Jacqueline B. Lewis memorial lectures

  30. Strang, Gil: Maximal flow through a domain. Math. Program. 26, 123–143 (1983)

    Article  MATH  MathSciNet  Google Scholar 

  31. Pock, T., Schoenemann, T., Graber, G., Bischof, H., Cremers, D.: A convex formulation of continuous multi-label problems. In: ECCV ’08, pp. 792–805 (2008)

  32. Bae, E., Lellmann, J., Tai, X.-C.: Convex relaxations for a generalized chan-vese model. In: 9th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 8081, pp. 223–236. Springer, New York (2013)

  33. Boykov, Y., Kolmogorov, V.: Computing geodesics and minimal surfaces via graph cuts. In: ICCV ’03: Proceedings of the Ninth IEEE International Conference on Computer Vision, p. 26. IEEE Computer Society, Washington, DC (2003).

  34. Geman, S., Geman, D.: Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. Readings in Uncertain Reasoning, pp. 452–472. Morgan Kaufmann Inc., San Francisco (1990)

    Google Scholar 

  35. Fulkerson, D., Ford, L.: Flows in Networks. Princeton University Press, Princeton (1962)

    MATH  Google Scholar 

  36. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)

    Article  Google Scholar 

  37. Kolmogorov, Vladimir, Rother, Carsten: Minimizing nonsubmodular functions with graph cuts: a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1274–1279 (2007)

    Article  Google Scholar 

  38. Hammer, P.L., Hansen, P., Simeone, B.: Roof duality, complementation and persistency in quadratic 01 optimization. Math. Program. 28, 121–155 (1984)

    Article  MATH  MathSciNet  Google Scholar 

  39. Rother, C., Kumar, S., Kolmogorov, V., Blake, A.: Digital tapestry. In: IEEE Proceedings of the Computer Vision and Pattern Recognition (2005)

  40. El-Zehiry, N.Y., Grady, L.: Combinatorial optimization of the discretized multiphase Mumford-Shah functional. Int. J. Comput. Vis. (2013) doi:10.1007/s11263-013-0617-0

  41. Berkels, B.: An unconstrained multiphase thresholding approach for image segmentation. In: SSVM, pp. 26–37 (2009)

  42. Hyman, J.M., Shashkov, M.J.: Natural discretizations for the divergence, gradient, and curl on logically rectangular grids. Comput. Math. Appl. 33(4), 81–104 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  43. Ito, Kazufumi, Kunisch, Karl: Lagrange Multiplier Approach to Variational Problems and Applications. Society for Industrial and Applied Mathematics, Philadelphia (2008)

    Book  MATH  Google Scholar 

  44. Ekeland, Ivar, Téman, Roger: Convex Analysis and Variational Problems. Society for Industrial and Applied Mathematics, Philadelphia (1999)

    Book  MATH  Google Scholar 

  45. http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/

  46. Chambolle, Antonin, Cremers, Daniel, Pock, Thomas: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 11131158 (2012)

    Article  MathSciNet  Google Scholar 

  47. Federer, H.: Geometric Measure Theory. Springer, New York (1969)

    MATH  Google Scholar 

  48. Vol’Pert, A.I.: Spaces BV and quasilinear equations. Mat. Sb. (N.S.) 73(115), 255302 (1967)

    MathSciNet  Google Scholar 

  49. Chambolle, Antonin: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1), 89–97 (2004)

    MathSciNet  Google Scholar 

  50. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  51. Boyle, J.P., Dykstra, R.: A method for finding projections onto the intersection of convex sets in hilbert spaces. In: Advances in Order Restricted Statistical Inference (Iowa City, Iowa, 1985). Lecture Notes in Statistics, vol. 37, p. 2847. Springer, Berlin, 1986 (1985)

  52. Hodneland, E.: Segmentation of digital images. Cand. Scient Thesis, Department of Mathematics, University of Bergen. Available online at www.mi.uib.no/tai (2003)

  53. Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the piecewise smooth Mumford-Shah functional. In: IEEE International Conference on Computer Vision (ICCV). Kyoto (2009)

  54. Yuan, J., Bae, E., Tai, X.-C., Boykov, Yuri: A continuous max-flow approach to potts model. In: European Conference on Computer Vision. LNCS, vol. 6316, pp. 379–392 (2010)

Download references

Acknowledgments

We thank the anonymous reviewers for valuable feedback. This research has been supported by the Norwegian Research Council eVita project 214889 and eVita project 166075.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Egil Bae.

Appendices

Appendix 1: A Proofs of Prop 2–5

1.1 Appendix 1.1: Proof of Prop 4

Proof

Let \(\frac{c_2+c_1}{2} \le I \le \frac{c_4+c_3}{2}\). Then

$$\begin{aligned} |c_2-I|^\beta \le |c_1-I|^\beta \;\;\; \text {and} \;\;\; |c_3-I|^\beta \le |c_4-I|^\beta , \end{aligned}$$

for any \(\beta \ge 1\). Therefore, adding these two inequalities

$$\begin{aligned} |c_2 - I|^\beta + |c_3 - I|^\beta \le |c_1 - I|^\beta + |c_4 - I|^\beta . \end{aligned}$$

\(\square \)

1.2 Appendix 1.2: Proof of Prop 2

When \(\frac{c_1+c_2}{2} \le I \le \frac{c_4+c_3}{2}\), the result follows from Prop 4. Consider \(I < \frac{c_2+c_1}{2}\), then

$$\begin{aligned} |I-c_1|^{\beta _0} \le |I-c_2|^{\beta _0} \le |I-c_3|^{\beta _0} \le |I-c_4|^{\beta _0}. \end{aligned}$$

Together with (64), this implies

$$\begin{aligned} 0 < |I-c_2|^{\beta _0} - |I-c_1|^{\beta _0} \le |I-c_4|^{\beta _0} - |I-c_3|^{\beta _0}. \end{aligned}$$

Therefore, there exists \(\theta _1 \ge \theta _2 > 1\) such that

$$\begin{aligned} |I-c_4|^{\beta _0} = \theta _1 |I-c_3|^{\beta _0} \quad \text {and} \quad |I-c_2|^{\beta _0} = \theta _2 |I-c_1|^{\beta _0}. \end{aligned}$$

For \(\beta \ge {\beta _0}\)

$$\begin{aligned}&|I-c_4|^\beta = \theta _1^{\beta - {\beta _0}} |I-c_3|^\beta \quad \text {and}\\&\quad |I-c_2|^\beta = \theta _2^{\beta - {\beta _0}} |I-c_1|^\beta , \end{aligned}$$

where \(\theta _1^{\beta - {\beta _0}} \ge \theta _2^{\beta - {\beta _0}} > 1\), hence

$$\begin{aligned}&|I-c_2|^\beta + |I-c_3|^\beta = \theta _2^{\beta - {\beta _0}} |I-c_1|^\beta + \frac{1}{\theta _1^{\beta - {\beta _0}}} |I-c_4|^\beta \\&\quad \le \theta _1^{\beta - {\beta _0}} |I-c_1|^\beta + \frac{1}{\theta _1^{\beta - {\beta _0}}} |I-c_4|^\beta \le \theta _1^{\beta - {\beta _0}} |I-c_1|^\beta \\&\quad + \frac{1}{\theta _1^{\beta - {\beta _0}}} |I-c_4|^\beta , \end{aligned}$$

where the last inequality follows from \(|I-c_1|^\beta \le |I-c_4|^\beta \). Exactly the same argument can be used to show Prop 2 when \(I > \frac{c_4+c_3}{2}\).

1.3 Appendix 1.3: Proof of Prop 3

Proof

Assume first \(I > c_3\), then

$$\begin{aligned} |c_1 - I| > |c_2 - I| > |c_3 -I|. \end{aligned}$$

Therefore, there exists a \(\theta > 1\) s.t.

$$\begin{aligned} |I-c_1| = \theta \, |c_2-I|. \end{aligned}$$

Pick \(\mathcal {C}_I^1 \in \mathbb {N}\) s.t.

$$\begin{aligned} \theta ^\beta \ge 2, \;\;\;\;\; \forall \beta \ge \mathcal {C}_I^1. \end{aligned}$$

Then

$$\begin{aligned}&|c_1-I|^\beta + |c_4-I|^\beta \ge |c_1-I|^\beta \ge 2 |c_2-I|^\beta \\&\quad > |c_2-I|^\beta + |c_3-I|^\beta \;\; \forall \beta \ge \mathcal {C}_I^1 \end{aligned}$$

If \(I<c_2\), then

$$\begin{aligned} |c_4 - I| > |c_3 - I| > |c_2 -I| \end{aligned}$$

and thus the same argument can be used to show there exists \(\mathcal {C}_I^2 \in \mathbb {N}\) such that

$$\begin{aligned} |c_4-I|^\beta + |c_1-I|^\beta > |c_2-I|^\beta + |c_3-I|^\beta , \;\;\; \forall \beta \ge \mathcal {C}_I^2. \end{aligned}$$

In case \(c_2 \le I \le c_3\), the existence of such a \(\mathcal {C}\) was proved in Prop 4, e.g. \(\mathcal {C} = 1\).

Therefore the condition (64) is satisfied for any \(I \in [0,L]\) by choosing \(\beta \ge \mathcal {C} = \max _{I \in [0,L]} \max \{ \mathcal {C}^1_I,\mathcal {C}^2_I \}\). \(\square \)

1.4 Appendix 1.4 Proof of Prop 5

We will show the condition holds for \(\beta = 1\), which by Prop 2 implies it holds for all \(\beta \ge 1\). Observe that if \(c_1,c_2\) and \(c_3,c_4\) are equidistant it follows that \(c_1 + c_4 = c_2 + c_3\). For \(I < c_2\)

$$\begin{aligned}&|I-c_2| + |I-c_3| = (c_2 - I) + (c_3 - I) \\&\quad = -2I + (c_2 + c_3)\\&\quad = -2I + (c_1 + c_4) = (c_1 - I) + (c_4 - I)\\&\quad \le |I-c_1| + |I-c_4|. \end{aligned}$$

For \(I \ge c_3\)

$$\begin{aligned}&|I-c_2| + |I-c_3| = (I - c_2) + (I - c_3)\\&\quad = 2I - (c_2 + c_3)\\&\quad = 2I - (c_1 + c_4) = (I - c_1) + (I - c_4) \\&\quad \le |I-c_1| + |I-c_4|. \end{aligned}$$

Appendix 2: Relation Between Dual Problem (63) and Discrete Max-flow Problem

Recall that for each \(p \in \mathcal {P}\) the data term was represented by the six edges (31). The flow on each of these edges are constrained by the capacities defined in (35) and (34), i.e.

$$\begin{aligned} P_s^1(p) \le c(s,v_{p,1})&\quad \text {flow} \le \; \text {capacity on}\; (s,v_{p,1}) \nonumber \\ P_s^2(p) \le c(s,v_{p,2})&\quad \text {flow} \le \; \text {capacity on} \;(s,v_{p,2}) \nonumber \\ P_t^1(p) \le c(v_{p,1},t)&\quad \text {flow} \le \; \text {capacity on}\; (v_{p,1},t) \nonumber \\ P_t^2(p) \le c(v_{p,2},t)&\quad \text {flow} \le \; \text {capacity on}\; (v_{p,2},t) \nonumber \\ \tilde{P}^{12}(p) \le c(v_{p,1},v_{p,2})&\quad \text {flow} \le \; \text {capacity on}\; (v_{p,1},v_{p,2}) \nonumber \\ \tilde{P}^{21}(p) \le c(v_{p,2},v_{p,1})&\quad \text {flow} \le \; \text {capacity on}\; (v_{p,2},v_{p,1}). \nonumber \end{aligned}$$

For notational convenience, we use capital letters \(P,Q\) to denote flow on discrete edges. The dual variables \(p_s^1, p_s^2, p_t^1, p_t^2, \tilde{p}^{12}, \tilde{p}^{21}\) over the continuous domain \({\varOmega }\) are inspired by these discrete flow functions. They are constrained by the capacities \(C_s^1(x), C_s^2(x), C_t^1(x), C_t^2(x), C^{12}(x)\) and \(C^{21}(x)\) respectively. Note that \(\tilde{p}^{12}\) and \(\tilde{p}^{21}\) should satisfy

$$\begin{aligned} 0 \le \tilde{p}^{12}(x) \le C^{12}(x), \;\;\; 0 \le \tilde{p}^{21}(x) \le C^{21}(x) \end{aligned}$$

for all \(x \in {\varOmega }\), but can be merged in \(p^{12} = \tilde{p}^{12} - \tilde{p}^{21}\). The above two constraints then transform into (51).

For each pair of neighboring pixels \((p,q) \in \mathcal {N}\), two edges were constructed in the discrete graph \(\mathcal {G}\): \((v_{p,1},v_{q,1})\) and \((v_{p_2},v_{q,2})\). Let \(Q^1\) denote the flow function on \((v_{p,1},v_{q,1})\) and \(Q^2\) the flow function on \((v_{p,2},v_{q,2})\). These flows are constrained by

$$\begin{aligned} 0 \le Q^1(v_{p,1},v_{q,1}) \le w_{pq}, \quad 0 \le Q^2(v_{p_2},v_{q,2}) \le w_{pq}, \; \end{aligned}$$

for all \((p,q) \in \mathcal {N}\). In the same vein, the two spatial flow fields \(q^i \in (C^\infty ({\varOmega }))^N\) \(i=1,2\) are defined in the continuous setting, and should satisfy (52). Observe that the continuous counterpart allows to measure the magnitude of \(q^1,q^2\) with 2-norm, which in turn produces rotationally invariant results.

Flow conservation should hold at each \(v_{p,1}\) and \(v_{p,2}\), i.e.

$$\begin{aligned}&\sum _{q \in \mathcal {N}_p^k} Q^1(v_{p,1},v_{q,1}) - P_s^1(p) + P_t^1(p) - \tilde{P}^{12}(p) + \tilde{P}^{21}(p) \\&\quad = 0, \;\;\; \\&\sum _{q \in \mathcal {N}_p^k} Q^2(v_{p,2},v_{q,2}) - P_s^2(p) + P_t^2(p) + \tilde{P}^{12}(p) - \tilde{P}^{21}(p) \\&\quad = 0, \, \end{aligned}$$

\(\forall p \in \mathcal {P}\). Here \(N^+(v)\) is defined as all \(w \in \mathcal {V}\) such that \((v,w) \in \mathcal {E}\) and \(N^-(v)\) is defined as all \(w \in \mathcal {V}\) such that \((w,v) \in \mathcal {E}\). The final two constraints (61) and (62) are generalizations of discrete flow conservation to the continuous setting. The total amount of discrete flow in the graph is in this case

$$\begin{aligned} \sum _{p \in \mathcal {P}} P_s^1(p) + P_s^2(p) \, . \end{aligned}$$

The objective functional (63) is a generalization of the above, and measures the total amount of flow that originates from the source.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bae, E., Tai, XC. Efficient Global Minimization Methods for Image Segmentation Models with Four Regions. J Math Imaging Vis 51, 71–97 (2015). https://doi.org/10.1007/s10851-014-0507-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-014-0507-2

Keywords

Navigation