Abstract
The Horn-Schunck (HS) optical flow method is widely employed to initialize many motion estimation algorithms. In this work, a variational Bayesian approach of the HS method is presented, where the motion vectors are considered to be spatially varying Student’s t-distributed unobserved random variables, i.e., the prior is a multivariate Student’s t-distribution, while the only observations available is the temporal and spatial image difference. The proposed model takes into account the residual resulting from the linearization of the brightness constancy constraint by Taylor series approximation, which is also assumed to be a spatially varying Student’s t-distributed observation noise. To infer the model variables and parameters we recur to variational inference methodology leading to an expectation-maximization (EM) framework with update equations analogous to the Horn-Schunck approach. This is accomplished in a principled probabilistic framework where all of the model parameters are estimated automatically from the data. Experimental results show the improvement obtained by the proposed model which may substitute the standard algorithm in the initialization of more sophisticated optical flow schemes.
Similar content being viewed by others
References
Alvarez, L., Weickert, J., Sanchez, J.: Reliable estimation of dense optical flow fields with large displacements. Int. J. Comput. Vis. 39, 41–56 (2000)
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. In: Proceedings of the International Conference of Computer Vision (ICCV), pp. 1–8 (2007)
Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)
Beal, M.J.: Variational algorithms for approximate Bayesian inference. Technical report, The Gatsby Computational Neuroscience Unit, University College, London (2003)
Beal, M.J.: The variational Bayesian EM algorithm for incomplete data with application to scoring graphical model structures. Bayesian Stat. 7, 453–464 (2003)
Ben-Ari, R., Sochen, N.: A general framework and new alignment criterion for dense optical flow. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 529–536 (2006)
Birchfield, S., Pundlik, S.: Joint tracking of features and edges. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (2008)
Bishop, C.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)
Black, M., Anandan, P.: A framework for the robust estimation of optical flow. In: Fourth International Conference on Computer Vision, pp. 231–236 (1993)
Black, M., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)
Black, M., Fleet, D.: Probabilistic detection and tracking of motion boundaries. Int. J. Comput. Vis. 38, 231–245 (2000)
Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Proceedings of the 8th European Conference on Computer Vision (ECCV), vol. 4, pp. 25–36 (2004)
Bruhn, A., Weickert, J., Schnorr, C.: Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)
Chantas, G., Galatsanos, N., Likas, A., Saunders, M.: Variational Bayesian image restoration based on a product of t-distributions image prior. IEEE Trans. Image Process. 17(10), 1795–1805 (2008)
Gkamas, T., Chantas, G., Nikou, C.: A probabilistic formulation of the optical flow problem. In: 21st International Conference on Pattern Recognition (ICPR), 11–15 November, Tsukuba, Japan (2012)
Glocker, B., Heibel, T.H., Navab, N., Kohli, P., Rother, C.: Triangleflow: optical flow with triangulation-based higher-order likelihoods. In: Proceedings of the 11th European Conference on Computer Vision Conference on Computer Vision: Part III, ECCV’10, pp. 272–285 (2010)
Heas, P., Herzet, C., Memin, E.: Robust optic-flow estimation with Bayesian inference of model and hyper-parameters. In: Scale Space and Variational Methods in Computer Vision. LNCS, vol. 6667, pp. 773–785 (2012)
Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)
Krajsek, K., Mester, A.: Bayesian inference and maximum entropy methods in science and engineering, aip conference proceedings. In: Scale Space and Variational Methods in Computer Vision. LNCS, vol. 872, pp. 311–318 (2006)
Krajsek, K., Mester, R.: A maximum likelihood estimator for choosing the regularization parameters in global optical flow methods. In: IEEE International Conference on Image Processing, pp. 1081–1084 (2006)
Krajsek, K., Mester, A.: Bayesian model selection for optical flow estimation. In: Proceedings of the 29th DAGM Conference on Pattern Recognition, Heidelberg (2007)
Lee, K.J., Kwon, D., Yun, I.D., Lee, S.U.: Optical flow estimation with adaptive convolution kernel prior on discrete framework. In: CVPR, pp. 2504–2511 (2010)
Liu, C., Rubin, D.B.: ML estimation of the t-distribution using EM and its extensions. Technical report, ECM and ECME, Statistica Sinica (1995)
Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intellegence (IJCAI), pp. 674–679 (1981)
McCane, B., Novins, K., Crannitch, D., Galvin, B.: On benchmarking optical flow. Comput. Vis. Image Underst. 84(1), 126–143 (2001)
Memin, E., Perez, P.: Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. Image Process. 7(5), 703–719 (1998)
Molina, R.: On the hierarchical Bayesian approach to image restoration. Applications to astronomical images. IEEE Trans. Pattern Anal. Mach. Intell. 16(11), 1122–1128 (1994)
Nagel, H., Enkelman, W.: An investigation of smoothness constraints for estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8(5), 565–593 (1986)
Paige, C., Saunders, M.: Solution of sparse indefinite systems of linear equations. SIAM J. Numer. Anal. 12, 617–629 (1975)
Ren, X.: Local grouping for optical flow. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (2008)
Roth, S., Black, M.: On the spatial statistics of optical flow. Int. J. Comput. Vis. 74(1), 33–50 (2007)
Sfikas, G., Nikou, C., Galatsanos, N.: Edge preserving spatially varying mixtures for image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, Alaska, USA (2008)
Sfikas, G., Nikou, C., Galatsanos, N., Heinrich, C.: Spatially varying mixtures incorporating line processes for image segmentation. J. Math. Imaging Vis. 36, 91–110 (2010)
Sfikas, G., Heinrich, C., Zallat, J., Nikou, C., Galatsanos, N.: Recovery of poliarimetric Stokes images by spatial mixture models. J. Opt. Soc. Am. A 28(3), 465–474 (2011)
Sfikas, G., Nikou, C., Galatsanos, N., Heinrich, C.: Majorization-minimization mixture model determination in image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2169–2176. Colorado Springs, Colorado (2011)
Sun, D., Roth, S., Lewis, J.P., Black, M.J.: Learning optical flow. In: Proceedings of the 10th European Conference on Computer Vision (ECCV): Part III, ECCV ’08, pp. 83–97 (2008)
Tzikas, D., Likas, A., Galatsanos, N.: The variational approximation for Bayesian inference. IEEE Signal Process. Mag. 25(6), 131–146 (2008)
Wedel, A., Cremers, D., Pock, T., Bischof, H.: Structure- and motion-adaptive regularization for high accuracy optic flow. In: International Conference on Computer Vision (ICCV), pp. 1663–1668 (2009)
Weickert, J., Schnörr, C.: A theoretical framework for convex regularizers in PDE based computation of image motion. Int. J. Comput. Vis. 45(3), 245–264 (2001)
Weickert, J., Schnörr, C.: Variational optical flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vis. 14(3), 245–255 (2001)
Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: CVPR, pp. 2464–2471 (2010)
Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)
Zhou, Z., Leahy, R.M., Qi, J.: Approximate maximum likelihood hyperparameter estimation for Gibbs priors. IEEE Trans. Image Process. 6(6), 844–861 (1997)
Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93(3), 368–388 (2011)
Zitnick, C., Jojic, N., Kang, S.: Consistent segmentation for optical flow estimation. In: Proceedings of the International Conference on Computer Vision (ICCV), vol. 2, pp. 1308–1315 (2005)
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
In what follows we present in detail the derivation of the update equations for the model variables and parameters.
In the fully Bayesian framework, the complete data likelihood, including the hidden variables and the parameters of the model, is given by:
where θ=[λ noise ,λ x ,λ y ,μ,ν x ,ν y ] gathers the parameters of the model. Estimation of the model parameters could be obtained through maximization of the marginal distribution of the observations p(d;θ):
However, in the present case, this marginalization is not possible, since the posterior of the latent variables given the observations \(p(\mathbf{u}, \tilde{\mathbf{A}}, \mathbf{b}|\mathbf{d})\) is not known explicitly and inference via the Expectation-Maximization (EM) algorithm may not be obtained. Thus, we resort to the variational methodology [5, 8] where we have to maximize a lower bound of \(p (\mathbf{u}, \tilde{\mathbf{A}}, \mathbf{b} )\):
This involves finding approximations of the posterior distribution of the hidden variables, denoted by q(u), \(q (\tilde{\mathbf{A}} )\), q(b) because there is no analytical form of the auxiliary function q for which the bound in (57) becomes equality. However, in the variational methodology, we employ the mean field approximation [8]:
and (57) becomes:
In our case, in the E-step of the variational algorithm (VE-step), optimization of the functional \(L (\mathbf{u}, \tilde{\mathbf{A}}, \mathbf{b}; \theta )\) is performed with respect to the auxiliary functions. Following the variational inference framework, the distributions q(u k ), k∈{x, y}, are Normal:
yielding
and
Therefore, this bound is actually a function of the parameters R k and m k , k∈{x,y} and a functional with respect to the auxiliary functions q(a k ),q(b). Using (58), the variational bound in our problem becomes:
where we have separated the parameters into two sets:
and
Thus, in the VE-step of the variational EM algorithm the bound must be optimized with respect to R k , m k , q(a k ) and q(b).
Taking the derivative of (63) with respect to m k , R k , q(α k ) and q(b) and setting the result equal to zero, we obtain the following update equations:
and
where
and
Notice that the final estimates for u x , u y are m x and m y , in (37) and (38), respectively.
After some manipulation, we obtain the update equations for the model parameters which maximize (63) with respect to q(a k ), q(b). The form of all q approximating-to-the-posterior functions will remain the same as the corresponding prior (due to the conjugate priors we employ) namely q(a k ), q(b) which approximate p(a k |u k ,λ k ,C k ;ν k ), p(b|u,λ noise ,F;μ) will follow Gamma distributions, ∀i=1,…,N,∀k∈{x,y}:
and
where the N×N matrix
the N×N matrix
and [⋅] i denotes the i-th element of the vector inside the brackets.
The size of matrices R x , R y and consequently C x , C y and F makes their direct calculation prohibitive. In order to overcome this difficulty, we employ the iterative Lanczos method [29] for their calculation. For matrices C x , C y and F only the diagonal elements are needed in (70) and (71) and they are obtained as a byproduct of the Lanczos method.
Note that as we can see from (66) and (67), there is a dependency between u x and u y , as it is the case in the standard Horn-Schunck method.
Notice also that since each q (t+1)(α k (i)) is a Gamma pdf, it is easy to derive its expected value:
and the same stands for the expected value of b(i):
where 〈.〉 q(.) denotes the expectation with respect to an arbitrary distribution q(⋅). These estimates are used in (66), (67), (68) and (69), where \(\hat{\mathbf{A}}_{k}^{(t)}\) and \(\hat{\mathbf{B}}^{(t)}\) are diagonal matrices with elements:
and
for i=1,…,N.
At the variational M-step, the bound is maximized with respect to the model parameters:
where
is calculated using the results from (66)–(69).
The update for λ noise is obtained after taking the derivative of \(L (q^{(t+1)} (\mathbf{u}_{k} ), q^{(t+1)} (\hat{\mathbf{A}}_{k} ), q^{(t+1)} (\hat{\mathbf{B}} ), \theta_{1}^{(t+1)}, \theta_{2} )\) in (63) with respect to it and setting it to zero:
By the same means we obtain the estimates for λ x and λ y :
with k∈{x,y}.
The degrees of freedom parameters ν k of the Student’s t-distributions are also computed accordingly through the roots of the following equation:
for ν k , k∈{x,y}, where Ϝ(x) is the digamma function (derivative of the logarithm of the Gamma function) and \(\nu_{k}^{(t)}\) is the value of ν k at the previous iteration.
Finally, by the same procedure we obtain estimates for the parameter μ of the noise distribution:
In our implementation Eqs. (79) and (80) are solved by the bisection method, as also proposed in [23].
Rights and permissions
About this article
Cite this article
Chantas, G., Gkamas, T. & Nikou, C. Variational-Bayes Optical Flow. J Math Imaging Vis 50, 199–213 (2014). https://doi.org/10.1007/s10851-014-0494-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10851-014-0494-3