Abstract
This paper tackles the simultaneous optimization of pose and Neural Radiance Fields (NeRF). Departing from the conventional practice of using explicit global representations for camera pose, we propose a novel overparameterized representation that models camera poses as learnable rigid warp functions. We establish that modeling the rigid warps must be tightly coupled with constraints and regularization imposed. Specifically, we highlight the critical importance of enforcing invertibility when learning rigid warp functions via neural network and propose the use of an Invertible Neural Network (INN) coupled with a geometry-informed constraint for this purpose. We present results on synthetic and real-world datasets, and demonstrate that our approach outperforms existing baselines in terms of pose estimation and high-fidelity reconstruction due to enhanced optimization convergence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Our use of invertibility strictly adheres to the well-established mathematical definition. Let f be a function whose domain is \(\mathcal {X}\) and codomain is \(\mathcal {Y}\). f is invertible iff there exists a function g from \(\mathcal {Y}\) to \(\mathcal {X}\) such that \(g(f(x))=x\) \(\,\forall x \in \mathcal {X}\) and \(f(g(y))=y\) \(\,\forall y \in \mathcal {Y}\) [14]. We use bijective and invertible interchangeably throughout our paper.
- 2.
This can be succinctly written as \(\textbf{r}^{(C)}(z) = z_{i,u} \textbf{d}\) as \(\textbf{o}^{(C)}\) is \([0,0,0]^{T}\) in camera coordinate space.
- 3.
- 4.
- 5.
For our proposed method, we evaluate the estimated global poses Eq. (5).
References
Allen-Zhu, Z., Li, Y., Liang, Y.: Learning and generalization in overparameterized neural networks, going beyond two layers. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Arora, S., Du, S., Hu, W., Li, Z., Wang, R.: Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In: International Conference on Machine Learning, pp. 322–332. PMLR (2019)
Behrmann, J., Grathwohl, W., Chen, R.T., Duvenaud, D., Jacobsen, J.H.: Invertible residual networks. In: International Conference on Machine Learning, pp. 573–582. PMLR (2019)
Bian, J.W., Bian, W., Prisacariu, V.A., Torr, P.: PoRF: pose residual field for accurate neural surface reconstruction. arXiv preprint arXiv:2310.07449 (2023)
Bian, W., Wang, Z., Li, K., Bian, J.W., Prisacariu, V.A.: NoPe-NeRF: optimising neural radiance field with no pose prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4160–4169 (2023)
Cai, H., Feng, W., Feng, X., Wang, Y., Zhang, J.: Neural surface reconstruction of dynamic scenes with monocular RGB-D camera. In: Advances in Neural Information Processing Systems, vol. 35, pp. 967–981 (2022)
Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Chen, Y., Lee, G.H.: DBARF: deep bundle-adjusting generalizable neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24–34 (2023)
Chen, Y., et al.: Local-to-global registration for bundle-adjusting neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8264–8273 (2023)
Chng, S.F., Ramasinghe, S., Sherrah, J., Lucey, S.: Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) European Conference on Computer Vision, pp. 264–280. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_16
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. arXiv preprint arXiv:1605.08803 (2016)
Guo, M., Fathi, A., Wu, J., Funkhouser, T.: Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503 (2020)
Jeffreys, H., Jeffreys, B.: Methods of Mathematical Physics. Cambridge University Press (1999)
Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 406–413. IEEE (2014)
Jeong, Y., Ahn, S., Choy, C., Anandkumar, A., Cho, M., Park, J.: Self-calibrating neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5846–5854 (2021)
Jiang, C., Huang, J., Tagliasacchi, A., Guibas, L.J.: ShapeFlow: learnable deformation flows among 3D shapes. In: Advances in Neural Information Processing Systems, vol. 33, pp. 9745–9757 (2020)
Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: LERF: language embedded radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19729–19739 (2023)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lei, J., Daniilidis, K.: Cadex: learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6624–6634 (2022)
Levoy, M.: Efficient ray tracing of volume data. ACM Trans. Graph. (TOG) 9(3), 245–261 (1990)
Li, Y., Liang, Y.: Learning overparameterized neural networks via stochastic gradient descent on structured data. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: BARF: bundle-adjusting neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5741–5751 (2021)
Mildenhall, B., et al.: Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (TOG) 38(4), 1–14 (2019)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)
Neyshabur, B., Tomioka, R., Srebro, N.: Norm-based capacity control in neural networks. In: Conference on Learning Theory, pp. 1376–1401. PMLR (2015)
Nguyen, Q.N., Mondelli, M.: Global convergence of deep networks with one wide layer followed by pyramidal topology. In: Advances in Neural Information Processing Systems, vol. 33, pp. 11961–11972 (2020)
Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Occupancy flow: 4D reconstruction by learning particle dynamics. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5379–5389 (2019)
Oymak, S., Soltanolkotabi, M.: Toward moderate overparameterization: global convergence guarantees for training shallow neural networks. IEEE J. Sel. Areas Inf. Theory 1(1), 84–105 (2020)
Park, K., Henzler, P., Mildenhall, B., Barron, J.T., Martin-Brualla, R.: Camp: camera preconditioning for neural radiance fields. ACM Trans. Graph. (TOG) 42(6), 1–11 (2023)
Park, K., et al.: Nerfies: deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865–5874 (2021)
Parra, A., Chng, S.F., Chin, T.J., Eriksson, A., Reid, I.: Rotation coordinate descent for fast globally optimal rotation averaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4298–4307 (2021)
Paschalidou, D., Katharopoulos, A., Geiger, A., Fidler, S.: Neural parts: learning expressive 3D shape abstractions with invertible neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3204–3215 (2021)
Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Tancik, M., et al.: Nerfstudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–12 (2023)
Truong, P., Rakotosaona, M.J., Manhardt, F., Tombari, F.: SPARF: neural radiance fields from sparse and noisy poses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4190–4200 (2023)
Wang, C., MacDonald, L.E., Jeni, L.A., Lucey, S.: Flow supervision for deformable NeRF. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21128–21137 (2023)
Wang, Q., et al.: Tracking everything everywhere all at once. arXiv preprint arXiv:2306.05422 (2023)
Wang, Z., Wu, S., Xie, W., Chen, M., Prisacariu, V.A.: NeRF–: neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 (2021)
Wu, T., et al.: Voxurf: voxel-based efficient and accurate neural surface reconstruction. arXiv preprint arXiv:2208.12697 (2022)
Xia, Y., Tang, H., Timofte, R., Van Gool, L.: SiNeRF: sinusoidal neural radiance fields for joint pose estimation and scene reconstruction. arXiv preprint arXiv:2210.04553 (2022)
Xu, C., et al.: NeRF-Det: learning geometry-aware volumetric representation for multi-view 3D object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23320–23330 (2023)
Xu, Q., et al.: Point-NeRF: point-based neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5438–5448 (2022)
Yang, G., Belongie, S., Hariharan, B., Koltun, V.: Geometry processing with neural fields. In: Advances in Neural Information Processing Systems, vol. 34, pp. 22483–22497 (2021)
Zhan, H., Zheng, J., Xu, Y., Reid, I., Rezatofighi, H.: ActiveRMAP: radiance field for active mapping and planning. arXiv preprint arXiv:2211.12656 (2022)
Zhao, F., et al.: HumanNeRF: efficiently generated human radiance field from sparse inputs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7743–7753 (2022)
Zhu, Z., et al.: Nice-slam: neural implicit scalable encoding for slam. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12786–12796 (2022)
Acknowledgement
We thank Chee-Kheng (CK) Chng for insightful discussions and technical feedback.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chng, SF., Garg, R., Saratchandran, H., Lucey, S. (2025). Invertible Neural Warp for NeRF. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15075. Springer, Cham. https://doi.org/10.1007/978-3-031-72643-9_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-72643-9_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72642-2
Online ISBN: 978-3-031-72643-9
eBook Packages: Computer ScienceComputer Science (R0)