Abstract
The Enhanced Video Coding (EVC) workgroup of the Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) organization aims at enhancing traditional video codecs by improving or replacing traditional encoding tools with AI-based counterparts. In this work, we explore enhancing MPEG Essential Video Coding (EVC) intra prediction with a learnable predictor: we recast the problem as a hole inpainting task that we tackle via masked convolutions. Our experiments in standard test conditions show BD-rate reductions in excess of 6% over the EVC baseline profile reference with some sequences in excess of 12%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ballé, J., Laparra, V., Simoncelli, E.P.: End-to-end optimized image compression. In: International Conference on Learning Representations (ICLR), Toulon, France (2017)
Ballé, J., Minnen, D., Singh, S., Hwang, S.J., Johnston, N.: Variational image compression with a scale hyperprior. In: International Conference on Learning Representations (ICLR), Vancouver, CA (2018)
Choi, K., Chen, J., Rusanovskyy, D., Choi, K.P., Jang, E.S.: An overview of the MPEG-5 essential video coding standard [standards in a nutshell]. IEEE Signal Process. Mag. 37(3), 160–167 (2020)
CISCO: Global 2021 forecast highlights (2021). https://www.cisco.com/c/dam/m/en_us/solutions/service-provider/vni-forecast-highlights/pdf/Global_2021_Forecast_Highlights.pdf
Dumas, T., Galpin, F., Bordes, P.: Iterative training of neural networks for intra prediction. IEEE Trans. Image Process. 30, 697–711 (2020)
Dumas, T., Roumy, A., Guillemot, C.: Context-adaptive neural network-based prediction for image compression. IEEE Trans. Image Process. 29, 679–693 (2019)
Goodfellow, I., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT Press, Cambridge (2016)
Helle, P., et al.: Intra picture prediction for video coding with neural networks. In: 2019 Data Compression Conference (DCC), pp. 448–457. IEEE (2019)
Hu, Y., Yang, W., Li, M., Liu, J.: Progressive spatial recurrent neural network for intra prediction. IEEE Trans. Multimedia 21(12), 3024–3037 (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: International Conference on Learning Representations (ICLR), Banff, CA (2014)
Li, W., Lin, Z., Zhou, K., Qi, L., Wang, Y., Jia, J.: Mat: Mask-aware transformer for large hole image inpainting. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10748–10758 (2022). https://doi.org/10.1109/CVPR52688.2022.01049
Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6
MPEG: JVET common test conditions and evaluation procedures for neural network-based video coding technology. In: Output Document of JVET (2023)
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting (2016)
Rippel, O., Bourdev, L.: Real-time adaptive image compression. arXiv preprint: arXiv:1705.05823 (2017)
Samuelsson, J., Choi, K., Chen, J., Rusanovskyy, D.: MPEG-5 EVC. In: SMPTE 2019, pp. 1–11. SMPTE (2019)
Schwarz, K., Wieschollek, P., Lensch, H.P.: Will people like your image? Learning the aesthetic space. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2048–2057. IEEE (2018)
Sze, V., Budagavi, M., Sullivan, G.J.: High Efficiency Video Coding (HEVC). Integrated Circuit and Systems, Algorithms and Architectures, vol. 39, p. 40. Springer, Cham (2014)
Toderici, G., et al.: Full resolution image compression with recurrent neural networks. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5435–5443, Honolulu, Hawaii, USA (2017)
Valenzise, G., Purica, A., Hulusic, V., Cagnazzo, M.: Quality assessment of deep-learning-based image compression. In: Multimedia Signal Processing, Vancouver, Canada (2018). https://www.hal.archives-ouvertes.fr/hal-01819588
Wan, Z., Zhang, J., Chen, D., Liao, J.: High-fidelity pluralistic image completion with transformers. CoRR abs/2103.14031 (2021). arxiv:2103.14031
Wang, L., Fiandrotti, A., Purica, A., Valenzise, G., Cagnazzo, M.: Enhancing HEVC spatial prediction by context-based learning. In: ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4035–4039. IEEE (2019)
Yu, Y., et al.: Diverse image inpainting with bidirectional and autoregressive transformers. CoRR abs/2104.12335 (2021). arxiv.org/abs/2104.12335
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Spadaro, G., Iacoviello, R., Mosca, A., Valenzise, G., Fiandrotti, A. (2023). A Learnable EVC Intra Predictor Using Masked Convolutions. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_45
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)