research-article

StableVQA: A Deep No-Reference Quality Assessment Model for Video Stability

Authors:

Ning LiuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 1066 - 1076

https://doi.org/10.1145/3581783.3611860

Published: 27 October 2023 Publication History

Abstract

Video shakiness is an unpleasant distortion of User Generated Content (UGC) videos, which is usually caused by the unstable hold of cameras. In recent years, many video stabilization algorithms have been proposed, yet no specific and accurate metric enables comprehensively evaluating the stability of videos. Indeed, most existing quality assessment models evaluate video quality as a whole without specifically taking the subjective experience of video stability into consideration. Therefore, these models cannot measure the video stability explicitly and precisely when severe shakes are present. In addition, there is no large-scale video database in public that includes various degrees of shaky videos with the corresponding subjective scores available, which hinders the development of Video Quality Assessment for Stability (VQA-S). To this end, we build a new database named StableDB that contains 1,952 diversely-shaky UGC videos, where each video has a Mean Opinion Score (MOS) on the degree of video stability rated by 34 subjects. Moreover, we elaborately design a novel VQA-S model named StableVQA, which consists of three feature extractors to acquire the optical flow, semantic, and blur features respectively, and a regression layer to predict the final stability score. Extensive experiments demonstrate that the StableVQA achieves a higher correlation with subjective opinions than the existing VQA-S models and generic VQA models. The database and codes are available at https://github.com/QMME/StableVQA.

References

[1]

Sebastiano Battiato, Giovanni Gallo, Giovanni Puglisi, and Salvatore Scellato. 2007. SIFT features tracking for video stabilization. In 14th international conference on image analysis and processing (ICIAP 2007). IEEE, 825--830.

Digital Library

[2]

Jinsoo Choi and In So Kweon. 2020. Deep iterative frame interpolation for full-frame video stabilization. ACM Transactions on Graphics, Vol. 39, 1 (2020), 1--9.

Digital Library

[3]

Yunlong Dong, Xiaohong Liu, Yixuan Gao, Xunchu Zhou, Tao Tan, and Guangtao Zhai. 2023. Light-VQA: A Multi-Dimensional Quality Assessment Model for Low-Light Video Enhancement. In Proceedings of the 31st ACM International Conference on Multimedia.

Digital Library

[4]

Yixuan Gao, Yuqin Cao, Tengchuan Kou, Wei Sun, Yunlong Dong, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai. 2023. VDPVE: VQA Dataset for Perceptual Video Enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1474--1483.

[5]

Deepti Ghadiyaram, Janice Pan, Alan C Bovik, Anush Krishna Moorthy, Prasanjit Panda, and Kai-Chieh Yang. 2017. In-capture mobile video distortions: A study of subjective behavior and objective algorithms. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 28, 9 (2017), 2061--2077.

[6]

Video Quality Experts Group et al. 2000. Final report from the video quality experts group on the validation of objective models of video quality assessment. In VQEG meeting, Ottawa, Canada, March, 2000.

[7]

Wilko Guilluy, Laurent Oudre, and Azeddine Beghdadi. 2021. Video stabilization: Overview, challenges and perspectives. Signal Processing: Image Communication, Vol. 90 (2021), 116015.

[8]

Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li, and Dietmar Saupe. 2017. The Konstanz natural video database (KoNViD-1k). In Proceedings of the 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1--6.

[9]

Jerin Geo James, Devansh Jain, and Ajit Rajwade. 2023. GlobalFlowNet: Video stabilization using deep distilled global motion estimates. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5078--5087.

[10]

Yeong Jun Koh, Chulwoo Lee, and Chang-Su Kim. 2015. Video stabilization based on feature trajectory augmentation and selection and robust mesh grid warping. IEEE Transactions on Image Processing, Vol. 24, 12 (2015), 5260--5273.

Digital Library

[11]

Jari Korhonen. 2019. Two-level approach for no-reference consumer video quality assessment. IEEE Transactions on Image Processing, Vol. 28, 12 (2019), 5923--5938.

[12]

Bowen Li, Weixia Zhang, Meng Tian, Guangtao Zhai, and Xianpei Wang. 2022. Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception. Transactions on Circuits and Systems for Video Technology, Vol. 32, 9 (2022), 5944--5958.

[13]

Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. 2023. AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment. arXiv preprint arXiv:2306.04717 (2023).

[14]

Dingquan Li, Tingting Jiang, and Ming Jiang. 2019. Quality assessment of in-the-wild videos. In Proceedings of the 27th ACM International Conference on Multimedia. 2351--2359.

Digital Library

[15]

Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun. 2013. Bundled camera paths for video stabilization. ACM transactions on graphics (TOG), Vol. 32, 4 (2013), 1--10.

[16]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012--10022.

[17]

Wei Lu, Wei Sun, Xiongkuo Min, Wenhan Zhu, Quan Zhou, Jun He, Qiyuan Wang, Zicheng Zhang, Tao Wang, and Guangtao Zhai. 2022. Deep Neural Network for Blind Visual Quality Assessment of 4K Content. IEEE Trans. Broadcast. (2022).

[18]

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012a. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing, Vol. 21, 12 (2012), 4695--4708.

Digital Library

[19]

Anish Mittal, Michele A Saad, and Alan C Bovik. 2015. A completely blind video integrity oracle. IEEE Transactions on Image Processing, Vol. 25, 1 (2015), 289--300.

Digital Library

[20]

Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012b. Making a ?completely blind" image quality analyzer. IEEE Signal processing letters, Vol. 20, 3 (2012), 209--212.

[21]

Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16. Springer, 184--201.

Digital Library

[22]

Luca Rossetto, Heiko Schuldt, George Awad, and Asad A Butt. 2019. V3C-a research video collection. In MultiMedia Modeling: 25th International Conference, MMM 2019, Thessaloniki, Greece, January 8-11, 2019, Proceedings, Part I 25. Springer, 349--360.

[23]

Wei Sun, Xiongkuo Min, Wei Lu, and Guangtao Zhai. 2022. A deep learning based no-reference quality assessment model for ugc videos. In Proceedings of the 30th ACM International Conference on Multimedia. 856--865.

Digital Library

[24]

Wei Sun, Xiongkuo Min, Guangtao Zhai, Ke Gu, Huiyu Duan, and Siwei Ma. 2019. MC360IQA: A multi-channel CNN for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing, Vol. 14, 1 (2019), 64--77.

[25]

Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer, 402--419.

[26]

Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin. 2022. Stripformer: Strip transformer for fast image deblurring. In Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX. Springer, 146--162.

[27]

Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C Bovik. 2021. UGC-VQA: Benchmarking blind video quality assessment for user generated content. IEEE Transactions on Image Processing, Vol. 30 (2021), 4449--4464.

[28]

Miao Wang, Guo-Ye Yang, Jin-Kun Lin, Song-Hai Zhang, Ariel Shamir, Shao-Ping Lu, and Shi-Min Hu. 2018. Deep online video stabilization with multi-grid warping transformation learning. IEEE Transactions on Image Processing, Vol. 28, 5 (2018), 2283--2292.

[29]

Yilin Wang, Sasi Inguva, and Balu Adsumilli. 2019. YouTube UGC dataset for video compression research. In Proceedings of the 2019 IEEE 21st International Workshop on Multimedia Signal Processing. IEEE, 1--5.

[30]

Zhongqiang Wang and Hua Huang. 2016. Pixel-wise video stabilization. Multimedia Tools and Applications, Vol. 75 (2016), 15939--15954.

Digital Library

[31]

Haoning Wu, Chaofeng Chen, Jingwen Hou, Liang Liao, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2022. Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling. In Proceedings of the Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI. Springer, 538--554.

Digital Library

[32]

Yufei Xu, Jing Zhang, Stephen J Maybank, and Dacheng Tao. 2022. DUT: learning video stabilization by simply watching unstable videos. IEEE Transactions on Image Processing, Vol. 31 (2022), 4306--4320.

Digital Library

[33]

Peng Ye, Jayant Kumar, Le Kang, and David Doermann. 2012. Unsupervised feature learning framework for no-reference image quality assessment. In 2012 IEEE conference on computer vision and pattern recognition. IEEE, 1098--1105.

[34]

Zhenqiang Ying, Deepti Ghadiyaram, and Alan Bovik. 2022. Telepresence Video Quality Assessment. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII. Springer, 327--347.

[35]

Jiyang Yu and Ravi Ramamoorthi. 2018. Selfie video stabilization. In Proceedings of the European Conference on Computer Vision (ECCV). 551--566.

Digital Library

[36]

Jiyang Yu and Ravi Ramamoorthi. 2020. Learning video stabilization using optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8159--8167.

[37]

Fang-Lue Zhang, Jue Wang, Han Zhao, Ralph R Martin, and Shi-Min Hu. 2015. Simultaneous camera path optimization and distraction removal for improving amateur video. IEEE Transactions on Image Processing, Vol. 24, 12 (2015), 5982--5994.

Digital Library

[38]

Lei Zhang, Qing-Zhuo Zheng, and Hua Huang. 2018. Intrinsic motion stability assessment for video stabilization. IEEE transactions on visualization and computer graphics, Vol. 25, 4 (2018), 1681--1692.

[39]

Zicheng Zhang, Chunyi Li, Wei Sun, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai. 2023 a. A Perceptual Quality Assessment Exploration for AIGC Images. arXiv preprint arXiv:2303.12618 (2023).

[40]

Zicheng Zhang, Wei Sun, Yingjie Zhou, Haoning Wu, Chunyi Li, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. 2023 b. Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluation. arXiv preprint arXiv:2307.02808 (2023).

[41]

Zicheng Zhang, Wei Wu, Wei Sun, Dangyang Tu, Wei Lu, Xiongkuo Min, Ying Chen, and Guangtao Zhai. 2023 c. MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos. arXiv preprint arXiv:2303.14933 (2023).

[42]

Minda Zhao and Qiang Ling. 2020. Pwstablenet: Learning pixel-wise warping maps for video stabilization. IEEE Transactions on Image Processing, Vol. 29 (2020), 3582--3595.

Digital Library

Cited By

Kou TLiu XZhang ZLi CWu HMin XZhai GLiu NCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Subjective-Aligned Dataset and Metric for Text-to-Video Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680868(7793-7802)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680868
Xiang JDang YChen PLiang RHuan RGao NCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Semantic-Aware and Quality-Aware Interaction Network for Blind Video Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680598(9970-9979)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680598
Franz GWegner DWiehn MKeßler S(2024)Evaluation of video stabilization metrics for the assessment of camera vibrationsInfrared Imaging Systems: Design, Analysis, Modeling, and Testing XXXV10.1117/12.3013654(12)Online publication date: 7-Jun-2024
https://doi.org/10.1117/12.3013654
Show More Cited By

Index Terms

StableVQA: A Deep No-Reference Quality Assessment Model for Video Stability
1. Computing methodologies
  1. Modeling and simulation
    1. Model development and analysis
      1. Modeling methodologies

Recommendations

A Deep Learning based No-reference Quality Assessment Model for UGC Videos
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Quality assessment for User Generated Content (UGC) videos plays an important role in ensuring the viewing experience of end-users. Previous UGC video quality assessment (VQA) studies either use the image recognition model or the image quality ...
Neural network solution for a real-time no-reference video quality assessment of H.264/AVC video bitstreams
Abstract
The ever-growing video streaming services require accurate quality assessment with often no reference to the original media. One primary challenge in developing no-reference (NR) video quality metrics is achieving real-timeliness while retaining ...
Reduced-Reference Video Quality Assessment Using Discriminative Local Harmonic Strength With Motion Consideration

This paper presents a reduced-reference objective picture quality measurement tool of compressed video. We have used a discriminative analysis of harmonic strength computed from edge-detected pictures to create harmonics gain and loss information that ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Shanghai Pujiang Program
China Postdoctoral Science Foundation

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
178
Total Downloads

Downloads (Last 12 months)162
Downloads (Last 6 weeks)12

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kou TLiu XZhang ZLi CWu HMin XZhai GLiu NCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Subjective-Aligned Dataset and Metric for Text-to-Video Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680868(7793-7802)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680868
Xiang JDang YChen PLiang RHuan RGao NCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Semantic-Aware and Quality-Aware Interaction Network for Blind Video Quality AssessmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680598(9970-9979)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680598
Franz GWegner DWiehn MKeßler S(2024)Evaluation of video stabilization metrics for the assessment of camera vibrationsInfrared Imaging Systems: Design, Analysis, Modeling, and Testing XXXV10.1117/12.3013654(12)Online publication date: 7-Jun-2024
https://doi.org/10.1117/12.3013654
Li CZhang ZWu HZhang KBai LLiu XZhai GLin W(2024)PAPS-OVQA: Projection-Aware Patch Sampling for Omnidirectional Video Quality Assessment2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558283(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558283
Li CWu HZhang ZHao HZhang KBai LLiu XMin XLin WZhai G(2024)Q-Refine: A Perceptual Quality Refiner for AI-Generated Image2024 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME57554.2024.10687390(1-6)Online publication date: 15-Jul-2024
https://doi.org/10.1109/ICME57554.2024.10687390
Zhou YZhang ZSun WLiu XMin XWang ZZhang XZhai G(2024)Thqa: A Perceptual Quality Assessment Database for Talking Heads2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10647507(15-21)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10647507
Qu BLiang XSun SGao W(2024)Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00659(6652-6660)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00659
Li XYuan KPei YLu YSun MZhou CChen ZTimofte RSun WWu HZhang ZJia JZhang ZCao LChen QMin XLin WZhai GSun JWang TLi LKong HWang WLi BLuo CWang HChen XMeng WPan XShi HZhu HXu XSun LChen ZLiu SKong FFan HXu YXu HYang MZhou JLi JWen SXu MLi DYao SDu JZuo WLi ZHe SMing AFu HMa HWu YXue FZhao GDu LGuo JZhang YZheng HChen JLiu YZhou DXu KXu QSun TDing ZHu Y(2024)NTIRE 2024 Challenge on Short-form UGC Video Quality Assessment: Methods and Results2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00643(6415-6431)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00643
Liu XMin XZhai GLi CKou TSun WWu HGao YCao YZhang ZWu XTimofte RPeng FFu HMing AWang CMa HHe SDou ZChen SZhang HXie HWang CChen BZeng JYang JWang WFang XLv XYan JZhi TZhang YLi YLi YXu JLiu JLiao YLi JYu ZGuan FLu YLi XMotamednia HHosseini-Benvidi SMahmoudi-Aznaveh AMansouri AGankhuyag GYoon KXu YFan HKong FZhao SDong WYin HZhu LWang ZHuang BSaha AMishra SGupta SSureddi RSaha OCelona LBianco SNapoletano PSchettini RYang JFu JZhang WCao WLiu LPeng HYuan WLi ZCheng YDeng YLi HQu BLi YLuo SWang SGao WLu ZConde MTimofte RWang XChen ZLiao RYe YWang QLi BZhou ZGeng MChen RTao XLiang XSun SMa XLi JYang MXu HZhou JZhu SYu BChen PXu XShen JDuan ZAsadi ELiu JYan QQu YZeng XWang LLiao R(2024)NTIRE 2024 Quality Assessment of AI-Generated Content Challenge2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00637(6337-6362)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00637
Li CKou TGao YCao YSun WZhang ZZhou YZhang ZZhang WWu HLiu XMin XZhai G(2024)AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00636(6327-6336)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00636
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents