Image Cropping under Design Constraints
Article No.: 40, Pages 1 - 7
Abstract
Image cropping is essential in image editing for obtaining a compositionally enhanced image. In display media, image cropping is a prospective technique for automatically creating media content. However, image cropping for media contents is often required to satisfy various constraints, such as an aspect ratio and blank regions for placing texts or objects. We call this problem image cropping under design constraints. To achieve image cropping under design constraints, we propose a score function-based approach, which computes scores for cropped results whether aesthetically plausible and satisfies design constraints. We explore two derived approaches, a proposal-based approach, and a heatmap-based approach, and we construct a dataset for evaluating the performance of the proposed approaches on image cropping under design constraints. In experiments, we demonstrate that the proposed approaches outperform a baseline, and we observe that the proposal-based approach is better than the heatmap-based approach under the same computation cost, but the heatmap-based approach leads to better scores by increasing computation cost. The experimental results indicate that balancing aesthetically plausible regions and satisfying design constraints is not a trivial problem and requires sensitive balance, and both proposed approaches are reasonable alternatives.
Supplementary Material
Appendix (Supplemental_NSS_ACMMMAsia-63.pdf)
- Download
- 5.83 MB
References
[1]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
[2]
Edoardo Ardizzone, Alessandro Bruno, and Giuseppe Mazzola. 2013. Saliency based image cropping. In Proc. International Conference on Image Analysis and Processing (ICIAP). Springer, 773–782.
[3]
Huarong Chen, Bin Wang, Tianxiang Pan, Liwang Zhou, and Hua Zeng. 2018. CropNet: Real-time thumbnailing. In Proc. ACM International Conference on Multimedia (ACMMM). 81–89.
[4]
Yi-Ling Chen, Tzu-Wei Huang, Kai-Han Chang, Yu-Chen Tsai, Hwann-Tzong Chen, and Bing-Yu Chen. 2017. Quantitative Analysis of Automatic Image Cropping Algorithms: A Dataset and Comparative Study. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV). 226–234.
[5]
Yi-Ling Chen, Jan Klopp, Min Sun, Shao-Yi Chien, and Kwan-Liu Ma. 2017. Learning to Compose with Professional Photographs on the Web. In Proc. ACM Conference on Multimedia (ACMMM). 37–45.
[6]
Bin Cheng, Bingbing Ni, Shuicheng Yan, and Qi Tian. 2010. Learning to photograph. In Proc. ACM International Conference on Multimedia (ACMMM). 291–300.
[7]
Seyed A Esmaeili, Bharat Singh, and Larry S Davis. 2017. Fast-At: Fast Automatic Thumbnail Generation Using Deep Neural Networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4622–4630.
[8]
Chen Fang, Zhe Lin, Radomir Mech, and Xiaohui Shen. 2014. Automatic Image Cropping Using Visual Composition, Boundary Simplicity and Content Preservation Models. In Proc. ACM International Conference on Multimedia (ACMMM). 1105–1108.
[9]
Chaoyi Hong, Shuaiyuan Du, Ke Xian, Hao Lu, Zhiguo Cao, and Weicai Zhong. 2021. Composing Photos Like a Photographer. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7057–7066.
[10]
Nora Horanyi, Kedi Xia, Kwang Moo Yi, Abhishake Kumar Bojja, Aleš Leonardis, and Hyung Jin Chang. 2022. Repurposing existing deep networks for caption and aesthetic-guided image cropping. Pattern Recognition 126 (2022), 108485.
[11]
Jingwei Huang, Huarong Chen, Bin Wang, and Stephen Lin. 2015. Automatic Thumbnail Generation Based on Visual Representativeness and Foreground Recognizability. In Proc. IEEE International Conference on Computer Vision (ICCV). 253–261.
[12]
Gengyun Jia, Huaibo Huang, Chaoyou Fu, and Ran He. 2022. Rethinking Image Cropping: Exploring Diverse Compositions From Global Views. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2446–2455.
[13]
Yueying Kao, Ran He, and Kaiqi Huang. 2017. Automatic image cropping with aesthetic map and gradient energy map. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1982–1986.
[14]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[15]
Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang. 2018. A2-RL: Aesthetics aware reinforcement learning for image cropping. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 8193–8201.
[16]
Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang. 2019. Fast A3RL: Aesthetics-Aware Adversarial Reinforcement Learning for Image Cropping. IEEE Transactions on Image Processing (TIP) 28, 10 (2019), 5105–5120.
[17]
Debang Li, Junge Zhang, and Kaiqi Huang. 2020. Learning to learn cropping models for different aspect ratio requirements. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12685–12694.
[18]
Jianxun Lou, Hanhe Lin, David Marshall, Dietmar Saupe, and Hantao Liu. 2022. TranSalNet: Towards perceptually relevant visual saliency prediction. Neurocomputing 494 (2022), 455–467.
[19]
Luca Marchesotti, Claudio Cifarelli, and Gabriela Csurka. 2009. A framework for visual saliency detection with applications to image thumbnailing. In Proc. IEEE International Conference on Computer Vision (ICCV). 2232–2239.
[20]
Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. AVA: A large-scale database for aesthetic visual analysis. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2408–2415.
[21]
Bingbing Ni, Mengdi Xu, Bin Cheng, Meng Wang, Shuicheng Yan, and Qi Tian. 2013. Learning to photograph: A compositional perspective. IEEE Transactions on Multimedia 15, 5 (2013), 1138–1151.
[22]
Masashi Nishiyama, Takahiro Okabe, Imari Sato, and Yoichi Sato. 2011. Aesthetic quality classification of photographs based on color harmony. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 33–40.
[23]
Masashi Nishiyama, Takahiro Okabe, Yoichi Sato, and Imari Sato. 2009. Sensation-Based Photo Cropping. In Proc. ACM International Conference on Multimedia (ACMMM). 669–672.
[24]
Zhiyu Pan, Zhiguo Cao, Kewei Wang, Hao Lu, and Weicai Zhong. 2021. TransView: Inside, Outside, and Across the Cropping View Boundaries. In Proc. IEEE/CVF International Conference on Computer Vision (ICCV). 4218–4227.
[25]
Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin, and Michael Cohen. 2006. Gaze-based interaction for semi-automatic photo cropping. In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI). 771–780.
[26]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[27]
Bongwon Suh, Haibin Ling, Benjamin B Bederson, and David W Jacobs. 2003. Automatic thumbnail cropping and its effectiveness. In Proc. ACM symposium on User interface software and technology (UIST). 95–104.
[28]
Wenguan Wang and Jianbing Shen. 2017. Deep cropping via attention box prediction and aesthetics assessment. In Proc. IEEE International Conference on Computer Vision (ICCV). 2186–2194.
[29]
Wenguan Wang, Jianbing Shen, and Haibin Ling. 2019. A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 41, 7 (2019), 1531–1544.
[30]
Zijun Wei, Jianming Zhang, Xiaohui Shen, Zhe Lin, Radomír Mech, Minh Hoai, and Dimitris Samaras. 2018. Good view hunting: Learning photo composition from dense view pairs. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5437–5446.
[31]
Hui Zeng, Lida Li, Zisheng Cao, and Lei Zhang. 2019. Reliable and Efficient Image Cropping: A Grid Anchor based Approach. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5949–5957.
[32]
Hui Zeng, Lida Li, Zisheng Cao, and Lei Zhang. 2020. Grid anchor based image cropping: A new benchmark and an efficient model. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 44, 3 (2020), 1304–1319.
[33]
Bo Zhang, Li Niu, Xing Zhao, and Liqing Zhang. 2022. Human-Centric Image Cropping with Partition-Aware and Content-Preserving Features. In Proc. European Conference on Computer Vision (ECCV). 181–197.
[34]
Lei Zhong, Feng-Heng Li, Hao-Zhi Huang, Yong Zhang, Shao-Ping Lu, and Jue Wang. 2021. Aesthetic-guided outward image cropping. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–13.
[35]
Zhihang Zhong, Mingxi Cheng, Zhirong Wu, Yuhui Yuan, Yinqiang Zheng, Ji Li, Han Hu, Stephen Lin, Yoichi Sato, and Imari Sato. 2022. ClipCrop: Conditioned Cropping Driven by Vision-Language Model. arXiv preprint arXiv:2211.11492 (2022).
Recommendations
Aesthetic-guided outward image cropping
Image cropping is a commonly used post-processing operation for adjusting the scene composition of an input photography, therefore improving its aesthetics. Existing automatic image cropping methods are all bounded by the image border, thus have very ...
Repurposing existing deep networks for caption and aesthetic-guided image cropping
Highlights- The core research question of this paper is how can we find the image part described by a user, such that the output image crop will represent and preserve the caption information meanwhile result in an aesthetically pleasing output?
- ...
AbstractWe propose a novel optimization framework that crops a given image based on user description and aesthetics. Unlike existing image cropping methods, where one typically trains a deep network to regress to crop parameters or cropping actions, we ...
Robust Image Cropping by Filtering Composition Irrelevant Factors
Image and GraphicsAbstractNumerous factors can impact the aesthetic quality of images: composition, resolution, exposure, color saturation and so on. Image cropping is to improve the aesthetic quality by recomposing the images. When the only consideration of an image ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
745 pages
ISBN:9798400702051
DOI:10.1145/3595916
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
MMAsia '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 59 of 204 submissions, 29%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 53Total Downloads
- Downloads (Last 12 months)53
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format