research-article

Public Access

PI2: End-to-end Interactive Visualization Interface Generation from Queries

Authors:

Eugene WuAuthors Info & Claims

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

Pages 1711 - 1725

https://doi.org/10.1145/3514221.3526166

Published: 11 June 2022 Publication History

Abstract

Interactive visualization interfaces are critical in data analysis. Yet creating new interfaces is challenging, as the developer must understand the queries needed for the desired analysis task, and then design the appropriate interface. Existing task models are too abstract to be used to automatically generate interfaces, and visualization recommenders do not take the queries nor interactions into account. PI2 is the first system to generate fully functional interactive visualization interfaces from a representative sequence of task queries. PI2 analyzes queries syntactically and proposes a novel Difftree representation that encodes the systematic variations between query abstract syntax trees. PI2 then poses interface generation as a schema mapping problem from each Difftree to a visualization that renders its results, and the variations encoded in each Difftree to interactions in the interface. Interface generation further takes the layout and screen size into account. Our user studies show that PI2 interfaces are comparable to or better than those designed by developers, and that PI2 can generate exploration interfaces that are easier to use than the state-of-the-art SQL notebook products. What's more, PI2 generates high-quality interfaces within a few seconds.

References

[1]

2021. Count -- The SQL notebook. https://count.co/.

[2]

2021. Hex Technologies. https://hex.tech/.

[3]

Johnny Accot and Shumin Zhai. 1997. Beyond Fitts' law: models for trajectory-based HCI tasks. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. 295--302.

Digital Library

[4]

Airtable. 2021. Airtable. https://airtable.com/.

[5]

Jillian Aurisano, Abhinav Kumar, Alberto Gonzalez, Jason Leigh, Barbara DiEugenio, and Andrew Johnson. 2016. Articulate2: Toward a conversational interface for visual data exploration. In IEEE Visualization.

[6]

Jacques Bertin. 1983. Semiology of graphics; diagrams networks maps. Technical Report.

[7]

Yngvi Bjornsson and Hilmar Finnsson. 2009. Cadiaplayer: A simulation-based general game player. IEEE Transactions on Computational Intelligence and AI in Games 1, 1 (2009), 4--15.

[8]

M. Bostock, Vadim Ogievetsky, and J. Heer. 2011. D3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17 (2011), 2301--2309.

Digital Library

[9]

Matthew Brehmer and Tamara Munzner. 2013. A multi-level typology of abstract visualization tasks. IEEE transactions on visualization and computer graphics 19, 12 (2013), 2376--2385.

Digital Library

[10]

Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. 2012. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligence and AI in Games 4 (2012), 1--43.

[11]

S. Card, T. Moran, and A. Newell. 1983. The psychology of human-computer interaction.

[12]

Winston Chang, Joe Cheng, J Allaire, Yihui Xie, Jonathan McPherson, et al. 2015. shiny: Web Application Framework for R, 2015. In CRAN.

[13]

Yiru Chen and Silu Huang. 2021. TSExplain: Surfacing Evolving Explanations for Time Series. Proceedings of the 2021 International Conference on Management of Data (2021).

Digital Library

[14]

Yiru Chen and Eugene Wu. 2020. Monte Carlo Tree Search for Generating Interactive Data Analysis Interfaces. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (2020).

Digital Library

[15]

Yiru Chen and Eugene Wu. 2021. PI2: Generating Visual Analysis Interfaces From Queries (technical report). CoRR abs/2107.08203 (2021). arXiv:2107.08203 https://arxiv.org/abs/2107.08203

[16]

Rémi Coulom. 2006. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In Computers and Games.

[17]

Kenneth Cox, Rebecca E Grinter, Stacie L Hibino, Lalita Jategaonkar Jagadeesan, and David Mantilla. 2001. A multi-modal natural language interface to an information visualization environment. International Journal of Speech Technology 4, 3 (2001), 297--314.

[18]

Figma. 2021. Figma: a collaborative interface design tool. https://www.figma.com.

[19]

FiveThirtyEight. 2021. All Posts Tagged Data Visualization. https://fivethirtyeight.com/tag/data-visualization/.

[20]

Krzysztof Z Gajos and Daniel S. Weld. 2004. SUPPLE: automatically generating user interfaces. In IUI '04.

[21]

Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2010. Automatically generating personalized user interfaces with Supple. Artif. Intell. 174 (2010), 910--950.

Digital Library

[22]

Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. arXiv preprint arXiv:2109.05153 (2021).

[23]

Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. In Proceedings of the 28th annual acm symposium on user interface software & technology. 489--500.

Digital Library

[24]

Google. 2021. Facets - Know Your Data. https://pair-code.github.io/facets/.

[25]

Hex. 2022. Hex Chart Cells. https://learn.hex.tech/docs/logic-cell-types/display-cells/chart-cells.

[26]

Hex. 2022. Hex Parameterize Sql. https://learn.hex.tech/tutorials/connect-to-data/parameterize-sql.

[27]

iCheck. 2021. iCheck. icheckuclaim.org.

[28]

H. V. Jagadish, Adriane Chapman, Aaron Elkiss, Magesh Jayapandian, Yunyao Li, Arnab Nandi, and Cong Yu. 2007. Making database systems usable. In SIGMOD.

[29]

Magesh Jayapandian and HV Jagadish. 2008. Automated creation of a forms-based database query interface. In PVLDB. VLDB Endowment.

[30]

Magesh Jayapandian and H. V. Jagadish. 2006. Automating the Design and Construction of Query Forms. In TKDE.

[31]

Bonnie E John and David E Kieras. 1996. The GOMS family of user interface analysis techniques: Comparison and contrast. In TOCHI.

[32]

JQuery. 2021. jQuery. https://jquery.com/.

[33]

Kaggle. 2021. Dataset: Supermarket Sales. https://www.kaggle.com/aungpyaeap/supermarket-sales.

[34]

S. Kandel, A. Paepcke, J. Hellerstein, and J. Heer. 2011. Wrangler: interactive visual specification of data transformation scripts. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2011).

Digital Library

[35]

D. Kieras. 1994. GOMS modeling of user interfaces using NGOMSL. In CHI '94.

Digital Library

[36]

Doris Jung-Lin Lee, Dixin Tang, Kunal Agarwal, et al. 20. Lux: Always-on Visualization Recommendations for Exploratory Data Science. VLDB (20).

[37]

Yuyu Luo, Xuedi Qin, Nan Tang, and Guoliang Li. 2018. Deepeye: Towards automatic data visualization. In 2018 IEEE 34th international conference on data engineering (ICDE). IEEE, 101--112.

[38]

I Scott MacKenzie and William Buxton. 1992. Extending Fitts' law to two- dimensional tasks. In Proceedings of the SIGCHI conference on Human factors in computing systems. 219--226.

Digital Library

[39]

Jock Mackinlay. 1986. Automating the design of graphical presentations of relational information. In Transactions On Graphics.

[40]

J. Mackinlay, P. Hanrahan, and Chris Stolte. 2007. Show Me: Automatic Presentation for Visual Analysis. IEEE Transactions on Visualization and Computer Graphics 13 (2007).

[41]

Jock Mackinlay, Pat Hanrahan, and Chris Stolte. 2007. Show me: Automatic presentation for visual analysis. In TVCG.

[42]

Metabase. 2021. Metabase. https://www.metabase.com.

[43]

Metabase. 2021. Metabase Documentation: SQL Parameters. https://www.metabase.com/docs/latest/users-guide/13-sql-parameters.html.

[44]

Dominik Moritz, Chenglong Wang, Greg L. Nelson, H. Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. 2018. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. In TVCG.

[45]

Dominik Moritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, and J. Heer. 2019. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Transactions on Visualization and Computer Graphics 25 (2019), 438--448.

Digital Library

[46]

Tamara Munzner. 2014. Visualization analysis and design. CRC press.

[47]

Daniel Murray. 2013. Tableau Your Data!: Fast and Easy Visual Analysis with Tableau Software.

[48]

Arpit Narechania, Arjun Srinivasan, and John Stasko. 2020. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 369--379.

[49]

Jeffrey Nichols, Brad A Myers, and Kevin Litwack. 2004. Improving automatic interface generation with smart templates. In IUI.

[50]

Plasmic. 2021. Plasmic: The visual builder for your tech stack. https://www.plasmic.app/.

[51]

Angel R Puerta, Henrik Eriksson, John H Gennari, and Mark A Musen. 1994. Model-based automated generation of user interfaces. In AAAI.

[52]

React. 2021. A JavaScript library for building user interfaces. https://reactjs.org/.

[53]

Retool. 2021. Build internal tools, remarkably fast. https://www.retool.com.

[54]

A. Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and J. Heer. 2017. Vega-Lite: A Grammar of Interactive Graphics. IEEE Transactions on Visualization and Computer Graphics 23 (2017), 341--350.

Digital Library

[55]

Maarten PD Schadd, Mark HM Winands, H Jaap Van Den Herik, Guillaume MJ-B Chaslot, and Jos WHM Uiterwijk. 2008. Single-player monte-carlo tree search. In International Conference on Computers and Games. Springer, 1--12.

Digital Library

[56]

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. arXiv preprint arXiv:2109.05093 (2021).

[57]

SDSS. 2021. SDSS Weblog SQL Search. http://skyserver.sdss.org/log/en/traffic/sql.asp?url=http://skyserver.sdss.org/log/en/traffic///help/download/.

[58]

SDSS. 2021. Sloan Digital Sky Survey, 2017. http://www.sdss.org/.

[59]

A. Sears. 1993. Layout Appropriateness: A Metric for Evaluating User Interface Widget Layout. IEEE Trans. Software Eng. 19 (1993), 707--719.

Digital Library

[60]

Vidya Setlur, Sarah E Battersby, Melanie Tory, Rich Gossweiler, and Angel X Chang. 2016. Eviza: A natural language interface for visual analysis. In Proceedings of the 29th annual symposium on user interface software and technology. 365--377.

Digital Library

[61]

D. Silver, Aja Huang, Chris J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, S. Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529 (2016), 484--489.

[62]

Arjun Srinivasan and John Stasko. 2017. Natural language interfaces for data analysis with visualization: Considering what has and could be asked. In Proceedings of the Eurographics/IEEE VGTC conference on visualization: Short papers. 55--59.

Digital Library

[63]

C. Stolte, Diane Tang, and P. Hanrahan. 2008. Polaris: a system for query, analysis, and visualization of multidimensional databases. Commun. ACM 51 (2008), 75--84.

Digital Library

[64]

Yiwen Sun, Jason Leigh, Andrew Johnson, and Sangyoon Lee. 2010. Articulate: A semi-automated model for translating natural language queries into meaningful visualizations. In International Symposium on Smart Graphics. Springer, 184--195.

Digital Library

[65]

Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J Ko. 2020. Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--13.

Digital Library

[66]

Tableau. 2012. Tableau Software. http://www.tableausoftware.com/.

[67]

Tableau. 2021. Connect to a Custom SQL Query. https://help.tableau.com/current/pro/desktop/en-us/customsql.htm.

[68]

Jeffrey Tao, Yiru Chen, and Eugene Wu. 2022. Demonstration of PI2: Interactive Visualization Interface Generation for SQL Analysis in Notebook. Proceedings of the 2022 ACM SIGMOD International Conference on Management of Data.

Digital Library

[69]

New York Times. 2021. Coronavirus (Covid-19) Data in the United States. https://github.com/nytimes/covid-19-data.

[70]

Jean Vanderdonckt. 1994. Automatic generation of a user interface for highly interactive business-oriented applications. In CHI.

[71]

Vega-lite. 2022. Vega-lite Mark Types. https://vega.github.io/vega-lite/docs/mark.html.

[72]

Chenglong Wang, Yu Feng, Rastislav Bodik, Isil Dillig, Alvin Cheung, and Amy J Ko. 2021. Falx: Synthesis-Powered Visualization Authoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.

Digital Library

[73]

Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2016. Voyager: Exploratory analysis via faceted browsing of visualization recommendations. In TVCG.

[74]

Eugene Wu and Samuel Madden. 2013. Scorpion: Explaining away outliers in aggregate queries. (2013).

Digital Library

[75]

J. S. Yi, Y. Kang, J. Stasko, and J. Jacko. 2007. Toward a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE Transactions on Visualization and Computer Graphics 13 (2007), 1224--1231.

Digital Library

[76]

Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 (2018).

[77]

Zehua Zeng, Minhui Xie, Matthew Gouzoulis, and Leilani Battle. 2021. Are We There Yet? A Review on Existing Perceptual Theory and Experiment Support for Visualization Recommendation Systems. arXiv preprint arXiv:2109.01271 (2021).

[78]

Qianrui Zhang, Haoci Zhang, Thibault Sellam, and Eugene Wu. 2019. Mining precision interfaces from query logs. In Proceedings of the 2019 International Conference on Management of Data. 988--1005.

Digital Library

[79]

Jonathan Zong, D. Barnwal, Rupayan Neogy, and A. Satyanarayan. 2021. Lyra 2: Designing Interactive Visualizations by Demonstration. IEEE Transactions on Visualization and Computer Graphics 27 (2021), 304--314.

Cited By

Xie YLuo YLi GTang N(2024)HAIChart: Human and AI Paired Visualization SystemProceedings of the VLDB Endowment10.14778/3681954.368199217:11(3178-3191)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.14778/3681954.3681992
Wang ZMunechika DLee SChau D(2024)SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational NotebooksExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650848(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650848
Vaithilingam PGlassman EInala JWang C(2024)DynaVis: Dynamically Synthesized UI Widgets for Visualization EditingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642639(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642639
Show More Cited By

Index Terms

PI2: End-to-end Interactive Visualization Interface Generation from Queries
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User interface design
  2. Visualization
    1. Visualization systems and tools
2. Information systems
  1. Data management systems

Recommendations

Demonstration of PI2: Interactive Visualization Interface Generation for SQL Analysis in Notebook
SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

We demonstrate PI2, the first notebook extension that can automatically generate interactive visualization interfaces during SQL-based analyses.
A Pen-Based Prosodic User Interface for Schoolchildren

A prosodic user interface is defined as an user interface that can deal with not only what is entered by the user but also how it is entered. A pen-based user interface provides more prosodic information than a mouse-based graphical user interface. The ...
A tangible user interface for interactive data visualization
CASCON '15: Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering

We present a prototype for a Tangible User Interface (TUI) designed to interactively query a database. While much work has been done on TUI, showing that they encourage collaboration and positively enhance user experience, few tangible systems have been ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data

June 2022

2597 pages

ISBN:9781450392495

DOI:10.1145/3514221

General Chair:
Zachary Ives
University of Pennsylvania (USA)
,
Program Chairs:
Angela Bonifati
Lyon 1 University (France)
,
Amr El Abbadi
University of California, Santa Barbara (USA)

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SIGMOD/PODS '22

Sponsor:

SIGMOD

SIGMOD/PODS '22: International Conference on Management of Data

June 12 - 17, 2022

PA, Philadelphia, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
766
Total Downloads

Downloads (Last 12 months)343
Downloads (Last 6 weeks)66

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xie YLuo YLi GTang N(2024)HAIChart: Human and AI Paired Visualization SystemProceedings of the VLDB Endowment10.14778/3681954.368199217:11(3178-3191)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.14778/3681954.3681992
Wang ZMunechika DLee SChau D(2024)SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational NotebooksExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650848(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650848
Vaithilingam PGlassman EInala JWang C(2024)DynaVis: Dynamically Synthesized UI Widgets for Visualization EditingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642639(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642639
Zhang WWang YSong YWei VTian YQi YChan JWong RYang H(2024)Natural Language Interfaces for Tabular Data Querying and Visualization: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.340082436:11(6699-6718)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3400824
Chen YTao JWu E(2023)DIGProceedings of the Workshop on Human-In-the-Loop Data Analytics10.1145/3597465.3605223(1-7)Online publication date: 18-Jun-2023
https://dl.acm.org/doi/10.1145/3597465.3605223
Li HYing LZhang HWu YQu HWang Y(2023)Notable: On-the-fly Assistant for Data Storytelling in Computational NotebooksProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580965(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580965
Pu XKay M(2023)How Data Analysts Use a Visualization Grammar in PracticeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580837(1-22)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580837
Heer JMoritz D(2023)Mosaic: An Architecture for Scalable & Interoperable Data ViewsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332718930:1(436-446)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3327189
Lin YLi HYang LWu AQu H(2023)InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational NotebooksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332717030:1(944-954)Online publication date: 25-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3327170
Pang RWang RNelson JBattle L(2022)How Do Data Science Workers Communicate Intermediate Results?2022 IEEE Visualization in Data Science (VDS)10.1109/VDS57266.2022.00010(46-54)Online publication date: Oct-2022
https://doi.org/10.1109/VDS57266.2022.00010

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents