Article

Free access

A visual medium for programmatic control of interactive applications

Authors:

Luke S. Zettlemoyer,

Robert St. AmantAuthors Info & Claims

CHI '99: Proceedings of the SIGCHI conference on Human Factors in Computing Systems

Pages 199 - 206

https://doi.org/10.1145/302979.303039

Published: 01 May 1999 Publication History

PDF eReader

Abstract

The VisMap system provides for visual manipulation of arbitrary off-the-shelf applications, through an applications graphical user interface. VisMaps API-independent control has advantages for tasks that can benefit from direct access to the functions of the user interface. We describe the design goals and architecture of the system, and we discuss two applications, a user-controlled visual scripting program and an autonomous solitaire-playing program, which together demonstrate some of the capabilities and limitations of the approach.

References

[1]

Gentner, D., and Nielsen, J. The Anti-Mac Interface, Communications of the ACM, 39:8 (August, 1996), 70- 82.

Digital Library

Google Scholar

[2]

Gonzales, R.C. and Woods, R.W. Digital linage Processing. Addison-Wesley, Reading, MA. 1992.

Digital Library

Google Scholar

[3]

Kieras, D. and Meyer, D. E. An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human- Computer Interaction.

Digital Library

Google Scholar

[4]

Lieberman, H. Integrating User Interface Agents with Conventional Applications. Proceedings of" Intelligent User Interfaces '98. (San Francisco, CA, January, 1998.) ACM Press, 39-46.

Digital Library

Google Scholar

[5]

Maes, P. Agents that Reduce Work and l~nformation Overload. Communications of the A CM, :37:7, (July 1994), 31-40.

Digital Library

Google Scholar

[6]

Myers, B. Demonstrational Interfaces: A Step Beyond Direct Manipulation, Watch What 1 Do: Programming by Demonstration, Alien Cypher, et. al., eds. MIT Press Cambridge, MA. 1993. pp. 485-512.

Digital Library

Google Scholar

[7]

Myers, B. Scripting Graphical Applications by Demonstration. Proceedings of CHI '98. (Los Angeles, CA, April, 1998.) 534-541.

Digital Library

Google Scholar

[8]

Penberthy, J. and Weld, D. UCPOP: A sound, complete, partial-order planner for ADL. Proceedings of the Third International Conference on Knowledge Representation and Reasoning. 1992. Morgan Kaufmann, 103-114.

Google Scholar

[9]

Potter, R. Triggers: Guiding Automation wil~a Pixels to Achieve Data Access. In Watch grhat I Do: Programming by Demonstration. M}{T Press, Cambridge, MA. 1993.

Digital Library

Google Scholar

[10]

Rich, C. and Sidner C. L. Adding a Collaborative Agent to Graphical User Interfaces, Proceedings of UIST'96, (1996), 21-30.

Digital Library

Google Scholar

[11]

Shneiderman, B. Direct Manipulation for comprehensible, predictable, and contro}Llable user interfaces. Proceedings of Intelligent User blterface '97. (Orlando, FL, January, 1997.) ACM Press, 33-39.

Digital Library

Google Scholar

[12]

Shneiderman, B., and Maes, P. Debate: Direct Manipulation vs. Interface Agents. Interactions, 4:6 (November and December, 1997), 42-61.

Digital Library

Google Scholar

[13]

Young, R. M., Green, T. R. G., and Simon, T. Programmable User Models for Predictive Evaluation of Interface Designs. Proceedings of CHI '89. 15-19.

Digital Library

Google Scholar

[14]

Zettlemoyer, L. S., St. Amant, R., and Dulberg, M. S. Application control through the user interface. Proceedings of Intelligent User Interfaces '99. (Redondo Beach, Los Angeles, CA, January, 1999.) To appear.

Digital Library

Google Scholar

Cited By

View all

Shaw PJoshi MCohan JBerant JPasupat PHu HKhandelwal ULee KToutanova KOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)From pixels to UI actionsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667612(34354-34370)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667612
Seneviratne SPenenco VKasthurirathna D(2023)AD-PU: A Novel Approach for Automated Identification of the Outliers in User Interface Testing (UAT)2023 IEEE AUTOTESTCON10.1109/AUTOTESTCON47464.2023.10296279(1-5)Online publication date: 28-Aug-2023
https://doi.org/10.1109/AUTOTESTCON47464.2023.10296279
Jiang YStuerzlinger WLutteroth CKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445043
Show More Cited By

Index Terms

A visual medium for programmatic control of interactive applications
1. Human-centered computing
  1. Collaborative and social computing
  2. Human computer interaction (HCI)

Recommendations

Transportable Applications Environment (TAE) Plus user interface designer WorkBench
CHI '92: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

TAE Plus was built at NASA's Goddard Space Flight Center to support the building of GUI user interfaces for highly interactive applications, such as realtime processing systems and scientific analysis system. TAE Plus is designed as a productivity tool ...
Graphical styles for building interfaces by demonstration
UIST '92: Proceedings of the 5th annual ACM symposium on User interface software and technology

Conventional interface builders allow the user interface designer to select widgets such as menus, buttons and scroll bars, and lay them out using a mouse. Although these are conceptually simple to use, in practice there are a number of problems. First, ...
Heuristics in real user interfaces
CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

It is the conventional wisdom in user interface design that direct manipulation is best and that interfaces should be predictable. This tends to argue against having a system “guess” or use heuristics or other AI approaches. However, an increasing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CHI '99: Proceedings of the SIGCHI conference on Human Factors in Computing Systems

May 1999

632 pages

ISBN:0201485591

DOI:10.1145/302979

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 1999

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CHI99

Sponsor:

SIGCHI

CHI99: Conference on Human Factors in Computing Systems

May 15 - 20, 1999

Pennsylvania, Pittsburgh, USA

Acceptance Rates

CHI '99 Paper Acceptance Rate 78 of 312 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
575
Total Downloads

Downloads (Last 12 months)46
Downloads (Last 6 weeks)12

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Shaw PJoshi MCohan JBerant JPasupat PHu HKhandelwal ULee KToutanova KOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)From pixels to UI actionsProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3667612(34354-34370)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3667612
Seneviratne SPenenco VKasthurirathna D(2023)AD-PU: A Novel Approach for Automated Identification of the Outliers in User Interface Testing (UAT)2023 IEEE AUTOTESTCON10.1109/AUTOTESTCON47464.2023.10296279(1-5)Online publication date: 28-Aug-2023
https://doi.org/10.1109/AUTOTESTCON47464.2023.10296279
Jiang YStuerzlinger WLutteroth CKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445043
Masson DMalacria SLank ECasiez GBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Chameleon: Bringing Interactivity to Static Digital DocumentsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376559(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376559
Wagner AGray J(2015)An Empirical Evaluation of a Vocal User Interface for Programming by VoiceInternational Journal of Information Technologies and Systems Approach10.4018/IJITSA.20150701048:2(47-63)Online publication date: 1-Jul-2015
https://doi.org/10.4018/IJITSA.2015070104
Lamberti FSanna ACarlevaris GDemartini C(2015)Adding Pluggable and Personalized Natural Control Capabilities to Existing ApplicationsSensors10.3390/s15020283215:2(2832-2859)Online publication date: 28-Jan-2015
https://doi.org/10.3390/s150202832
Dixon MLaput GFogarty JJones MPalanque PSchmidt AGrossman T(2014)Pixel-based methods for widget state and style in a runtime implementation of sliding widgetsProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/2556288.2556979(2231-2240)Online publication date: 26-Apr-2014
https://dl.acm.org/doi/10.1145/2556288.2556979
Rungta ASumma BDemir DBremer PPascucci V(2013)ManyVisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2013.17419:12(2878-2885)Online publication date: 1-Dec-2013
https://dl.acm.org/doi/10.1109/TVCG.2013.174
Alegroth ENass MOlsson H(2013)JAutomateProceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation10.1109/ICST.2013.61(439-446)Online publication date: 18-Mar-2013
https://dl.acm.org/doi/10.1109/ICST.2013.61
Alegroth EFeldt ROlsson H(2013)Transitioning Manual System Test Suites to Automated TestingProceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation10.1109/ICST.2013.14(56-65)Online publication date: 18-Mar-2013
https://dl.acm.org/doi/10.1109/ICST.2013.14
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Transportable Applications Environment (TAE) Plus user interface designer WorkBench

Graphical styles for building interfaces by demonstration

Heuristics in real user interfaces