Back

“Put-that-there”: Voice and gesture at the graphics interface

Richard A. Bolt

article1980SIGGRAPH2,101 citations
Listen

Despite commercial advances in voice recognition and pen- or tablet-based input, most computer graphics systems still rely heavily on standard mice and keyboards. Traditional single-device interaction forces a trade-off between ease of execution and expressiveness, whereas graphical design tasks require both rapid manipulation and flexible commands. The article evaluates why and how combining voice and gestural inputs creates more intuitive, efficient graphical interfaces, and it outlines core design principles to guide developers.

The article synthesizes findings from foundational multimodal systems, experimental user studies, and technical integration frameworks developed across academia and industry. It reviews performance metrics, architectural methods such as frame-based parsing and typed feature structure unification, and user interaction patterns across various spatial, numerical, and verbal tasks.

The findings demonstrate clear performance and usability advantages for multimodal interaction. Combining voice and gesture reduces task completion errors by 36 percent, cuts spoken word counts by 23 percent, and accelerates completion times by 10 percent compared to voice-only interfaces. In specialized design tasks like computer-aided modeling, adding speech commands improved user productivity by 108 percent because users maintained visual focus without interrupting hand movements. Users overwhelmingly favor combined inputs, preferring multimodal interaction in up to 89 percent of numerical tasks and 100 percent of spatial map tasks. Furthermore, the two inputs serve complementary roles: natural language excels at descriptive and non-visual commands, while gestures provide direct visual and spatial manipulation, allowing contextual cross-referencing to automatically resolve ambiguous or incomplete user input.

These results show that multimodal interfaces significantly reduce cognitive load by allowing users to process visual-spatial tasks and verbal commands simultaneously without cognitive interference. For graphics developers and organizations, implementing well-structured multimodal controls can substantially improve operator efficiency and lower input error rates in complex environments, such as computer-aided design and command-and-control operations.

To successfully implement these systems, interface designers should partition tasks based on modality strengths: assign non-graphical command and control to speech, and dedicate direct spatial input—both pointing and iconic motion—to gestures. Systems should maintain multiple underlying representations of the scene (algebraic, visual, and metric) and integrate inputs using type-constrained unification or multi-tier temporal and contextual fusion. Developers must also evaluate user interaction iteratively at each stage of design.

Technical challenges remain regarding how to standardize metric representations for spatial relationships (such as defining concepts like "between" or "on top of") and how to eliminate the need for fixed vocabularies. While current evidence strongly confirms the productivity and accuracy benefits of multimodal input, organizations should treat adaptive, self-learning multimodal systems as an evolving research area that requires targeted prototyping before full-scale deployment.

design taskskeyboardmap taskGestureSpeechInput
Bolt (1980).pdf
  • Paper: The Recognition of Human Movement Using Temporal Templates, A. Bobick et al. (2001). Reviewing this foundational template-matching study provides essential background on processing temporal motion streams without full 3D reconstruction before examining integrated multimodal graphics.
  • Paper: Multimodal Machine Learning: A Survey and Taxonomy, Tadas Baltrušaitis et al. (2017). This survey extends the foundational concepts of combining voice and gesture by organizing subsequent advances in representation, translation, alignment, and fusion into a cohesive taxonomy.

Topics

design taskskeyboardmap taskmulti-modal interactionsmulti-modal interfacesmulti-tiermultimodal systemparsing algorithmprototypingtrade

Keywords

GestureSpeechInputInterfaceGraphicsVoiceMultimodalInterfacesGraphics InterfaceComputer

Table of Contents

  • Introduction
  • Why Multimodal Interfaces?
  • Lessons from Previous Interfaces
  • Design Recommendations
  • Future Research
  • References

Knowls

  1. Knowl 1 — Interface Design Guidelines for Multimodal Voice and Gesture Systems

    model/method

    To construct intuitive multimodal graphics interfaces combining speech and gesture, interface design should adhere to general structural principles as well as modality-specific guidelines:

    General Multimodal Architecture Guidelines

    • Modality Allocation: Assign non-graphical command-and-control tasks (such as menu selections and mode switches) to speech, and visuo-spatial manipulation tasks (such as pointing, positioning, sizing, and orienting) to gestures.
    • Ambiguity Resolution: Employ contextual knowledge from the application state, graphical scene, or dialogue discourse to disambiguate incomplete or imperfectly recognized inputs.
    • Multi-Representational Modeling: Maintain multiple representations of scene objects (algebraic, visual, and metric) to support disparate modal expressions.
    • Input Integration: Apply formal unification grammars or type-constrained unification mechanisms rather than ad-hoc heuristics to fuse simultaneous or sequential inputs.

    Modality-Specific Guidelines

    • Speech Input:
      • Design an acoustically distinct command vocabulary to minimize phonetic recognition errors.
      • Provide continuous system feedback indicating recognizer state and interpretation status.
      • Decouple speech recognition processing from graphics rendering pipelines to avoid latency bottlenecks.
    • Gesture Input:
      • Utilize hand tension or deliberate posture changes to explicitly signal the onset and termination of gestural commands.
      • Ensure gestural interactions are fast, incrementally responsive, and easily reversible.
      • Prefer natural, established human gestures to minimize user cognitive burden and learning time.

  2. Knowl 2 — Multi-Representational Knowledge Framework for Multimodal Scene Understanding

    model/method

    Multimodal graphics interfaces require scene objects to be represented across three complementary knowledge encodings to accommodate the differing semantics of speech and gesture:

    1. Algebraic Representations: Encodes categorical, hierarchical, and abstract relationships (such as artificial intelligence frame structures). This supports speech inputs expressing propositional or non-visual relational semantics.
    2. Visual Representations: Encodes object geometry, surface features, and appearance primitives (such as 3D mesh models and CAD primitives). This supports direct visual rendering and appearance-based referencing.
    3. Metric Representations: Encodes continuous spatio-temporal coordinates, relative orientations, and geometric constraints between entities (such as spatial prepositions like "on top of" or "between").

    Multimodal Interpretation Pipeline

    Interpreting multimodal user input executes in three successive stages:

    1. Feature Identification: Extract key linguistic and kinematic features from the raw speech and gesture data streams.
    2. Multi-Domain Input Encoding: Translate extracted features into intermediate algebraic, visual, and metric representations.
    3. Cross-Domain Mapping: Map the multi-domain input representations onto the corresponding scene representations to resolve referents and execute system actions.

  3. Knowl 3 — Unification-Based Multimodal Input Integration via Typed Feature Structures

    algorithm

    A robust method for integrating simultaneous and partial multimodal inputs uses type-constrained feature structure unification across input modalities.

    Input: Speech feature structures S={s1,s2,… }S = \{s_1, s_2, \dots\}, Gesture feature structures G={g1,g2,… }G = \{g_1, g_2, \dots\}, each annotated with type TT, completeness status C∈{complete,partial}C \in \{\text{complete}, \text{partial}\}, time stamp interval [tstart,tend][t_{\text{start}}, t_{\text{end}}], and recognition probability PP
    Output: Integrated semantic command structure R∗R^*
    IntegratedCandidates = []
    for each speech structure s∈Ss \in S:
        for each gesture structure g∈Gg \in G:
            if TimeIntervalsOverlapOrCorrelate(s, g) then
                if TypesAreCompatible(s.T, g.T) then
                    merged_structure = UnifyFeatureStructures(s, g)
                    if merged_structure is not NULL and IsSemanticallyComplete(merged_structure) then
                        joint_prob = s.P×g.Ps.P \times g.P
                        IntegratedCandidates.append((merged_structure, joint_prob))
    if IntegratedCandidates is empty then
        return NULL
    R∗R^* = candidate in IntegratedCandidates with maximum joint_prob
    return R∗R^*

    In this architecture, speech phrases (e.g., "Barbed wire") generate typed semantic frames with open argument slots (such as a missing location slot of type line), while concurrent gestures generate candidate geometric structures (such as point coordinates or coordinate lists). Unification combines the partial speech frame with the matching typed gesture structure to produce a complete executable command. Incompatible types cannot unify, preventing invalid cross-modal mergers and allowing high-confidence information in one modality to compensate for recognition uncertainty in the other.

  4. Knowl 4 — Hierarchical Temporal and Contextual Multimodal Fusion Engine

    model/method

    In multiagent multimodal graphical interfaces, input events from independent input streams (such as speech recognizers, datagloves, and eye-trackers) are merged into unified system commands by a generic fusion engine executing three sequential fusion stages:

    1. Microtemporal Fusion: Merges input tokens and semantic fragments that occur simultaneously within a tightly defined temporal window (such as a spoken deictic pronoun "there" combined with an active pointing coordinate).
    2. Macrotemporal Fusion: Merges sequential input events that belong to the same interaction dialogue or composite command sequence across a broader time interval (such as a speech command followed sequentially by a gestural parameter adjustment).
    3. Contextual Fusion: Merges inputs whose association is established through shared contextual features, discourse history, or current scene focus rather than explicit temporal synchronization alone.

  5. Knowl 5 — Top-Down Semantic Contextual Interpretation of Iconic Gestures

    model/method

    Iconic gestures are physical hand and arm motions that depict the physical shape, orientation, or movement trajectory of an entity (for instance, rotating a hand to illustrate the target orientation of an object). Because an iconic hand motion in isolation is ambiguous—it could represent translation, rotation, scaling, or pointing—it cannot be parsed bottom-up from kinematics alone.

    Multimodal understanding of iconic gestures requires a top-down contextual interpretation model:

    1. Spoken natural language provides the governing semantic frame and operation context (e.g., the command "Rotate the chair like this").
    2. The semantic frame specifies which kinematic features of the concurrent hand gesture to extract (e.g., angular rotation around an axis while ignoring translational wrist position).
    3. The extracted parameter values from the gesture are bound directly to the transformation parameters of the graphical entity identified in the speech utterance.

  6. Knowl 6 — Functional Partitioning of Speech and Gesture Semantics

    model/method

    Multimodal graphical interface performance depends on partitioning user interaction semantics according to the intrinsic affordances of each input channel:

    • Speech Semantics: Suited for discrete, symbolic, descriptive, and non-spatial operations (such as invoking menu actions, specifying object types, issuing macro commands, and accessing occluded or non-visible graphical entities). Speech relieves the user's hands from switching between interaction modes.
    • Gesture Semantics: Suited for continuous, spatial, and direct-manipulation operations (such as setting parameter magnitudes, positioning, rotating, drawing paths, and pointing).
    • Synergistic Execution: Users operate both channels simultaneously or sequentially without cognitive interference between verbal and visuo-spatial task processing modes. For example, speech issues the command type while hand gestures (via dataglove or stylus) continuously specify transformation values on the surface without breaking contact.

  7. Knowl 7 — Multimodal Representational Requirements by Modality

    data/table

    Multimodal graphic interfaces require specific scene representation formats depending on the communication channel and gesture type utilized by the user.

    Modality Representational Needs
    Speech - Categorical / prepositional
    - Shape / visual appearance
    - Spatio-temporal information
    Gesture - Iconic: shape
    - Deictic: spatial

    This classification indicates that speech carries broad semantic functions including abstract categories, descriptive shape queries, and temporal/spatial descriptions, whereas gesture is split into deictic actions (pointing, requiring spatial/coordinate representation) and iconic actions (depicting object shape or motion path, requiring shape representation).

  8. Knowl 8 — Limitations in Metric Spatial Knowledge Representation and Multimodal Adaptation

    limitation

    Two foundational limitations restrict voice and gesture graphics interfaces:

    1. Absence of Standardized Metric Spatial Representations: There is no generally accepted formal representation for encoding complex metric and relative spatial relationships between objects (such as unambiguous mathematical definitions of relational concepts like "between those objects", "on top of", or paths and boundary regions spanning multiple irregular objects).
    2. Lack of User-Adaptive Interaction Models: Most multimodal interfaces enforce fixed, pre-programmed speech vocabularies and rigid gesture lexicons. They lack adaptive machine learning mechanisms capable of learning user-specific speech phrasing, individualized gestural styles, or dynamic user-defined associations between descriptive words and geometric parameters (such as shape and color).

Coverage note — No substantial contributed material was omitted; the paper is a survey and architectural synthesis, and all core architectural models, design guidelines, data tables, and future research limitations have been captured.

References

  1. 1.Baudel, T. and M. Beaudouin-Lafon. "Charade: Remote Control of Objects Using Free-Hand Gestures," Communications of the ACM, 36(7), 1993, pp. 28-35.
  2. 2.Bolt, R.A. "Conversing with Computers," In R. Baecker, W. Buxton, (Eds.). Readings in Human-Computer Interaction: A Multidisciplinary Approach, California: Morgan-Kaufmann, 1987.
  3. 3.Clay, S. and J. Wilhelms. "Put: Language-Based Interactive Manipulation of Objects," IEEE Computer Graphics and Applications, 16(2), March 1996, pp. 31-39.
  4. 4.Cohen, P. "The Role of Natural Language in a Multimodal Interface," Proceedings of the UIST '92 Conference, 1992, pp. 143-149.
  5. 5.Coutaz, J., D. Slaber and B. Balbo. "Towards Automatic Evaluation of Multimodal User Interfaces," Knowledge-Based Systems, 6(4), December 1993, pp. 258-266.
  6. 6.Hauptmann, A.G. and P. McAvinney. "Gestures with Speech for Graphics Manipulation," Intl. J. Man-Machine Studies, 38, 1993, pp. 231-249.
  7. 7.Johnston, M., P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman and I. Smith. "Unification-based multimodal integration," Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, 1997.
  8. 8.Jones, D., K. Hopeshi and C. Frankish. "Design Guidelines for Speech Recognition Interfaces," Applied Ergonomics, 20(1), 1989, pp. 47-52.
  9. 9.Kendon, A. "Gesticulation and Speech: Two Aspects of the Process of Utterance," In M. Key (Ed.) The Relation between Verbal and Nonverbal Communication. The Hague: Mouton, 1980, pp. 207-227.
  10. 10.Koons, D.B. "Capturing and Interpreting Multi-Modal Descriptions with Multiple Representations," AAAI Spring 1994 Symposium, Intelligent Multi-Modal Multi-Media Interface Systems, Stanford, March 21-23, 1994.
  11. 11.Koons, D. B., C. J. Sparrell, and K. R. Thorisson. "Integrating Simultaneous Output from Speech, Gaze, and Hand Gestures," In M. Maybury, (Ed.). Intelligent Multimedia Interfaces, Menlo Park: AAAI/MIT Press, 1993, pp. 243-261.
  12. 12.Lucente, M., G. Zwart and A. George. "Visualization Space: A Testbed for Deviceless Multimodal User Interface," Proceedings of the Intelligent Environments Symposium, AAAI Spring Symposium Series, March 23-25, 1998, Stanford University.
  13. 13.Marsh, E., K. Wauchope and J. Gurney. "Human-Machine Dialogue for Multi-Modal Decision Support Systems," AAAI Spring Symposium Series on Intelligent Multi-Media Multi-Modal Systems, Stanford University, 1994, http://www.aic.nrl.navy.mil/papers/1994/m4.html.
  14. 14.Martin, G.L. "The Utility of Speech Input in User-Computing Interfaces," Intl. J. Man-Machine Studies, 30, 1989, pp. 355-375.
  15. 15.Neal, J. and S. Shapiro. "Intelligent multimedia interface technology," In Intelligent User Interfaces, J. Sullivan, S. Tyler (Eds.) ACM Press, New York, New York, 1991, pp. 45-68.
  16. 16.Nigay, L. and J. Coutaz. "A Generic Platform for Addressing the Multimodal Challenge," In Conference on Human Factors in Computing Systems (CHI '95), May 1995, ACM Press, pp. 98-105.
  17. 17.Oviatt, S. "Multimodal Interfaces for Dynamic Interactive Maps," Proceedings of the Conference on Human Factors in Computing Systems: CHI '96, Vancouver, Canada. ACM Press, New York, 1996, pp. 95-102.
  18. 18.Oviatt, S. "User-Centered Modeling for Spoken Language and Multimodal Interfaces," IEEE Multimedia, Winter 1996, pp. 26-35.
  19. 19.Roy D. and A. Pentland. "Multimodal Adaptive Interfaces: Vision and Modeling," Tech. Report #438, Media Laboratory, MIT, Cambridge, MA, 1997, http://www-white.media.mit.edu/cgi-bin/tr_page_maker/.
  20. 20.Salisbury, M., J. Henderson, T. Lammers, C. Fu and S. Moody. "Talk and Draw: Bundling Speech and Graphics," IEEE Computer, 1990, 23(8), pp. 59-65.
  21. 21.Sparrell, C. and D. Koons. "Interpretation of Coverbal Depictive Gestures," Proceedings of Intelligent Multi-Modal Multi-Media Interface Systems, AAAI Spring Symposium Series, March 21-23, 1994, Stanford University.
  22. 22.Sturman, D. and D. Zeltzer. "A design method for "whole-hand" human-computer interaction," ACM Trans. on Information Systems, 11(3), July 1993, pp. 219-238.
  23. 23.Thorisson, K., D. Koons and R. Bolt. "Multi-Model Natural Dialogue," CHI 92 Video Proceedings, 1992, p. 653.
  24. 24.Treisman, A. and A. Davies. "Divided Attention to Ear and Eye," In S. Kornblum, (Ed.). Attention and Performance IV, New York: Erlbaum, 1973, pp. 101-117.
  25. 25.Vergo, J. "A Statistical Approach to Multimodal Natural Language Interaction," Proceedings of the AAAI '98 Workshop on Representations for Multi-modal Human Computer Interaction, July 1998, http://tigger.cs.uwm.edu/wrkshp/.
  26. 26.Vo, M. and A. Waibel. "Modeling and Interpreting Multimodal Inputs: A Semantic Integration Approach," Technical Report CMU-CS-97-192, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 1997.
  27. 27.Weimer, D. and S. K. Ganapathy. "A Synthetic Visual Environment with Hand Gesturing and Voice Inputs," In Proceedings of Human Factors in Computing Systems (CHI'89), ACM Press, 1989, pp. 235-240.

Citation

MLA
Bolt, R. A. ““Put-that-there””. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH '80, 1980, pp. 262–70, https://doi.org/10.1145/800250.807503.
APA
Bolt, R. A. (1980). “Put-that-there”. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH '80, 262–270. https://doi.org/10.1145/800250.807503
Chicago
Bolt, R. A. 1980. ““Put-that-there””. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH '80, 262–70. https://doi.org/10.1145/800250.807503.
Harvard
Bolt, R.A. (1980) ““Put-that-there””, Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH '80. ACM Press, pp. 262–270. Available at: https://doi.org/10.1145/800250.807503.
Vancouver
1. Bolt RA (1980) “Put-that-there”. In: Proceedings of the 7th annual conference on Computer graphics and interactive techniques - SIGGRAPH '80. ACM Press, pp 262–270

BibTeX

@inproceedings{Bolt_1980, series={SIGGRAPH ’80}, title={“Put-that-there”: Voice and gesture at the graphics interface}, url={http://dx.doi.org/10.1145/800250.807503}, DOI={10.1145/800250.807503}, booktitle={Proceedings of the 7th annual conference on Computer graphics and interactive techniques  - SIGGRAPH ’80}, publisher={ACM Press}, author={Bolt, Richard A.}, year={1980}, pages={262–270}, collection={SIGGRAPH ’80} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF