Once the data was collected, the researchers reviewed all the videos of the gestures and classified them according to a detailed lexicon defined by the research team.
To help maintain consistency across researchers, all unique gestures were catalogued and uploaded to an online gesture glossary. Tables 1 and 2 below show the data collected for each unique gesture in the gesture glossary.
Early researchers uploaded many unique gestures to the online gesture glossary, while later researchers were able to compare the gesture made by their participants to the gesture pictures in the glossary and re-use the same gesture ID and description.
All gestures were reviewed and entered into the results spreadsheet, the structure of which is shown in Table 3. Because most of the detail describing the gesture is contained in the online gesture glossary, the values entered into the results spreadsheet were limited to the minimum number of items to classify the gesture and enable the data analyses.
Table 1. Lexicon for describing unique gestures.
| Term | Values | Explanation | Example |
| ID | Number | A unique sequential number for the gesture and action |
1 |
| Gesture Name | Text | A unique name for the gesture that will be copied to the RESULTS spreadsheet | Swipe left from object to neutral space |
| Gesture Image | Image | A representative image of the gesture in 160×120 resolution | |
| Added By | Text | IUP name | HumanCentric |
| Type | See Table 2 | General gesture category: Direct Manipulation, Symbolic or Combination Gesture. Type determines which group of fields should be entered for each geasture (e.g., do not enter ‘Symbolic’ information for ‘Direct Manipulation’ gestures) | Direct Manipulation |
| Direct Manipulation Fields – Participant affects an object or white space on the display | |||
| Motion | See Table 2 | A simplified description of the direct manipulation gesture | Swipe |
| Direction | See Table 2 | General direction of the motion | left |
| Adverb 1 | See Table 2 | (optional) A modifier describing how the motion and direction relate to the location | from |
| Location 1 | See Table 2 | The start location of the gesture. | Object |
| Adverb 2 | See Table 2 | (optional) A modifier describing how the motion and direction relate to the location 2 | to |
| Location 2 | See Table 2 | The end location of the gesture, if different from the start. | Neutral space |
| Repeating | Yes/No | Identifies if the gesture is performed a non-specific or arbitrary number of times | No |
| Symbolic Fields – Participant draws an iconic representation on the display | |||
| Symbol Description | Text | A detailed description of the symbolic gesture, if not described adequately by the Gesture name | Letter A |
| Nature | See Table 2 | Alphanumeric, Symbolic or Abstract | Alphanumeric |
| Character | Text | The specific character drawn by the participant, if available on a keyboard | A |
| Language | Text | If alphanumeric, the language of the character | English |
| Combination Fields – Participant performs multiple different direct Manipulation or Symbolic gestures | |||
| Gesture Combination | Text | A detailed description of the combination gesture, if not described adequately by the Gesture name | Press and hold index finger in neutral space while swiping vertically with the middle finger |
| Order | See Table 2 | Identifies if combination gestures are perfomed in series (one after the other) or simultaneously (at the same time) | Simultaneous |
| Number of Gestures | Number | The number of direct-manipulation or symbolic gestures used for the combination. This number should always be 2 or greater. |
2 |
| Fingers | Number | Total number of different fingers used to make the combination gesture |
2 |
| Comments | Text | Any additional comments about the gesture | |
Table 2. Lexicon values.
| Term | Values | Explanation |
| Type | Direct Manipulation | Participant acts directly on an object or white space on the surface |
| Symbolic | Participant draws a symbol (e.g., letter, number, picture) on the surface | |
| Combination | Participant combines multiple Direct Manilulation and/or Symbolic gestures | |
| Motion | Tap | A short touch on the surface (similar to a mouse “click”) |
| Double tap | Two short touches on the surface in the same spot with the same finger | |
| Triple tap | Three short touches on the surface in the same spot with the same finger | |
| Quadruple tap | Four short touches on the surface in the same spot with the same finger | |
| Swipe | A straight line drawn on the surface with one finger | |
| Double swipe | A straight line drawn on the surface with two fingers | |
| Curve | A curved line with just one bend (i.e., a “C” but not an “S” bend) drawn on the surface with one finger | |
| Pinch | Two or more fingers moving toward a single point | |
| Spread | Two or more fingers moving away from a single point | |
| Press & hold | Participant deliberately holds finger on the surface for more than a second (subjective, may require the participant voicing intent) | |
| Spiral | Curve circling around a central point but at increasing distances from the center | |
| Loop | A full circle where the ends overlap but end at different places | |
| Circle | A complete circle shape | |
| Square | A complete square shape | |
| X | The letter X | |
| Release | The end of a Press & Hold when the participant removes the finger from the surface | |
| Micro roll | Participant presses a finger on the surface and “rolls’ it from side to side | |
| Direction | Right | Applies to Swipes |
| Left | Applies to Swipes | |
| Up | Applies to Swipes | |
| Down | Applies to Swipes | |
| Diagonal | Applies to Swipes | |
| Clockwise | Applies to Curves, Spirals, Loops and Circles | |
| Counter-clockwise | Applies to Curves, Spirals, Loops and Circles | |
| Vertical | Applies to Pinches and Spreads | |
| Horizontal | Applies to Pinches and Spreads | |
| Adverb | to | Optional modifier for Swipes and Curves, can describe ending a “Drag and Drop” |
| from | Optional modifier for Swipes and Curves, can describe beginning a “Drag and Drop” | |
| around | Optional modifier for Circles, Curves, Spirals, Loops, Squares | |
| across | Optional modifier for any motion | |
| in | Optional modifier for any motion | |
| on | Optional modifier for any motion | |
| Location | Object | Any icon, image, word on the slide |
| Multiple objects | More than one icon, image, word on the slide | |
| Between objects | The space between two Objects on the slide. Commonly used in Move and Paste actions | |
| Neutral space | Any space on the slide that does not contain information, inlcuding the corners and edges | |
| Framework | Visible borders (lines) on the edges of the surfaces | |
| Nature | Alphanumeric | A letter or number |
| Symbolic | A shape that is easily described (e.g., arrow, question mark) | |
| Abstract | A shape that is not easily described or is not normally seen. Gestures that are not Alphanumeric or Symbolic | |
| Order | Simultaneous | More than one gesture performed at the same time |
| Series | More than one gesture performed one after the other |
Table 3. Gesture results spreadsheet
| Gesture study results | ||||||||||
| Company | Participant |
Group |
Trial # |
Action |
Fingers |
Gesture ID |
Gesture Name |
Menu Requested |
Button Requested |
Comments |
Dan is the Director of Human Factors & Research at HumanCentric in the United States. Dan received his Ph.D. in Human Factors in 1994 and has been designing and evaluating user interfaces for a variety of companies since then. When Dan is not studying gestures for work, he is the receipient of them from his 2 children. 