Many mobile devices have touch-sensitive screens that people interact with using fingers or thumbs. However, such interaction is difficult because targets become occluded, and because fingers and thumbs have low input resolution. Recent research has addressed occlusion through visual techniques. However, the poor resolution of finger and thumb selection still limits selection speed. We address the selection speed problem through a new target selection technique called Escape. In Escape, targets are selected by gestures cued by icon position and appearance.

Fig. 1 shows in more detail how the Escape selection technique works. The user presses his thumb close to (but not necessarily on) the target icon (more specifically, within the area of a "Parhi" box, explained later), and then makes a linear gesture in the direction that the target icon points (we call this pointer "a beak"). Icons can be packed close together, but are still easily distinguished as long as each icon is well-separated from the other icons that have the same gesture. We say that no two identical icons can share the same "Parhi box," in reference to the previously-mentioned finding by Parhi et al. that, to keep error rates low, targets should be at least 9.2mm x 9.2mm square. Although the minimum-area shape of such a target is, in practice, not likely a box, we ignore this distinction here.

An advantage of this approach is that it relies less on the user's visual feedback loop. In traditional target selection, the user moves a cursor closer to the desired target, looks to see if the cursor lies within the target, and then repeats these steps until the cursor is properly positioned. This process can take several hundred milliseconds for small targets. With Escape, the user need only use their visual ability to recognize the position and appearance of the icon. After this, they need only tap their thumb in the 9.2 mm box around the icon position and make the gesture. Their visual system is used only to guide their thumb to the first point of contact, not to direct a cursor after the initial contact. Also, there is no need for the user to reorient to any other visual changes, such as the position of the Offset Cursor or the dynamically appearing inset of Shift. Explained in terms of user interaction techniques, Escape replaces the visually demanding and time-consuming target-selection task that follows the initial thumb press with a much coarser selection task followed by a crossing task of making a sufficiently-long gesture. For the demonstration of Escape, please watch the video.

Our user study showed that Escape is significantly faster than but is comparably accurate to Shift, an alternative technique, on a similar task. For more details, Please see our paper.

.

Fig. 2. By carefully assigning icons, overlaps and unnecessary icon proximities can be avoided. (a) Random assignment; (b) Our overlap-avoidance algorithm. The circled region shows a case where the algorithm avoids placing identical icons together, and the squared region shows how the algorithm avoids icon overlap.

The algorithm's primary task is to find an assignment that allows icons to be well-separated from other icons with the same gesture. Additionally, the system should minimize icon overlap, especially of the beak. This problem is similar to graph coloring, which is known to be NP-complete even for planar graphs. Thus, there is no known efficient optimal algorithm. Here we describe a heuristic algorithm that appears to work well in practice. However, we implemented a greedy algorithm based a basic graph coloring algorithm for this purpose. For more details, Please see our paper.

Fig. 2 illustrates how our algorithm improves upon a random icon assignment. For a high rate of success, the algorithm can handle the cases, where the density of icons is 2.3 icons per square centimeter.The algorithm calculates the arrangement of 100 items in around three seconds on a Windows Mobile emulator. Note that icon assignments can be precomputed offline in some applications. Moreover, Escape can also be useful for manually-designed user interfaces, in which case the maximum density can be predictably achieved.