Gary Hardock, Gordon Kurtenbach and William Buxton
Department of Computer Science
University of Toronto
Toronto, Ontario
Canada M5S 1A4
The intent is to match the flow of paper documents that we observed in the everyday world, and how they are marked up. The system is very much modeled on Wang Freestyle (Perkins, Blatt, Workman and Ehrlich, 1989; Francik and Akagi, 1989; & Levine and Ehrlich, in press). With Freestyle, annotations are made with a stylus, very much as would be made with paper and pencil and by voice. In what follows, we are only concerned with stylus annotations.
With Freestyle, the copy of the document that is distributed is only a "dumb" snap-shot of the original. Annotations are integrated as a separate layer to the "snap-shot." There is no recognition of the markings. To edit the final document, then, the user needs two versions: the "dumb" annotated copy and the original.
______________________________
* Proceedings of UIST'93, 259-266.
Our contribution is to incorporate mark recognition into the system and to explore some novel navigation tools that are enabled by the higher-level data structures that we use. In what follows, we describe the system and the results of our limited user-testing.
Figure 1: The Asynchronous Collaborative Writing
The principal author creates a document and then sends copies of it to collaborators. The collaborators then annotate their copies and send them back to the principal author. The principal author reviews the suggestions and possibly incorporates them in a revised version.
(b)
Figure 2: MATE in "Edit Mode".
(a) A user draws a move command. (b) After lifting the pen, the editing command is performed.
Copies of this document can then be emailed to the collaborators. Each reviews the their copy using MATE in "annotation mode". In this mode a collaborator can specify suggested changes by marking up the document. As with paper documents and Freestyle, markings are not interpreted as commands. They are treated strictly as annotations. These marks do not necessarily have to be editing commands. For example, a marking could be something as vague as "reword this paragraph". An example of various annotations is illustrated in Figure 3.
Marked up documents are returned to the primary author, who does the actual revision. This is done using two views of the document (Figure 4). The left view shows the marked up document. Each reviewr's annotations appear in a different colour. Additional annotations can be made in this window, but the underlying text will not change. The right view shows the current version of the document. No annotations are visible since any marks made in this window are interpreted as commands and executed immediately. This view is similar to the stand-aloneeditor window than may have been used in creating the original document.
The two views are iwork in concert, however. A user can point to an annotation in the left view and ask the system to perform it. The resulting changed document appears in the right view. The important characteristics of this design are that:
* the editing, annotation, viewing and incorporation tasks are integrated in a consistent, seamless manner.
Figure 3: MATE in "Annotation Mode".
In annotation mode, users mark up a document in MATE just as if they were marking up a paper document.
* the annotations are visually persistent, even after they are incorporated into the document, thus providing a mechanism to identify and select them at any time.
Other reasons for this split view design are identified later when the specific design issues are discussed.
* marks are visible
* people spend many years learning how to make and understand marks
* marks allow a very flexible protocol
* marks are spatially laid out.
All of the above properties make marks well suited for annotating documents. The fact that most people can make and understand marks gives a compelling reason to build mark-based interfaces to computer applications. But it is because marks possess all of these properties that enables the same marks to be used as both an annotation and as an editing command.
Figure 4: MATE in "Incorporation Mode".
In incorporation mode, a user can view the annotated document, and select which annotations to incorporate. Annotations t(seen in the left window) that have been "executed" appear as thin lines (eg., "ed"). Annotations that have not been executed appear as thick lines. Annotations are colour coded according to who made them. Annotations that represent commands can be executed by selecting them with the stylus. Annotations that have been executed can be "undone" by selecting them. The current state of the document appears in the right hand window. The user can navigate (scroll) independently or synchronously in each window.
One way of comparing the annotation process and editing a document on a computer is that in the first case a person is communicating with another person, whereas in the second case a person is communicating with a computer. The goal then becomes to design a method in which a person can communicate to both another person and to a computer application.
The fact that marking commands to the computer application are visible is the key. If, instead of immediately interpreting a marking command, the computer does not process the mark but simply leaves the mark visible, the mark can be thought of as an annotation. From this point of view, annotations are deferred edit commands. In terms of the different modes of MATE, annotation mode can also be called deferred mode, and edit mode is immediate mode.
(a)
(b)
(c)
(d)
Figure 5: Undoing irrespective of temporal order.
The initial text is shown in (a). (b) is the result after the deletion is performed. (c) is the result after the move is performed. After undoing the delete command, the desired result is (d).
* everyday use brings certain expectations, namely the maker of the annotation expects the person interpreting the annotation to understand the markings.
* The reader may not meet the writer's expectations in terms of understanding the writer's annotations.
* But the computer can give immediate feedback to the writer to verify the writer's expectations about the computer's understanding.
Table 1 shows that the addition of computer recognizing commands can aid in the communication between the annotation's creator and reader.
A preliminary study using transparencies was conducted to identify the issues and advantages of overlaying the annotations. The results indicated that the usefulness of this feature depends upon the density of annotations on the page. In some cases a single transparency was cluttered, and overlaying several transparencies made the annotations illegible.
As the problems of clutter are also found in a single set of annotations, we decided to address the more general problem of reducing the density of the markings. This is accomplished by providing support functions such as "Hide Set of Marks", "Show Set of Marks", "Hide Mark", and "Show Hidden Marks".
Understanding Computer Reader Understands MisInterprets Does not Understand Understands mutual understanding user will ignore computer neither aids computer's nor hinders user misinterpretation interpretation MisInterprets Computer understands, user and computer may may help the reader have different understand the marking misinterpretations, which may cause the user to possibly try a different interpretation Does not User may accept the Understand computer's misinterpretation
Table 1: Enhancing the Understanding between the creator and reader of an annotation.
If the reader does not understand or misinterprets the meaning of a marking, the computer may be able to understand it
Another way of thinking about this command, besides as a broken move, is as two commands, move to buffer with ID, and move from buffer with ID. This allows multiple text buffers, each with a unique placeholder symbol to identify it. It is important to note that the user chooses which placeholder to move text into, thus providing a strong connection between the placeholder and the text it contains. Also, because the placeholders containing text are visible, it is trivial to determine what text is contained in a placeholder buffer.
(a)
(b)
Figure 6: Moving text across pages using "move to star" and "move from star".
Figure 7: Marking menu for operations on annotation markings.
Navigation mode is actually implemented as a modified marking menu, shown in figure 8. Page flicks are always too fast for the menu to pop up, but the page push commands can also be selected by the menu. In addition to the page push and page flick commands, there are the GoTo and GoBack commands which are discussed in more detail in the next section.
Figure 8: Marking menu for navigation.
* how should scrolling and other types of navigation be coordinated between the two views?
* what should happen, and what should appear after an editing command is performed, or an annotation is selected to be done or undone.?
* what support is needed to aid the user in understanding the relationship between the two views?
GoTo Annotation is similar to GoTo Text, except that it is applied only to annotations. This means that GoTo Annotation can only be used in the annotation view. When GoTo Annotation is applied to a mark, the mark is interpreted, and if the interpretation is successful, the Edit view is aligned with the annotation view and the text affected by the command is highlighted.
(a)
(b)
Figure 9: Attempting to Update a Single View.
(a) shows the original view, before the update. (b) shows the result of an attempted update. Such an update is very difficult as almost all of the marks need to be changed or moved, including marks strictly meant to be treated as comments. Note that the placement of "Poor" is unknown.
These arguments do not mean that the two view approach is necessarily better. It is an alternative with its own issues and problems. It would be useful in the future to design the alternative interfaces and compare the advantages and disadvantages to each approach.
Five users were placed in a mock scenario, in which they were to pretend that they were the principal author given a marked-up document to edit. What they chose to incorporate was entirely up to them.
The results of the study showed that users choose to incorporate annotations by selection rather than by manually doing the edit themselves. Issues concerning histories, undo, and previewing were inconclusive as the users did not use these features.
The task was not complicated enough to bring out the issues regarding the coordination of the two views. However, there was some confusion between the GoTo Text and the GoTo Annotation commands.
The navigation testing was confounded by the stylus which was unreliable. In particular this made page flicks very difficult to specify. Several users preferred to use the scrollbar, instead of page flicks. Another problem was the response time of the system for page movements, which often confused users about whether it understood their command or not. These results do not conclude that page flicks do not work, but that more fine-tuning will be needed if they are to be useful and usable.
Francik, E. & Akagi, K. (1989). Designing a computer pencil and tablet for handwriting. Proceedings of the Human Factors Society 33rd Annual Meeting, 445-449.
GO corp. (1991a). The Point of the Pen. Byte, February, pp. 211 - 221.
GO corp. (1991b). Video Presentation at Stanford University, by Robert M. Carr, February 13.
Hardock, G. (1991) Design Issues for Line Driven Text Editing/Annotation Systems. In Graphics Interface `91, Calgary, June, 1991. pp 77 - 84
Kosarek, J. L., Lindstrom, T. L., Ensor, J. R., & Ahuja, S. R. (1990). A Multi-User Document Review Tool. In S. Gibbs, & A. A. Verrijn-Stuart (Ed.), Proceedings of Multi-User Interfaces and Applications (pp. 207 - 215). North-Holland: Elsevier Science Publishers.
Kurtenbach, G. & Buxton, W. (1991). Marking Menus In UIST: Proceedings of the ACM Symposium on User Interface Software and Technology. pp. 137 - 144.
Levine, S.R. & Ehrlich, S.F. (in press). The Freestyle system: a design perspective. In A. Klinger (Ed.). Human-Machine Interactive Systems. New York: Plenum Press.
Perkins, R., Blatt, L., Workman, D. & Ehrlich, S. (1989). Iterative tutorial design in the product development cycle. Proceedings of the Human Factors Society 33rd Annual Meeting, 268-272.
Posner, I.R. (1991). A Study of Collaborative Writing. Master's Thesis, University of Toronto.
Welbourn, L.K. & Whitrow, R.J. (1988). A Gesture Based Text Editor, in D. Jones & R. Winder (Eds.), People and Computers IV, Proceedings of the Fourth Conference of the British Computer Society Human-Computer Interaction Specialist Group. Cambridge, Cambridge University Press, pp. 363 - 371.
Wolf, C.G., Rhyne, J.R. & Ellozy, H.A. (1989). The Paper-Like Interface, IBM Technical Report RC 14615 (64399) 2/3/89, also in Designing on Using Human Computer Interfaces and Knowledge-Based Systems, G. Salvendy & M.J. Smith (Eds.), Elsevier Science Publ, Amsterdam, 1989, pp. 494 - 501.