Pen and Baton Interface for a Digitized Orchestral Score
A CSC490 Project

Introduction

This page documents the research project I conducted for CSC490. My project explored the possible use of overloading a pen-and-table interface with additional 3D gestural input interface.

In particular, I designed a prototype of a digitized music score for an orchestra. The digital score will be displayed on a tablet computer that is built into the music stand. The conductor would use the pen to edit and navigate through the digital score, and the same pen would double as a baton for conducting. The movements of the baton are tracked real-time using the Vicon motion tracking system, and the current location in music is transmitted to the digital score. If multiple musicians are involved, each of their digital score will be synchronized, and the current location and the tempo will be transmitted to all units.

Most of the prototype was written in Java. Some parts were written in C++, wrapped with a JNI interface in order to communicate with the Java side of things. The reason for having some parts written in C++ was because I wanted to use the available C++ sample code for interacting with the Vicon system.

The prototype was designed in a modular way. There is the main display module which displays the music and the current progress in the music. There is also the baton reader module, which interprets the baton movements and translates them into tempo data.

The modular design was important for keeping the project on schedule. At the start of the project, I knew that it would take me a while to learn to use the Vicon system, and to integrate it into my project. By using a modular approach, I was able to write most of the prototype application independently from the Vicon system. Initially, I wrote a placeholder baton reader module, which simulated the baton movements with the mouse. After the foundation of the prototype was written and tested this way, I was able to focus on getting the Vicon portion of the prototype up and running. I eventually replaced the placeholder baton reader moduel with a real, working version.

Challenges

One of the biggest challenges in developing this prototype was learning about the Vicon motion tracking system. Since I was totally unfamiliar with the technology, it took me a while to get used to the workflow of setting up the cameras and calibrating them, and capturing data. However, I found that once I learned how it works, it was fairly easy to use and I did not have any problems after that.

Another challenge was writing an interface to the Vicon system that will read the real-time movements of the objects. There was sample code available, but it was written in C++. Compiling this into a DLL and wrapping it with a JNI interface for my Java code took a bit of time. At the end, it was a good learning experience, and it was quite satisfying to see everything up and running.

After overcoming the technical difficulties, I concentrated on improving the HCI aspect of the prototype. For example, the material I used as the baton was too flexible, and everytime you swung it, the tip would bounce due to the reflex in the baton. This effecitvely resulted in noise data, which interfered with determining the correct progress in the music. The easy way to fix this was to filter out miniscule movements, but it was a hard to decide the exact threshold for the filter.

Another challenge was to design a usable representation of the musical score. I originally looked at third-party Java packages that display musical staves and notes, but I could not find one that was stable and flexible enough to be adopted into my application. In the end, I decided that for the scope of this prototype, it is enough to simply display the beat numbers along with the names of the notes (e.g. C, C#, D, Eb, E, F, etc...). See the screenshot below for more details on what the final prototype looks like.

Conclusion and Future Considerations

All in all, I think the prototype worked as expected. There is still some issues regarding the sensitivity of the motion tracking. Perhaps there is an algorithmic way of predicting the next beat, by monitoring the acceleration of the baton and detecting when it slows down. This would help in dynamically filtering out unwanted vibrations in the baton.

It would also be helpful to gather some feedback from potential users. In order to do this, I would need to add some sort of a peer-networking module, so that multiple copies of the application running on separate machines can synchronize with each other. There would be one Master copy that is controlled by the conductor, and several Slaves that are controlled by the musicians. As the conductor waves the baton, the Master will notify the Slaves of the current tempo. This was part of the original project plan, but unfortunately, I ran out of time before I can conduct any real experiments.

Overall, I found the project a rewarding experience, and I would consider following it up with further research and development if given the opportunity.

Videos

Screenshot

Click on the picture to view a larger version.

Pictures

Presentations

The following are PowerPoint presentations, prepared for CSC490 group meetings.