Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection

Overview

This paper discusses frustrated total internal reflection as a simple, inexpensive, and scalable technique for enabling high-resolution multi-touch sensing on rear-projected interactive surfaces. In more detail, they go over previous applications, provide implementation details, discuss results from their initial prototype, and outline future directions.

Previous Applications

This technique was used in application such as finger prints since at least the 1960’s, a painting application in the 1970’s, robotics, and other applications. This technique is widely known and has been around for a long time. The principle is also used for fiber optics. This is a pro since the technology is familiar and has been studied and implemented already. This brings down the cost for using this technique.

Implementation Details

Simple imaging techniques such as rectification, background subtraction, noise removal, and connected component analysis are used for each frame of data. Since these algorithms are widely known and used, this is a pro when it comes to implementing this technology. Since this technique is camera-based, this implies that there will be a drawback with various backgrounds and background lighting. This is a con for this technique compared to a capacitor based system that is not affected by background noise.

Results from Initial Prototype

There was a disparity from using a ¼” waveguide for their prototype, but they said there was no reason, other than ease of implementation, to use a smaller one. Their surface became contaminated easily after usage which caused problems. However, with the adaptable background algorithm, it will learn the noise and not consider it as a touch. Also, the system depends on the optical qualities of the object being sensed and this will cause the system to not detect non-human touch objects such as a mug or hand with glove. Also, a user with dry skin will have to push harder on the screen to achieve the success of a user without dry skin. The surface will have to be cleaned periodically to maintain accuracy. Also noted, is that some of these issues could be remedied by engineering a compliant surface overlay.

Future Directions

They discuss upgrading the system to differentiate between different points of contact. Currently, there is no way to distinguish if two finger came from the same hand.

Discussion

• Low-level algorithms to detect touch, light, background, etc...
• How does this technique compare to 3M & MS Surface systems?
• Adaptive algorithms to target user attributes: dry skin vs. non-dry skin

Design and Validation of Two-Handed Multi-Touch Tabletop Controllers for Robot Teleoperation

Overview

This paper describes a virtual controller for a multi-touch surface. They then compare this remote control to a physical controller in an experiment that was conducted on a made up replication of a disaster area where they had to locate victims. Finally, the results are explored and presented.

DREAM Controller

The DREAM controller is a multi-hand controller that is similar to a video game controller, but is wrapped around flat on a screen. The controller automatically appears and follows a hand that is placed on the surface. The biggest drawback of this controller versus a physical controller is that there is no immediate feedback. A user would have to memorize the controller and learn that the button was pressed without physically feeling it in their hands as is such with a physical controller. If a user pushes the ‘A’ button on a physical controller with their eyes closes, it is evident. And, with a virtual controller on a surface, it is not evident; the user must trust that the system got the command.

Experiment

There is no mention of making the “training” of using the joystick and DREAM controller. It may be true that the experimenters favored “training” the subjects with the DREAM controller over the physical controller. Since the experimenters are from UMass and the DREAM controller was designed there, they may unknowingly put more emphasis on training people to use the DREAM controller versus the physical controller. A safety net should have been set (it may have been but not mentioned in the paper) that put an equal amount of training and enthusiasm on each of these two methods. With the results being fairly statistically insignificant and this unknown, the experiment & results are a bit skeptical to me.

Results

The experiment was a bit sceptical. There was a very weak significance in the user response (travelling further, finding more victims) to the advantages of using the virtual remote.

Another interesting result was that they were able to travel so far that they lapped around and found the same victim more than once.

Discussion

• Bias in training operators to use physical controller versus using the DREAM controller
• The operators may have seen some favoring in the DREAM controller training versus the physical controller that the experiment conductors did unknowingly.
• The statistical significance was very small: this may show that there is not much of a difference between the two.
• It would be interesting to see an experiment comparing the DREAM controller to an actual controller that is similar to the controller from a Playstation or XBox.

Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces

Overview

This paper goes over freehand gestural interaction with direct-touch computation surfaces. The paper then goes over the design principles: gesture registration, gesture relaxation, and gesture & tool reuse. They tested the design principles using the annotate gesture, wipe gesture, cut/copy & paste gesture, and pile-n-browse gesture and they conducted these in a user evaluation.

Design Principles

One drawback, mentioned in the gesture relaxation section, was that the height of a table - have it be a coffee table or desk height - may impact the system’s ability to read the gesture. If a user attempts to put down three fingers, the coffee table height while standing may only read the finger tips and the desk table height when sitting down may read the finger tips and much of the rest of the fingers. If these were two different gestures, the system would not be able to differentiate them.

For the “gesture and tool reuse section”, they are hinting towards using the same gesture to mean different things in different contexts. This is similar to having modes, which is typically not a good thing according to most HCI studies. This could confuse a user and have them doing a certain gesture expecting a certain result, but having a different result. However, there must be cases where it is clearly evident that using the same gesture while in a different case would yield an expected result, but designers would have to be careful.

Another note, a user may be in the middle of one gesture, post gesture registration, and have the urge to perform another gesture, without finishing the first gesture. This could cause some confusion amongst users and yield unpredicted results. Possibly, some system feedback could keep user aware that the prior gesture has not finished yet.

It is good that reusing gesture primitives, allowing users to learn a smaller vocabulary, is possible. Designers must be careful to not use these primitives for completely different tasks.

User Evaluation

In the study, the users appeared to have learned the gestures fairly easy, but some struggled with the “piling” gestures. It would be interesting to see if users remembered these gestures after not using them for some period of time.

Discussion

• A common vocabulary of gesture primitives
• Using them in different contexts should be for similar tasks and yield similar results. For example, dragging one finger should be used for moving something in all contexts, not resizing in one context and moving the screen in another context. To me, that would be confusing. There should be a common vocabulary where, say, four fingers would drag the screen and not ever move an item on the screen.
• Memory of gestures
• It would be interesting to see a study of how users memorize these common gestures and gesture primitives over time. They may become second nature or users would keep having to refer back to some guide.
• Primitives as letters, and gestures as words
• It is interesting to compare the gestures & primitives to a full human language
• Each primitive would be like a letter or word and gestures would be like a sentence. Users would be able to “speak” to the computers with this language.

User-Defined Gestures for Surface Computing

Overview

This paper was about a study involving user-defined gestures for surface computing and they discuss developing a vocabulary based on the common gestures that users perform without any outside influence. They performed a study where they would show an action on the screen and the user would try to guess the gesture that caused the action. Further, they created a taxonomy, analyzed the results in detail, made observations, and detailed discussions about the experiment & findings that they discovered.

Developing a User-Defined Gesture Set

One idea presented was that users are not designers. They emphasize that care must be taken when building a gesture set using user defined gestures. Testing on a large number of people, they will get the general consensus as to what is the most popular gesture to cause a specific action, but it may not be the best gesture to cause that specific action. Another interesting point is that the feedback from a computer might change the next gesture, or the next primitive step of a gesture. For example, when a user puts his or her hand down, a ripple appears. If the ripple did not appear, they may have taken a different course. This may impact the experiment, but they stated that they removed feedback.

Discussion

• It would be interesting to see a study where the subjects were unaware that the testers were looking for them to provide the most logical gesture to cause an action
• User’s could be giving a UI and told to take make some gesture to cause an action (such as delete, move, cut, etc...) in a “dummy” app, then collect that data-this way the user would naturally try to do some gesture to cause the action
• Integrating past knowledge - learned from single point device - will bias the most “natural” form of gestures: it would be interesting to see if results are the same when people that are not computer literate were tested.
• Mixed UIs:
• Standard computer input along with multi-touch screen: this would be the most practical application for typical computer use. An example would be a small box in the corner with common tasks like cut, copy, paste, etc... (another type of shortcut) or drawing a question mark on the screen for help, drag and drop an item with fingers if a user finds it more convenient than doing it with a mouse, etc...