I really like Donald Norman’s essay »Natural User Interfaces Are Not Natural« that he wrote for the ACM interactions magazine. In this essay Norman shares his view on the role of natural user interfaces (for him represented, e.g., by »speech, gesture, or the tapping of the body’s electrical signals for “thought control,”«) in terms of interaction with future computer systems.

I have got the impression that marketing guys love to pronounce the end of mouse and keyboard interfaces that will in their opinion (or at least in their words) become obsolete as their place will be absorbed by natural user interfaces that are based on gestural interaction provided by systems like the Kinect or by multi-touch screens. I do not share this opinion. As it comes to me, (nearly) every interaction approach has its right to exist and usually is best for certain types of tasks. I doubt, for instance, that there soon will be a better way to enter text than with a standard keyboard based on haptic keys.

So, Norman provides one of the few pieces of literature that keeps cool about the natural user interface hype and gives profound arguments where he sees their potentials and pitfalls. I strongly recommend to read the whole essay. In the following, I cite some strong points his essay makes.

It is also unlikely that complex systems could be controlled solely by body gestures because the subtleties of action are too complex to be handled by actions–it is as if our spoken language consisted solely of verbs. We need ways of specifying scope, range, temporal order, and conditional dependencies. As a result, most complex systems for gesture also provide switches, hand-held devices, gloves, spoken command languages, or even good old-fashioned keyboards to add more specificity and precision to the commands.
…
Gesture and touch-based systems are already so well accepted that I continually see people making gestures to systems that do not understand them: tapping the screens of non-touch-sensitive displays, pinching and expanding the fingers or sliding the finger across the screen on systems that do not support these actions, and for that matter, waving hands in front of sinks that use old-fashioned handles, not infrared sensors, to dispense water.
…
All new technologies have their proper place. All new technologies will take a while for us to figure out the best manner of interaction as well as the standardization that removes one source of potential confusion. None of these systems is inherently more natural than the others. The mouse and keyboard are not natural. Speech utterances will have to be learned and gestures carefully developed and standardized through time. The standards don’t have to be the best of all possibilities. The keyboard has standardized upon variations of qwerty and azerty throughout the world even though neither is optimal–standards are more important than optimization.

Are natural user interfaces natural? No. But they will be useful.

Donald Norman – »Natural User Interfaces Are Not Natural«

Throng comes in two flavors:

I. Throng provides…

1.) … a Tuio proxy (or bridge) that is necessary for applications that may only access network sources on the local machine (like Adobe Flash). If other devices (like iPhones) want to access such an application via Tuio over the network, a proxy will be necessary.

2.) … multiplexing of Tuio alive messages: If Tuio messages from multiple sources access one client application (or as it is called in OSC: a server application), all sources use alive messages that only contain their alive ids. If the client application does not support different source providers (which is the case for many tuio libraries), each Tuio source will remove the alive objects and cursors of each other Tuio source causing, e.g., touches to flicker.

3.) … individualization of the session ids of each source: if different Tuio message sources use the same session id range, the results can be … awkward. Hence, Throng individualizes them.

4.) … one consistent Tuio Fseq message order for all arriving packets.

5.) … adding of a Tuio source message to each Tuio package or, if such a message already exists, modification of this message after the “@” to contain source_ip:source_port to allow for discerning of sources in the client application.

II. Throng Custom provides an application in which the x and y values of Tuio set messages of different sources can be cropped, shrunk and moved in order to allow for, e.g., a two camera – two tracker system on two computers for one interactive surface.

Throng is hosted (with full code available) on Google Code.

Throng is based on Flosc by Ben Chun the illposed OSC classes and the TUIO Java implementation by Martin Kaltenbrunner.

Read more at Engadget and Microsoft.

English:
umami is an intelligent interactive media system which adds a hitherto unknown dimension to your kitchen.

A multitouch surface integrated into the kitchen and the integration of your everyday cooking gear with the interaction process enable you to operate umami intuitively – automatic recognition of food articles helps with the selection of recipes.

With umami experienced chefs now have the opportunity to take on advanced recipes in a new manner as well as to discover additional sources of inspiration. Beginners are being guided through meal preparation by means of step-by-step instructions and tips & tricks from the kitchen world in general.
When cooking with a group of people, everyone that likes to can easily participate.

Deutsch:
umami ist ein intelligentes, interaktives Mediensystem, dass den Lebensraum Küche um eine bisher unbekannte Dimension erweitert und das Kocherlebnis neu definiert. Es unterstützt seine Nutzerinnen und Nutzer, indem es die Arbeit in der Küche erleichtert und zusätzliche Anreize schafft Speisen zuzubereiten. Die Küchenoberfläche selbst wird zum Interface und hilft bei der Planung und Durchführung des Kochvorgangs. Das System kann von mehreren Akteuren gleichzeitig bedient werden und soll den Spaß am gemeinsamen Kochen fördern. Dabei stellt sich umami als neuartiges System dar. Durch die Zusammenführung mehrerer Aufgaben und die Möglichkeit der zentralen Verwaltung von Informationen, wird der Kochprozess erleichtert.

Bei der Entwicklung wurden zwei Zielgruppen besonders berücksichtigt: Erstens: Ambinitionierte Hobbyköche, die ihre Kreativität und das vielfältige Ausprobieren beim Kochen in den Mittelpunkt stellen: umami unterstützt diesen Kochprozess durch Tipps und Anstöße.

Zweitens: Gesellige Hobbyköche, die gerne mit mehreren Leuten zusammen kochen und das Kocherlebnis genießen. Da das System die Zubereitung der Gerichte und die Durchführung der einzelnen Arbeitsschritte ausführlich anleitet, können auch ungeübte Köche mit Spaß und Genuss an einem Kochabend teilnehmen.

Zusätzlich kann umami dem Benutzer helfen, spezielle Bedürfnisse zu beachten, z.B. Allergien, welche im Benutzerkonto des Systems hinterlegt werden können.

Außerdem unterstützt es die Phantasie der Nutzerinnen und Nutzer, indem es ihnen neue Wege aufzeigt Lebensmittel zuzubereiten. Das Ganze wird ergänzt durch ein Multitouch- Interface, welches dazu anregt, mit dem System zu interagieren. Die Bedienelemente werden nicht mit einem abstrakten Zeigegerät, einem virtuellen Fadenkreuz oder Mauszeiger berührt, sondern mit dem eigenen Finger, was die haptisch-sinnliche Dimension des Projekts unterstreicht.

Fancy: Special drinking glasses, which allow to recognize the amount of fluid in the glasses.

After being able to compile the Touchlib I quickly grew curious about what all those blobconfig parameters in the Touchlib’s config.xml REALLY do.

First of all, some parameters are really useful and it is a shame that the functionality is (nearly) nowhere documented. Even the pros in the nuigroup seem not to know for which purpose they are being used.

If someone wants to know why I am using the Touchlib and not tBeta aka CCV that is because CCV keeps crashing on my computers and I do not have Visual Studio 2008, which is necessary to compile CCV on Windows. Additionally, I had a student adding fiducial support to the Touchlib (by adding the reacTIVision code). Thus, I currently stick to the Touchlib.

Disclaimer: The code version of the Touchlib I used is the one that my student tweaked while adding reacTIVision fiducial support. You can find the Touchlib’s source code at the Touchlib code repository in Google code. However, many users are constantly submitting their code changes to this repository. E.g. the newest file of CBlobTracker, which does the actual tracking of blobs in the camera’s image, does not include the setup function anymore in which certain values of config.xml have been set for the tracker. Thus, those values are not being used in the code version anymore. However, they should still be used in the binary version that you can download here. I will use this older version of CBlobTracker to explain some parameter’s functionalities of config.xml. However, I think that in the Touchlib’s binary version those parameters are being used.

blobconfig
Touchlib searches in subsequent captured frames from the camera for the same blobs. I.e. if a blob in the previous frame has been found at position (x=250, y=300) and in the current frame a blob has been found at position (x=251,y=299) Touchlib interprets this as the same blob. However, tracking is not that simple that Touchlib could search for a blob in the direct surrounding but for each blob of a previous frame all blobs in the current frame are being compared for their distance. Touchlib thinks that the blob in the current frame with the shortest distance is the same blob (It is not that simple but let’s stick to this for now).

1. Attribute minDimension (very handy!):
This is the minimum size that a blob must have. If you e.g. have lots of small white dots in the RectifyFilter that are smaller than minDimension they must not be tracked. (see CBlobTracker line 209). Default value of minDimension is 2. I use a minDimension of 5. However, the appropriate minDimension depends on your camera’s resolution and your table’s size.

2. Attribute maxDimension (very handy!):
This is the maximum size that a blob can have. (see CBlobTracker line 209). Default is 250, which is way too big. I use a maxDimension of 20. However, the appropriate maxDimension depends on your camera’s resolution and your table’s size. This value is extremely handy if you want to prevent that a big object (like a cup or a sheet of paper) that is being put onto the touch surface is recognized as a touch.

3. Attribute distanceThreshold:
With distanceThreshold you can configure with which distance another blob in a consecutive frames might still be interpreted as the same blob. E.g. if distanceThreshold is big the distance between the two blobs can be big and vice versa. (see CBlobTracker lines 755 ff.). I currently do not see the point of this parameter.

4. Attribute minDisplacementThreshold (does nothing so far):
Should do basically the same as distanceThreshold but inverted: minDisplacementThreshold configures how far a blob must have been moved in order that an update is being sent. However, it is being set in CBlobTracker in line 51 but nowhere used in the code.

5. Attribute ghostFrames:
I do not know exactly what ghostFrames does. However, I guess that it regulates the amount of frames in which a blob is not being found in the current frame (for the count of ghostFrames frames) but not deleted as well out of the list of blobs. Thus, if a ghost frame is being sent by the camera (maybe an empty image because of capturing problems) not all blobs are being deleted but the ghost frame can be ignored. However, as I said I do not know the exact functionality. For more detail see CBlobTracker lines 717 – 732.

As already mentioned in this post, we presented Vispol at the booth of the state of Hessen/Germany at the fair “Moderner Staat” in Berlin on 24. and 25. November 2009. Vispol has been shown on the round multi-touch and TUI table from Sensory Minds.

Here are a few photos from the fair. Please apologize the poor quality of the photos as my camera did not see the sense in making proper use of its auto-focus functionality.

I love those nail tattoos of this young Vispol user:

My software Vispol will be shown at the booth of the state of Hessen/Germany at the fair “Moderner Staat” in Berlin on 24. and 25. November 2009.

It will be shown on the new really cool round [!] multi-touch and TUI table from sensory minds.

I will be there on both days.

In terms of the NUI research project Squidy at the Konstanz University the TUIO overlay SurfaceToTuio has been developed for Microsoft Surface. Download the SurfaceToTuio component from the sourceforge project site (or via direct download).

SurfaceToTuio is available as source code only (there is no executable). However, you can just upload the SurfaceToTuio folder to your Surface, doubleclick the Visual Studio project and compile it in the Surface’s Visual Studio right away.

Finally, a decent software interface for a decent hardware.

Update: The project of the Konstanz University has a bug, which has been resolved by Julian. Check out the stuff in his repository.

Figure 1: 3D scatter plot of Fisher’s dataset assorted with the k-means algorithm

In VotkA users can interact with a three-dimensional cluster analysis visualization via a Tangible User Interface (TUI), the iPhone app OSCRemote or both in parallel. The TUI is provided by a small box that is technically based on Seth Sandler’s cheap multi-touch pad (you can see the box in the youtube video at the end of this article). On top of the box fiducials from reacTIVision are being tracked. Whenever a fiducial is being recognized a TUIO message with the fiducial’s id, position and rotation will be sent to VotkA.

In terms of the cluster analysis the k-means algorithm has been implemented. The underlying multivariate data that is being visualized is Fisher’s iris dataset. In this dataset different aspects of iris blossoms are being stored (e.g. petal length/width, sepal length/width). Within the visualization this data is being dispersed in a cube according to different aspects (e.g. on the x-axis petal length, on the y axis sepal width and on the y-axis sepal height).

Figure 2: Scatter plot projected on the dimenions sepal length / petal length

The implemented visualization methods 3D scatter plot (see figure 1), scatter plot matrix / Grand Tour have been slightly adjusted to visualize Fisher’s dataset. The interaction with the Grand Tour visualization method is being carried out in the TUI with a real cube, which incorporates the projection of the three-dimensional data on the dimensional layers in the graph. By tilting the cube in the application the appropriate projection side will be displayed on the screen (see figure 2 and video).

When interacting via fiducials an additional sound feedback mechanism has been provided by converting TUIO messages in VotkA to midi signals that are being sent to a software sequencer. Midi channels are mapped to fiducial ids and midi signals are produced according to position and rotation of the fiducials. Actually it should not have been necessary to convert TUIO messages in VotkA to midi because reacTIVision supports both outputs, midi and TUIO. However, only one output channel at a time can be handled by reacTIVision.

Video of interaction with VotkA. Please note that colors are being poorly displayed and that midi sound output has been disabled in this video:

Employed technologies:

• Java
• Processing for VotkA’s visualization part and for TUIO message processing
• reacTIVision for fiducial tracking
• mmj-Library for midi production in VotkA
• k-Means for cluster analysis
• Fisher’s Iris Dataset
• OSCRemote App as iPhone TUIO Client (for the production of the OSC messages)