Kinect Investigations

Microsoft Kinect Sensor and OpenNI Framework

Originally known as Project Natal, the Kinect is a motion-sensing device used as a hands free controller with Microsoft’s X-Box games console. The kinect uses motion sensing data that includes depth data (via infrared) in order to allow users to control gameplay through gestures, motion and speech.

In 2010, open source drivers started to become available for the kinect and it can now be used with multiple open source libraries as a sensing device, much like a webcam, that will output data relating to X, Y and Z positions and movement velocity via OSC (Open Sound Control) (http://opensoundcontrol.org/).

Thus the data captured from the kinect can be used to control, manipulate and interact with space and place by routing the data accordingly.

For the purposes of these three tests, I was mainly interested in how to unpack the kinect OSC data using Puredata and re-represent that data in a number of different ways in order to control virtual objects and interact with the spaces.

TEST #1: This model was based on the “picture cube” that I remember from the 1970s, and was developed to be shown at the annual digital storytelling festival in Aberystwyth in June 2011. In the demonstration we were attempting to describe how new media applications and new technology might impact on the “traditional” form of digital storytelling.

The original application consisted of two picture cubes controlled by two users (in this case myself and a member of Dr. David Frohlichs’ team) where the concept was to locate and tell the story related to a picture that evoked a particular memory. By “stepping out” of the space and then re-entering, the second user was able to take control of the first cube and vice-versa, encouraging a collaborative storytelling effort based on memories of the same picture.

(The “picture cubes” at DS6, June 2011)

For the purposes of demonstrating this application the following is a single user version of the picture cube. Using Apple’s Quartz Composer (http://developer.apple.com/graphicsimaging/quartzcomposer/) to build a 3D cube and render a picture on each of the six sides, and the openNI framework for capturing the kinect data, this application allows a user to control the rotation of the picture cube across multiple axes using their hands as controllers.

(OpenNI client on the left and the Quartz picture cube on the right)

TEST #2: OpenNI2TUIO and Puredata. This is a proof of concept aimed at extracting and using the depth data from the kinect to control virtual objects (in this case a random film player screen). The data is unpacked and routed in Puredata via TUIO protocol.

(OpenNI2TUIO interface on the right and PureData output on the left)

TEST #3: OSCeleton and Puredata. OSCeleton is another kinect library based on the OpenNI build that provides a much more comprehensive data set that translates each joint position on a tracked skeleton. In this example the texture of the joints is mapped from the random video player in TEST #2.

Conclusions, Issues and Development

Over the last two years the kinect has become a large focus of interest for those involved in pervasive computing and a simple Google search will reveal a plethora of projects, innovations and libraries that can really open the possibilities of this device as an interactive controller that can re-map and re-visualise space and place.

The main focus of interest in this device for me has been mainly to provoke an insight into its potential applications within a moving image and new media context. The kinect is very adept at communicating with other devices and other software in order to produce work that goes far beyond “blob tracking” (as in CVC and other tracking software), even allowing users to appear and interact in virtual spaces such as OpenSim and Second Life (http://www.youtube.com/watch?v=YDVDQJLStYo).

Throughout these experiments it seems to me that the most relevant client would be OSCeleton with its ability to motion track multiple users in 3D, however, all of these OSC clients have presented their own issues around unpacking and routing the data appropriately in Puredata. Having said that, once proof of concept is established, the Puredata patches can easily be saved as abstractions and called into future projects very easily. So in this way I am building a repository of kinect patches which can be modified and re-used dependent on the project needs.

In terms of development of the application of the kinect in interactive spaces, there are more libraries and clients appearing almost weekly as the online community develop new and innovative uses for the kinect. One of many examples that I am currently investigating involves the interaction with a 3D video wall using only kinect based gestures: as the user “touches” the video screen the movement of the hand can interact with and change the playback speed, audio track and so on (http://www.keyboardmods.com/2010/12/howto-kinect-openninite-skeleton.html).

The kinect seems to really represent a shifting idea of perception of experience that seems to be blurring the lines between “real” and “virtual” as actions performed in “real” spaces (both individually and collaboratively) become the triggers of “virtual” experience in “virtual” places.