Error: no such theme PatternSkinTheme
You are here: Foswiki>Glimmer Web>GleamWiki>GettingStarted (2012-08-19, JennelleNystrom)EditAttach

Getting Started with GLEAM

What is GLEAM?

GLEAM is the Grinnell Livescripting Environment for Art and Music. It is a Kinect library implemented in Scheme that relies on Sensebloom's OSCeleton program. OSCeleton sends the coordinates of the skeleton's joints from the Kinect as OSC messages. The GLEAM Kinect library interprets the messages received by OSCeleton and provides the user with a gesture-based interface, in which user-defined handlers can be attached to gestures to create art and music.

The Kinect Scheme library was written to be used by Andrew Sorensen's Impromptu application, a live coding environment for Scheme that focuses on audio production and includes support for OSC messages. Directions for setting up OSCeleton with Impromptu and GLEAM are included below.

Installation

System requirements

Right now, only MacOSX 10.6 (Snow Leopard) and above with a 64-bit Intel based CPU can run GLEAM. You will also need a Microsoft Kinect camera.

Downloading OSCeleton and its dependencies

  • Follow the directions on avin2's SensorKinect github page to install the software and all its dependencies into a Kinect directory.
  • After installing OpenNi, NITE and SensorKinect, use the samples provided in the NITE and OpenNI directories to test that they work as expected.

Enabling more than one hand

  • In the terminal window, move into the NITE subdirectory of your Kinect directory. You should see five directories named with names of the form 'Hands_1_3_0'.
  • For each directory named Hands, navigate to the Data subdirectory. Open the Nite.ini file within the Data directory in a text editor. You should see something like this:
  1. [HandTrackerManager]
  2. ;AllowMultipleHands=1
  3. ;TrackAdditionalHands=1
  • In each Nite.ini file for each Hands directory, remove the semicolons to uncomment the lines AllowMultipleHands=1 and TrackAdditionalHands=1.
  • Run NITE/Sample-Pointviewer again to make sure both hands are being tracked.

Patching and Installing OSCeleton

  • Download OSCeleton from the Sensebloom github page into your Kinect directory. Enter the OSCeleton directory and run make.

Test OSCeleton: make sure the Kinect is plugged into a power source and plug the camera into a USB drive. In the OSCeleton directory, run ./osceleton -w -r
OSCeleton automatically sends OSC messages to port 7110. To make sure OSC messages are being sent, open another terminal window and navigate to the OSCeleton directory. 
Run ./osc2text 7110. Make sure messages are being printed.

  • If no messages are being received by osc2text, go back through the previous steps and to ensure proper installation before moving on.
  • Once OSCeleton has been tested, download the patch from the GLEAM github page into your Kinect directory or the OSCeleton subdirectory.
  • Navigate to the OSCeleton directory and run the following command, modifying the path to the oscfilter.patch as necessary:

 patch -p1 < ./oscfilter.patch

  • run make. This adds a new executable called oscfilter to the osceleton directory. Oscfilter reads oscmessages from a port and forwards only specific joint messages to another port.

Test the oscfilter: With the Kinect plugged in, run ./osceleton -r -p 8000 to send OSC data to port 8000. 
In another terminal window, run ./oscfilter 8000 7110 "l_hand" "r_hand" 
This reads OSC messages from port 8000 (the port the Kinect is sending message to) and forwards only "l_hand" and "r_hand" messages to port 7110.
Open another terminal window and run ./osc2text 7110 to print all the messages being sent to port 7110. 

  • If only "l_hand" and "r_hand" messages are being received, setup of OSCeleton with oscfilter is complete.

Intalling Impromptu and adding the gestures library

  • Download the .dmg file from Impromptu's downloads page and install the file in the Applications folder
  • Follow the instructions on the downloads page to set up Apple Keybindings
  • Download the kinectlib.scm file from the GLEAM github page.
  • Move kinectlib.scm to the ~/Library/Application Support/Impromptu directory
  • Open Impromptu. Make sure a "Loading... kinectlib.scm" message appears in the log window on start up.

Starting Impromptu and GLEAM

To set up Impromptu and OSCeleton to use with GLEAM, complete the following steps:
  • Navigate to the Sensebloom OSCeleton directory. Run the following command to start OSCeleton sending messages to port 8000:
  • ./osceleton -r -p 8000
  • Wait for calibration. Once that is complete, open a new terminal window and navigate again to the OSCeleton directory. Run the following command to have the oscfilter read from port 8000 and forward specific joint messages to port 7110. After naming the write port, provide a list of OSCeleton strings to state the joints that you would like to be forwarded. For a basic session, use the hands. A full list of joints can be found on the OSCeleton github page.
  • ./oscfilter 8000 7110 "l_hand" "r_hand"
  • Now, open the Impromptu window. Since the Kinect library should load automatically, GLEAM procedures should be active as soon as Impromptu is opened. To start gesture tracking, run (kinect-start! 7110) then (gestures-start!).
  • If the library is not set up to automatically load, use Scheme's load procedure to load kinectlib.scm

Programming in Scheme

A basic understanding of the Scheme programming language is necessary to use the GLEAM Scheme library. To familiarize yourself with Scheme, we recommend reading an introductory tutorial. A list of Scheme procedures available in Impromptu is available on Impromptu's WIKI. For more in depth procedure information, check out the Racket Guide. However, please note that Impromptu does not have a Racket Scheme compiler, so not all Racket functions are supported. However, the Racket guide provides thorough documentation of most basic Scheme procedures.

It's also helpful to read the GleamAPI to understand GLEAM-specific procedures.

A Short Introduction to Impromptu

Impromptu is a live coding environment for the Scheme functional programming language. Live or interactive coding is a performative method of coding in which programs are manipulated during run time. We chose Impromptu for its scheduling features and its support for MIDI and OSC. We recommend that you familiarize yourself with Impromptu by reading the tutorials and watching the videos on Impromptu's tutorial page. Because an understanding Impromptu's timing festures is also essential to understanding the environment, we also recommend that new users check out the timing tutorial.

Joints

A user's joints are referenced in the GLEAM library by the string assigned to them by OSCeleton. A list of these strings can be found on OSCeleton's github page.

By default, only two joints are enabled at the start of each GLEAM session: the left hand and the right hand. To add more joints, the programmer must call joint-add! and provide the osceleton string for the new joint. Note that the added joint must also be added to the list of joints passed to Impromptu by the oscfilter. If the joint's information is not being sent to Impromptu, joint-add! will not have any effect.

To remove a joint, the programmer calls joint-remove! with the joint's osc string. Note that the right hand cannot be removed. To change this default, one must modify the joints list in the kinectlib.scm library.

When a joint is enabled and gesture tracking is on, the GLEAM library stores the last one hundred positions of that joint along each directional axis. Whenever a new position is stored, it is paired with the Impromptu's frame count at the time of storage. By using the included functions like axis-get-currrent-pos and axis-get-previous-time, programmers can determine the velocity of a joint along an axis or track the joint's movement.

In addition to the positions of a joint, GLEAM also stores the direction of the joint along each axis. This can be accessed with axis-get-direction. If the direction for the axis is negative, the joint is moving along that axis towards the origin. If the direction is zero, the joint is not moving along that axis at all.

Since functions like axis-get-current-pos and axis-get-direction must access the storage information for a joint, the user must pass the storage information as a parameter to these procedures. The provided function oscstring->joint takes the joint's osceleton string as a parameter and returns this storage information. Thus it is often necessary for the programmer to call oscstring->joint and then pass this value directly to one of the information-getter procedures.

Recognizing Gestures

Gesture recognition is written into the GLEAM library so that users do not have to manually check for gestures. However, it is important to understand how gestures work to take full advantage of the library. Basically, a gesture is a simple scheme function that may take a joint as a parameter and returns true or false based on the current behavior of the specific joint. Each included gesture investigates a particular aspect of a joint's behavior. For example, the gesture procedure gesture-joint-up checks whether a particular joint is moving up and returns true even if the joint is moving in another direction while moving up. Likewise, gesture-joint-up-right returns true if the joint is moving up and right at the same time.

When a gesture function returns true, it signals the handler attached to that specific gesture case to evaluate. If a gesture does not have any handlers attached to it, nothing will happen even if the gesture function returns true. Thus, in order for a gesture to evoke a handler, that gesture must (1) return true and (2) have a handler attached to it. For more information on handlers, see below.

Currently, there are two main types of gestures: single-joint gestures and two-handed gestures. Single-joint gestures are gesture functions that take a single joint as a parameter and examine only that joint's behavior. Within the category of single-joint gestures, there are two subcategories: simple single-joint gestures and multdirectional single-joint gestures. The procedure gesture-joint-up from above is an example of a simple gesture since it only examines one direction of the joint and may return true even if the joint is moving in another direction while also moving up. Multiple simple single-joint gestures can be evaluated at once if handlers are defined for each gesture. For example, the handlers attached to gesture-joint-down and gesture-joint-left will evaluate if the joint in question is moving down and left at once.

On the other hand, multidirectional single-joint gestures are intended to execute only one handler even if the joint is moving in more than one direction at once. Gestures like gesture-joint-right-down and gesture-joint-right-up-forward will only evaluate if the joint is moving in all the specified directions at once. These multidirectional gestures also keep the simple-joint gestures attached to these directions from evaluating. So, in the example above, gesture-joint-down and gesture-joint-left will not evaluate their handlers if gesture-joint-left-down evaluates its handler by returning true. Thus, in a way, multidirectional gestures are prioritized above simple gestures. Likewise, more complex, three-dimensional multidirectional gestures are prioritized above two-dimensional gestures. This means that if gesture-joint-right-up-forward evaluates its handler, the handlers associated with gesture-joint-right-up or gesture-joint-right-forward will not be executed. It is important to keep this behavior in mind when planning gestures.

The last type of gestures, two-handed gestures, deals with the behavior of the user's hands. Since the joints are already defined for these gestures, two-handed gestures do not take parameters yet still return true or false like single-joint gestures. Like multidirectional single-joint gesutres, two-handed gestures are capable of prohibiting other gestures from evaluating their handlers. If a two-handed gesture returns true, no other gesture handler associated with a hand will evaluate. Furthermore, two-handed gestures with a higher priority can prevent other two-handed gestures from evaluating. For example, if gesture-both-hands-right-up evaluates its handler, gesture-both-hands-up will not be checked. Once again, an understanding how gestures are prioritized can help programmers and performers to coordinate gestures.

When tracking gestures is enabled with the gestures-start! procedure, a loop will continuously check for gestures in order of priority until gestures-stop! is called or an error arises. If a gesture returns true but does not have a handler attached to it, gesture-tracking will continue checking all gestures lower in priority. To change how often the gesture tracker loops through the gesture list, modify the refresh rate with the context-set-refresh-rate! procedure.

Available Gestures

A list of gestures, organized by category, can be found in the GleamAPI. To help alleviate the challenge of learning so many new gestures, a naming convention has been enforced. All single-joint gesture names start with "gesture-joint-" while most two-handed gestures begin with "gesture-both-hands-". An exception to this pattern are the two-handed gestures that allow the left and right hand to move independently of each other. An example of such a gesture is gesture-right-hand-up-left-hand-down. For these types of gestures, note that the right hand is always listed before the left hand.

Writing Your Own Gestures

As mentioned above, gestures are functions that may take a joint as a parameter and return true or false based on the joint's behavior. The presence of the parameter is determined by the gesture type. Single-joint gestures take a joint as a parameter while two-handed gestures do not. Thus the first step in writing a gesture is deciding which category the gesture belongs to.

The body of each gesture function is an and statement composed of gesture procedures and a ~simultaneous? procedure call. Using other gesture procedures within a more complex gesture ensures that the and statement will work as expected. The ~simultaneous? function takes four parameters: a joint1, an axis1, joint2 and axis2. Axis1 and axis2 are either x, y or z, and ~simultaneous? returns true if joint1 is moving along axis1 at the same time as joint2 is moving along axis2. When writing a single-joint gesture, joint1 and joint2 will be the same joint. For example (~simultaneous? (oscstring->joint "l_hand") x (oscstring->joint "l_hand") y) returns true if the left hand is moving vertically and horizontally at once. Like wise, (~simultaneous? (oscstring->joint "r_hand") x (oscstring->joint "l_hand") x) returns true if the right and left hand are both moving horizontally.

In the examples above, it is necessay to call oscstring->joint on the osc string associated with the joint. This is because the ~simultaneous? function must access the internal storage of the joint rather than just the joint's name. For single-joint gestures, it is not necessary to call oscstring->joint on the joint argument because the gesture tracking procedures do this internally. However, if one wishes to write a gesture function that corresponds to a specific joint or a specific set of joints, oscstring->joint must be called. In these cases, it may be helpful to use a let statement to avoid having to reconvert the oscstring at every instance. Because two-handed gestures work specifically for the left and right hands, each two-handed gesture declaration uses a let statement with oscstring->joint. For example, here's what the implementation of gesture-both-hands-up looks like:
(define gesture-both-hands-up
   (lambda ()
      (let ((left (oscstring->joint "l_hand"))
            (right (oscstring->joint "r_hand")))
         (and (gesture-joint-up left)
              (gesture-joint-up right)
              (~simultaneous? left y right y)))))

Note the calls to other gesture functions within the and statement and the ~simultaneous? statement. Here's an example of a single-joint, multidirectional gesture:
(define gesture-joint-down-forward 
  (lambda (joint)
    (and (gesture-joint-forward joint)
         (gesture-joint-down joint)
         (~simultaneous? joint y joint z))))

When writing gestures, keep in mind the naming conventions outlined above the Available Gestures section. Adhering to a naming convention will help you remember the gestures available when coding live.

Once a gesture function is written, the final step is to add the gesture to one of the gesture lists in the kinectlib.scm. Currently, the library has three lists: simple gestures, multidirectional gestures and two-handed gestures. These lists affect how these gestures are evaluated so it is important to add your gesture to the correct list to avoid errors. When adding your gesture to a list, keep in mind gesture priorities. The position of your gesture should ensure its proper execution and stop the execution of lower priority gestures if need be.

Handlers

Handlers are functions that allow gestures to trigger changes in Impromptu. They are connected to gestures with the function gesture-change-handler!. Each gesture may have multiple handlers, one for each enabled joint. When the gesture returns true for a particular joint, only the handler associated with that joint will be evaluated.

Because of the architecture of GLEAM, gesture handlers cannot take unique parameters. Instead, the GLEAM library can pass joint information to handlers as they are executed. This parameter-passing functionality is not enabled at the start of a GLEAM session. To pass joint data to handlers, the programmer must call context-toggle-parameter-passing! to switch the state of parameter passing. If parameter passing is off, handlers are called without any arguments.

When parameter passing is on, only the osceleton string associated with that joint is passed to the handler. Thus when writing handlers that access joint information, it is important to call oscstring->joint on the argument before using procedures like axis-get-current-pos. In the case of two-handed-gestures, both the left and right hand strings are passed.

Writing Your Own Handlers

When writing handlers, the programmer must keep in mind that the handler can be called with a varying number of parameters. To account for this, the programmer can either specify an optional number of parameters or make sure that parameter passing is constantly on or off for the entire programming session.

Handlers that are intended to take an optional number of parameters should be declared in the following way:
(define <em>function-name
<span style="white-space: pre;">   </span>(</em>lambda _vals
_ <span style="white-space: pre;">      </span>(....

Excluding the the parentheses after the lambda statement signifies that the handler can be run with any number of parameters. To change the behavior of the procedure based on the number of parameters passed, the programmer should first check whether or not vals is null. This test will return true whenever the handler is called without parameters. If the handler has been called with parameters, vals is the list of arguments. To access these arguments, calls to car or list-ref must be made.

Accessing Joint Information

If a joint string has been passed to a handler, this string can be used with oscstring->joint and the joint information getting procedures to change the behavior of the handler based on the current state of the joint. For example, if a programmer is writing a handler that modifies volume, it makes sense that the volume would increase as the joint for the handler moves towards the origin. To determine whether the joint is behaving this way, the programmer must pass the result of (oscstring->joint joint) as an argument to axis-get-direction.

It is also possible for handlers to depend on the behavior of other joints besides the one passed as an argument. For example, if a programmer wants a handler that sets the volume to the current position of the left hand whenever the right hand is moving up, "l_hand" can be passed to oscstring->joint instead of the right hand. Unfortunately, procedures with this kind of functionality require the programmer to hard code the alternate joint.

Getting Handlers to Behave the Way You Want

Because of the way gestures are tracked and evaluated in GLEAM, a handler can be evaluated many times while the gesture is true. The rate at which the handler evaluates is determined by the global refresh rate.

Constant handler evaluation was a design choice to allow GLEAM to reflect the process of playing an instrument as closely as possible. However, this functionality can also cause handlers to behave unexpectedly for new users. For example, if a programmer wants to write a function that plays a note whenever a hand is moving forward, he or she should bear in mind that the procedure will play notes constantly as long as the hand is moving forward. If the current refresh rate is high enough, this could generate some horrible sounding noise. To limit how often the handler plays a note, the programmer can rewrite the function to only play the note when the position of the joint on the forward motion passes a certain point with axis-get-current-pos. Likewise, the programmer could write the handler to play notes on specific intervals, such as whenever the rounded current position is a factor of 10.

-- JennelleNystrom - 2012-07-25
Topic revision: r8 - 2012-08-19, JennelleNystrom
 

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback