Hello World With Leap Motion & Python

This article demonstrates how to connect to the Leap Motion controller and access basic tracking data. After reading this article and following along with your own basic program, you should have a solid foundation for beginning your own application development.

First, a little background...

How the Leap Motion Controller Works

The Leap Motion controller encompasses both hardware and software components.

The Leap Motion hardware consists primarily of a pair of stereo infrared cameras and illumination LEDs. The camera sensors look upward (when the device is in its standard orientation). The following illustration shows how a user’s hands look from the perspective of the Leap Motion sensor:

The Leap Motion software receives the sensor data and analyzes this data specifically for hands, fingers, and arms. The software maintains an internal model of the human hand and compares that model to the sensor data to determine the best fit. Sensor data is analyzed frame-by-frame and the service sends each frame of data to Leap Motion-enabled applications. The Frame object received by your application contains all the known positions, velocities and identities of tracked entities, such as hands and fingers. For an overview for the tracking data provided by the controller, read API Overview.

The Leap Motion software runs as a service (Windows) or daemon (Mac and Linux) on the client computer. Native Leap Motion-enabled applications can connect to this service using the API provide by the Leap Motion dynamic libraries (provided as part of the Leap Motion SDK). Web applications can connect to a WebSocket server hosted by the service. The WebSocket provides tracking data as a JSON-formatted message – one message per frame of data. A JavaScript library, LeapJS, provides an API wrapping this data. For more information read System Architecture.

Set Up the Files

This tutorial also uses command line compilers and linkers (where needed) in order to focus on the code rather than the environment.

1. If you haven’t already, download and unzip the latest Leap Motion SDK from the developer site and install the latest Leap Motion service.
2. Open a terminal or console window and navigate to the SDK samples folder.
3. Sample.py contains the finished code for this tutorial, but to get the most out of this lesson, you can rename the existing file, and create a new, blank Sample.py file in this folder.
4. In your new Sample.py program, add code to import the Leap Motion libraries. The following code detects whether you are running 32- or 64-bit Python and loads the proper library:

   import os, sys, inspect, thread, time
   src_dir = os.path.dirname(inspect.getfile(inspect.currentframe()))
   # Windows and Linux
   arch_dir = '../lib/x64' if sys.maxsize > 2**32 else '../lib/x86'
   # Mac
   #arch_dir = os.path.abspath(os.path.join(src_dir, '../lib'))
   
   sys.path.insert(0, os.path.abspath(os.path.join(src_dir, arch_dir)))
   
   import Leap
   

5. Add the “structural” code to define a Python command-line program:

   def main():
   
       # Keep this process running until Enter is pressed
       print "Press Enter to quit..."
       try:
           sys.stdin.readline()
       except KeyboardInterrupt:
           pass
   
   if __name__ == "__main__":
       main()
   

Note that the statement: sys.stdin.readline() does not play nicely with IDLE and possibly other IDEs. To use IDLE, you must implement a different way to prevent the program from reaching the end of the main() subroutine and thus exiting immediately.

This code simply prints a message and then waits for keyboard input before exiting. See Running the Sample for instructions on running the program.

Get Connected

The next step is to add a Controller object to the program – which serves as our connection to the Leap Motion service/daemon.

def main():
     controller = Leap.Controller()

     # Keep this process running until Enter is pressed
     print "Press Enter to quit..."
     try:
         sys.stdin.readline()
     except KeyboardInterrupt:
         pass

When you create a Controller object, it automatically connects to the Leap Motion service and, once the connection has been established, you can get tracking data from it using the Controller.frame() method.

The connection process is asynchronous, so you can’t create the Controller in one line and expect to get data in the next line. You have to wait for the connection to complete. But for how long?

To Listen or not to Listen?

You can add a Listener object to the Controller, which provides an event-based mechanism for responding to important Controller state changes. This is the approach used in this tutorial – but it is not always the best approach.

The Problem with Listeners: Listener objects use independent threads to invoke the code that you implement for each event. Thus, using the listener mechanism can introduce the complexities of threading into your program. It becomes your responsibility to make sure that the code you implement in your Listener subclass accesses other parts of your program in a thread-safe manner. For example, you might not be able to access variables related to GUI controls from anything except the main thread. There can also be additional overhead associated with the creation and cleanup of threads.

Avoiding Listeners: You can avoid using Listener objects by simply polling the Controller object for frames (or other state) when convenient for your program. Many programs already have a event- or animation-loop to drive user input and animation. If so, you can get the tracking data once per loop – which is often as fast as you can use the data anyway.

The Listener class in the API defines the signatures for a function that will be called when a Controller event occurs. You create a listener by creating a subclass of Listener and implementing the callback functions for the events you are interested in.

To continue this tutorial, add the SampleListener class to your program:

class SampleListener(Leap.Listener):

    def on_connect(self, controller):
        print "Connected"


    def on_frame(self, controller):
        print "Frame available"

def main():
    #...

If you have already taken a look at the finished file, you may have noticed that several more callback functions are present. You can add those too, if you wish, but to keep things simple, we will concentrate on |Listener_onConnect|_ and |Listener_onFrame|_.

Now create a SampleListener object using your new class and add it to your controller:

def main():
    # Create a sample listener and controller
    listener = SampleListener()
    controller = Leap.Controller()

    # Have the sample listener receive events from the controller
    controller.add_listener(listener)

    # Keep this process running until Enter is pressed
    print "Press Enter to quit..."
    try:
        sys.stdin.readline()
    except KeyboardInterrupt:
        pass
    finally:
        # Remove the sample listener when done
        controller.remove_listener(listener)

Now is a good time to test your sample program. Follow the directions in: Running the Sample.

If everything is correct (and your Leap Motion hardware is plugged in), then you should see the string “Connected” printed to the terminal window followed by an rapid series of “Frame available”. If things go wrong and you cannot figure out why, you can get help on our developer forum at developer.leapmotion.com.

Whenever you run into trouble developing a Leap Motion application, try opening the diagnostic visualizer. This program displays a visualization of the Leap Motion tracking data. You can compare what you see in your program to what you see in the visualizer (which uses the same API) to isolate and identify many problems.

On Connect

When your Controller object successfully connects to the Leap Motion service/daemon AND the Leap Motion hardware is plugged in, The Controller object changes its is_connected property to true and invokes your |Listener.onConnect|_ callback (if there is one).

When the controller connects, you can set controller properties using such methods as Controller.set_policy().

On Frame

All the tracking data in the Leap Motion system arrives through the Frame object. You can get Frame objects from your controller (after it has connected) by calling the Controller.frame() method. The |Listener_onFrame|_ callback of your Listener subclass is called when a new frame of data becomes available. When you aren’t using a listener, you can compare the id value to that of the last frame you processed to see if there is a new frame. Note that by setting the history parameter of the frame() function, you can get earlier frames than the current one (up to 60 frames are stored). Thus, even when polling at a slower rate than the Leap Motion frame rate, you can process every frame, if desired.

To get the frame, add this call to frame() to your |Listener_onFrame| callback:

def on_frame(self, controller):
    frame = controller.frame()

Then, print out some properties of the Frame object:

def on_frame(self, controller):
    frame = controller.frame()

    print "Frame id: %d, timestamp: %d, hands: %d, fingers: %d" % (
          frame.id, frame.timestamp, len(frame.hands), len(frame.fingers))

Run your sample again, put a hand or two over the Leap Motion device and you should now see the basic statistics of each frame printed to the console window.

I’ll end this tutorial here, but you can look at the rest of the code in the original sample program for examples on how to get all the Hand, Finger, Arm, and Bone objects in a frame.

Running the Sample

Type the following at your command prompt (with the current directory set the SDK samples folder):

python Sample.py

You should see the message “Connected” printed to standard output when the application initializes and connects to the Leap. You should then see frame information printed each time the Leap dispatches the onFrame event.