Monday, September 24, 2012

Webcam Swiper

I've spent a little time lately working on an experiment that uses the new getUserMedia() method.  I'm calling it the Webcam Swiper.  Chrome and Firefox now support Webcam access, although Firefox < 20 requires turning on a flag in about:config.

It's a page that displays a book.  If you allow webcam access it will watch for you swiping your hand left or right through the air, which triggers a page turn in the appropriate direction.

I was inspired to try this when I saw the cool Magic Xylophone.  I thought of all the cases where someone might want to make a simple directional gesture and have something happen on the page.  Image Carousels could rotate, the next item in a list could be displayed, book pages could be turned, etc...

To implement this I had to figure out how to detect when a hand had moved from one side of the screen to the other. I started by detecting motion the same way that the Magic Xylophone does. First turn the image from the webcam to greyscale so I only have to worry about brightness and not color changes. Then, comparing one frame to the next, watch for any pixels that change intensity more than some given threshold.  Next I determine what I call the "motion weight" for the given frame. Basically any pixel on the left side of the screen which is considered to have movement, subtracts from the weight. Any pixel on the right side of the screen which has movement adds to the weight. Total them all up and we get a number that summarizes where the most movement is occurring on the screen. Now all we need to do is compare the "motion weight" over the last series of frames. If we have a highly negative weight which then quickly shifts to a highly positive weight, we know the user just swiped right. If it is positive and goes negative, the user just swiped left. A custom event is fired on the body element of the page. Then whoever is listening for the event can do whatever they want.  In my case I do a couple CSS transitions and transforms to turn the page of the book.

That's all there is to it. I had some problems with varying light level, so I sample every 50 frames and calibrate the sensitivity based on the ambient light level in the room. Also I adjust for slower hardware by timing each frame analysis. If it is taking too long I lower the scan frequency. Otherwise it stays high.

I've made this all into a library which should be fairly easy to use.

Include the webcam-swiper-0.1.js with a script tag or the loader of your choice.

Then bind the swipe events however you choose.  Example with jQuery:

  $("body").bind("webcamSwipeLeft", yourLeftEventHandler);
  $("body").bind("webcamSwipeRight", yourRightEventHandler);

Start the webcam access with a call to the global initializeWebcamSwiper function like this:


Now it is running!  If you choose to stop it call


Play with it and feel free to improve it or give any feedback.