Gesture Controlled Pitch Bend

Here's a demo of a project I recently completed as part of my Cognitive Video class: Gesture Controlled Pitch Bend.

The motivation behind this project stems from my interest for the variety of sounds a guitar can produce, although I am mainly a keyboard player.
One of the expressions I have always wanted to reproduce on a keyboard are bends and vibratos. The effects are subtle, but they definitely add something to licks. Especially when a nice long bend slowly, asymptotically pass the Blue note and lands on the 5th..

There are ways to do that by using the pitch wheel featured on some keyboards. But it's a bit awkward. It's like using a whammy bar to bend a note. So I thought, why not make a keyboard a 2-dimensional device?
We use one dimension to run through all the notes - what about extracting information about the position of the player's hands along the perpendicular?

In the setup described in the video, a camera continuously tracks the position of the player's hands and sends displacement positions along one dimension to an arduino, which handles MIDI communication. The result is fairly intuitive. Slide your hand away and the bend goes up, slide towards your body and the band goes opposite.

Improvements to come:
1) I had a new idea for quick tracking of the hand
2) Make this a standalone device by using a Raspberry Pi and doing the image processing on the RPi's GPU
3) Automatic calibration to register displacements occurring over the keyboard region only.

Work at Tandent Vision Science

I started working at Tandent Vision Science about a month ago. I absolutely love the work, the people, and the office! The company focuses on computer vision, and recently published a release called Lightbrush. The software is amazing: in a nutshell, you can separate all the shadows in an image with one click:

I think this is incredible: so many computer vision tasks are limited by illumination variations. Lightbrush provides a fundamental first step by making the image illumination-invariant. As such, it can be used as a natural preprocessing step in all computer vision pipelines, and makes most recognition applications much simpler. Anyone with experience dealing with images for the purpose of the 3 R's (Recognition, Reconstruction, Registration) will see the incredible benefit of this software.
Besides, I am a big fan of photoshop, and I know a feature such as this one would be revolutionary! I can't count how many times I've seen awful photoshopped images due to incoherent lighting.
Apparently Lightbrush gained huge attention from texture artists and high level graphics artists (at Pixar notably) so the product is geared towards a very specialized crowd. Too bad, I personally think the average user would have a blast playing with this product.
I've been working on improving the machinery - can't talk about it! - and it still blows my mind that this is possible with minimal user input... There are many aspects that I have to take into account in my tasks, such as keeping the user in mind with respect to user interaction complexity, speed (texture artists work with gigantic images).

I'm working with 2 very friendly Carnegie Mellon alumni. We each have our separate offices (with a REAL DOOR - something I missed at other workplaces I spent time at) and has a very cozy homey feel, much different from the old functional cubicle. I'm definitely loving the work here.

Automated caller

It's been a while since I updated anything here. My schedule for this new semester is fairly busy: 4 classes, research and an internship barely leave enough time to cook.

The script below is something I wrote back in December when contact information from a certain religious hate group was hacked then published online. It successively and continuously calls all the numbers on a text file (in the file below, the numbers are just stored in an array). I used mechanize to fill forms on website such as findmyphone.com and dial the numbers.

Call_These_Numbers.rb

I was debugging the script using my number and had to leave in the middle of it running... I came back to 57 missed calls and voice messages.