The idea was to mix algorithmic music with "real-people" music by making a robot that could listen to other human players and use their playing to seed a variety of response algorithms. A variety of call-and-response routines were developed, some simply rearranging input, others that created more dense or sparse responses to complement the human’s playing. We dabbled a bit into beat matching and using fractal math to drive the robot.
Here’s a video early prototype of just the right arm:
Technical Challenges
Listening:
Hits were detected as sharp changes in the spectrum. Simply looking for amplitude spikes would miss soft hits quickly following louder ones. A max/msp external was written that was based on a tweaked version of bonk
Communications / Microcontrollers:
Three pics were used. One received commands via ethernet from a mac running max/msp. Two other pics were each responsible for one arm.
Right Arm:
The right arm was fairly simple, just a solenoid attached to a thin piece of aluminum. Amplitude was a function of pwm and “on” time. PID control was used on the slide positioning system.
Left Arm:
One goal of making a robot (as opposed to just doing everything in software with speakers and a microphone) is that the visuals can help the human interact. Typically we associated large motions with louder sounds, and a big anticipatory hand-lift not only tells you that a big hit is coming, but when it is coming. The left arm was designed to have a much larger range of vertical motion for this purpose.
This was substantially harder than the right arm, since a fair amount of timing anticipation was required to allow the arm to lift slightly before the beat, and then make contact precisely wen the beat should occur at the right velocity. Further complicating the situation, previous hits could have the arm already moving at a high velocity. If you want to play a soft note right after a really loud one, you’d have to push down hard to compensate for the bounce energy. A giant lookup table was used in the end that took into account the current velocity, time since the last hit and the next note’s velocity. A linear motor drove the hand up and down, while another belted slide system moved it in and out.
Body
The body was cut out of plywood on CNC machines and contains metal clamps in all the joints so that it can be adjusted for different drums and taken apart to travel.
Beat Detection Challenges:
The beat detection algorithm was mostly based on Eric Sheirer’s paper called “Tempo and Beat Analysis of Acoustic Musical Signals.” In short, it filters the audio for on sets by differentiating a few bands of the spectrum, and then runs those “onset” signals through two banks of comb filters: one for tempo and one for phase. You can sort of think about it like trying to lay two patterns on top of each other. The best matching comb filter will resonate the most. The algorithm works surprisingly well against pre-recorded music, but when you try to play along with the robot, it’s hard to avoid a circular chase. As the robot tries to lock to your tempo, it’s almost impossible to avoid adjust your own tempo slightly to match the robot’s.
Links:
More videos and detailed publications can be found here: http://www.cc.gatech.edu/gilwein/Haile.htm
