I’ve started working with Zimmy on creating an multi-faceted installation based around audio visual feedback. It stems from Zimmy’s experiments with pointing video cameras at projections of what the video camera is looking at… thus creating a feedback loop.

It is hugely fun, and awe inspiring in a way- I really love things that make you stop and think about all the technology we’re using day-to-day. Normal usage doesn’t have this effect. With the video feedback it really makes you stop and think about how it can happen, and therefore how the components in the loop can actually function.

Collaboration

I think our (my & Zimmy’s) aim is to create an interactive installation, whereby everything you can see and hear is created by feedback; yet both audio and visuals will be controllable by the audience (or at least influenced by them). I’ve created a system whereby an audio feedback loop is created and then modulated by movements of a Wii remote controller- I’m pretty pleased with it! It sounds good (reminded to self, make a video for demonstrating this…)

Some Research

There is a plethora of many existing Wii hacks, and even more examples of using feedback principals to augment or create audio (and video) – I’ve done quite a bit of research into GlovePIE. It is a simple programming language that allows you to program the information from the Wii remote controller. I am only using two factors from the Wii remote – the “roll” and the “pitch” of the device. For my purposes I then convert that to a MIDI (MIDI is a standardized protocol for communication between electronic music devices) signal which is finally sent into an audio program – I am using one called Ableton. Ableton then processes the signals and changes various internal parameters to control the feedback.

I’ve also done quite a lot of practical research experimenting with shepard (or risset) tone generators. According to WikiPedia a shepard tone is;

A Shepard tone, named after Roger Shepard, is a sound consisting of a superposition of sine waves separated by octaves. When played with the base pitch of the tone moving upwards or downwards, it is referred to as the Shepard scale. This creates the auditory illusion of a tone that continually ascends or descends in pitch, yet which ultimately seems to get no higher or lower.

Although not a feedback element itself – its something that Zimmy wanted to use in the audio-element of the work. In my feedback machine using the Wii remote, a shepard tone modulates any audio signal it creates… which will then be fed-back. I’m pleased with the result, although it can sound quite harsh if you aren’t careful.

So far I’ve only used the sensors in the Wii controller that sense whether it is pointing toward the floor or the ceiling and whether it is on its front, back or side. They also have accelerometers, sensing what direction it is moving in – although I find this hard to work with. Finally, they also have an infra-red based ability to tell exactly what the remote is pointing at. In games the infra-red sensor is used for things like pointing guns at enemies or drawing a circle round something. Using the GlovePIE software I talked about earlier you can use this to control the cursor on a computer screen.

In theory (and practice, it turns out) it should be possible to “paint” on-screen with the remote. Combine the on-screen painting, with Zimmy’s video feedback loop, with my audio-feedback loop (which is controlled by the same remote control as the on-screen painting) and there you have it… I’ll put this to Zimmy, and see what he thinks.

Whilst researching other Wii painting projects and things on the internet, I discovered Johnny Chung Lee‘s website. He has done quite a lot of really amazing work with the Wii controllers – much more sophisticated than what I’m trying to do (but I think we have different aims, after all). What I learned from his site was that the Wii controller actually has a high-resolution infrared sensor on it (which works in conjunction with a little infrared emitter that you put on top of your TV, for gaming). Its very high resolution, and very sophisticated. Johnny Lee’s videos demonstrate various things including “painting” but also using the controller as a so called “multi-touch screen” and interactive white board applications – all at a fraction of the cost of the normal hardware required for these applications. One of the coolest is using your fingers as points for the sensor to see.. see it to believe it. Brilliant!

Fingers

Interactive Whiteboard