Weird Mirrors in TouchDesigner
I’m doing this ‘Design for Emerging Technologies’ class, and the first big project (after teaching them how to use git!) is in TouchDesigner. This is a very intimidating piece of software that drives big visuals - immersive light installations, stage shows, etc. It’s programming visuals but with an incredibly complicated ways to transform live data. And it’s got a STEEP learning curve.
Like; this is a simple project. It’s not easy!
But it’s also got an MCP. Which means now our robot friends can help!
So with Claude in our pocket, the project becomes: how do you take a totally unfamiliar technlogy and apply your interaction design toolkit here? What are the affordances? What actions can a user take? What kind of feedback can you give the user?
When I was in school, we had an Elder Statesman professor who loved to harangue us that the tools of design were universal, and could be applied across disciplines. He didn’t care that we were in graphic, or industrial. He’d YELL at us: Design is Design… Design is Design is Design is Design.
So I am trying to embody that professor and incept my students with this idea. But with less shouting.
Here’s the brief I gave them, more or less:
TouchDesigner can do all kinds of complicated things and run complicated hardware. But you can run it on your laptop, and your laptop has a screen and a microphone and a webcam and a keyboard and speakers. Your job is to make a **Weird Mirror**: create something that allows a user walk up to your project, see themselves *reflected* in some way, literally or metaphorically. And take them on a journey: make the user FEEL something.
And we just wrapped the project, and it’s SO fun to see what they made!
Many stuck with a warped representation of you, on the webcam - using hand tracking to let you manipulate something on the screen, virtually. But the class got creative.
- We had a 3D-pong game you play with head tracking; your nose is the cursor.
- We had a jump scare.
- We had a bashful ghost you had to befriend by staying very still and then petting it.
- We had a lively koi pond where you could feed the fish.
- We had a climate change timeline you could control, but if you went too far forward you melted.
Here’s a few visual examples:
Sicily made a little interface where you plant a seed, which grows into a flower. When you smile the sun shines; your hands can create snow.
Cami made a volleyball game - a ‘keepy-uppy’ where you use your hands pretty naturally to play:
Quin had the idea to use it teach the participant how to ‘Wave the Wheat’ - the celebration that Jayhawks do at the University of Kansas when we win a basketball game or (more rarely) score a touchdown. The user naturally positions themselves in the center of the crowd, because there’s a visible space there, a natural affordance. When you raise your hands above your shoulders, the crowd starts imitating you. And when you successfully Wave the Wheat, fireworks go off and the Fight Song plays.
Saj made an abstract instrument - you don’t see yourself but you see your effect on the world. There’s an abstract multicolored sphere, and as you zoom in you can ‘pluck’ the strands of the sphere making music. (In the demo video below, the bottom-left quadrant is what the user actually sees -the other quadrants are for troubleshooting)
Madi eschewed visual tracking altogether, and leaned into the idea that any data source could ‘reflect’ the user. She vibecoded an app for her Apple Watch; you put it on, you see your own heartbeat reflected. Your natural urge is to try to spike your heartrate by doing jumping jacks or something… and then her tool gives you worrying stats about sudden heart failure. It’s both impressive and (imho) pretty funny.
Anyway: this was maybe the most fun I’ve had teaching a single project. I’ll definitely do it again if I can!
