Hands Behind the Light: Experiencing the Forgotten Art of Shadow Puppet Manipulation

This project invites users to step into the role of a shadow puppet performer, using real-time color tracking to control digital puppets. Rather than reinterpreting the artform, it focuses on recreating the tactile experience of manipulation, letting users feel what’s usually hidden behind the screen/stage.

Project Video

Abstract

What does it feel like to be behind the screen of a traditional Chinese shadow play—not as a viewer, but as the puppeteer? This project asks a simple but often overlooked question: Can we design an experience where users don't just watch shadow puppetry, but actually feel what it's like to control it? While Chinese shadow puppetry is widely admired for its visual elegance, what’s often forgotten is the person behind the screen—the performer who brings the puppet to life through careful coordination. The expressive power of the performance depends not only on the puppet’s appearance but on the subtlety of the hand movements manipulating it. This project focuses on that often-ignored experience: the physical artistry of control.

Using webcam-based color tracking, users hold up three colored sticks—red for the head, blue for the left hand, green for the right—and move them in space to animate a virtual shadow puppet. The system translates these physical gestures into movement, giving users a tactile and visual experience of what it means to become the puppeteer.

A surprising moment that continues to stay with me came during early testing. Watching users fumble, focus, and finally delight as they moved the puppet evoked an idea I hadn’t expected: If Western wizards use a single magic wand to control the world, what kind of person in the East uses three magic sticks to control a human figure? The answer: a shadow puppeteer. That realization reframed the project for me—not just as a tech demo, but as a playful cultural mirror.

This project is not a nostalgic digital replica. It’s a live interface that gives participants a sense of the performer’s role. By shifting interaction from passive watching to active manipulation, it opens up a richer understanding of an art form that often gets flattened into ornament or spectacle. At its core, this project is about empathy through embodiment: preserving not just how shadow puppetry looks, but how it feels to perform. It’s a small but meaningful step toward sustaining the living, physical knowledge that powers traditional performance in the digital age.

Photos