Research.  

The research stage of this project focused on exploring how hand tracking technology could be used as a performative tool for sound design. Initial inspiration came from a series of TouchDesigner tutorials demonstrating MediaPipe integration, Torin Blankensmith, was one YouTuber who gave me insight into how this project could be achieved, also Tutorials that showed how individual hand landmarks could be isolated and used as control data. For example, introductory tutorials on MediaPipe hand tracking illustrated how specific points on the hand can be selected and manipulated within TouchDesigner using CHOP networks, forming the basis for interactive systems.   These examples were key in understanding how raw tracking data could be translated into usable control signals. a tutorial from alwayscodingsomething helped me find these datapoints and how i could use the networks in a different way. 

Further investigation into MediaPipe revealed that the system outputs 21 3D landmarks per hand, each representing a specific joint or fingertip. This level of detail suggested strong potential for expressive control, as each point could be mapped to different parameters within a sound design environment. However, tutorials also highlighted the importance of selecting and refining data rather than using all available points, reinforcing the need for a structured approach to interaction design.  

In addition to hand tracking, research into TouchDesigner’s VST integration provided insight into how audio plugins could be controlled using CHOP data. The ability to map real-time data streams directly to plugin parameters allowing for a seamless connection between movement and sound, forming the conceptual core of the project. 

YouTubers, video references and artists further influenced the development process by demonstrating practical workflows, creators like Torin and Dom Scott gave insight on how Media Pipe can be used including selecting specific finger data, remapping normalized values, and building modular systems for interaction. These examples emphasised the importance of data scaling and organisation when working with real-time inputs. Whereas artists like Imogen heap uses the  Mi.Mu gloves, using real time emotion and physical movement to create and synthesise sound.  

Overall, this research stage allowed both the technical and conceptual framework for the project to come together. It confirmed that combining Media Pipe's detailed tracking data with Touch Designer's node-based system could enable a highly responsive and expressive method of controlling VST chains through physical movement. 

The prototyping stage began with an initial idea based around gesture recognition, using predefined hand shapes to trigger and manipulate parameters within a VST chain. This was influenced by early experiments MediaPipe, where gesture classification can be used to generate certain outputs. The idea was that different hand shapes would control different parameters of different VST plug ins. However, this would have limited my number of data outputs there were eight preset gestures per hand. Which felt quite restrictive, I chose not to fully develop this version, as it seemed capable of offering the level of control needed for more detailed sound manipulation. 

The second prototype moved towards using positional tracking data instead, mapping hand movement directly to parameters in TouchDesigner. This felt closer to the intended outcome, as it allowed for continuous control rather than fixed triggers. The overall structure of the network quickly became difficult to manage. Even when working with a single hand, the setup felt cluttered and confusing, and it wasn’t always clear how the tracked points were being processed or mapped. This made it harder to troubleshoot and slowed down development. 

In the third prototype, the focus shifted towards simplifying and organising the system. By grouping elements into Base COMPs and separating different stages of the process, the network became much easier to follow. This made it quicker to make changes and understand how the data was flowing. Compared to earlier versions, this setup felt more stable and practical to use, and it ultimately formed the basis for the final system. I have linked below the gradual shift through the increase of reactivity in my YouTube videos,  

The final system uses MediaPipe to control VST plugins in real time, transforming physical movement captured into audio manipulation, the tracking data consists of 21 points per hand, each outputting normalised X,Y and Z values between 0 and 1, rather than using all the available data, the system was refined to focus only on fingertip tracking. This decision reduces the complexity of the pathway created and improves control, as fingertip movement is more intentional and expressive in a performance context. 

The selected data is organised and processed within Touchdesigner using a CHOP network. A key stage of this process is the use of a Math Chop to remap the normalised tracking data to parameter ranges within the Math CHOP allowing fluidity and more control over the VST plugins. This ensures that physical movement translates into data that will allow meaningful sonic movement.   

This set up is duplicated for both hands, creating a mirrored structure that increases the number of controllable anchor points. While maintain consistency. This enables multiple elements of the sound to be changed simultaneously.The pathway has a output for performance too, using the ‘movieout’ operator and the pathway in green at the bottom of the picture above i can record the outcome and movement of my hands shown in several clips in this blog. 

The VST chains have two pathways, one being built for percussive and rhythmic material and another for ambient textures. This separation allows different types of movement and mapping to increase the expressivity of the final piece. For the ambient chain I will be using plug ins such as Valhalla Supermassive and Raum, Saturn 2 and phasis all being simple plug ins, but with tremendous amounts of manipulation capacity, For the percussive pathway I used Phasis, Thrash, and Replika. This allowed for more harsher and quicker manipulation, allowing the percussive areas to stay in the mix. and a when creating my material i will then record it into logic Pro, allowing further refinement. Below is a walkthrough on how it works with clips of extracts used in the compositions, the composition consists of all percussive ad ambient parts played and manipulated through my Pathway. The bass and kick are the only material not affected by this pathway.