Towards a Generalized Mapping Framework For Gesture-Responsive Music and Speech Generation

This project proposes a modular, real-time gesture-to-sound mapping framework for the GestureCap system. It will enable users to translate hand gestures into expressive audio control for music and speech applications. The system introduces a mapping engine that sits between gesture recognition (via MediaPipe in Python) and sound generation (in Max/MSP). The engine will allow users to define how gestures control sound using deterministic mapping (rule-based), learned mappings (customizable, ML-based), and hybrid mappings combining the two. It will emphasize real-time performance, flexibility, user-friendliness, and reusability with mappings that can be saved/loaded as JSON files. This project will enable rapid prototyping of gesture-controlled musical instruments and other sound generation for creative as well as research-oriented objectives.