Imagine humans as complex dynamical systems: systems that are characterized by multiple interacting layers of hidden states (e.g., internal processes involving functions of cognition, perception, production, emotion, and social interaction) producing measurable multimodal behavior signals (e.g., body gestures, facial expressions, physiology, and speech). This abstraction of humans with a signals and systems framework naturally brings a synergy between engineering and behavioral sciences. Behavioral signal processing (BSP) offers a new frontier of interdisciplinary research between these communities. The core research in BSP is to mathematically model human behaviors with observational data, i.e., use computational methods grounded in statistics, signal processing, and machine learning to provide quantitative evidence that characterizes and tracks human internal states. The tangible outcome of BSP, i.e., behavioral analytics, offers new quantitative methods for enhancing the capabilities of domain experts in facilitating better decision making. The relevant application domains are diverse, ranging from behavior sciences, mental health, education, security, neuroscience, and personalized human-machine interfaces to user-centric commercial applications.
The challenges of BSP lie in the complexities of modeling heterogeneous human behaviors. Sources of variability in human behaviors originate from the differences in mechanisms of information encoding (behavior production) and decoding (behavior perception). An additional layer of complexity exists because human behaviors occur largely during interactions with the environment and agents therein. This interplay, which causes a coupling effect between humans’ behaviors, is the essence of interaction dynamics that has been at the core of human communication studies. Further challenges lie in the contextualization of such research in a meaningful and domain-aware manner. This involves translating the knowledge into a range of domains (e.g., the arts, education, and healthcare.
These challenges represent a multitude of exciting research opportunities in signals and systems. This includes research in designing and instrumenting behavior measurement techniques in ecologically-meaningful ways, developing novel signal processing techniques for capturing objective behavior information from noisy observational data (i.e., feature extraction from raw recording data), creating new algorithms for modeling behavioral interaction dynamics in the presence of confounding inter and intra speaker variability, and contextualizing the entire effort in a domain-aware manner.