Behavioral signal processing (BSP) is an emerging cross-cutting research and development field that aims at providing novel signal processing algorithms to model observable complex human behavior as modulating by internal construct contextualized in applications of societal significance (e.g., health and education related). The advancement of BSP technologies centers on developing advanced computational frameworks that address both algorithmic-robustness and in-domain-interpretability.

In this 3-year proposal, we will specifically develop algorithms to derive the pain-related multimodal behavior analytics of on-boarding emergency room (ED) patients during triage sessions from multimodal, i.e., audio-video and physiology sensors, recordings. The proposal lays out a three-year research plan in the following two main technical directions:

a. Advancement in multimodal pain-related behavior modeling of audio-video signal data handing idiosyncratic, environmental, clinically-induced variabilities
b. Investigation of multitask learning framework in incorporating clinically-relevant information in order to support the use of pain behavior-analytics at triage classification for physicians

The proposal involves technical components in developing multimodal behavior feature analyses and extraction algorithms with properly contextualized machine learning framework to quantify the intensity of pain objectively and automatically. At the same time, it includes a research initiative into a large-scale audio-video and vital sign collection of real patients during emergency triage sessions.

The expected research outcome of the proposal is two folds: 1) the technical developments of these behavior analytics will be useful one day in transforming the current status-quo in ED triage classification (TTAS) by providing an objective and quantifiable measure on the intensity of pain – additionally bringing potential novel insights to scientifically understand this fundamental and yet complex aspect of human internal attribute (pain) through cross-boundary collaborative research. 2) We anticipate the impact and contribution of such research effort can inspire additional interdisciplinary effort for human-centered research (including industry and academia): advancing and contextualizing existing speech, language, even computer vision technologies, sparking new capabilities of a variety interdisciplinary research effort of substantial societal implications.