Studies have shown that people with different personalities would result in a different physiological reaction when encountering emotional stimulus. In this work, we propose an attribute-invariance loss embedded variational autoencoder (AI-VAE) to learn the personality-invariant physiological signal representation. The AI-VAE includes an additional loss aiming to perturb features from different personality polarity to obtain emotion discriminative representation. We evaluate our framework on a large emotion corpus of physiological data. Our method achieves a state of the art unweighted accuracy of 68.8% and 67.0% in a binary classification of arousal and valence, which improves over the baseline vanilla VAE by 5.5% and 6.5%. Further analysis reveals that several EEG features are statistically relevant between different personalities types across emotional states, and ECG features are also specifically correlated to personality dimension of “Creativeness”, underscoring the importance of personality in modulating psychophysiological processes.