There has been increasing attention drawn to modelling inter- rater ambiguity in Continuous Emotion Recognition (CER) sys- tems using probability distributions for arousal and valence. However, the relationship between modelling label ambiguity and robustness to noise, and more broadly, the impact of real- world noise on CER systems remains insufficiently explored. In this study, we argue that incorporating inter-rater ambigu- ity during training can regularize the noise response, leading to noise robustness. To this end, we propose a novel loss function that incorporates inter-rater ambiguity into model training. Ex- periments conducted on the RECOLA dataset demonstrate that our proposed method achieves a maximum Concordance Cor- relation Coefficient (CCC) improvement of 0.117 and 0.077 for mean and standard deviation predictions, respectively, across all noise conditions. We further integrate traditional noisy augmen- tation strategies with our proposed method and observe promis- ing results.