Advanced wearable tracking shows potential for identifying psychological and emotional stress relevant to the mental health of high-intensity emergency responders. Heart rate variability (HRV) captured by wearable devices can indicate the correlation between intra-subject daily variations and stress. HRV also varies due to various demographic attributes, representing inter-subject relationships. This work introduces an individual-aware metric learning approach that leverages HRV features to train intra-subject representations, considering inter-subject effects based on attribute similarity through stress label clustering. We use the multi-similarity loss within the metric learning framework to consider various personal attributes, thereby improving discriminability. Evaluation of the TILES-2018 and Firefighter database shows promising results in binary stress classification: F1 score of 68.15% with BACC of 59.13% and MCC of 0.186, and F1 score of 73.07% with BACC of 56.52% and MCC of 0.136, respectively.