Physiological emotion recognition leverages biosignals to interpret human emotional states under multimedia stimuli. While recent graph-based methods offer promising performance by modeling interdependencies among subjects, most focus on global structural learning, overlooking localized interactions critical for nuanced emotion understanding. In this work, we present LocalGSL, a graph structure learning framework that performs targeted local connectivity refinement to enhance the discriminative capacity of physiological graphs. By introducing learnable and differentiable strategies to adjust edge and node connectivity, our approach captures subtle structural variations that boost discriminative ability. Experimental results show that our refined local structure method yields notable gains in prediction accuracy. Further analyses reveal that local refinement strengthens node-level expressiveness and increases centrality variation.