Tumor detection and classification in medical images are critical for guiding patient management and treatment decisions. However, accurate segmentation and classification of tumors remain challenging due to their small size relative to the overall image. Existing approaches often face difficulties with limited data and potential segmentation errors, resulting in suboptimal performance in real-world applications. To address these challenges, we propose a novel approach leveraging the Segmentation Mask Augmentation (SMA) framework. Our framework enhances the robustness of tumor classification models by generating diverse and imprecise segmentation masks during training, thereby simulating real-world scenarios. Experimental results across four distinct datasets demonstrate the effectiveness of our approach. Our framework presents a promising solution for robust tumor classification, with potential implications for improving clinical diagnosis and patient management.