Background Esophageal abnormalities cause severe health complications. Diagnosis is traditionally performed on endoscopic imaging, but the method is highly subjective and prone to inter-observer variability. Consequently, there is a critical need for an advanced deep learning model that enables early detection of esophageal diseases with high accuracy, integrated with robust interpretability techniques. The goal is to develop and evaluate multiple hybrid algorithms for the early detection of esophageal abnormalities and to identify the optimal architecture, and integrate interpretability techniques.Methods This study utilized a publicly available dataset of endoscopic images to evaluate four deep learning architectures, Vision Transformer (ViT), EfficientNetB0 with ViT, EfficientNetV2M with ViT, ResNet50V2. The methodology utilized the Synthetic Minority Over-Sampling Techniques (SMOTE) for class balancing and Contrast Limited Adaptive Histogram Equalization (CLAHE) for image enhancement.Results ViT achieved validation accuracy of 94.31%, Hybrid EfficientNetB0 with ViT reached 97.40%, and ResNet50V2 obtained 99.69%. The hybrid EfficientNetV2M with ViT model surpassed these with the highest accuracy of 99.75%, alongside a 100% recall, 98.44% precision, a F1-score of 99.22%, Cohen’s Kappa score of 99% and Matthews Correlation Coefficient (MCC score) of 99%.Conclusion The proposed framework demonstrates that fusing sophisticated preprocessing pipelines with hybrid deep learning architectures significantly enhances the automated recognition of esophageal abnormalities. Moreover, the developed novel Sea Shell SHAP and its integration alongside traditional Sharply Additive exPlanations offers transparency and Novel HCAM alongside traditional Gradient-weighted Class Activation Mapping (Grad CAM) and Integrated Gradient, resolved the “black box” limitations of AI and enabling its trust for clinical decision-support system.