Acute stroke lesions evolve rapidly, making it essential to model both temporal progression signals and anatomical constraints. DME-Net incorporates complementary temporal and spatial memory banks that collaboratively stabilize predictions across different lesion stages. The temporal memory captures evolving intensity and shape patterns from diffusion-weighted and perfusion-weighted imaging, while the spatial memory preserves stable anatomical structures that prevent spurious expansion. A gating mechanism adaptively balances the influence of both memory sources based on lesion characteristics. Evaluated on ISLES2018 (3,263 slices; 228 subjects), DME-Net achieves a Dice of 0.893, outperforming ConvLSTM-UNet (0.813, +9.8%) and 3D-UNet (0.846, +4.7%). HD95 declines from 14.7 mm to 8.9 mm (−39.5%), and false positives decrease by 13.6%.