Federated test-time adaptation (FTTA) enables privacy-preserving model adaptation to unlabeled target data during inference, yet it struggles with dynamic source client availability and uncertain test samples under distribution shifts. Existing methods overlook offline client impacts and rely on noisy pseudo-labels, leading to error accumulation. To address these challenges, we propose a novel FTTA framework, termed \underline{Fed}erated test-time adaptation under \underline{D}ynamic client collaboration and \underline{C}ategory-aware uncertainty (FedDC), which integrates dynamic client weighting initialization and class-aware margin thresholds to improve adaptation robustness. During source training, clients are aggregated adaptively based on participation history to reduce bias. At test time, category-specific thresholds separate confident and uncertain samples, preserving prediction uncertainty to mitigate noise. Extensive experiments on CIFAR100, Tiny-ImageNet, PACS, and CarlaTTA show that FedDC outperforms baseline methods under various distribution shifts, with clear improvements in adaptation accuracy and stability. The proposed design supports robust deployment in real-world decentralized visual systems. The source code is publicly available at https://github.com/ycarobot/FedDC.