Investigating CoTTA: Validating Real-Time Neural Network Adaptations

Continual Test-Time Adaptation (CoTTA) extends TTAPL by updating model parameters continuously with each new data point, rather than waiting for large batches of unlabeled data. CoTTA's approach typically includes:

• Consistency loss: Encourages stable predictions across augmented views of the same input.
• Teacher-student updates: A “teacher” model refines the “student” model iteratively, reducing catastrophic forgetting.

By blending self-training with cautious weight updates, CoTTA aims to preserve the model's original knowledge while adapting to new conditions. In practice, this can be challenging when the distribution shift is severe or the model architecture is especially large. Our research examines how different consistency losses (e.g., KL, PolyLoss, Cosine Similarity) affect CoTTA's ability to adapt across multiple architectures.

• Pseudo-label drift: Incorrect pseudo-labels reinforce model biases, leading to cascading errors.
• Overfitting to test samples: Excessive adaptation to a small set of test data can degrade model generalization.

Our evaluation of Continual Test-Time Adaptation (CoTTA) was conducted using the standard CIFAR-10-to-CIFAR-10C image classification task. CIFAR-10 consists of 60,000 32x32 color images across 10 classes, while CIFAR-10C is a corrupted version of the test set with severe distortions such as noise, blur, and digital artifacts.

In this task, a model's ability to correctly classify a large number of corrupted images reflects its capacity to adapt to distribution shifts. We tracked four key metrics—accuracy, precision, recall, and F1 score—to measure improvements relative to a baseline model that receives no adaptation.

Experiment Structure: For each combination of neural network architecture and consistency loss function, we ran the adaptation process on CIFAR-10C (Severity 5 corruptions only). This allowed us to isolate the effects of both model complexity and loss function choice on adaptation performance.

Model Choice: Our experiments used models selected from RobustBench, ensuring a diverse range of architectures. We evaluated variants of ResNet (PR), WideResNet (WR), and PreActResNet (R), including models that have been trained with robust strategies such as RLAT and AugMix. Models are labeled M1 through M7 in order of increasing depth and complexity, which helped us assess the influence of architecture on test-time adaptation.

Loss Functions: In addition to a baseline with no adaptation and Normal TTA, we evaluated TENT (which minimizes entropy) and several CoTTA variants using different consistency losses:

• Cross-Entropy: Serves as our baseline since CoTTA was originally designed with CE loss.
• Cosine Similarity: Measures the angular difference between prediction vectors, potentially leading to smoother updates.
• PolyLoss: An extension of cross-entropy with a polynomial term (ε = 0.1) to reduce overconfidence.
• KL Divergence: Compares the full probability distributions between teacher and student, supporting gradual adaptation.
• Self-Training Cross-Entropy: Uses the student model's own predictions as pseudo-labels, though it may reinforce initial errors.

Our aim was to determine whether alternative loss functions can provide performance comparable to or better than the cross-entropy baseline, and to understand how different architectures respond under these various adaptation schemes.

Our experimental results indicate that when model architecture is held constant, CoTTA variants using Cross-Entropy, KL Divergence, and PolyLoss deliver similar improvements across the key metrics—typically within a ±1% range.

This consistency suggests that the choice of consistency loss can be flexibly substituted without a drastic impact on overall prediction quality, provided the model architecture remains unchanged.

In contrast, when we fix the loss function, model architecture emerges as a critical factor. For example, models trained with robust methods like RLAT or AugMix often show significant improvements under CoTTA, whereas certain high-capacity models exhibit instability or even reduced performance relative to the source model.

Among the loss functions tested, CoTTA with KL Divergence and PolyLoss consistently matches or exceeds the performance of the cross-entropy baseline, while the self-training variant frequently underperforms. This disparity underscores the challenges of relying solely on self-generated labels in a continual adaptation framework.

Overall, our findings emphasize that no single TTA strategy is optimal across all scenarios. Instead, the effectiveness of test-time adaptation is governed by the interplay between the chosen consistency loss and the underlying model architecture. For a more detailed view of the numerical improvements, please refer to the interactive table below.