Documented runs
6
Best observed ECE
0.0283
Attention overhead
<1 ms
The question
Can learned attention over a kNN memory improve accuracy, calibration, or robustness beyond uniform and distance-weighted neighbors without making retrieval impractical?
The research trail
Each run changed one part of the system, then kept only the evidence that survived the next experiment.
Experiments 1–3
Establish the baseline
Built: ResNet18 embeddings, a FAISS memory bank, and a single-head attention prototype.
Learned: On CIFAR-10, attention reached 88.54% accuracy versus 88.61% for uniform kNN. It was viable, not a clear win.
Experiment 4
Strengthen the representation
Built: ResNet50, contrastive learning, hard-negative mining, bounded temperature, and 50 training epochs.
Learned: Training was more stable and all kNN variants reached about 89.7%. The gap between distance weighting and attention stayed negligible.
Experiment 5
Test the full method
Built: 256-dimensional embeddings, four attention heads, MixUp, label smoothing, TTA, a k-ensemble, k-sweeps, and noise tests.
Learned: TTA was the signal: ECE fell from 0.1297 to 0.0283 (78%) and accuracy reached 87.20%. Attention alone remained comparable to the baselines.
Experiment 6
Validate the result
Built: A final held configuration and the same calibration, robustness, and efficiency checks.
Learned: The pattern held: TTA reduced ECE to 0.0379, while attention alone reached 0.1359. The project concluded with a more precise claim.
Final validation
Experiment 6 is the final check. It separates the learned-attention result from the TTA result rather than treating them as the same contribution.
| Method | Accuracy | ECE | NLL |
|---|---|---|---|
| Uniform kNN | 85.57% | 0.1336 | 2.028 |
| Distance kNN | 85.55% | 0.1169 | 2.028 |
| Attention kNN | 85.48% | 0.1359 | 2.058 |
| Attention kNN + TTA | 85.54% | 0.0379 | 0.931 |
What held up
TTA consistently improved calibration; the strongest run reduced ECE by 78%. FAISS kept retrieval practical, and performance remained stable across the k sweep.
What did not
Attention did not establish a reliable improvement over distance weighting. The calibrated TTA result is real, but it is not proof that learned attention was the cause.
Evidence, not decoration
Calibration, robustness, and parameter sensitivity were all inspected—not inferred from a single headline metric.
What the work leaves behind
A modular research prototype: ResNet embeddings, a FAISS memory bank, multi-head neighbor attention, training losses aligned to neighbor aggregation, and a documented record of the experiments that changed the conclusion.