Runs
6
Lowest ECE
0.0283
Extra query cost
<1 ms
What I tested
Could a model learn which neighbors to trust without making a simple retrieval system expensive?
How it unfolded
Four passes, with the useful result separated from the interesting idea.
- 01
Start with a fair baseline
I paired a ResNet18 embedder with a FAISS memory bank. Attention reached 88.54% accuracy; uniform kNN reached 88.61%.
- 02
Make the representation stronger
ResNet50, contrastive learning, and hard negatives moved every variant to about 89.7%. Attention still did not pull ahead.
- 03
Try the full recipe
Multi-head attention, MixUp, and test-time augmentation (TTA) cut ECE from 0.1297 to 0.0283.
- 04
Run the check again
The result held: TTA reached 0.0379 ECE; attention alone reached 0.1359.
The final comparison
Experiment 6 · lower ECE means better-calibrated confidence.
| Method | Accuracy | ECE | NLL |
|---|---|---|---|
| Uniform kNN | 85.57% | 0.1336 | 2.028 |
| Distance kNN | 85.55% | 0.1169 | 2.028 |
| Attention kNN | 85.48% | 0.1359 | 2.058 |
| Attention kNN + TTA | 85.54% | 0.0379 | 0.931 |
What helped
TTA improved calibration
TTA was the clear win. The strongest run cut ECE by 78% and added less than 1 ms per query.
What did not
Attention did not beat the baselines
Learned weights stayed close to uniform and distance-weighted kNN. The evidence is not strong enough to call it an accuracy improvement.
The evidence
I kept these plots because they show how the model behaved, not just its best score.
Bottom line
This is a working prototype and a useful negative result. If the goal is better calibration, start with TTA. Adding attention made the system more complex without a clear gain.