Elamur Research⌕
☰
home / papers / 2026 / Similarity-Distance-Magnitude Activations

Similarity-Distance-Magnitude Activations

Paper · 2026 · Allen Schmaltz

Summary

Language models often give highly confident wrong answers. Softmax-based uncertainty estimates also break down under covariate shift or on out-of-distribution inputs. This paper proposes the SDM (Similarity-Distance-Magnitude) activation as a drop-in replacement for softmax at the final layer.

A small 1-D CNN exemplar adaptor is trained on the frozen hidden states of a pre-trained LM. From its compressed representations, three signals are derived:

  • SIMILARITY: how many consecutive nearest training neighbours were correctly predicted and share the test prediction.
  • DISTANCE: the nearest-neighbour distance, normalised through class-wise empirical CDFs on a calibration set.
  • MAGNITUDE: the usual logit, measuring distance to the decision boundary.

Building on this activation, the SDM estimator adapts label-conditional conformal prediction. It partitions the class-wise eCDFs to control class- and prediction-conditional accuracy in selective classification.

Experiments with pre-trained LMs show that the estimator stays more robust under distribution shift than existing post-hoc calibration methods while remaining informative in-distribution. Because predictions can be traced back to concrete training and calibration examples, the method also offers interpretability-by-exemplar.

Key points

  • The activation has the form (2+q)^(d·z′), normalised over classes. Setting q = e−2 and d = 1 recovers the standard softmax. When d = 0, the output is uniform, signalling maximal uncertainty.
  • A HIGH-RELIABILITY region is defined by a threshold q_min. For MIXTRAL 8X7B+SDM on the FACTCHECK calibration set, no finite q_min was found at α = 0.95. At α = 0.94, q_min was 134.0.
  • The adaptor input concatenates a mean-pool of final-layer hidden states with the hidden state of the last sequence position. This allows the inputs to be cached for efficient training.
  • At test time, the method needs the training exemplar vectors. The authors compare the extra cost to typical dense retrieval.
  • Experiments use open-weight models only. Applying the method to black-box commercial APIs, for example via cross-encoding with an open-weight model, is left to future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Allen Schmaltz. 2026. Similarity-Distance-Magnitude Activations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1109

@inproceedings{schmaltz2026similarity,
  title = {{Similarity-Distance-Magnitude Activations}},
  author = {Allen Schmaltz},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1109},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1109}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-… — Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
  2. Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. 2021. Uncertainty Sets for Image Classifiers using Conformal Prediction. In International Conference on Learning Representations. find
  3. Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore. Association for Computational Linguistics.
  4. Glenn W. Brier. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1 – 3. find
  5. C. K. Chow. 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4):247–254. find
  6. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and accurate deep network learning by exponential linear units (elus). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  7. T. Cover and P. Hart. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1):21–27. find
  8. A. P. Dawid. 1982. The well-calibrated bayesian. Journal of the American Statistical Association, 77(379):605–610. find
  9. Luc Devroye, László Györfi, and Gábor Lugosi. 1996. A Probabilistic Theory of Pattern Recognition. In Stochastic Modelling and Applied Probability. find
  10. A. Dvoretzky, J. Kiefer, and J. Wolfowitz. 1956. Asymptotic Minimax Character of the Sample Distribution Function and of the Classical Multinomial Estimator. The Annals of Mathematical Statistics, 27(3):642 – 669. find
  11. Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. 2020. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455–482. find
  12. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA. PMLR. find
  13. Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  14. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321– 1330. PMLR. find
  15. Chirag Gupta and Aaditya Ramdas. 2022. Top-label calibration and multiclass-to-binary reductions. In International Conference on Learning Representations. find
  16. James Harrison, John Willes, and Jasper Snoek. 2024. Variational bayesian last layers. In The Twelfth International Conference on Learning Representations. find
  17. J. T. Gene Hwang and A. Adam Ding. 1997. Prediction intervals for artificial neural networks. Journal of the American Statistical Association, 92(438):748–757. find
  18. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  20. Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. 2019. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  21. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  22. Jing Lei and Larry Wasserman. 2014. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):71–96. find
  23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  24. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon…
  25. P. Massart. 1990. The Tight Constant in the Dvoretzky-Kiefer-Wolfowitz Inequality. The Annals of Probability, 18(3):1269 – 1283. find
  26. Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volum… find
  27. John C. Platt. 1999. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, pages 61–74. MIT Press. find
  28. Yaniv Romano, Matteo Sesia, and Emmanuel Candes. 2020. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, pages 3581–3591. Curran Associates, Inc. find
  29. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics.
  30. Allen Schmaltz. 2021. Detecting local insights from global labels: Supervised and zero-shot sequence labeling via a convolutional decomposition. Computational Linguistics, 47(4):729–773. find
  31. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. find
  32. Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. 2019. Evaluating model calibration in classification. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learn… find
  33. Start your response with Yes or No. find
  34. L. G. Valiant. 1984. A theory of the learnable. Commun. ACM, 27(11):1134–1142. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  36. Vladimir Vovk. 2012. Conditional validity of inductive conformal predictors. In Proceedings of the Asian Conference on Machine Learning, volume 25 of Proceedings of Machine Learning Research, pages 475– 490, Singapore Management University, Singapore. PMLR. find
  37. We replace DOCUMENT with the corresponding text for each instance. Factcheck. Similarly, for the factcheck datasets, we prompt the LMs for a binary classification (Yes or No) as follows: Here is a statement that may contain errors. \<statement\> DOCUMENT \</statement\> Is the statement true? Answer Yes… find
  38. Vladimir Vovk, Alex Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Springer-Verlag, Berlin, Heidelberg. find
  39. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
  40. As above, we replace DOCUMENT with the corresponding text for each instance. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmaltz-2026-similarity-distance-magnitude-activations/