# Symmetrical Attention Mechanisms in Transformers

| **Title** | **Link** | **Distinctive Trait** |
|-----------|----------|------------------------|
| **Symmetric Dot-Product Attention for Efficient Training of BERT Language Models** | [arXiv](https://arxiv.org/abs/2406.06366) | Introduces symmetric dot-product attention that reduces training steps by 50% and improves BERT's efficiency without performance loss. |
| **The Underlying Structures of Self-Attention: Symmetry, Directionality, and Emergent Dynamics in Transformer Training** | [arXiv](https://arxiv.org/abs/2502.10927) | Theoretical study showing how bidirectional vs. autoregressive training affects attention symmetry and convergence dynamics. |
| **A Lightweight Vision Transformer with Symmetric Modules for Vision Tasks (SFormer)** | [SAGE Journal](https://journals.sagepub.com/doi/full/10.3233/IDA-227205) | Proposes symmetric attention and FFN blocks to build lightweight vision Transformers optimized for edge devices. |
| **SCATT: Transformer Tracking with Symmetric Cross-Attention** | [ACM / Springer](https://dl.acm.org/doi/10.1007/s10489-024-05467-1) | Replaces traditional correlation with symmetric cross-attention for enhanced global context in visual tracking. |
| **Symmetric Power Transformers** | [Manifest AI](https://manifestai.com/articles/symmetric-power-transformers/) | Uses even-power dot product attention to enforce symmetry, outperforming softmax in long-sequence modeling tasks. |
