# fast-norm-cuda

A fast, yet specialized, RMSNorm/LayerNorm implementation

This library is under development. Currently, only some special cases are supported, and the performance is not yet fully optimized.

- [x] RMSNorm
- [ ] LayerNorm
- [x] Float16 and BFloat16
- [ ] More data types
- [x] More shapes
- [ ] Accelerate if no wgrad
- [ ] Performance tuning


## Statement

This work was independently completed by me at home using my personal RTX 3080.
