GitXplorerGitXplorer
h

candle-layer-norm

public
4 stars
8 forks
0 issues

Commits

List of commits on branch main.
Unverified
94c2add7d94c2d63aebde77f7534614e04dbaea1

relax candle requirement

OOlivierDehaene committed a year ago
Unverified
5533f25f3a7f9c3e93c3ba8758b000872587f135

feat: return result of the residual add

OOlivierDehaene committed a year ago
Unverified
0dd5bdceb9ba7cded921c62f9ddd66e7726327ba

chore: relax dependencies

OOlivierDehaene committed a year ago
Unverified
f66ff482dcc20404918f71627020cba9210cb52a

fix: blanket match for storage and device

OOlivierDehaene committed a year ago
Unverified
d417e18f5476203119270e290553fb3fa1161dba

feat: remove mandatory beta

OOlivierDehaene committed a year ago
Unverified
5ed96012a693dff9685320765dd55a57fdaecdd6

Revert to single compute cap

OOlivierDehaene committed a year ago

README

The README file for this repository.

Candle Cuda Layer Norm

Layer Norm fused operation for the Candle ML framework.

This Layer was adapted from https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm.

It implements fused dropout + residual + LayerNorm, building on Apex's FastLayerNorm.

Major changes:

  • Add residual.
  • Make it work for both pre-norm and post-norm architecture.
  • Support more hidden dimensions (all dimensions divisible by 8, up to 8192).
  • Implement RMSNorm as an option.