GitXplorerGitXplorer
h

candle-layer-norm

public
4 stars
8 forks
0 issues

Commits

List of commits on branch main.
Unverified
e5858babe3ecf1a256ddce388db54c6c8c56b8cf

Fixing build.rs

NNarsil committed a year ago
Unverified
72e6fbdf727095492949fc6adae1e2204bb53c77

first commit

OOlivierDehaene committed a year ago

README

The README file for this repository.

Candle Cuda Layer Norm

Layer Norm fused operation for the Candle ML framework.

This Layer was adapted from https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm.

It implements fused dropout + residual + LayerNorm, building on Apex's FastLayerNorm.

Major changes:

  • Add residual.
  • Make it work for both pre-norm and post-norm architecture.
  • Support more hidden dimensions (all dimensions divisible by 8, up to 8192).
  • Implement RMSNorm as an option.