GitXplorerGitXplorer
h

candle-layer-norm

public
4 stars
8 forks
0 issues

Commits

List of commits on branch main.
Unverified
887beca59aaa36cbbfa3b36796111193eacd762f

Remove warning

OOlivierDehaene committed a year ago
Unverified
01a55934cb172036a1458712969523cfa6563fbe

Support CUDA_COMPUTE_CAPS env var

OOlivierDehaene committed a year ago
Unverified
3e8e997655d774f93c083a70a5f25efd9808c22f

Update candle version

OOlivierDehaene committed a year ago
Unverified
8c2370dc2c80a4543502ef1a4dd51a65a6e7d567

Add licenses

OOlivierDehaene committed a year ago
Unverified
66ac750a41b41b39989278aca46fc805dbca869e

format

OOlivierDehaene committed a year ago
Unverified
458af4619fe5bd64f5793bcabfe39ac0b5fa328f

Add tests

OOlivierDehaene committed a year ago

README

The README file for this repository.

Candle Cuda Layer Norm

Layer Norm fused operation for the Candle ML framework.

This Layer was adapted from https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm.

It implements fused dropout + residual + LayerNorm, building on Apex's FastLayerNorm.

Major changes:

  • Add residual.
  • Make it work for both pre-norm and post-norm architecture.
  • Support more hidden dimensions (all dimensions divisible by 8, up to 8192).
  • Implement RMSNorm as an option.