Skip to content
GitLab

[not to be merged] bunch of changes that I have been using to train latest models

Open Administrator requested to merge ngoyal_sequence_parallel_cpu_offload into main

Created by: ngoyal2707

this branch is what I am using to train latest model with following changes:

I will not merge this and will piece out separate PRs for each of these following:

  1. Seq_parallel
  2. ability to only checkpoint MHA
  3. disable bias
  4. disable LN affine,
  5. act cpu_offload experimental
  6. merging of gelu with FC2 to save on act memory of gelu
  7. bunch of cleanup