Galore pytorch

[非官方]盛大：通过梯度低级投影进行记忆效率的LLM培训

WIP非官方实施GALORE：通过梯度低级投影的记忆效率LLM培训

路线图

• 层面训练技巧
• 样品训练循环
• 在玩具数据上添加培训日志
• 培训真实*数据

参考

 @article{zhao2024galore,
  title   = {GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection},
  author  = {Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2403.03507}
}