Galore pytorch

[非官方]盛大：通過梯度低級投影進行記憶效率的LLM培訓

WIP非官方實施GALORE：通過梯度低級投影的記憶效率LLM培訓

路線圖

• 層面訓練技巧
• 樣品訓練循環
• 在玩具數據上添加培訓日誌
• 培訓真實*數據

參考

 @article{zhao2024galore,
  title   = {GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection},
  author  = {Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2403.03507}
}