Russian technology giant Yandex has open sourced its self-developed YaFSDP tool, an optimization method for large language model (LLM) training, whose efficiency leads the industry. YaFSDP can significantly increase the LLM training speed by up to 26%, and significantly save GPU resource costs, which is undoubtedly a great benefit for AI developers and enterprises. This tool performs particularly well when the training parameter size reaches 30 billion to 70 billion, providing more possibilities for small and medium-sized enterprises and individual developers to train LLM independently.
Yandex, a technology giant from Russia, has recently open sourced its independently developed YaFSDP tool to the global AI community, which is currently the most efficient large language model (LLM) training optimization method in the industry. Compared with the FSDP technology widely used in the industry, YaFSDP can increase LLM training speed by up to 26%, which is expected to save a lot of GPU resources for AI developers and enterprises.
YaFSDP (Yandex Full Sharded Data Parallel) is an enhanced version of Yandex based on FSDP. It focuses on optimizing GPU communication efficiency and memory usage, eliminating bottlenecks in the LLM training process. In communication-intensive tasks such as pre-training, alignment and fine-tuning, YaFSDP shows excellent performance improvements, especially when the training parameter size reaches 30 billion to 70 billion.
Mikhail Khruschev, senior development expert at Yandex and member of the YaFSDP team, said: "YaFSDP is best suited for widely used open source models based on the LLaMA architecture. We are still continuing to optimize and expand its versatility across different model architectures and parameter sizes, with a view to its wider use Improve training efficiency in various scenarios."
It is estimated that, taking training a model with 70 billion parameters as an example, using YaFSDP can save about 150 GPU resources, which is equivalent to saving US$500,000 to US$1.5 million in computing power costs per month. This cost saving is expected to make autonomous LLM training more feasible for SMEs and individual developers.
At the same time, Yandex also promises to continue to contribute to the development of the global AI community. YaFSDP open source is a reflection of this commitment. Previously, the company has shared a number of highly regarded open source AI tools, such as CatBoost high-performance gradient boosting library, AQLM extreme model compression algorithm and Petals model training simplification library.
Industry analysts point out that as the scale of LLM continues to expand, improving training efficiency will become the key to the development of artificial intelligence. Technical breakthroughs such as YaFSDP are expected to help the AI community advance large model research faster and explore its application prospects in natural language processing, computer vision and other fields.
The open source of YaFSDP demonstrates Yandex's positive attitude and contribution in promoting the development of AI technology. It also provides a powerful tool for the global AI community, further lowering the threshold for large model training and accelerating the popularization and application of AI technology.