Recently, a research team from the Tubingen Ellis Institute, the University of Maryland and Lawrence Livermore National Laboratory successfully developed a new language model called Huginn. This model adopts a unique recursive architecture that significantly improves its inference ability in complex tasks. Unlike traditional language models, Huginn does not need to rely on special "inference chain" training, but can independently reason within the "latent space" of the neural network and output the results. This innovative design opens up new directions for the development of language models.
The training process of the Huginn model was carried out on the Frontier supercomputer, and the researchers used 4096 AMD GPUs for large-scale training. Its training method is unique and adopts a strategy of variable calculation number of iterations. The system can randomly determine the number of repeated calculation modules, so that the model can better adapt to the complexity of different tasks. This flexible training method lays the foundation for Huginn's efficient reasoning ability.
Huginn performed particularly well in math and programming tasks during the test. In GSM8k and MATH benchmarks, Huginn's performance even surpasses open source models with both parameter size and training data volume several times higher than its own. The researchers found that Huginn was able to dynamically adjust the depth of computation based on the complexity of the task and independently develop inference chains within the “potential space.” Further analysis shows that the model forms complex computational patterns in the "latent space", such as presenting a circular trajectory when solving mathematical problems. This discovery proves that Huginn has the ability to learn independently and is able to reason in novel ways.
While Huginn's absolute performance still has room for improvement, it has already shown amazing potential as a proof-of-concept model. Researchers believe that as the reasoning time is extended and the ability is further improved, large models using Huginn architecture are expected to become an alternative to traditional inference models. The team also emphasized that Huginn's approach may capture some indescribable types of reasoning and plans to continue to study in depth in the future to explore scaling methods such as reinforcement learning to further improve the performance of the model.