In the field of artificial intelligence, cross-domain fine-tuning technology is becoming a key solution to protect model property rights and data privacy. Recently, a joint research team from Ant Digital Technology, Zhejiang University, Liverpool University and East China Normal University proposed an innovative framework called ScaleOT at the AAAI2025 conference. While keeping the model performance lossless, this framework improves privacy protection by 50%, and reduces computing power consumption by 90% compared with traditional knowledge distillation technology. This breakthrough result provides an efficient and lightweight solution for cross-domain fine-tuning of the ten-billion-level parameter model. Its paper was selected as the oral report of AAAI for its innovation. The number of submissions at this conference was close to 13,000, and the proportion of oral reports was only 4.6%.
Cross-domain fine-tuning technology converts the big model into an emulator through lossy compression. The data holder trains the adapter based on it and returns it to the big model to complete the tuning, thus ensuring that neither the data nor the model is out of the domain and protecting the privacy of both parties. However, the prior art has some limitations: first, the “uniform building block” approach can easily lead to the loss of key layers of the model, thereby significantly reducing performance; second, the use of distillation technology to compensate for performance losses will lead to high computational costs; in addition, existing methods lack flexibility in privacy protection.
The ScaleOT framework proposed by the Ant Digital Technology technical team successfully balances model performance and privacy security through three innovative ideas. First, the framework uses reinforcement learning scanning to automatically identify key layers by evaluating the importance of the smart layer of the big model, thereby reducing performance losses. Secondly, the retained original layer is "coded" to prevent attackers from restoring the original model and improving privacy protection strength with almost lossless performance. Finally, the ScaleOT framework can be flexibly assembled according to different scenarios to achieve adjustable privacy strength.
Solving the privacy and security issues of data and models is an important issue for big models to implement in the industry, especially the financial industry. This innovative algorithm of Ant Digital has been integrated into its Moss big model privacy protection product, and has become one of the first products in China to pass the Institute of Information and Communications Technology to perform special tests on environmental products that are trustworthy. The successful application of this technology not only provides a safer model deployment solution for the financial industry, but also sets a benchmark for large-scale model applications in other industries.
With the continuous development of artificial intelligence technology, cross-domain fine-tuning technology will play an important role in more fields. The proposal of the ScaleOT framework not only provides new ideas for the balance between model performance and privacy security, but also lays a solid foundation for the widespread application of large models in the future. This innovative achievement of Ant Digital Technology and its partners will undoubtedly promote the further development of artificial intelligence technology in the field of privacy protection.