Researchers from the National University of Singapore and Purdue University have developed a breakthrough technology called PAB that enables real-time processing of video generation based on diffusion transformation. This technology is based on the Diffusion Transformer (DiT) model and improves video generation speed by 10.6 times by reducing redundant attention calculations, reaching an astonishing 21.6 frames per second. PAB can be applied to multiple popular DiT video generation models, such as Open-Sora, Open-Sora-Plan and Latte, without additional training, laying a solid foundation for future real-time video generation technology. It not only greatly improves the processing speed, but also significantly reduces the communication overhead between multiple GPUs while ensuring video quality, providing more efficient distributed inference capabilities for real-time video generation. The following is a more detailed explanation of PAB technology.
Recently, researchers from the National University of Singapore and Purdue University successfully proposed PAB technology to achieve real-time processing of video generation based on diffusion conversion.
Product entrance: https://top.aibase.com/tool/pab
This technology is the first attempt at a video generation model based on Diffusion Transformer (DiT), achieving a generation speed of up to 21.6 frames per second by reducing redundant attention calculations, a 10.6x acceleration, without sacrificing quality. Works with several popular DiT video generation models, including Open-Sora, Open-Sora-Plan and Latte. PAB is a training-free method that can empower future DiT video generation models with real-time generation capabilities. PAB requires no training and can give any future diffusion transformation-based video generation model the ability to process in real time.
Important features:
PAB attention broadcasting significantly improves the speed of video generation by reducing redundant attention calculations and achieves real-time generation.
Based on the stability and difference of attention, PAB sets different broadcast ranges for different types of attention, thereby minimizing quality loss while ensuring computational efficiency.
By improving sequence parallel processing technology, PAB reduces the communication overhead between multiple GPUs and further improves the speed and efficiency of video generation.
The researchers found that the attention mechanism in the video diffusion transformation model has obvious differences between time steps. Through this discovery, PAB was proposed to alleviate unnecessary attention calculations. In the stable middle part, PAB broadcasts the attention output of one diffusion step to multiple subsequent steps, thereby significantly reducing the computational cost. In addition, for more efficient computation and minimizing quality loss, different broadcast ranges are set for different attention types.
In order to further improve the speed of video generation, the researchers improved the parallel processing method based on dynamic sequence parallelism (DSP), which eliminated most of the communication overhead by broadcasting time attention, achieving more than 50% reduction in communication overhead, and provided a better solution for real-time video generation. Provides more efficient distributed inference capabilities.
Highlight:
⭐ PAB technology enables real-time video generation and accelerates processing speed by 10.6 times.
⭐ By observing the difference in the attention mechanism of the video diffusion conversion model, PAB is proposed to alleviate unnecessary attention calculations.
⭐ By improving the parallel processing method, communication overhead is greatly reduced, providing more efficient distributed inference capabilities for real-time video generation.
The emergence of PAB technology marks a major breakthrough in real-time video generation technology, providing powerful real-time processing capabilities for future video generation models based on diffusion conversion, and further expanding the application prospects of artificial intelligence in the video field. It is believed that PAB technology will play an increasingly important role in the field of video generation in the future.