The SAM model of Meta performed well in the field of image segmentation, but there is insufficient in the tracking of video objects, especially in the complex scenes. Researchers at the University of Washington have developed the Samurai model for this to improve SAM2, which significantly improves the performance of video object tracking. Samurai cleverly combines time movement clues and motion perception memory selection mechanisms. Like a martial arts warrior, they can accurately predict the motion trajectory of objects and choose the most reliable mask.
The "segmentation of all" model SAM launched by META can be described as invincible in the field of image segmentation, but when it comes to the tracking of video objects, it is a bit unsatisfactory, especially in the scene of crowds of people, fast movement or playing "hiding cats". SAM will be confused. This is because the memory mechanism of the SAM model is like a "fixed window", which only cares about the recent screen and ignores the quality of the memory content, which leads to errors in the video and the tracking effect is greatly reduced.
In order to solve this problem, researchers at the University of Washington "thought hard" and finally developed a model called Samurai, which made "Devil Reform" for SAM2, which was specially used to track video objects. Samurai's name is very domineering, and it does have two brushes: it combines the time motion clue and the newly proposed sports perception memory choice mechanism, just like a martial arts warrior, which can accurately predict the motion trajectory of the object and improve the improvement of the movement of the object, and improve the improvement Cover selection, and eventually achieve stable and accurate tracking without re -training or fine -tuning.
The secret of Samurai lies in the two innovations:
First move: Sport modeling system. This system is like the "Eagle Eye" of the samurai, which can more accurately predict the location of the objects in the complex scene, thereby optimizing the choice of mask, so that Samurai will not be confused by similar objects.
Second trick: Sports perception memory selection mechanism. Samurai abandoned the simple "fixed window" memory mechanism of SAM2, and instead adopted a hybrid scoring system, combined with the original mask similarity, objects and movement scores, just like the warrior carefully selected weapons, only retains the most related historical information, so as to thus Improve the overall tracking reliability of the model and avoid errors.
Samurai is not only strong in martial arts, but also agile and can run in real time. More importantly, it shows a strong zero -sample performance on various benchmark datasets, which means that it can adapt to various scenes without special training and show strong generalization capabilities.
In actual combat tests, Samurai has improved significantly in terms of success rate and accuracy compared to existing trackers. For example, on the Lasotext dataset, it obtains a 7.1% AUC gain; on the GOT-10K dataset, it obtains a 3.5% AO gain. What is even more surprising is that it has achieved the result of the LASOT dataset that is comparable to the complete supervision method, which fully proves its powerful strength in the complex tracking scene and the great potential of practical application in the dynamic environment.
The success of Samurai is due to its clever use of sports information. Researchers combine the traditional Carman filter with SAM2. By predicting the location and size of the object, the model is used to select the most reliable mask from multiple candidate masks. In addition, they also designed a memory selection mechanism based on three scores (mask similarity scores, object scores and motion scores). Only when these three scores reach the threshold, the frame screen will be selected into the frame. Memory library. This selective memory mechanism effectively avoids the interference of irrelevant information and improves the accuracy of tracking.
The emergence of Samurai has brought new hope to the field of video object tracking. It not only surpasses the existing tracker in performance, but also does not need to re -train or fine -tuning, which can be easily applied to various scenarios. It is believed that in the future, Samurai will play an important role in autonomous driving, robotics, video surveillance and other fields to bring us a more intelligent life experience.
Project address: https://yangchris11.github.io/samurai/
Thesis address: https: //arxiv.org/pdf/2411.11922
All in all, the Samurai model has made breakthroughs in the field of video object tracking. Its efficient, accurate and robust performance provides strong technical support for future intelligent applications. Its innovative memory mechanism and sports modeling system are worthy of in -depth research and reference.