This paper proposes rapid extraction algorithm for identifying the main direction of smoke, according to internal small local movement feature of initial rising smoke and the overall presentation presents upward and both sides diffusion trend. The algorithm adopts Meanshift Kernel function of histogram to model movement characteristics and estimates the smoke pieces of motion vector direction by using Bhattachyarya coefficient. Smoke data block motion accumulation are analyzed statistically to estimate the proportion of each suspected smoke region in main movement to realize the identification of inside smoke moving direction. The obtained results from the thick black smoke experiment, gray smog experiment, human body movement interference experiment show that the algorithm is better able to extract the main direction features of smoke and has a strong ability to eliminate interference. It provides criterion for the further study of smoke recognition algorithm.