Most of the traditional block-matching algorithms for motion estimation (ME) can only yield local optimal motion vectors (MVs). In this paper, the autoregressive moving average process (ARMA) model is selected to formulate the correlation of neighboring blocks in a frame, and the adaptive Kalman filtering algorithm is applied to refine the MVs. The horizontal and vertical ARMA models are constructed to utilize the filtering algorithm twice to get a better performance. Our method can also be extended to realize disparity estimation (DE) in order to apply it in a multi-view video coding (MVC) system. The experiment results show the effectiveness of our method to improve the accuracy of conventional fast block matching algorithms.