A novel automatic target detection and tracking algorithm for tracking targets based on ship borne infrared imagery is proposed in this paper. The algorithm has two modes: detection and tracking. In detection mode, the proposed algorithm utilized difference between the consecutive frames and top-hat filter to determine the target position. In tracking mode, the Intensity Variation Function (IVF) is utilized. Before tracking and location the new position, the algorithm should determine the IVF whether reliable. If unreliable, the mode turns to detection. Experimental results by using real-life infrared image sequence are shown to validate the robustness of the proposed technique.