A method of abnormal motion detection for intelligent video surveillance is presented, which includes object intrusion detection, object overlong stay detection and object overpopulation detection. Background subtraction algorithm is used to detect moving objects in video streams. Kalman filter is applied for object tracking. By the construction of relation matrix, the tracking process is divided into five statuses for prediction and estimation, which are object disappearing, object separating, new object appearing, object sheltering and object matching. The object parameters and predictive information in the next frame which is used to track moving objects is established by Kalman filter. Then, three types of abnormal motion detection are implemented. The relative position of alarm area or guard line with the rectangle boxes of the moving objects is used to detect whether the object is invading. The existing time of the moving objects in monitor area is counted to detect whether the object is staying too long. Moving objects in the monitor area are classified and counted to detect whether the objects are too much. Alarm will be triggered when abnormal motion detection as defined is detected in the monitor area.