Human action recognition has been widely researched and applied in intelligent visual surveillance fields nowadays. Most work on action recognition has been visible-spectrum oriented over the past decade, while the persistence of visual surveillance system increases the demand for night-time action recognition. This paper deals with the problem of night action recognition using thermal infrared imagery. A novel algorithm based on the human action silhouettes energy histograms is proposed. The algorithm first makes use of the statistical background model and background subtraction method to extract the human action silhouettes, while calculating the silhouette energy images for the action sequences. Then, the histograms of oriented gradients are computed from the silhouette energy images. Finally, the human action is represented by the energy histograms features, and recognized by using the Euclidean distance and nearest neighbor classifier. An infrared human action database was built to provide a foundation for night action recognition. Experimental results using the infrared thermal action data show the effective of this method.