Error exists in almost every measurement, which often affects the application of measurement. In traditional method, system error is estimated by combine several measurements, but in some cases, we can get only one measurement and should estimate the system error of this measurement. This paper studies the estimation of high frequency system error of one measurement element. In this method, we assume the frequency of signal is lower than the frequency of system error. Under this assumption, we can constrain the derivative of the signal to be small because of its low frequency, and this we help us to establish a model to separate system error with high frequency from signal. According to this thought, this paper gives a regularization model for system error estimation based on signal smoothness constraint. Simulated results demonstrated the efficiency of new method.