Priors for Time Series Forecasting

Abstract:

Regularization is a method for improving the solution of ill-posed problems with neural networks. In regularization, a penalty term, called regularizer or prior, is added to the performance function. The penalty term is weighted with a regularization parameter, , to balance the trade-off between model bias and model variance. We have compared the performances of different priors on several different time series data sets, to see if there is any consistent difference in performance between priors. The conclusions from our study on real world time series data has weight decay the best performance and the Bishop smoother is the worst choice.

• Periodical: Applied Mechanics and Materials (Volumes 263-266)
• Pages: 171-174
• DOI: https://doi.org/10.4028/www.scientific.net/AMM.263-266.171
• Online since: December 2012
• Authors: Jing Jing Zhou
• Keywords: Ill-Posed, Neural Network (NN), Prior, Regularization, Time Series Forecasting

References

[1] G. E. Hinton. Connectionist learning procedure. Artificial Interlligence,40:185–234, 1989.

[2] p. M. Williams. Bayesian regularization and pruning using a laplace prior.Neural Computation, 7:117–143, 1995.

DOI: 10.1162/neco.1995.7.1.117

[3] A. Weigend, D. Rumelhart, and B. A. Humberman. Generalization byweight-elimination applied to currency exchange rate prediction. In Proc.Int. Joint Conf. Neural Networks, pages 2374–2379, Singapore, 1991.

DOI: 10.1109/ijcnn.1991.170743

[4] C. M . Bishop. Curvature-driven smoothing: A learning algorithm for feedforward networks. IEEE Transactions on Neural Networks, 4(5):882–884, 1993.

DOI: 10.1109/72.248466

[5] S. H. J. van Vuuren. Neural network correlates with generalization. Master's thesis, University of Pretoria.

[6] J. E. Moody and T. S. R¨ognvaldsson. Smoothing regularizers for projective basis function networks. Advanceds in Neural Information Processing Systems, 9:585–591, 1997.

[7] P. Craven and G. Wahba. Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized crossvalidation. Numer. Math., 31:377–403, 1979.

DOI: 10.1007/bf01404567

[8] G. Golub, H. Heath, and G.Wahba. Generalized cross validation as a method for choosing a good ridge parameter. Technometrics, 21:215–224, 1979.

DOI: 10.1080/00401706.1979.10489751

[9] H. Akaike. Statistical predictor identification. Ann. Inst. Statist. Math., 22:203–217, 1970.

[10] J. E. Moody. The effective number of parameters: an analysis of generalization and regularization in nonlinear learning systems. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems, volume 4, pages 847–854. Morgan Kaufmann Publishers, San Mateo, CA, 1992.

[11] C. T. Leung and T.W. S. Chow. Adaptive regularization parameter selection method for enhancing generalization capability of neural networks. Artificial Intelligence, 107:347–356, 1999.

DOI: 10.1016/s0004-3702(98)00115-5