An Artificial Intelligence Approach for Online Optimization of Flexible Manufacturing Systems


Article Preview

This paper addresses the problem of efficiently operating a flexible manufacturing machine in an electricity micro-grid featuring a high volatility of electricity prices. The problem of finding the optimal control policy is formulated as a sequential decision making problem under uncertainty where, at every time step the uncertainty comes from the lack of knowledge about fu-ture electricity consumption and future weather dependent energy prices. We propose to address this problem using deep reinforcement learning. To this purpose, we designed a deep learning architecture to forecast the load profile of future manufacturing schedule from past production time series. Combined with the forecast of future energy prices, the reinforcement-learning algorithm is trained to perform an online optimization of the production ma-chine in order to reduce the long-term energy costs. The concept is empirical-ly validated on a flexible production machine, where the machine speed can be optimized during the production.



Edited by:

Jörg Franke, Michael Scholz and Annika Höft




J. Bakakeu et al., "An Artificial Intelligence Approach for Online Optimization of Flexible Manufacturing Systems", Applied Mechanics and Materials, Vol. 882, pp. 96-108, 2018

Online since:

July 2018




* - Corresponding Author

[1] Faktenpapier atypische Netznutzung. DIHK und VEA - Bundesverband der Energie-Abnehmer e. V. Hannover, April (2015).

[2] Roesch M., Brugger M., Braunreuther S., Reinhart G.: Klassifizierung von Energieflexibilitätsmaßnahmen. ZWF. (2017).


[3] R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning,, 1st ed. Cambridge, MA, USA: MIT Press, (1998).

[4] Gholian, H. Mohsenian-Rad and Y. Hua, Optimal Industrial Load Control in Smart Grid,,IEEE Trans. Smart Grid, vol. 7, no. 5, pp.2305-2316, (2016).


[5] R. Chen, H. Sun, Q. Guo, H. Jin, W. Wu and B. Zhang, Profit-seeking energy–intensive enterprises participating in power system scheduling: Model and mechanism,,Appl. Energy , vol. 158, pp.263-274, (2015).


[6] Z. Wang, F. Gao, Q. Zhai, X. Guan, K. Liu and D. Zhou, An integrated optimization model for generation and batch production load scheduling in energy intensive enterprise,, in Proc. IEEE Power Eng. Soc. Gen. Meet., 2012, pp.1-8.


[7] A. Middelberg , J. F. Zhang and X. H. Xia, An optimal control model for load shifting – with application in the energy management of a colliery,,Appl. Energy, vol. 86, pp.1266-1273, (2009).


[8] J. Wang et al, Optimal vehicle batching and sequencing to reduce energy consumption and atmospheric emissions in automotive paint shops,, Int. J. Sustainable Manuf., vol. 2, (2011).


[9] F. Shrouf et al, Optimizing the production scheduling of a single machine to minimize total energy consumption costs,, J. Cleaner Prod., vol. 67, p.197–207, (2014).


[10] G. Chen, L. Zhang, J. Arinez, and S. Biller, Energy-efficient production systems through schedule-based operations,, IEEE Trans. Autom. Sci. Eng., vol. 10, no. 1, p.27–37, Jan. (2013).


[11] Z. Sun and L. Li, Opportunity estimation for real-time energy control of sustainable manufacturing systems,, IEEE Trans. Autom. Sci. Eng., vol. 10, no. 1, p.38–44, Jan. (2013).


[12] L. Li and Z. Sun, Dynamic energy control for energy efficiency improvement of sustainable manufacturing systems using Markov decision process,, IEEE Trans. Syst., Man, Cybern. Syst, vol.43, no.5, p.1195–1205, Sep (2013).


[13] J. R. Duflou et al, Towards energy and resource efficient manufacturing: A processes and systems approach,, CIRP Annals—Manuf. Technol., vol. 61, no. 2, p.587–609, (2012).

[14] Pechmann and I. Schler, Optimizing energy costs by intelligent production scheduling,, in Glocalized Solutions for Sustainability Manufacturing. New York, NY, USA: Springer, (2011).


[15] Bruzzone et al, Energy aware scheduling for improving manufacturing process sustainability: A mathematical model for flexible flow shops,, CIRP Annals—Manuf. Technol. , vol. 61, (2012).


[16] K. Fang, et al, A new approach to scheduling in manufacturing for power consumption and carbon footprint reduction,, J. Manuf. Syst., (2011).

[17] S. Ashok, Peak-load management in steel plants,, Appl. Energy, vol. 83, p.413–424, (2006).

[18] Middelberg, J. Zhang, and X. Xia, An optimal control model for load shifting with application in the energy management of a colliery,, Appl. Energy , vol. 86, p.1266–1273, (2009).


[19] Z. Sun, L. Li, M. Fernandez, and J. Wang, Inventory control for peak electricity demand reduction of manufacturing systems considering the tradeoff between production loss and energy savings,, J. Cleaner Prod., vol. 82, p.84–93, (2014).


[20] M. R. Alam, M. St-Hilaire, and T. Kunz, Computational methods for residential energy cost optimization in smart grids: A survey,, ACM Comput. Surv., vol. 49, p.2:1–2:34, Apr. (2016).


[21] Barbato and A. Capone, Optimization models and methods for demand-side management of residential users: A survey,, Energies, vol. 7, no. 9, p.5787 – 5824, (2014).


[22] E. Loukarakis, C. J. Dent, and J. W. Bialek, Decentralized multi-period economic dispatch for real-time flexible demand management,, IEEE Transactions on Power Systems, vol. 31, (2016).


[23] H. Mohsenian-Rad and A. Leon-Garcia, Optimal residential load control with price prediction in real-time electricity pricing environments,, IEEE Transactions on Smart Grid, (2010).


[24] N. G. Paterakis et al, Coordinated operation of a neighborhood of smart households comprising electric vehicles, energy storage and distributed generation,, IEEE Transactions on Smart Grid, vol. 7, no. 6, p.2736–2747, Nov (2016).


[25] J. Vardakas et al, A survey on deman response programs in smart grids: Pricing methods and optimization algorithms,, IEEE Comunication Surveys Tutorials, vol. 17, no. 1, p.152–178, (2015).


[26] L. A. Hurtado et al, Comfort-constrained demand flexibility management for building aggregations using a decentralized approach,, in International Conference on Smart Cities and Green ICT Systems, May 2015, p.1–10.


[27] Liu, X. Xu and D. Hu, Multiobjective reinforcement learning: A comprehensive overview,, IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 45, no. 3, pp.385-398, (2015).

[28] R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning,, Third ed. Cambridge, MA, USA: MIT Press, (2017).

[29] V. Mnih et al, Human-level control through deep reinforcement learning,, Nature, (2015).

[30] S. Gu, E. Holly, E. Timothy, P. Lillicrap and S. Levine, Mastering the game of Go without human knowledge,, Springer Nature, (2016).

[31] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, Control of a Quadrotor with Reinforcement Learning,, IEEE Robotics And Automation Letters, (2017).

[32] W. Xia, H. Li and B. Li, A Control Strategy of Autonomous Vehicles Based on Deep Reinforcement Learning,, 9th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, 2016, pp.198-201.


[33] N.N, Measurement instruction to determine the energy- and resource demand ofmachine tools for mass production,, VDMA 34179, Beuth Verlag GmbH, (2015).

[34] Arens, T. Hopfgartner, T. Jensen, M. Lamping, M. Pieper, Packaging Machine Language V3.0 Mode & States Definition Document,,, (2006).