Iranian Journal of Mechanical Engineering Transactions of ISME

Iranian Journal of Mechanical Engineering Transactions of ISME

A New Energy Management Method Based on Reinforcement Q-learning for Minimizing the Operational Cost of an Energy System

Authors
Sepehr Sanaye
Ahmadreza Sarrafi
Iran University of Science and Technology-Faculty of mechanical engineering.
10.30506/ijmep.2022.129773.1706
Abstract
This paper presents a new method of energy management (EM). The proposed method was defined as a reinforcement learning problem for a system consisting of solar panels, wind turbines, gas engines and batteries. The best corresponding energy managements for all the possible system states were found by employing the Q-learning algorithm to solve the EM problem. These results were then used for managing the power supply of four residential buildings in Negin Island Bushehr located in the southwest of Iran. The simulation was performed for 8760 hours of a year. The proposed energy management method was compared with the load following energy management, which resulted in a 2.4% reduction in annual operational costs and a 3.7% reduction in CO2 emissions. With the same quality of the results, the new EM method took about 2.5 times less computational time compared to the optimum energy management performed by genetic algorithm. The effect of iteration number on the convergence of algorithm was also investigated.
Keywords
Energy management
Reinforcement learning
Hybrid energy system
Renewable energy
Gas engine

Subjects

[1] Gupta, S.K., and Purohit, P., “Renewable Energy Certificate Mechanism in India: A Preliminary Assessment”, Renewable and Sustainable Energy Reviews, Vol. 22, pp. 380-392, (2013).
 
[2] Tian, Z., and Seifi, A.A., “Reliability Analysis of Hybrid Energy System”, International Journal of Reliability, Quality and Safety Engineering, Vol. 21(03), Article Number. 1450011, (2014).
 
[3] Olatomiwa, L., Mekhilef, S., Ismail, M.S., and Moghavvemi, M., “Energy Management Strategies in Hybrid Renewable Energy Systems: A Review”, Renewable and Sustainable Energy Reviews, Vol. 62, pp. 821-835, (2016).
 
[4] Azuatalam, D., Paridari, K., Ma, Y., Förstl, M., Chapman, A.C., and Verbič, G., “Energy Management of Small-scale PV-battery Systems: A Systematic Review Considering Practical Implementation, Computational Requirements, Quality of Input Data and Battery Degradation”, Renewable and Sustainable Energy Reviews, Vol. 112, pp. 555-570, (2019).
 
[5] Kadri, A., Marzougui, H., Aouiti, A., and Bacha, F., “Energy Management and Control Strategy for a DFIG Wind Turbine/Fuel Cell Hybrid System with Super Capacitor Storage System”, Energy, Vol. 192, Article Number. 116518, (2020).
 
[6] Sun, C., Sun, F., and Moura, S.J., “Data Enabled Predictive Energy Mّmanagement of a PV Battery Smart Home Nanogrid”, American Control Conference (ACC), IEEE, Chicago, USA, pp. 1023-1028. (2015).
 
[7] Dursun, E., and Kilic, O., “Comparative Evaluation ofّ Different Power Management Strategies of a Stand-alone PV/Wind/PEMFC Hybrid Power System”, International Journal of Electrical Power and Energy Systems, Vol. 34(1), pp. 81-89, (2012).
 
[8] Castañeda, M., Cano, A., Jurado, F., Sánchez, H., and Fernández, L.M., “Sizing Optimization, Dynamic Modeling and Energy Management Strategies of a Stand-alone PV/Hydrogen/Battery-based Hybrid System”, International Journal of Hydrogen Energy, Vol. 38(10), pp. 3830-3845, (2013).
 
[9] Askarzadeh, A., “A Memory-based Genetic Algorithm for Optimization of Power Generation in a Microgrid”, IEEE Transactions on Sustainable Energy, Vol. 9(3), pp.1081-1089, (2018).
[10] Qi, W., Liu, J., Chen, X., and Christofides, P.D., “Supervisory Predictive Control of Standalone Wind/Solar Energy Generation Systems”, Control Systems Technology, IEEE Transactions on Control Systems Technology, Vol. 19(1), pp. 199-207, (2011).

 

[11] Yahyaoui, I., Yahyaoui, A., Chaabene, M., and Tadeo, F., “Energy Management for a Stand-alone Photovoltaic-wind System Suitable for Rural Electrification”, Sustainable Cities and Society, Vol. 25, pp. 90-101, (2016).
 
[12] Kim, S., and Lim, H., “Reinforcement Learning Based Energy Management Algorithm for Smart Energy Buildings”, Energies, doi.org/10.3390/en11082010, Vol. 11(8), (2018).
 
[13] Leo, R., Milton, R.S., and Sibi, S., “Reinforcement Learning for Optimal Energy Management of a Solar Microgrid”, Global Humanitarian Technology Conference- South Asia Satellite (GHTC-SAS), IEEE, Trivandrum, India, pp. 183-188, (2014).
 
[14] Dalamagkidis, K., Kolokotsa, D., Kalaitzakis, K., and Stavrakakis, G.S., “Reinforcement Learning for Energy Conservation and Comfort in Buildings”, Building and Environment, Vol. 42(7), pp. 2686-2698, (2007).
 
[15] Saloux, E., Teyssedou, A., and Sorin, M., “Explicit Model of Photovoltaic Panels to Determine Voltages and Currents at the Maximum Power Point”, Solar Energy, Vol. 85(5), pp. 713-722, (2011).
 
[16] Luque, A., and Hegedus, S., (Eds.), “Handbook of Photovoltaic Science and Engineering”, 2nd Edition, John Wiley and Sons, West Sussex, England, United Kingdom, (2011).       
 
[17] Fesharaki, V.J., Dehghani, M., Fesharaki, J.J., and Tavasoli, H., “The Effect of Temperature on Photovoltaic Cell Efficiency”, In Proceedings of the 1st International Conference on Emerging Trends in Energy Conservation–ETEC, Tehran, Iran, pp. 20-21, (2011).
 
[18] Manwell, J.F., McGowan, J.G., and Rogers, A.L., “Wind Energy Explained: Theory, Design and Application”, 2nd Edition, John Wiley and Sons, West Sussex, England, United Kingdom (2010).       
   
[19] Chang, T.P., Liu, F.J., Ko, H.H., Cheng, S.P., Sun, L.C., and Kuo, S.C., “Comparative Analysis on Power Curve Models of Wind Turbine Generator in Estimating Capacity Factor”, Energy, Vol. 73, pp. 88-95, (2014).
 
[20] Approximate Natural Gas Generator Fuel Consumption Chart | Diesel Service. (2017), Dieselserviceandsupply.com, Retrieved 12 January 2017, from http://www.dieselserviceandsupply.com/Natural_Gas_Fuel_Consumption.aspx.     
 
[21] Barley, C.D., and Winn, C.B., “Optimal Dispatch Strategy in Remote Hybrid Power Systems”, Solar Energy, Vol. 58(4-6), pp. 165-179, (1996).
 
[22] Aghaei, J., and Alizadeh, M. I., “Multi-objective Self-scheduling of CHP (Combined Heat and Power)-based Microgrids Considering Demand Response Programs and ESSs (Energy Storage Systems)”, Energy, Vol. 55, pp. 1044-1054, (2013).
[23] Sutton, R.S., and Barto, A.G., “Introduction to Reinforcement Learning”, Cambridge: MIT Press, Massachusetts, London, England, Vol. 135, (1998).       
 
[24] Meteonorm, (2022), Meteonorm.com. Retrieved 23 December 2022, from http://meteonorm.com/.
 
[25] Ministry of Energy, Renewable Energy and Energy Efficiency Organization (SATBA), (2022), Satba.gov.ir, Retrieved 23 December 2022, from https://www.satba.gov.ir/en/news/1098/Renewable-Energy-Reports2017.
 
[26] Multicrystalline Solar Panels | Yingli Solar, (2016), Yinglisolar.co, Retrieved 20 September 2016, from http://www.yinglisolar.com/.
 
[27] Jiangsu NaiEr Wind Power Technology Development Co., Ltd. (2016), Wxnaier.com, Retrieved 22 August 2016, from http://www.wxnaier.com.     
 
[28] EC POWER. (2017), Ecpower.eu., Retrieved 22 December 2017, from https://www.ecpower.eu/en/.
 
[29] KOHLER Generators, (2017), Retrieved 22 December 2017, from https://kohlerpower.com/en/residential/generators.
 
[30] User, S. Amitisenergy.com, Retrieved 20 september 2016, from http://amitisenergy.com/.
 
[31] Hendron, R., Anderson, R., Christensen, C., Eastment, M., and Reeves, P., “Development of an Energy Savings Benchmark for All Residential End-uses”, Proceedings of SIMBUILD, August 4-6, Boulder, Colorado, (2004).     
 
[32] Household Consumption Management Software, (2016), Pishruncontrol.com, Retrieved 19 December 2016, from http://www.pishruncontrol.com/post/9.
 
[33] Modern Afrough Company, (2017), Modernafrough.pcn.ir, Retrieved 11 July 2017, from http://www.modernafrough.pcn.ir. 
 
[34] TOTALGAZ, (2016), Total.com.b., Retrieved 10 December 2016, from http://www.total.com.bd/lpg-for-consumers/about-lpg.
 
[35] National Iranian Gas Company, (2016), Nigc.ir, Retrieved 27 June 2016, from http://www.nigc.ir/Portal/Home/.
 
[36] Zambolin, E., and Del Col, D., “An Improved Procedure for the Experimental Characterization of Optical Efficiency in Evacuated Tube Solar Collectors”, Renewable Energy, Vol. 43, pp. 37-46, (2012).  
 
[37] Jiangsu NaiEr Wind Power Technology Development Co., Ltd., (2016), Wxnaier.com, Retrieved 22 August 2016, from http://www.wxnaier.com.
  
[38] HOPPECKE - Power from Innovation: HOPPECKE, (2018), Hoppecke.com, Retrieved 3 August 2018, from http://www.hoppecke.com.
[39] HOMER - Hybrid Renewable and Distributed Generation System Design Software, (2019), Retrieved 6 March 2019, from https://www.homerenergy.com.

  • Receive Date 24 June 2020
  • Revise Date 21 October 2021
  • Accept Date 26 October 2021