Wu, Shuang; Shi, Ling; Wang, Jun; Tian, Guangjian; (2022) Understanding Policy Gradient Algorithms: A Sensitivity-Based Approach. In: Chaudhuri, K and Jegelka, S and Song, L and Szepesvari, C and Niu, G and Sabato, S, (eds.) Proceedings of the 39th International Conference on Machine Learning. (pp. pp. 24131-24149). Proceedings of Machine Learning Research (PMLR): Baltimore, MD, USA.

Preview

Text wu22i.pdf - Published Version Download (703kB) | Preview

Abstract

The REINFORCE algorithm from Williams is popular in policy gradient (PG) for solving reinforcement learning (RL) problems. Meanwhile, the theoretical form of PG is from Sutton et al. Although both formulae prescribe PG, their precise connections are not yet illustrated. Recently, Nota and Thomas (2020) have found that the ambiguity causes implementation errors. Motivated by the ambiguity and implementation incorrectness, we study PG from a perturbation perspective. In particular, we derive PG in a unified framework, precisely clarify the relation between PG implementation and theory, and echo back the findings by Nota and Thomas. Diving into factors contributing to empirical successes of the existing erroneous implementations, we find that small approximation error and the experience replay mechanism play critical roles.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as JSONCSVXML

Downloads by country - last 12 months

Export as XMLJSONCSV

Archive Staff Only

View Item