PFPN: Continuous Control of Physically Simulated Characters using Particle Filtering Policy Network

Pei Xu, Ioannis Karamouzas

Clemson University

In ACM SIGGRAPH Conference on Motion, Interaction and Games, 2021.
Also NeurIPS Deep Reinforcement Learning Workshop, 2021.

Abstract

Data-driven methods for physics-based character control using reinforcement learning have been successfully applied to generate high-quality motions. However, existing approaches typically rely on Gaussian distributions to represent the action policy, which can prematurely commit to suboptimal actions when solving high-dimensional continuous control problems for highly-articulated characters. In this paper, to improve the learning performance of physics-based character controllers, we propose a framework that considers a particle-based action policy as a substitute for Gaussian policies. We exploit particle filtering to dynamically explore and discretize the action space, and track the posterior policy represented as a mixture distribution. The resulting policy can replace the unimodal Gaussian policy which has been the staple for character control problems, without changing the underlying model architecture of the reinforcement learning algorithm used to perform policy optimization. We demonstrate the applicability of our approach on various motion capture imitation tasks. Baselines using our particle-based policies achieve better imitation performance and speed of convergence as compared to corresponding implementations using Gaussians, and are more robust to external perturbations during character control.

Video

Poster

Bibtex

@inproceedings{pfpn,
    author = {Xu, Pei and Karamouzas, Ioannis},
    title = {{PFPN}: Continuous Control of Physically Simulated Characters Using Particle Filtering Policy Network},
    year = {2021},
    isbn = {9781450391313},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    doi = {10.1145/3487983.3488301},
    booktitle = {Motion, Interaction and Games},
    articleno = {7},
    numpages = {12},
    location = {Virtual Event, Switzerland},
    series = {MIG '21}
}