Convolutions are paramount to the success of neural network based learning for image, speech and video processing. This is due to their adequacy with the properties of such signals, which turns them into excellent feature extractors. For humanoids, whose state is usually represented as a vector of positions, angles and velocities (with hiearchical relations between them), we do not seem to know any good, standard feature extractor that could replace convolutions in a neural network based Reinforcement Learning (RL) algorithm. To improve RL in this context, it could also be important to choose carefully how we represent the inputs and outputs, and there exist a lot of model-based approaches for humanoid robot control from which inspiration could be drawn. Indeed, a lot of simplified dynamic models have been designed and used over the past decades, along with specific movement primitives, motion planners, low dimension state representations, basic task solvers and task composition methods, etc. These are tools that could potentially help RL algorithms to better exploit the particular structure and dynamics of humanoid robots, thus improving the efficiency of the learning. On the other hand, attention should be paid to the fact that for some problems, methods that try to exploit structure a lot have been outperformed by more "brute-force" learning techniques that make very few assumptions on the specificities of the problem. The questions that will be adressed (but not answered) during this presentation are the following:

- What tools could be used to specifically improve humanoid robot RL? 

- Do these tools already exist in the literature, or should they be invented? 

- Is it actually a good idea to try to develop specific RL techniques for humanoid robots (and more generally complex robots), or in the long run will generic methods be a better solution?

10:00 Emmanuel Rachelson -- A few topics in Reinforcement Learning (an related ideas).

There are two stories in this presentation and according to the audience's interests we shall invest more time in some parts. I’ll briefly narrate some twists and turns of my short career… and their consequences, in a scientific plot where the guiding thread is the field of Reinforcement Learning and the (current) happy ending is the creation of SuReLI (Supaero Reinforcement Learning Initiative). Then we shall delve into a specific chapter. We shall attempt to understand why, when I had to change my pasty shop, I kept preferring croissants to start off my day, why it is a Mathematical problem with implications in Reinforcement Learning and how it affected my research on learning strategies in stochastic one-player games.

slides (PDF)

11:00 Justin Carpentier -- Optimal Control and Learning in Robotics - Application to the control of Humanoid Robots