Is a good representation sufficient for sample efficient reinforcement learning?
Abstract: Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied only with respect to (worst-case) approximation error, in the more classical approximate dynamic programming literature. With regards to the statistical viewpoint, this question is largely unexplored, and the extant body of literature mainly focuses on conditions which permit sample efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning.
This work shows that, from the statistical viewpoint, the situation is far subtler than suggested by the more traditional approximation viewpoint, where the requirements on the representation that suffice for sample efficient RL are even more stringent. Our main results provide sharp thresholds for reinforcement learning methods, showing that there are hard limitations on what constitutes good function approximation (in terms of the dimensionality of the representation), where we focus on natural representational conditions relevant to value-based, model-based, and policy-based learning. These lower bounds highlight that having a good (value-based, model-based, or policy-based) representation in and of itself is insufficient for efficient reinforcement learning, unless the quality of this approximation passes certain hard thresholds. Furthermore, our lower bounds also imply exponential separations on the sample complexity between 1) value-based learning with perfect representation and value-based learning with a good-but-not-perfect representation, 2) value-based learning and policy-based learning, 3) policy-based learning and supervised learning and 4) reinforcement learning and imitation learning.
Paper: https://arxiv.org/pdf/1910.03016.pdf
#reinforcement_learning #representation_learning
Abstract: Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied only with respect to (worst-case) approximation error, in the more classical approximate dynamic programming literature. With regards to the statistical viewpoint, this question is largely unexplored, and the extant body of literature mainly focuses on conditions which permit sample efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning.
This work shows that, from the statistical viewpoint, the situation is far subtler than suggested by the more traditional approximation viewpoint, where the requirements on the representation that suffice for sample efficient RL are even more stringent. Our main results provide sharp thresholds for reinforcement learning methods, showing that there are hard limitations on what constitutes good function approximation (in terms of the dimensionality of the representation), where we focus on natural representational conditions relevant to value-based, model-based, and policy-based learning. These lower bounds highlight that having a good (value-based, model-based, or policy-based) representation in and of itself is insufficient for efficient reinforcement learning, unless the quality of this approximation passes certain hard thresholds. Furthermore, our lower bounds also imply exponential separations on the sample complexity between 1) value-based learning with perfect representation and value-based learning with a good-but-not-perfect representation, 2) value-based learning and policy-based learning, 3) policy-based learning and supervised learning and 4) reinforcement learning and imitation learning.
Paper: https://arxiv.org/pdf/1910.03016.pdf
#reinforcement_learning #representation_learning
Meta Reinforcement Learning: An Introduction
Intro: a good meta-learning model is expected to generalize to new tasks or new environments that have never been encountered during training. The adaptation process, essentially a mini learning session, happens at test with limited exposure to the new configurations. Even without any explicit fine-tuning (no gradient backpropagation on trainable variables), the meta-learning model autonomously adjusts internal hidden states to learn. Training RL algorithms can be notoriously difficult sometimes. If the meta-learning agent could become so smart that the distribution of solvable unseen tasks grows extremely broad, we are on track towards general purpose methods — essentially building a “brain” which would solve all kinds of RL problems without much human interference or manual feature engineering. Sounds amazing, right?
Blog: https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html
#reinforcement_learning #meta_learning #research_paper
Intro: a good meta-learning model is expected to generalize to new tasks or new environments that have never been encountered during training. The adaptation process, essentially a mini learning session, happens at test with limited exposure to the new configurations. Even without any explicit fine-tuning (no gradient backpropagation on trainable variables), the meta-learning model autonomously adjusts internal hidden states to learn. Training RL algorithms can be notoriously difficult sometimes. If the meta-learning agent could become so smart that the distribution of solvable unseen tasks grows extremely broad, we are on track towards general purpose methods — essentially building a “brain” which would solve all kinds of RL problems without much human interference or manual feature engineering. Sounds amazing, right?
Blog: https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html
#reinforcement_learning #meta_learning #research_paper
Lil'Log
Meta Reinforcement Learning
Meta-RL is meta-learning on reinforcement learning tasks. After trained over a distribution of tasks, the agent is able to solve a new task by developing a new RL algorithm with its internal activity dynamics. This post starts with the origin of meta-RL and…
What is an agent?
Intro: A thought-provoking essay which sheds new light on the agent-environment boundary and philosophy behind the current definition of agent, especially in the field of reinforcement learning.
https://anna.harutyunyan.net/wp-content/uploads/2020/09/What_is_an_agent.pdf
#reinforcement_learning #philosophy
Intro: A thought-provoking essay which sheds new light on the agent-environment boundary and philosophy behind the current definition of agent, especially in the field of reinforcement learning.
https://anna.harutyunyan.net/wp-content/uploads/2020/09/What_is_an_agent.pdf
#reinforcement_learning #philosophy
Machine Learning & Computational Statistics Course
Course Intro: This course covers a wide variety of topics in machine learning and statistical modeling. While mathematical methods and theoretical aspects will be covered, the primary goal is to provide students with the tools and principles needed to solve the data science problems found in practice.
https://davidrosenberg.github.io/ml2016/#home
#machine_learning #statistics #course
Course Intro: This course covers a wide variety of topics in machine learning and statistical modeling. While mathematical methods and theoretical aspects will be covered, the primary goal is to provide students with the tools and principles needed to solve the data science problems found in practice.
https://davidrosenberg.github.io/ml2016/#home
#machine_learning #statistics #course
New Deep Learning Course by Yann LeCun & Alfredo Canziani (Recommended)
Course Intro: This course concerns the latest techniques in deep learning and representation learning, focusing on supervised and unsupervised deep learning, embedding methods, metric learning, convolutional and recurrent nets, with applications to computer vision, natural language understanding, and speech recognition.
Additional Info: This course is available in 11 languages such as Persian, and I personally translated some of the materials of this course to Persian :).
https://atcold.github.io/pytorch-Deep-Learning/
#deep_learning #course
Course Intro: This course concerns the latest techniques in deep learning and representation learning, focusing on supervised and unsupervised deep learning, embedding methods, metric learning, convolutional and recurrent nets, with applications to computer vision, natural language understanding, and speech recognition.
Additional Info: This course is available in 11 languages such as Persian, and I personally translated some of the materials of this course to Persian :).
https://atcold.github.io/pytorch-Deep-Learning/
#deep_learning #course
From CAPTCHA to Commonsense: How Brain Can Teach Us About Artificial Intelligence
Abstract: Despite the recent progress in AI-powered by deep learning in solving narrow tasks, we are not close to human intelligence in its flexibility, versatility, and efficiency. Efficient learning and effective generalization come from inductive biases, and building Artificial General Intelligence (AGI) is an exercise in finding the right set of inductive biases that make fast learning possible while being general enough to be widely applicable in tasks that humans excel at. To make progress in AGI, we argue that we can look at the human brain for such inductive biases and principles of generalization. To that effect, we propose a strategy to gain insights from the brain by simultaneously looking at the world it acts upon and the computational framework to support efficient learning and generalization. We present a neuroscience-inspired generative model of vision as a case study for such an approach and discuss some open problems about the path to AGI.
URL: https://www.frontiersin.org/articles/10.3389/fncom.2020.554097/full
#neuroscience #artificial_general_intelligence
Abstract: Despite the recent progress in AI-powered by deep learning in solving narrow tasks, we are not close to human intelligence in its flexibility, versatility, and efficiency. Efficient learning and effective generalization come from inductive biases, and building Artificial General Intelligence (AGI) is an exercise in finding the right set of inductive biases that make fast learning possible while being general enough to be widely applicable in tasks that humans excel at. To make progress in AGI, we argue that we can look at the human brain for such inductive biases and principles of generalization. To that effect, we propose a strategy to gain insights from the brain by simultaneously looking at the world it acts upon and the computational framework to support efficient learning and generalization. We present a neuroscience-inspired generative model of vision as a case study for such an approach and discuss some open problems about the path to AGI.
URL: https://www.frontiersin.org/articles/10.3389/fncom.2020.554097/full
#neuroscience #artificial_general_intelligence
Frontiers
Frontiers | From CAPTCHA to Commonsense: How Brain Can Teach Us About Artificial Intelligence
Despite the recent progress in AI powered by deep learning in solving narrow tasks, we are not close to human intelligence in its flexibility, versatility, a...
The owner of this channel has been inactive for the last 17 months. If they remain inactive for the next 27 days, they may lose their account and admin rights in this channel. The contents of the channel will remain accessible for all users.