TensorFlow Probability: Learning with confidence

TensorFlow Probability (TFP) is a Python library built on TensorFlow that makes it easy to combine probabilistic models and deep learning on modern hardware (TPU, GPU). It's for data scientists, statisticians, and ML researchers/practitioners who want to encode domain knowledge to understand data and make predictions with uncertainty estimates. In this talk we focus on the "layers" module and demonstrate how TFP "distributions" fit naturally with Keras to enable estimating aleatoric and/or epistemic uncertainty.

Website: https://www.tensorflow.org/probability

Introduction Video: https://www.youtube.com/watch?v=BrwKURU-wpk
#tensorflow #machine_learning

Are we done with ImageNet?

Abstract: Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

https://arxiv.org/abs/2006.07159
#benchmark #image_net #computer_vision

Mathematics for Machine Learning

Summary: The fundamental mathematical tools needed to understand machine learning include linear algebra, analytic geometry, matrix decompositions, vector calculus, optimization, probability and statistics. These topics are traditionally taught in disparate courses, making it hard for data science or computer science students, or professionals, to efficiently learn the mathematics. This self-contained textbook bridges the gap between mathematical and machine learning texts, introducing the mathematical concepts with a minimum of prerequisites. It uses these concepts to derive four central machine learning methods: linear regression, principal component analysis, Gaussian mixture models and support vector machines. For students and others with a mathematical background, these derivations provide a starting point to machine learning texts. For those learning the mathematics for the first time, the methods help build intuition and practical experience with applying mathematical concepts. Every chapter includes worked examples and exercises to test understanding. Programming tutorials are offered on the book's web site.

https://mml-book.github.io/book/mml-book.pdf
#machine_learning #mathematics

SIREN: Implicit Neural Representations with Periodic Activation Functions

Abstract: Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

Paper: https://arxiv.org/abs/2006.09661

Website: https://vsitzmann.github.io/siren/

Explanatory Video: https://youtu.be/Q5g3p9Zwjrk
#deep_learning #neural_network

Grounding Language in Play: A scalable approach for controlling robots with natural language

https://language-play.github.io/
#nlp #reinforcement_learning #deep_learning

Synthesis and Stabilization of Complex Behaviors through Online Trajectory Optimization

Abstract: We present an online trajectory optimization method and software platform applicable to complex humanoid robots performing challenging tasks such as getting up from an arbitrary pose on the ground and recovering from large disturbances using dexterous acrobatic maneuvers. The resulting behaviors, illustrated in the attached video, are computed only 7x slower than real time, on a standard PC. The video also shows results on the acrobot problem, planar swimming and one-legged hopping. These simpler problems can already be solved in real time, without pre-computing anything

Video of their experiments: https://youtu.be/anIsw2-Lbco

Paper: https://homes.cs.washington.edu/~todorov/papers/TassaIROS12.pdf
#model_predictive_control #optimal_control #robotics

PyTorch Internals

Summary: This article is for those of you who have used PyTorch, and thought to yourself, "It would be great if I could contribute to PyTorch," but were scared by PyTorch's behemoth of a C++ codebase. I'm not going to lie: the PyTorch codebase can be a bit overwhelming at times. The purpose of this talk is to put a map in your hands: to tell you about the basic conceptual structure of a "tensor library that supports automatic differentiation", and give you some tools and tricks for finding your way around the codebase. I'm going to assume that you've written some PyTorch before, but haven't necessarily delved deeper into how a machine learning library is written.

https://blog.ezyang.com/2019/05/pytorch-internals/
#pytorch #deep_learning

An operator view of policy gradient methods

Abstract: We cast policy gradient methods as the repeated application of two operators: a policy improvement operator I, which maps any policy π to a better one Iπ, and a projection operator P, which finds the best approximation of Iπ in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as Reinforce and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of I and P to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how Reinforce and the Bellman optimality operator, for example, can be seen as two sides of the same coin.

https://arxiv.org/pdf/2006.11266.pdf
#reinforcement_learning #policy_iteration #value_iteration

Neural Architecture Search without Training

Abstract: The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be extremely slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be remedied if we could infer a network's trained accuracy from its initial state. In this work, we examine how the linear maps induced by data points correlate for untrained network architectures in the NAS-Bench-201 search space, and motivate how this can be used to give a measure of modelling flexibility which is highly indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU.

Explanatory Video: https://www.youtube.com/watch?v=a6v92P0EbJc

GitHub Repo: https://github.com/BayesWatch/nas-without-training

Paper: https://arxiv.org/abs/2006.04647
#deep_learning #neural_architecture_search

An Introduction to Deep Reinforcement Learning

Abstract: Deep reinforcement learning is the combination of reinforcement learning (RL) and deep learning. This field of research has been able to solve a wide range of complex decisionmaking tasks that were previously out of reach for a machine. Thus, deep RL opens up many new applications in domains such as healthcare, robotics, smart grids, finance, and many more. This manuscript provides an introduction to deep reinforcement learning models, algorithms and techniques. Particular focus is on the aspects related to generalization and how deep RL can be used for practical applications. We assume the reader is familiar with basic machine learning concepts.

Paper: https://arxiv.org/pdf/1811.12560.pdf
#reinforcement_learning
#deep_learning

Yann LeCuN advice for an undergraduate student who aspires to become a Machine Learning Scientist in the field of Deep Learning

(0) take all the continuous math and physics class you can possibly take. If you have the choice between “iOS programming” and “quantum mechanics”, take “quantum mechanics”. In any case, take Calc I, Calc II, Calc III, Linear Algebra, Probability and Statistics, and as many physics courses as you can. But make sure you learn to program.
(1) Take an AI-related problem you are passionate about.
(2) think about it on your own
(3) once you have formed your own idea of it, start reading the literature on the problem
(4) you will find that (a) your ideas were probably a bit naive but (b) your view of the problem is slightly different from what was done before.
(5) Find a professor in your school that can help you make your ideas concrete. It might be difficult. Professors are busy and don’t have much time for undergrads. The ones with the most free time are the very junior, the very senior, and the ones who are not very active in research.
(6) If you don’ find a professor with spare time, hook up with a postdoc or PhD student in his/her lab.
(7) ask the professor if you can attend his/her lab meetings and seminars or sit in his/her class.
(8) Before you graduate, try to write a paper about your research or release a piece of open source code.
(9) Now apply to PhD programs. Forget about the “ranking” of the school for now. Find a reputable professor who works on topics that you are interested in. Pick a person whose papers you like or admire.
(10) Apply to several PhD programs in the schools of the above-mentioned professors and mention in your letter that you’d like to work with that professor but would be open to work with others.
(11) ask your undergrad professor to write a recommendation letter for you. It’s maximally efficient if your undergrad professor is known by your favorite PhD advisor.
(12) if you don’t get accepted in one of your favorite PhD programs, get a job at Facebook or Google and try to get a gig as an engineer assisting research scientists at FAIR or Google Brain.
(13) publish a papers with the research scientists in question. Then re-apply to PhD programs and ask the FAIR or Google scientists you work with to write a recommendation letter for you.

https://www.quora.com/What%E2%80%99s-your-advice-for-undergraduate-student-who-aspires-to-be-a-research-scientist-in-deep-learning-or-related-field-one-day
#machine_learning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020)

Abstract: In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.

Paper: https://arxiv.org/abs/2005.01643
#reinforcement_learning #offline_reinforcement_learning

Backward Feature Correction: How Deep Learning Performs Deep Learning

Summary: How does a 110-layer ResNet learn a high-complexity classifier using relatively few training examples and short training time? We present a theory towards explaining this in terms of hierarchical learning. We refer hierarchical learning as the learner learns to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning efficiently and automatically by applying SGD. On the conceptual side, we present, to the best of our knowledge, the FIRST theory result indicating how deep neural networks can be sample and time efficient on certain hierarchical learning tasks, when NO KNOWN non-hierarchical algorithms (such as kernel method, linear regression over feature mappings, tensor decomposition, sparse coding, and their simple combinations) are efficient. We establish a principle called "backward feature correction", where training higher layers in the network can improve the features of lower level ones. We believe this is the key to understand the deep learning process in multi-layer neural networks.

Paper: https://arxiv.org/pdf/2001.04413.pdf
#theory #deep_learning

Is a good representation sufficient for sample efficient reinforcement learning?

Abstract: Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for sample efficient reinforcement learning? This question has largely been studied only with respect to (worst-case) approximation error, in the more classical approximate dynamic programming literature. With regards to the statistical viewpoint, this question is largely unexplored, and the extant body of literature mainly focuses on conditions which permit sample efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning.
This work shows that, from the statistical viewpoint, the situation is far subtler than suggested by the more traditional approximation viewpoint, where the requirements on the representation that suffice for sample efficient RL are even more stringent. Our main results provide sharp thresholds for reinforcement learning methods, showing that there are hard limitations on what constitutes good function approximation (in terms of the dimensionality of the representation), where we focus on natural representational conditions relevant to value-based, model-based, and policy-based learning. These lower bounds highlight that having a good (value-based, model-based, or policy-based) representation in and of itself is insufficient for efficient reinforcement learning, unless the quality of this approximation passes certain hard thresholds. Furthermore, our lower bounds also imply exponential separations on the sample complexity between 1) value-based learning with perfect representation and value-based learning with a good-but-not-perfect representation, 2) value-based learning and policy-based learning, 3) policy-based learning and supervised learning and 4) reinforcement learning and imitation learning.

Paper: https://arxiv.org/pdf/1910.03016.pdf
#reinforcement_learning #representation_learning

Meta Reinforcement Learning: An Introduction

Intro: a good meta-learning model is expected to generalize to new tasks or new environments that have never been encountered during training. The adaptation process, essentially a mini learning session, happens at test with limited exposure to the new configurations. Even without any explicit fine-tuning (no gradient backpropagation on trainable variables), the meta-learning model autonomously adjusts internal hidden states to learn. Training RL algorithms can be notoriously difficult sometimes. If the meta-learning agent could become so smart that the distribution of solvable unseen tasks grows extremely broad, we are on track towards general purpose methods — essentially building a “brain” which would solve all kinds of RL problems without much human interference or manual feature engineering. Sounds amazing, right?


Blog: https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html
#reinforcement_learning #meta_learning #research_paper

What is an agent?

Intro: A thought-provoking essay which sheds new light on the agent-environment boundary and philosophy behind the current definition of agent, especially in the field of reinforcement learning.

https://anna.harutyunyan.net/wp-content/uploads/2020/09/What_is_an_agent.pdf
#reinforcement_learning #philosophy

Machine Learning & Computational Statistics Course

Course Intro: This course covers a wide variety of topics in machine learning and statistical modeling. While mathematical methods and theoretical aspects will be covered, the primary goal is to provide students with the tools and principles needed to solve the data science problems found in practice.

https://davidrosenberg.github.io/ml2016/#home
#machine_learning #statistics #course

New Deep Learning Course by Yann LeCun & Alfredo Canziani (Recommended)

Course Intro: This course concerns the latest techniques in deep learning and representation learning, focusing on supervised and unsupervised deep learning, embedding methods, metric learning, convolutional and recurrent nets, with applications to computer vision, natural language understanding, and speech recognition.

Additional Info: This course is available in 11 languages such as Persian, and I personally translated some of the materials of this course to Persian :).

https://atcold.github.io/pytorch-Deep-Learning/
#deep_learning #course

From CAPTCHA to Commonsense: How Brain Can Teach Us About Artificial Intelligence

Abstract: Despite the recent progress in AI-powered by deep learning in solving narrow tasks, we are not close to human intelligence in its flexibility, versatility, and efficiency. Efficient learning and effective generalization come from inductive biases, and building Artificial General Intelligence (AGI) is an exercise in finding the right set of inductive biases that make fast learning possible while being general enough to be widely applicable in tasks that humans excel at. To make progress in AGI, we argue that we can look at the human brain for such inductive biases and principles of generalization. To that effect, we propose a strategy to gain insights from the brain by simultaneously looking at the world it acts upon and the computational framework to support efficient learning and generalization. We present a neuroscience-inspired generative model of vision as a case study for such an approach and discuss some open problems about the path to AGI.

URL: https://www.frontiersin.org/articles/10.3389/fncom.2020.554097/full
#neuroscience #artificial_general_intelligence