Reversible RNNs

Paper about how to reduce memory costs of GRU and LSTM networks by 10-15x without loss in performance. Also 5-10x for attention-based architectures. New paper with Matt MacKay, Paul Vicol, and Jimmy Ba, to appear at NIPS.

Link: https://arxiv.org/abs/1810.10999

#dl #RNN #NIPS2018

Faster R-CNN and Mask R-CNN in #PyTorch 1.0

Another release from #Facebook.

Mask R-CNN Benchmark: a fast and modular implementation for Faster R-CNN and Mask R-CNN written entirely in @PyTorch 1.0. It brings up to 30% speedup compared to mmdetection during training.

Webcam demo and ipynb file are available.

Github: https://github.com/facebookresearch/maskrcnn-benchmark

#CNN #CV #segmentation #detection

Mask R-CNN Benchmark Demo

A set of best where-to-start-with-python resources.

This is the collection of beginners resources from a tweet by fast.ai cofounder, covering any resource which helped people to learn python from scratch.

https://telegra.ph/A-collection-of-where-to-start-python-resources-11-05

#beginner #novice #CS #python #tutorial

Making floating point math highly efficient for AI hardware.

Fresh FB article on accelerating ML.

https://code.fb.com/ai-research/floating-point-math/

#facebook

Analyzing Experiment Outcomes: Beyond Average Treatment Effects

Cool piece from Uber's engineering department about why you can't just use the average customer experience to see if product changes are worth it. You have to consider the DISTRIBUTIONAL changes of the customer experience.

Link: https://eng.uber.com/analyzing-experiment-outcomes/

#statistics #uber #abtest

​​TL-GAN: transparent latent-space GAN

GANs to generate photo-realistic faces with some control over characteristics.


Demo: https://www.kaggle.com/summitkwan/tl-gan-demo
Medium Post: https://blog.insightdatascience.com/generating-custom-photo-realistic-faces-using-ai-d170b1b59255
Github: https://github.com/SummitKwan/transparent_latent_gan#1-instructions-on-the-online-demo

#GAN #cv

Amazon’s SageMaker Object2Vec, a highly customizable algorithm that can learn embeddings of various types high-dimensional objects.

Link: https://aws.amazon.com/ru/blogs/machine-learning/introduction-to-amazon-sagemaker-object2vec/

#Object2Vec #Amazon #Embeddings

Prototypical Clustering Networks for Dermatological Disease Diagnosis

Paper will be presented at the ML4D workshop at #NIPS2018

Link: https://arxiv.org/abs/1811.03066

#nn #bio #medical

Monitor Your PyTorch Models With Five Extra Lines of Code



Ever felt like manually managing your Visdom / TensorBoard server and logs is a pain across experiments, projects and teams?
Weights & Biases provides a simple cloud-based experiment logging and plotting system, with easy integration for PyTorch models.

Link: https://www.wandb.com/blog/monitor-your-pytorch-models-with-five-extra-lines-of-code

#pytorch

New paper on Lipschitz neural net architectures. Uses sorting as an activation function, with matrix norm constrained weights. Universal Lipschitz function approx. Enforce adversarial robustness (margin) using hinge loss.

Link: https://arxiv.org/abs/1811.05381

#nn #lipschitz

​​Neural network 3D visualization framework. Very nice in-depth visualizations.

Now you can actually see how the layers look.

Github: https://github.com/tensorspace-team/tensorspace
LiveDemo (!): https://tensorspace.org/html/playground/vgg16.html

#visualization #nn

​​Really interesting talk at MLconfSF by Franziska Bell on how #Uber uses NLP for customer experience. Most of what was described are recent advances in their COTA platform.

Link: https://eng.uber.com/cota/

​​DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution

Using GANs to generate MasterFingerPrints that unlock 22-78% phones sensors (dep. on security level of sensor). It doesn't get much more "adversarial" than that.

This work can be potentially used to create fingerprint which can be used to match 22-78% of fingerprints in the wild, creating Skeleton key, fitting any security system, including home alarm or phone lock.

ArXiV: https://arxiv.org/pdf/1705.07386.pdf

#GAN #security #fingerprint

Sptoify announced its new Data Science Challenge

Spotify Sequential Skip Prediction Challenge is a part of #WSDM Cup 2019. The dataset comprises 130M Spotify listening sessions, and the task is to predict if a track is skipped. The challenge is live today, and runs until Jan 4.

Link: https://www.crowdai.org/challenges/spotify-sequential-skip-prediction-challenge

#kaggle #CompetitiveDataScience #Spotify

ImageNet/ResNet-50 Training speed dramatically (6.6 min -> 224 sec) reduced

ResNet-50 on ImageNet now (allegedly) down to 224sec (3.7min) using 2176 V100s. Increasing batch size schedule, LARS, 5 epoch LR warmup, synch BN without mov avg. (mixed) fp16 training. "2D-Torus" all-reduce on NCCL2, with NVLink2 & 2 IB EDR interconnect.

1.28M images over 90 epochs with 68K batches, so the entire optimization is ~1700 updates to converge.

ArXiV: https://arxiv.org/abs/1811.05233

#ImageNet #ResNet

New open source suite of ODE solvers in PyTorch

Everything happens on the GPU and is differentiable. Now you can use ODEs in your deep learning models.

Github: https://github.com/rtqichen/torchdiffeq

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Paper shows that the loss of two-layer neural networks can be optimized to zero in polynomial time using gradient descent.

ArXiV: https://arxiv.org/pdf/1810.02054.pdf

#nn #dl

And the same for #ResNet, #RNN and feed-forward #nn without residual connections.

Gradient Descent Finds Global Minima of Deep Neural Networks
ArXiV: https://arxiv.org/pdf/1811.03804.pdf

On the Convergence Rate of Training Recurrent Neural Networks
ArXiV: https://arxiv.org/pdf/1810.12065.pdf

A Convergence Theory for Deep Learning via Over-Parameterization
ArXiV: https://arxiv.org/pdf/1811.03962.pdf

#dl