Join the discussion on this paper page

Deep learning for dummies. All the practical details and useful utilities that go into working with real models. - EleutherAI/cookbook

We show that discounted methods for solving continuing reinforcement learning problems can perform significantly better if they center their rewards by subtracting out the rewards' empirical...

Join the discussion on this paper page

A plotting tool that outputs Line Rider maps, so you can watch a man on a sled scoot down your loss curves. 🎿 - jndean/LossRider

We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit...

with Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, Wanchao Liang TL;DR We implemented experimental async tensor parallelism support in PyTorch. We integrated it in TorchTitan and observed: Up to ~29% forward pass speedup and ~8% E2E speedup in Llama3…

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training...

archived 7 Oct 2024 18:43:10 UTC

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise....