archived 7 Oct 2024 18:43:10 UTC

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise....

A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its...

The rise of intelligent autonomous systems, especially in robotics and autonomous agents, has created a critical need for robust communication middleware that can ensure real-time processing of...

Meta Motivo, an open-source behavioral foundation model designed to control virtual, physics-based humanoid agents.

It aims to significantly simplify the creation of general-purpose humanoid agents for robotics and virtual avatars.

Try the demo: https:…

KV caching in LLMs, clearly explained (with visuals):

To make big investments in scaling robotic learning, we need to understand what the scaling laws for robotics data actually are.

Claude Opus 4.1 Plays Pokemon!

Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better...

I am excited to open-source PipelineRL - a scalable async RL implementation with in-flight weight updates. Why wait until your bored GPUs finish all sequences? Just update the weights and continue inference!

Code: https://t.co/AgEyxXb7Xi
Blog: https://t.co/n4FRxiEcrr