with Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, Wanchao Liang TL;DR We implemented experimental async tensor parallelism support in PyTorch. We integrated it in TorchTitan and observed: Up to ~29% forward pass speedup and ~8% E2E speedup in Llama3…

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training...

archived 7 Oct 2024 18:43:10 UTC

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise....

A key property of reasoning systems is the ability to make sharp decisions on their input data. For contemporary AI systems, a key carrier of sharp behaviour is the softmax function, with its...

The rise of intelligent autonomous systems, especially in robotics and autonomous agents, has created a critical need for robust communication middleware that can ensure real-time processing of...

Meta Motivo, an open-source behavioral foundation model designed to control virtual, physics-based humanoid agents.

It aims to significantly simplify the creation of general-purpose humanoid agents for robotics and virtual avatars.

Try the demo: https:…