📃 Energy-Based Transformers are Scalable Learners and Thinkers

Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes.

@Machine_learn

"Transcendence" is when an LLM, trained on diverse data from many experts, can exceed the ability of the individuals in its training data.

This paper demonstrates three types: when AI picks the right expert skill to use, when AI has less bias than experts & when it generalizes.

📚 Read

@Machine_learn

با عرض سلام ما برای یکی از مقالاتمون در حوزه ی پزشکی نیاز به نفر ۴ ام داریم با قبولی شرایط پرداخت میتونیم اضافه کنیم.

@Raminmousa