2014 - Invention of modern content-based neural attention, added to RNNs:
Neural Machine Translation by Jointly Learning to Align and Translate - Dzmitry Bahdanau
Jacobs University Bremen Germany,
KyungHyun Cho and Yoshua Bengio at University Μ of Montreal
βIn this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoderβdecoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.β
2017 - Elimination of the RNN and just using attention, i.e. transformers:
Attention is All You Need - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
βWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.β
Neural Machine Translation by Jointly Learning to Align and Translate - Dzmitry Bahdanau
Jacobs University Bremen Germany,
KyungHyun Cho and Yoshua Bengio at University Μ of Montreal
βIn this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoderβdecoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.β
2017 - Elimination of the RNN and just using attention, i.e. transformers:
Attention is All You Need - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
βWe propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.β
π13π2π₯°1
Long-list tipping point phenomenon
Where asking for longer lists gets you more and more results β up until a certain point, where suddenly it flips to giving you far fewer usable results than if you had asked for a short list.
It just gives up on even trying.
Where asking for longer lists gets you more and more results β up until a certain point, where suddenly it flips to giving you far fewer usable results than if you had asked for a short list.
It just gives up on even trying.
π37π8π€‘8π₯2
This media is not supported in your browser
VIEW IN TELEGRAM
Future of Labor
π₯21π19π15π€£14π€¨4β€2
Bard demo disaster tanks Google stock price
Microsoft up, Google down. The Bard demo flopped.
Microsoft up, Google down. The Bard demo flopped.
π5