NeurIPS 2023 posters (day 0, 1, and 2)

На нулевом дне NeurIPS я был на конференции-спутнике NeurIPS: ML4Health. Я немного занимался medical NLP вместе с MIT/Harvard и знакомые позвали меня поучаствовать в research roundtable как junior chair (извиняюсь я не знаю как это переводить).

Вот пара интересных статей с ML4Health:
1. MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (arxiv)
1. A Multimodal Dataset of 21,412 Recorded Nights for Sleep and Respiratory Research (arxiv)
1. Multimodal in-context learning enables rationale generation (aka MedFlamingo) (arxiv)

А теперь поток статей с NeurIPS:
1. Трансформеры в начале учат эмбеддинги под равномерным attention, после чего уже учат attention (arxiv)
1. Explainability at scale: сделали новый метод объяснения нейросетей и попробовали на Alpaca-7B. Смогли интерпретировать что для промпта "Please say yes only if it costs between [X.XX] and [X.XX] dollars, otherwise no" модель использует конкретный (и очень простой) алгоритм который можно увидеть на одной из картинок (arxiv)
1. То где в трансформере находится информация на удивление никак не связано с тем какие слои надо корректировать для knowledge editing (arxiv)
1. MLM отлично заходит для мультимодального предобучения (картинки, аудио, видео) даже если вы используете просто L2 лосс. Всё что вам нужно это скейлинг (arxiv)
1. Mathematical Capabilities of ChatGPT (arxiv)
1. Можно делать мультимодальные модели из кучи одномодальных без тренировки. Всё что надо это немного пар (базисных) данных из разных модальностей. Идея: строить фичи на основе схожести к вашим базисным данным (arxiv)
1. Трансформеры тренируются постепенно повышая ранк KQ^T. Эта статья очень зацепила тк частично доказывает мою гипотезу что нейросетки тренируются locally-low rank, и больше мотивируют то что ReLoRA – это правильный подход для тренировки нейросетей. (arxiv)

(Из-за лимита символов картинки будут в следующем посте)

Интересный пост Томаша Миколова

"Yesterday we received a Test of Time Award at NeurIPS for the word2vec paper from ten years ago. I'm really happy about it! I think it's the first "best paper" type of award I ever received. In fact, the original word2vec paper was rejected at the first ICLR conference in 2013 (despite the acceptance rate of around 70%), so it made me think how difficult it is for reviewers to predict future impact of research papers.

I've heard a lot of comments - both positive and negative - about word2vec during those years, and did not really comment online about it. Somehow I felt the research community is constantly flooded by propaganda-style PR from certain researchers who are hacking this way the citation counts and attention of others, and I did not want to be part of this. But after ten years, I think it could be entertaining to share some stories associated with this paper.

One frequent comment I've heard was that the code was difficult to understand to the point that some people thought I made it unreadable intentionally. But no, I'm not so evil :D The code ended up being over-optimized because I was waiting for many months for approval to publish it, and meanwhile I was trying to make it both faster and shorter. In fact, looking back, if there were not Greg and Jeff in the Brain team, I doubt I would ever get that approval - I think word2vec was likely the first widely known AI project that Google open-sourced.

There was also significant controversy around the GloVe project from Stanford NLP group that was published more than a year after word2vec. While it copied many tricks from our project, GloVe always felt like a step back to me: it was slower, required more memory, and the resulting vectors had lower quality than the original word2vec. However, it was published with word vectors pre-trained on much more data and thus gained a lot of popularity - although the comparison was really apples-to-oranges. We anyways did fix this later in the fastText project, where we did show that word2vec is much better than GloVe when trained on the same data.

I also received a lot of comments on the word analogies - from "I knew that too but forgot to publish it!" (Geoff Hinton, I believe you :) happens to everyone, and anyways I think everybody knows what the origin of Distributed Representations is) to "it's a total hack and I'm sure it doesn't work!" (random guys who didn't bother to read the papers and try it out themselves - including Ian Goodfellow raging about it on Twitter).

Despite word2vec being my most cited paper, I did never think of it as my most impactful project. In fact, word2vec code originally started as a subset of my previous project - RNNLM - which I think ended up forgotten too quickly. In my eyes, it was at least as revolutionary as AlexNet. Just to name ideas that were for the first time ever demonstrated within RNNLM already in 2010 (when it was still dark ages for deep learning): scalable training of recurrent neural networks (as I invented gradient clipping), first ever text generation from neural language model (I was showing examples of this since 2007), dynamic evaluation, character and sub-word level neural language modeling, neural language model adaptation (nowadays called fine-tuning), first publicly available LM benchmark (the modified Penn Treebank dataset - there really was nothing like this on the web when I started my PhD). I published the first ever study showing that neural nets beat n-gram language models increasingly more with more training data when everything is done correctly (today this sounds obvious, but back in the days this was widely considered impossible - even most Google guys did think that the more data you have, the more futile is to work on anything besides n-grams and smoothing techniques).