This media is not supported in your browser
VIEW IN TELEGRAM
Figure x OpenAI 2024
This status update just blows my mind. Really amazing progress. Especially the level of hand dexterity.
Figure combines a vision language model from openAI with a control system.
They did it in 13 days after signing a partnership🔥
For more details: X post
This status update just blows my mind. Really amazing progress. Especially the level of hand dexterity.
Figure combines a vision language model from openAI with a control system.
They did it in 13 days after signing a partnership🔥
For more details: X post
❤5🔥2
Grok 1 by X. 314B MoE.
Model Details
- Base model trained on a large amount of text data, not fine-tuned for any particular task.
- 314B parameter Mixture-of-Experts model with 25% of the weights active on a given token.
- Trained from scratch by xAI using a custom training stack on top of JAX and Rust in October 2023.
Link: https://x.ai/blog/grok-os
Model Details
- Base model trained on a large amount of text data, not fine-tuned for any particular task.
- 314B parameter Mixture-of-Experts model with 25% of the weights active on a given token.
- Trained from scratch by xAI using a custom training stack on top of JAX and Rust in October 2023.
Link: https://x.ai/blog/grok-os
❤3
#10 Summary
Humanoid Locomotion as Next Token Prediction
What.
They trained causal decoder to predict next action (and observation)
Data.
Normally, you'd need a bunch of data that shows both what the robot sees (observations) and what it does (actions). But that's tough to get . The authors used videos - some with the actions laid out and some without. This way, the robot can learn even from videos where we don't know what the actions were supposed to be.
In case there’re not action, they replace with [MASK] token. Very simple and straightforward
My thoughts
- I love how this paper makes the robot predict its next move and what it'll see next. It's like it's planning its future steps.
- For the robot to guess what's going to happen next accurately, it needs to have a mini understanding of physics and how the world works. This concept, called a 'world model,' is super intriguing.
- What's next? You can add condition with cross attention and train to understand commands, like VIMA paper.
More examples
Humanoid Locomotion as Next Token Prediction
What.
They trained causal decoder to predict next action (and observation)
Data.
Normally, you'd need a bunch of data that shows both what the robot sees (observations) and what it does (actions).
In case there’re not action, they replace with [MASK] token. Very simple and straightforward
My thoughts
- I love how this paper makes the robot predict its next move and what it'll see next. It's like it's planning its future steps.
- For the robot to guess what's going to happen next accurately, it needs to have a mini understanding of physics and how the world works. This concept, called a 'world model,' is super intriguing.
- What's next? You can add condition with cross attention and train to understand commands, like VIMA paper.
More examples
❤4
Brain-To-Text Competition 2024
This is the most fascinating BCI competition yet, organized by Stanford. Everyone has one month to develop the world's best brain-to-speech decoder!
Task: Predict attempted speech from brain activity.
Deadline: June 2, 2024
Dataset: They've recorded 12,100 sentences from a patient who can no longer speak intelligibly due to amyotrophic lateral sclerosis (ALS).
Just letting you know we're jumping into this challenge!
Together with @Altime, @kovalev_alvi and the team of ALVI Labs, we're going to create something interesting.
Like this post if you want to follow our updates❤️
This is the most fascinating BCI competition yet, organized by Stanford. Everyone has one month to develop the world's best brain-to-speech decoder!
Task: Predict attempted speech from brain activity.
Deadline: June 2, 2024
Dataset: They've recorded 12,100 sentences from a patient who can no longer speak intelligibly due to amyotrophic lateral sclerosis (ALS).
For each sentence, we provide the transcript of what the participant was attempting to say, along with the corresponding time series of neural spiking activity recorded from 256 microelectrodes in speech-related areas of cortex.
Just letting you know we're jumping into this challenge!
Together with @Altime, @kovalev_alvi and the team of ALVI Labs, we're going to create something interesting.
Like this post if you want to follow our updates❤️
👍9❤6🔥4🦄2
Neuralink Update - July 10, 2024
I watched their latest update and prepared some notes about it.
Link for the full
Control of the cursor.
Best cursor control without eye tracking.
How it works: participant imagines hand movements, translated to cursor control. I expect that they predict velocity of the cursor.
They achieved amazing speed and quality of cursor control on Day 133. For example, I couldn't beat this record for the first time. Let's try!
That's link: https://neuralink.com/webgrid/
My result: 7.88 BPS
Device usage up to 70 hours per week. That's almost half their waking life!
Future Plans and Goals:
Main goal is to give people "superpowers", not just restore functionality
Pretty damn cool
- Minimize device size, close gap between implant and brain. So they want to make device feel less external.
- Implant deeper for better signal diversity
- Decode text, enable faster typing (brain-to-text)
Holy Shit
- Develop visual prostheses (currently studying in monkeys)
- Repair paralysis (tested on animals with spinal cord implants)
- Use Optimus (Tesla) arms and legs, controlled by Neuralink implants
Future Implantation Process:
- Aim for fully automated implantation
- Quick procedure (10 minutes, "like cyberpunk")
My thoughts
In my view, the progress in hardware is truly exciting. I think that solving paralysis is possible; there are no restrictions from the laws of physics. However, as Elon Musk said, it's a super hard technical problem. There are plenty of papers on restoring movement and speech in the lab.
For example in brain-to-text competition, we also could decode sentences from brain signals. Really inspired story.
So stay tuned and cyborg might be real very soon.
I watched their latest update and prepared some notes about it.
Link for the full
Control of the cursor.
Best cursor control without eye tracking.
How it works: participant imagines hand movements, translated to cursor control. I expect that they predict velocity of the cursor.
They achieved amazing speed and quality of cursor control on Day 133. For example, I couldn't beat this record for the first time. Let's try!
That's link: https://neuralink.com/webgrid/
My result: 7.88 BPS
Device usage up to 70 hours per week. That's almost half their waking life!
Future Plans and Goals:
Main goal is to give people "superpowers", not just restore functionality
Pretty damn cool
- Minimize device size, close gap between implant and brain. So they want to make device feel less external.
- Implant deeper for better signal diversity
- Decode text, enable faster typing (brain-to-text)
Holy Shit
- Develop visual prostheses (currently studying in monkeys)
- Repair paralysis (tested on animals with spinal cord implants)
- Use Optimus (Tesla) arms and legs, controlled by Neuralink implants
Future Implantation Process:
- Aim for fully automated implantation
- Quick procedure (10 minutes, "like cyberpunk")
My thoughts
In my view, the progress in hardware is truly exciting. I think that solving paralysis is possible; there are no restrictions from the laws of physics. However, as Elon Musk said, it's a super hard technical problem. There are plenty of papers on restoring movement and speech in the lab.
For example in brain-to-text competition, we also could decode sentences from brain signals. Really inspired story.
So stay tuned and cyborg might be real very soon.
❤5🔥1🤔1🐳1
Cursor controlled by a brain implant.
Wireless, compatible with almost all devices using BLE.
You can even use it during a flight!
Wireless, compatible with almost all devices using BLE.
You can even use it during a flight!
🤯4🔥3👍1👻1
Muscle-To-Motion Competition 2024
I’m happy to tell you about new competition!
You've got 21 days to cook up the world's best prosthetic control system. This is the BCI competition, organized by ALVI Labs and BCI Initiative.
Task: Predict finger movements from the muscle activation(EMG).
Deadline: August 5, 2024
Dataset: We've recorded 10+ hours of data from both healthy and amputee subjects.
This isn't just classification - it's a full regression challenge. We'll test your model on a real amputee. Your code could directly impact someone's life.
Time to create some next-level shit and maybe, just maybe, revolutionize prosthetics. Let’s fun.❤️
kaggle: https://www.kaggle.com/c/bci-initiative-alvi-hci-challenge/overview
Github with code: https://github.com/BCI-I/BCI_ALVI_challenge
I’m happy to tell you about new competition!
You've got 21 days to cook up the world's best prosthetic control system. This is the BCI competition, organized by ALVI Labs and BCI Initiative.
Task: Predict finger movements from the muscle activation(EMG).
Deadline: August 5, 2024
Dataset: We've recorded 10+ hours of data from both healthy and amputee subjects.
We asked people to make different finger movements (simple moves, gestures, static moves). We recorded muscle activity using Myo Thalmic Armband ( 8 electrodes, 200Hz). Plus, we captured hand positions with 20 DoF at 25 Hz.
This isn't just classification - it's a full regression challenge. We'll test your model on a real amputee. Your code could directly impact someone's life.
Time to create some next-level shit and maybe, just maybe, revolutionize prosthetics. Let’s fun.❤️
kaggle: https://www.kaggle.com/c/bci-initiative-alvi-hci-challenge/overview
Github with code: https://github.com/BCI-I/BCI_ALVI_challenge
🔥9👍1👻1
the last neural cell
Muscle-To-Motion Competition 2024 I’m happy to tell you about new competition! You've got 21 days to cook up the world's best prosthetic control system. This is the BCI competition, organized by ALVI Labs and BCI Initiative. Task: Predict finger movements…
Muscle-To-Motion Competition 2024: Final Week Countdown
Only 7 days remain in our prosthetic control system competition.
kaggle link: https://kaggle.com/c/bci-initiative-alvi-hci-challenge/overview
🔬 Challenge: Predict finger movements from EMG data
🏆 Impact: Your model will be tested on a real amputee
🚀 Leaderboard is tight!
Submissions close August 5th, 2024.
Only 7 days remain in our prosthetic control system competition.
kaggle link: https://kaggle.com/c/bci-initiative-alvi-hci-challenge/overview
🔬 Challenge: Predict finger movements from EMG data
🏆 Impact: Your model will be tested on a real amputee
🚀 Leaderboard is tight!
Submissions close August 5th, 2024.
🔥5👻2🐳1
DeepPhase: periodic autoencoders for learning motion phase manifolds | 2022
Я сейчас занимаюсь LFP decomposition и наткнулся на старую, но интересную статью.
Они получают компоненты с помощью свёрток(могут быть и RNN). Затем берут синусоиду и подгоняют ее под эти компоненты.
В конце, они проецируют эти синусы в пространство сигнала. Тоже свертками. И решают задачу реконструкции. Всё работает, потому что все операции дифференцируемые.
Зачем это нужно?
Чтобы понять внутреннюю структуру временных сигналов. Генерировать, интерполировать и так далее. Они применяли для движений. Я считаю что можно пробовать для других сигналов: EEG, LFP, ECoG.
Например, построить фазовые манифолды.
code
P. S. If you do not speak Russian, use the top "Translate to English" button.
#review
Я сейчас занимаюсь LFP decomposition и наткнулся на старую, но интересную статью.
let's force model to learn periodic latents
Они получают компоненты с помощью свёрток(могут быть и RNN). Затем берут синусоиду и подгоняют ее под эти компоненты.
В конце, они проецируют эти синусы в пространство сигнала. Тоже свертками. И решают задачу реконструкции. Всё работает, потому что все операции дифференцируемые.
Зачем это нужно?
Чтобы понять внутреннюю структуру временных сигналов. Генерировать, интерполировать и так далее. Они применяли для движений. Я считаю что можно пробовать для других сигналов: EEG, LFP, ECoG.
Например, построить фазовые манифолды.
code
P. S. If you do not speak Russian, use the top "Translate to English" button.
#review
1🔥6👍5🦄1
Genie: Generative Interactive Environments
What: learn latent actions from videos (only) of games.
- predict future frames based on previous and latent actions.
- they trained actions to help model make transition between frames.
- just let’s AI model figures out commands by yourself.
SAM 2: Segment Anything in Images and Videos
What: SAM now works well with videos.
- annotate big dataset of videos.
- add memory block to ensure temporal consistency of predicted mask.
Mixture of A Million Experts
What: expand MoE for lots of experts.
- store low rank approx of experts.
- works better than dense FFN.
The Road Less Scheduled
What: propose schedule-free optimizer.
- one more thing that beats AdamW.
- easy to drop in your training pipeline.
Rolling Diffusion Models
What: incorporating temporal info in generative diffusion process for videos.
- let’s make denoising and predict next frames at the same time.
- hard math, but idea is interesting.
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
What: step into merging local and global planning.
Our approach is shown to combine the strengths of next-token prediction models, such as variable-length generation, with the strengths of full-sequence diffusion models, such as the ability to guide sampling to desirable trajectories.
#digest
Please open Telegram to view this post
VIEW IN TELEGRAM
🔥4👍2🐳1
Tasty Neuro Papers | 01-31 July 2024
Brain decoding
🔘 Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution
Кратко: pretrained transformer for spikes.
- Single-spike resolution, никакого rate coding.( На самом деле есть, но на маленьких бинах)
- Придумали multi task masking MtM - модель учится, попеременно маскируя и восстанавливая активность во времени, между нейронами и областями мозга.
- Обучаемый токен подсказывает модели текущую схему маскирования.
🔘 Semantic encoding during language comprehension at single-cell resolution
Кратко: Нашли нейроны, активирующиеся на конкретные значения слов во время речи.
- Реагируют на конкретные семантические категории слов (еда, животные и т.д.)
- Активность этих нейронов зависит от контекста предложения, а не просто от звучания слов
- Нейронная сеть может предсказать значение слова по активности этих клеток
Мне лично нравится тенденция использовать все события(каждый спайк). Потому что при rate coding мы например имеем задержку и не можем некоторые быстро изменяющиеся вещи улавливать(саккады).
Соберите больше данных, а модель сама разберется -> пока что работает почти везде.
#digest
Brain decoding
Кратко: pretrained transformer for spikes.
- Single-spike resolution, никакого rate coding.( На самом деле есть, но на маленьких бинах)
- Придумали multi task masking MtM - модель учится, попеременно маскируя и восстанавливая активность во времени, между нейронами и областями мозга.
- Обучаемый токен подсказывает модели текущую схему маскирования.
Кратко: Нашли нейроны, активирующиеся на конкретные значения слов во время речи.
- Реагируют на конкретные семантические категории слов (еда, животные и т.д.)
- Активность этих нейронов зависит от контекста предложения, а не просто от звучания слов
- Нейронная сеть может предсказать значение слова по активности этих клеток
Single-neuronal recordings were obtained from the prefrontal cortex of the language-dominant hemisphere in a region centred along the left posterior middle frontal gyrusМысли вслух. Скоро сделаем обзор на первую статьи(universal translator) и сравним с предыдущей "foundation" моделью POYO.
Мне лично нравится тенденция использовать все события(каждый спайк). Потому что при rate coding мы например имеем задержку и не можем некоторые быстро изменяющиеся вещи улавливать(саккады).
Соберите больше данных, а модель сама разберется -> пока что работает почти везде.
#digest
Please open Telegram to view this post
VIEW IN TELEGRAM
🔥4🕊1👻1
Geometric Machine Learning.
GREEN (Gabor Riemann EEG Net). Звучит мудрено, но идея простая: взяли обучаемые фильтры, добавили риманову геометрию и получили мощный инструмент для работы с ЭЭГ.
paper
code
Как это работает:
1. Пропускаем ЭЭГ через обучаемые фильтры. По факту свертки.
2. Смотрим, как каналы и частоты связаны. Строим ковариационную матрицу. (кстати, похоже на скоры в self-attention до софтмакса).
3. Потом идет магия с римановой геометрией. Фишка в том, что работают с целыми матрицами, а не векторами.
В итоге получаем крутые фичи для MLP, которые дают хорошие результаты.Интересно конечно попробовать сделать претрейн такой штуки с помощью contrastive learning. Может и получше потом задачи решать будет.
P.S. В сохраненках нашел статью про геометрию в ML с крутыми картинками:
Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures
Если работаете с нейроданными или просто любите ML - сохраните. Может пригодиться.
#review
GREEN (Gabor Riemann EEG Net). Звучит мудрено, но идея простая: взяли обучаемые фильтры, добавили риманову геометрию и получили мощный инструмент для работы с ЭЭГ.
paper
code
Как это работает:
1. Пропускаем ЭЭГ через обучаемые фильтры. По факту свертки.
2. Смотрим, как каналы и частоты связаны. Строим ковариационную матрицу. (кстати, похоже на скоры в self-attention до софтмакса).
3. Потом идет магия с римановой геометрией. Фишка в том, что работают с целыми матрицами, а не векторами.
В итоге получаем крутые фичи для MLP, которые дают хорошие результаты.Интересно конечно попробовать сделать претрейн такой штуки с помощью contrastive learning. Может и получше потом задачи решать будет.
P.S. В сохраненках нашел статью про геометрию в ML с крутыми картинками:
Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures
Если работаете с нейроданными или просто любите ML - сохраните. Может пригодиться.
#review
3👍8🔥2👨💻2
Forwarded from Нейроинтерфейсы (Sergei Shishkin)
Сравнение разных преобразований:
Stefan Scholl. Fourier, Gabor, Morlet or Wigner: Comparison of Time-Frequency Transforms. arXiv, 17 Jan 2021, https://doi.org/10.48550/arXiv.2101.06707
В том числе красивые картинки:
a gallery of time-frequency representations of numerous signals from different fields of applications to allow for visual comparison
Stefan Scholl. Fourier, Gabor, Morlet or Wigner: Comparison of Time-Frequency Transforms. arXiv, 17 Jan 2021, https://doi.org/10.48550/arXiv.2101.06707
В том числе красивые картинки:
a gallery of time-frequency representations of numerous signals from different fields of applications to allow for visual comparison
❤4👍2🔥1