#09 Summary
🤖 RT-1: Robotics Transformer for Real-World Control at Scale

project page → link

⚡️Long story short:
RT-1 can efficiently perform everyday tasks using a hand manipulator based on text instructions.

🤓Methodology:
The research employs imitation learning to train the agent, which is composed of pre-trained language and image models, along with a decoder for predicting actions.

📌Key Components:
1. The model receives text instructions and derives sentence embeddings using a pre-trained T5 model.
2. It processes six images (the robot's environment) via EfficientNet, with text embedding integration as detailed in the paper.
3. Subsequently, RT-1 processes multimodal (text + images) features using a decoder-only model.

📊Training:
RT-1 was trained in a supervised setting with the aim of predicting the next action, as a human annotator would. The dataset consists of 130k demonstrations across 744 tasks. During training, RT-1 is given six frames, resulting in 48 tokens (6x8) from image and text instructions.

💃Intriguing insights:
1. Auto-regressive methods tend to slow down and yield poorer performance.
2. Discretizing the action space enables solving classification problems rather than regression, and allows sampling from the prediction distribution.
3. Continuous actions perform worse in comparison.
4. Input tokens are computed only once, with overlapped inference applied.
5. Data diversity proves to be more critical than data quantity.

😅My thoughts:
The RT-1 model demonstrates impressive results in accomplishing everyday tasks based on text instructions.
AI models like RT-1, with their increasing abilities in complex tasks, may soon deserve human names, such as "Robert" for RT-1, to highlight their advancements.

ML papers | 01-13 June 2023

💎 Video + Text

Probabilistic Adaptation of Text-to-Video Models

What: Finetune large pretrain text to video model on small domain specific videos.

Complicated but interesting. You can finetune pretrain diffusion model on your domain with small additional block.

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

What: Finetune LLM for understanding video+audio.

Use Q-Former for getting audio and video features. Then add it to pretrained llama model.

🧬Diffusion

Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model

What: propose simple implementation and intuition of diffusion model.

Good start to dive into the field and try on your data.

💎Audio Transformers

Simple and Controllable Music Generation

What: propose decoder for text 2 audio based on latent audio features.

They use vq quantization. Check it if you don't hear about it.
It allows to represent data with a limited number of vectors.


💎If you like this format please write in comments.
#digest

🧬 Tasty papers | 13-20 June 2023

Multimodal

🟣LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

Add visual information to LLM using trainable adapters.

Expand LLaMA Adapters V1 to vision.
+ Apply early fusion for visual tokens.
+ Add calibration of norm, bias of the LLM model.
+ Finetune on image-text dataset.

Audio

🟣High-Fidelity Audio Compression with Improved RVQGAN

Compress natural audio to discrete tokens with VQ technique.

Train universal compression model on all audio data: speech, music, noise.
+ add vector quantization.
+ add adversarial loss (GAN loss).

🟣Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

Audio generative "diffusion" model trained on 50k hours data.

Use Flow Matching, similar w/ diffusion, but better ✌
Masked train setting with context information. The model can synthesize speech, noise removal, content editing,

Neuro

🟢Decoding and synthesizing tonal language speech from brain activity

Decode tonal language from ECoG data with CNN-LSTM models.

Adapt multi-stream model -> looks unnecessary complicated.
Record small datasets. Overall 10 minutes per patient for 8 different syllables.

#digest

Introducing motor interface for amputee

That is the first AI model for decoding precise finger movements for people with hand amputation. It uses only 8 surface EMG electrodes.

ALVI Interface can decode different types of moves in virtual reality
🔘finger flexion
🔘finger extension
🟣typing
🟣some more

💎Full demo: YouTube link

Subscribe and follow the further progress of ALVI Labs:
Twitter: link
Instagram: link