📌 Practical exercises and additional materials for the book "Build a Large Language Model (From Scratch)"

A Github repository with practical exercises, notebooks with code for developing, pre-training, and fine-tuning a GPT-type LLM model based on one of the best books on building an LLM from scratch.

▶️ About the book:
In this book, you will learn and understand how large language models work from the inside, creating your own LLM step by step, with a detailed explanation of each stage in clear language, diagrams and examples.

The method described in the book demonstrates the approach used to create large fundamental models such as those underlying ChatGPT.

In the repository, each chapter of the book has several (3-4) applied examples in ipynb format or as an executable python script. The code is aimed at a wide audience, is designed to run on regular laptops and does not require specialized equipment.

▶️ The main value of the repository is additional practical materials that will help you to study in more depth the subtleties and nuances of the process of setting up and learning LLM:

Setting

🟢 Tips on Setting Up Python
🟢 Installing Python Packages and Libraries
🟢 Docker Environment Setup Guide

Chapter 2: Working with Text Data

🟠 Comparison of different implementations of Byte Pair Encoding (BPE)
🟠 Understanding the difference between embedding and line layers
🟠 Dataloader Intuition with Prime Numbers

Chapter 3: Code of Attention Mechanisms

🟢 Comparison of Effective Implementations of Multi-Head Attention
🟢 PyTorch Buffers

Chapter 4: Implementing the GPT Model from Scratch

🟠 FLOPS Analysis

Chapter 5: Pre-training on unlabeled data

🟢 Alternative Loading of HuggingFace Scales Using Transformers
🟢 Pre-training GPT on the Project Gutenberg dataset
🟢 Adding more features to the learning cycle
🟢 Hyperparameter optimization for pretraining
🟢 Creating a user interface for interacting with LLM
🟢 Convert GPT to Llama
🟢 Llama 3.2 from scratch
🟢 Memory-efficient model loading

Chapter 6: Fine-tuning for Classification

🟠 More experiments on fine-tuning the different layers and using larger models
🟠 Fine-tuning various models based on a 50K row IMDB movie review dataset.
🟠 Building a User Interface for Interacting with a GPT-Based Spam Classifier

Chapter 7: Fine-tuning to Follow Instructions

🟢 Dataset utilities for finding close duplicates and creating passive voice entries
🟢 Evaluating responses to instructions using OpenAI and Ollama APIs
🟢 Creating a dataset for fine-tuning instructions
🟢 Improving the dataset for fine-tuning instructions
🟢 Creating a Preference Dataset with Llama 3.1 70B and Ollama
🟢 DPO for LLM Alignment procedure
🟢 Creating a user interface for interacting with a GPT model with fine-tuning of instructions

🖥 Github

https://t.iss.one/DataScienceT ✅

Explore "Pretraining LLMs," a short course developed with upstageai.

The course covers pretraining from scratch, continuing pretraining on custom data, and how using smaller open-source models can reduce costs.

Take the course for free: https://hubs.la/Q02YFKyx0

https://t.iss.one/DataScienceT ✅