Build Your Own Reasoning Model: A Full Video Course
Shiliew VPN - Focuses on providing stable network services'}
I have written a lot about reasoning models here over the last two years on Ahead of AI, including a recent article on mechanisms to control reasoning effort levels in contemporary LLMs. Reasoning models are still as relevant as ever. In fact, almost all current-generation LLMs today are reasoning models or offer a reasoning mode.
For those who are interested in how reasoning models work, including inference scaling, reinforcement learning, and distillation, I wrote my recent Build Reasoning Model (From Scratch), which implements these from the ground up in PyTorch, which, in my experience, is still the best way to understand and learn how things really work.
Now, in the last few months, I also sat down and recorded accompanying lectures I wanted to share with you. So, below, you will find eight reasoning model-from-scratch coding videos (11 hours of content).
We start with a pre-trained LLM and implement methods to evaluate and improve it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up course last year, this is essentially the sequel (but you can also watch it without the prior course as a prerequisite).
I’m currently taking a short break and will be back soon with more exciting LLM research roundups next month. In the meantime, I hope you enjoy coding along!
As a small side note, and since I often get asked how reasoning models relate to today’s AI agents, here’s a short section on this before getting started with the videos.
A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to other problem domains as well.
A reasoning model then functions as the engine inside an agent harness, where the agent harness is a software or tooling framework around the LLM/reasoning model to tackle even more challenging problems (bigger coding tasks, travel booking, and all other sorts of personal assistant tasks). The task of the harness is sophisticated context management and giving the LLM access to tools.
Figure 1: An engine analogy for conventional LLMs, reasoning models, and agent harnesses. Figure from my Components of a Coding Agent article.
In this analogy depicted in the figure above, the model is the engine, and the agent harness adds the machinery that lets us use it for various things (as an upgrade from a simple chatbot interface). An agent can use either a conventional LLM or a reasoning model, but reasoning models are the more typical choice as they are more capable. (As mentioned above, most current-gen LLMs are reasoning models with variable reasoning effort settings.)
So, the bottom line is that reasoning models, as the “engine,” remain an important part of understanding modern LLM agents.
The resources accompanying these topics shown above are:
The Build a Large Language Model (From Scratch)book and videos
The Build A Reasoning Model (From Scratch)book (and videos below)
The Components of a Coding Agentarticle on Substack
The videos below focus on developing the reasoning model that can power such a system.
The reasoning-from-scratch videos focus on a small pre-trained Qwen3 model, where we implement the reasoning methods ourselves in Python and PyTorch. Starting with a pre-trained model lets us focus on the interesting part: evaluation, inference-time scaling, reinforcement learning, and distillation.
Regarding the prerequisites, I recommend being comfortable with Python and PyTorch. I designed the reasoning materials as standalone content, so the previous Build A Large Language Model (From Scratch) content is recommended but not strictly required.
Compute-resource-wise, you can work through the early text-generation and evaluation examples on a CPU. For the later experiments, especially training, a GPU makes the work much more practical. The first video covers the setup and some more recommendations; the repository also includes hardware and environment recommendations.
Figure 2: An overview of the topics being covered.
These videos accompany my book, Build a Reasoning Model (From Scratch). If you like to read through the explanation in more detail and at your own pace, work through exercises, and so on, I recommend the book as a companion here.
For the most in-depth learning experience, my suggestion is to read a chapter, work through its code alongside the corresponding video, and then try the exercises.
The book is available from Manning and Amazon, and the print edition is in color!
Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) print book.
Thank you to everyone who has picked up a copy or recommended it to a friend. Your support helps me keep creating these materials.
As a last, quick side note before getting to the main videos below: For a shorter, more conceptual overview, I also recommend my AI Reasoning Models course on LinkedIn Learning (it should be free for 24h when accessing it via the link there, which shouldn’t then require a LinkedIn Premium subscription).
The course is about 90 minutes and explains how reasoning models work, covering the main ideas behind inference-time scaling, reinforcement learning, and distillation. It’s a good optional preview if you’d like the main ideas without working through the full implementations below.
To get to the main part of this article, here are the eight main lecture videos.
The first video starts with an explanation of the relationship between conventional LLMs, reasoning models, and agents. Next, the main topic of the video is setting up the coding environment. This includes installing the Python and PyTorch dependencies (with uv), among others. I also discuss hardware choices and common setup issues.
Topics covered
The relationship between conventional LLMs, reasoning models, and agent harnesses
On the benefits of learning through code implementations
The scope of this video series
Installing Python dependencies
Choosing hardware and checking CUDA or Apple Silicon support
Using a remote GPU with VS Code and troubleshooting the environment
Watch video 1 · Setup instructions
Before adding reasoning methods in the upcoming videos, we need a working base model. Instead of training it from scratch, since that would be a whole other book (see Build A Large Language Model (From Scratch)), we start with a pre-trained model.
Here, we load the pre-trained weights of a Qwen3 0.6B model. Then, we implement the text-generation process via a streaming text generator and add KV caching and model compilation to make text generation more efficient.
These implementations give us the baseline and base components we will evaluate and improve in the following videos.
Topics covered
Encoding text as tokens and decoding it back into text
Downloading Qwen3 weights and loading the model on the selected compute device
Going over tensor shapes, logits, and next-token predictions
Implementing greedy decoding and streaming text generation
Handling end-of-sequence tokens and measuring generation speed
Adding a KV cache to reuse attention keys and values
Combining KV caching with
torch.compileand comparing hardware performance
Watch video 2 · Chapter 2 code
In this lecture, we build a math-answer verifier step by step, from answer extraction to normalization and checking mathematical equivalence. We then use it to evaluate the model on MATH-500.
We use this verifier to evaluate the base model’s performance on the MATH-500 evaluation dataset.
However, the verifier also has a second job later in the series: providing the reward signal for reinforcement learning with verifiable rewards (RLVR).
Topics covered
Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)
Reusing answer verification as a reward signal for RLVR (more on that in chapter 6)
The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence
Building an answer grader and an evaluation loop for MATH-500
Comparing base and reasoning models on MATH-500
Watch video 3 · Chapter 3 code
Now we explore ways to improve answers while keeping the model weights fixed. But before implementing specific inference-scaling techniques, we first implement methods to generate diverse answers from the same prompt (this is standard practice in proprietary LLM chatbots, such as ChatGPT).
Concretely, we implement temperature scaling, multinomial sampling, and top-p filtering to generate diverse responses. Then we combine multiple responses through self-consistency, using majority voting over their final answers.
We also examine the tradeoff between answer accuracy and the extra computation needed to generate more candidates.
Topics covered
Comparing how additional compute benefits model performance during training and during inference
A gentle intro to inference-scaling through chain-of-thought prompts
Implementing temperature scaling and multinomial sampling to generate diverse answers
Adding top-p filtering to control the quality of the diverse answers
Combining candidate answers through self-consistency, one of the most popular inference scaling techniques
Evaluating MATH-500 accuracy against the cost of additional inference compute
Watch video 4 · Chapter 4 code
Building on the previous video, we implement scoring functions to compare multiple answers generated from the same prompt. We can also use these scoring methods to break ties in self-consistency (covered in the previous lecture).
The scoring methods covered here include computing token and sequence log probabilities (this will also be relevant in the later reinforcement learning chapters). We then implement a self-refinement loop in which the model critiques an answer and attempts a revision.
The evaluation lets us check how useful those additional steps are for our model.
Topics covered
Using scoring functions for comparing multiple answers and breaking ties
Implementing a simple heuristic scorer
Computing token probabilities and sequence likelihoods (also relevant for the following lectures)
Using log probabilities for improved numerical stability
Building a self-refinement loop for generating critiques and revised answers
Evaluating self-refinement on MATH-500
Watch video 5 · Chapter 5 code
In this lecture, we move on to implementing a training technique called Reinforcement Learning with Verifiable Rewards (RLVR). We first implement a simplified version of Group Relative Policy Optimization (GRPO), generate groups of responses, score them with our verifier, and use their relative rewards for the model weight updates during training.
Topics covered
Generating reasoning traces and understanding the relationship between answer lengths and problem-solving abilities
Discussion and comparison of two important LLM post-training methods: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
Turning verifier scores into rewards and relative advantages
Computing sequence log probabilities and a simplified GRPO loss
Assembling the training loop with logging and checkpoint saves
Evaluating checkpoints and checking training stability and memory needs
Watch video 6 · Chapter 6 code
Now that we implemented the training loop in the previous lecture, we can revisit it and analyze the training runs in more detail.
Concretely, this video adds more training metrics for inspection and analysis, including entropy and advantage statistics, clipped policy ratios, KL loss term values, and formatted rewards.
We then review the training logs and evaluation results to understand what these additions do and whether additions such as clipped policy ratios and KL loss terms are helpful.
Topics covered
Running longer training experiments and interpreting their logs
Tracking advantages and entropy to understand training behavior
Implementing clipping in the policy loss and analyzing its effects
Adding a KL loss term and discussing reward hacking
Understanding limitations of the simplified KL formulation
Implementing format rewards and conditioning them on answer correctness
Comparing checkpoints on MATH-500 and exploring further GRPO research
Watch video 7 · Chapter 7 code
Finally, we explore another way to train a small reasoning model, that is, learning from answers generated by a stronger (and bigger) teacher model. We discuss different forms of distillation, prepare reasoning data, and implement the supervised training loop.
We finish this lecture and course by evaluating the distilled model on MATH-500 and comparing the results. This rounds out the series with an alternative to developing reasoning capabilities through reinforcement learning.
Topics covered
Comparing the different combination and approaches to distillation, including soft and hard distillation, and on-policy and off-policy distillation
Creating and collecting teacher-generated answers for the MATH dataset (using DeepSeek R1)
Preparing the distillation dataset, including tokenization and training/validation/test splits
Computing answer-token log probabilities and understanding the cross-entropy loss
Implementing supervised training and tracking training and validation losses
Evaluating checkpoints on MATH-500 and comparing distillation results
Watch video 8 · Chapter 8 code
As you watch these lectures, I encourage you to pause occasionally and try a small experiment of your own if something interesting comes to mind.
For example, feel free to change the sampling settings, inspect a few failed answers, or try variations of different ideas similar to the suggestions in the exercises.
And if you would like the full written walkthrough, figures, and exercises alongside the videos, you can find Build a Reasoning Model (From Scratch) at Manning and Amazon.
Thanks for reading and supporting Ahead of AI. I am taking a short break to regenerate, and I look forward to being back with the next round of LLM research articles next month.
In the meantime, happy viewing and tinkering!
Shiliew VPN - Focuses on providing China stable network services'}