Blog

Notes & deep dives

Long-form, intuition-first writeups: the idea in plain language first, then the math, a worked example, and runnable code. Mostly reinforcement learning and LLM systems for now.

• May 5, 2026

  Reinforcement Learning from First Principles (and the Math You Actually Need)

  What RL really is, why 'how good' means 'how good on average', and the five pieces of math (expectation, the Markov property, variance, running averages, and discounting) that the rest of the field is built on. Intuition first, one clean derivation each, a worked example, then code with Gymnasium.

  • #reinforcement-learning
  • #fundamentals
  • #probability
  • #gymnasium

• May 12, 2026

  MDPs and the Bellman Equation: The Recursion Behind Every RL Algorithm

  How to formalize an environment as a Markov Decision Process, why the value of a state splits into V and Q, and the Bellman equation that ties them together recursively. Derive it once, verify it by hand, then solve it exactly on FrozenLake in Gymnasium.

  • #reinforcement-learning
  • #mdp
  • #bellman-equation
  • #gymnasium

• May 19, 2026

  DP, Monte Carlo, and TD: Three Ways to Solve the Bellman Equation

  Three algorithms (Dynamic Programming, Monte Carlo, and TD(0)) all solve the same Bellman equation. You'll implement each on a custom Mars Rover gridworld and watch them converge to the same values from different starting assumptions.

  • #reinforcement-learning
  • #dynamic-programming
  • #monte-carlo
  • #temporal-difference
  • #gymnasium

• May 26, 2026

  SARSA, Q-learning, and DQN: From a Table to a Network That Plays Atari

  Control = prediction + a max. We turn the TD update into Q-learning and SARSA, watch them disagree on a cliff, then swap the table for a neural network and train a DQN that learns Pong from raw pixels.

  • #reinforcement-learning
  • #q-learning
  • #sarsa
  • #dqn
  • #deep-rl
  • #gymnasium

• Jun 2, 2026

  Policy Gradients: Learning the Policy Directly, from a Bandit to Actor-Critic

  Stop learning values and reading a policy off them: parameterize the policy and climb expected reward directly. We derive the score-function trick on a one-state Archer bandit, watch one softmax update touch all nine logits, kill the variance with a baseline, add states and credit assignment, and end on an Actor-Critic that solves the Archer MDP.

  • #reinforcement-learning
  • #policy-gradients
  • #reinforce
  • #actor-critic
  • #advantage
  • #gymnasium

• Jun 9, 2026

  TRPO and PPO: The Largest Safe Step a Policy Can Take

  The gradient gives a direction, but who chooses the step size? This post explores the step-size crisis in RL, builds importance sampling from scratch, derives the surrogate objective, fences it with TRPO's KL constraint, then clips it with PPO, all grounded in a continuous-control example where PPO learns to swing a pendulum upright.

  • #reinforcement-learning
  • #ppo
  • #trpo
  • #importance-sampling
  • #trust-region
  • #gymnasium

• Jun 16, 2026

  RLHF: Teaching a Language Model What "Good" Means

  A language model only ever predicts the next token. This post earns the sentence "PPO aligned ChatGPT": how generating text becomes an MDP, how human comparisons become a Bradley-Terry reward model, how a value head turns one end-of-answer score into a per-token signal, and how the KL leash keeps PPO from gaming a flawed reward, grounded in a hands-on build that aligns a tiny GPT-2 assistant.

  • #reinforcement-learning
  • #rlhf
  • #reward-model
  • #bradley-terry
  • #ppo
  • #kl-penalty

• Jun 23, 2026

  GRPO: Teaching a Model to Reason by Comparing It to Itself

  RLHF aligned a model with two heavy passengers: a learned reward model and a value-head critic. This post throws both away. You will see why the critic was only ever a baseline and how a group of sampled answers replaces it for free, why a verifiable check beats a hackable reward model for math and code, how GRPO wraps that group-relative advantage in PPO's clip, and how this exact recipe produced DeepSeek-R1, all grounded in a hands-on run that trains Llama-3.2-3B to reason on grade-school math.

  • #reinforcement-learning
  • #grpo
  • #verifiable-rewards
  • #rlvr
  • #deepseek-r1
  • #reasoning
  • #ppo

• Jun 30, 2026

  DPO and Agentic RL: Align Without a Reward Model, Then Step Into the World

  Two moves close the series. First we simplify: Direct Preference Optimization reaches the same aligned model as RLHF while firing both the reward model and the RL loop, and SimPO and KTO trim it further. Then we expand: point the same group-relative gradient at a multi-step world and the model stops answering and starts acting, calling tools, reading observations, and earning a reward only at the end. You will see the DPO loss derived from the RLHF objective, every number of a worked preference, and a runnable multi-turn agent that learns to use a tool from reward alone.

  • #reinforcement-learning
  • #dpo
  • #simpo
  • #kto
  • #preference-optimization
  • #agentic-rl
  • #rl-environments
  • #llm

No posts with that tag yet.