Machine learning
rl_warmups
From-scratch PyTorch implementations of core deep reinforcement learning algorithms, written to learn them properly and explain them clearly.
The problem
Reinforcement learning is easy to read about and surprisingly hard to get working. I wanted to understand the fundamentals by building them myself, rather than importing a framework that hides the interesting parts.
The approach
The library implements four algorithms that show the main design choices in the field, all sharing one training loop, replay buffer and TensorBoard logging setup, with Optuna wired in for hyperparameter search.
- Deep Expected SARSA: on-policy temporal-difference learning with an expectation over next actions
- DQN: off-policy value learning with a target network and an epsilon-greedy policy
- REINFORCE: a Monte Carlo policy-gradient method with no value function
- Actor-Critic: a policy and a value function learning together to reduce variance
The README doubles as a short course: the Bellman equation, discounted returns, Monte Carlo versus temporal-difference learning, on-policy versus off-policy, exploration versus exploitation, replay buffers and baselines, each with a concrete example.
Where it stands
I paused the project to work on other things, and I say so plainly in the README: it is untested and not finished, and is best used as a reference implementation or study aid rather than a production library.