Fantasy RL

Fantasy RL asks whether an agent can learn to manage a fantasy football squad — buying, selling, balancing a budget, and choosing when to stop — purely from the reward it receives at the end of each week. It is a custom Gym environment plus four reinforcement-learning algorithms trained against it.

The environment

The simulation models a league closely enough to be interesting and loosely enough to be trainable. An agent holds eleven players with a minimum of eight, faces a market of twenty-five, starts with a budget of 100,000, and can take at most twenty-one actions before the week closes.

The action space is discrete and deliberately small: indices 0–10 sell a player from the squad, 11–35 buy one from the market, and index 36 ends the week. The state is a flat normalised vector — eleven squad players and twenty-five market players, each described by seventeen performance metrics, plus the current budget, everything scaled to [0, 1].

Rewards come from three places. Buying pays out relative to how the new player compares with the best player already in the squad, selling costs on the same scale, invalid actions are penalised at -1, and finishing the week pays according to the squad’s total points.

Algorithms and tracking

Four agents are implemented against the same environment: DQN, PPO, REINFORCE, and SARSA — value-based and policy-gradient, on-policy and off-policy. Every training run logs episode rewards, loss, squad performance, and budget utilisation to MLflow, and writes reward plots and checkpoints to a timestamped directory.

What I learned

The clearest lesson was that the environment is the experiment. I started by comparing algorithms and spent most of my time fixing the reward function, because an agent will optimise exactly what you wrote down rather than what you meant. My first reward scheme produced agents that hoarded budget and never traded — technically optimal, entirely useless. Reward shaping is where the real modelling happens, and the algorithm choice mattered far less than I assumed going in.

Normalising the state space was the other thing that had to be right before anything else could work. Player prices, points, and form live on wildly different scales, and networks fed raw values simply never converged. Squeezing everything into [0, 1] was not a tidying step; it was the difference between learning and not.

Running four algorithms side by side taught me what the textbook descriptions don’t: the practical cost of on-policy methods that throw away every sample, the variance of REINFORCE without a baseline, and how much of PPO’s reputation comes from being forgiving of imperfect hyperparameters. And using MLflow from the beginning meant I could actually answer “was that better?” instead of trusting my memory of a plot from three days earlier.

Fantasy RL also produced Fantasy API, which started as this project’s data layer and grew into something worth maintaining on its own.