@NeilSlater hI! Do you know any ways to debug an RL algo? For exmaple in ML we generally do gradient checking using small increments in weight and observing its effect on the cost. Similarly if I wanted to implement a Model Free control using MC sampling and GLIE policy improvement how should I go around to make sure there are no mistakes in my implementation?

To those ends you can:

a) Test your algorithm on any of the many toy problems with know solutions or where simpler agents already work well, to verify that the results are the same

b) After finding a policy and converging, use simple Monte Carlo estimates of value functions and compare to the agent's value function.

I've used (b) a fair bit myself to check implementations of DQN on new environments, and verify that action values are accurate.

To get the MC estimates and compare, you can simply run the agent with the target policy (i.e. optimal policy, no exploration for off-policy) in the environment after training, and store all the value predictions and rewards. At the end of each episode - or after long enough that discount factor or horizon make further rewards unimportant - you can use the actual rewards to produce unbiased returns.

Optionally you can average those returns to get the unbiased MC estimate for each state/acton pair.

Or you can just take them as raw data (which will make more sense in large or continuous state spaces).

Then you can compare the predicted values from the agent with the MC values and get MSE.

If the environment is fairly deterministic you should expect a low value

If it is not, then you might want to compare between agents to see whether your new agent is producing reasonable results

@NeilSlater thanks for the insight! Actually as a beginner it is very tough to understand why an agent is behaving the way it is behaving. For example I implemented a modified Gridworld with a step penalty (to ensure speed of agent) and the agent will get a bonus if it reaches a certain co-ordinate and a penalty if it reaches a un-desired co-ordinate.

Both desired and un-desired co-ordinates are terminal, I noticed that is penaty = -reward at desired/un-desired state and at a ceratin step penalty, the agent quickly tries to go to the un-desired state if and only if it is closer to the initial state.

It was fairly unintuitive until I realised since the un-desired state is terminal the agent quickly completes its journey and the bonus at a desired state is not enough to compensate the agent to go to that state.

So I noticed there is actually a lot of importance in the penalties and rewards you assign. Do you know any such resource which discusses these aspects. I recently posted a question similar to this too.

*(Both desired and un-desired co-ordinates are terminal, I noticed that if penalty = --reward at desired and un-desired state, and at a certain step penalty, the agent quickly tries to go to the un-desired state if and only if it is closer to the initial state.

@DennisSoemers do you know some resources which somewhat discusses this problem in a fairly simple way?

Although I know it sounds fairly obvious to people experienced in RL it took me quite a while to get to the problem, so i wanted to take a quick look if some resources are available