Chapter 10 - Advantage computed the wrong way?

[ghost]

https://ai.stackexchange.com/questions/21172/advantage-computed-the-wrong-way

ipdb> adv_v.shape                                                                                                                            
torch.Size([128, 128])

ipdb> vals_ref_v.shape                                                                                                                       
torch.Size([128])

ipdb> values_v.detach().shape                                                                                                                
torch.Size([128, 1]) 

[ghost]

I changed the line adv_v = vals_ref_v - value_v.detach() to adv_v = vals_ref_v - value_v.squeeze(-1).detach(). It seems the convergence is much faster. According to the A2C algorithm, it is just logic to apply Q(a, s) - V(s) where Q(a, s) and V(s) with the same shape.

The call to detach() is important here as we don't want to propagate the PG into our value approximation head.

[ghost]