Posts 13. Advanced Actor-Critic(A2C)
Post
Cancel

13. Advanced Actor-Critic(A2C)

May 6, 20212021-05-07T02:20:00+09:00 by Jungi Lee
2 min

목차

  1. Advanced Actor Critic
  2. Algorithm
  3. Reference

Advanced Actor Critic

∇θJθ≃∑∞t=0∫st,at∇θlnPθ(at|st)Q(st,at)Pθ(st,at)dst,at

위 식에서 Q 대신 action의 함수가 아닌 b(st)를 넣어보자

∑∞t=0∫st,at∇θlnPθ(at|st)b(st)Pθ(st,at)dst,at =∑∞t=0∫st,at∇θlnPθ(at|st)b(st)Pθ(at|st)P(st)dst,at =∑∞t=0∫st,at∇θPθ(at|st)Pθ(at|st)b(st)Pθ(at|st)P(st)dst,at =∑∞t=0∫st,at∇θPθ(at|st)b(st)P(st)dst,at =∑∞t=0∇θ∫st,atPθ(at|st)b(st)P(st)dst,at =∑∞t=0∫st∇θ∫atPθ(at|st)datb(st)P(st)dst =∑∞t=0∫st∇θ1b(st)P(st)dst=0

즉, b(st)가 Q에 대한 함수가 아니면 0 이기 때문에 Q-b=Q가 성립한다.

  • V[X] = V[X-a] but V[X[Q]] ≠ V[X[Q-V]]

  • Q-V는 optimal은 아니나 variance가 낮아진다.

  • Advantage=Q(st,at)−V(st)
∇θJθ≃∑∞t=0Est,at[∇θlnPθ(at|st)(Q−V)]

Q,V actor 3개의 network를 사용해야하므로 Q를 V로 표현

Q(st,at)=Est+1[Rt+γV(st+1)|st,at]

E[X] -a = E[X-a] (a= 상수)

Q−V=Est+1[Rt+γV(st+1)−V(st)|st,at] ∇θJθ≃∑∞t=0Est,at[∇θlnPθ(at|st)(Q−V)] =∑∞t=0Est,at[∇θlnPθ(at|st)Est+1[Rt+γV(st+1)−V(st)|st,at]] =∑∞t=0∫st,at∇θlnPθ(at|st)∫st+1(Rt+γV(st+1)−V(st))P(st+1|st,at)dst+1Pθ(st,at)dst,at =∫st,at,st+1∑∞t=0∇θlnPθ(at|st)(Rt+γV(st+1)−V(st))Pθ(st)Pθ(at|st)P(st+1|st,at)dst,at,st+1

marginalization으로 인하여 Pθ(st)가 생겼으므로 원래대로 돌이키면 P(st|st+1,at+1)를 써도 될 것같다.

=∫st,at,st+1∑∞t=0∇θlnPθ(at|st)(Rt+γV(st+1)−V(st))P(st|st+1,at+1)Pθ(at|st)P(st+1|st,at)dst,at,st+1

Algorithm

  1. Initialize θ,w

  2. Collect N samples ( 1 sample: {si,ai,si+1})

  3. Actor update: θ←θ+α∑ti=t−N+1∇θlnPθ(at|st)(Ri+γVw(si+1)−Vw(si))

  4. Critic update: w←w−β∇w∑ti=t−N+1(Ri+γVw(si+1)−Vw(si))2

  5. Clear Batch

  6. repeat 1 ~ 4

Reference

  1. 혁펜하임 유투브
Reinforcement Learning, Single-Agent
RL Single-Agent
This post is licensed under CC BY 4.0 by the author.
Recent Update

13. Actor-Critic

14. (Mnih 2016 ICML) Asynchronous Methods for Deep Reinforcement Learning