Off-policy Learning with Eligibility Traces: A Survey
type: Dataset
description: In the framework of Markov Decision Processes, off-policy learning, that is the problem of learning a linear approximation of the value function of some fixed policy from one trajectory possibly generated by some other policy. We briefly review on-policy learning algorithms of the literature (gradient-based and least-squares-based), adopting a unified algorithmic view. Then, we highlight a systematic approach for adapting them to off-policy learning with eligibility traces. This leads to some known algorithms - off-policy LSTD(λ), LSPE(λ), TD(λ), TDC/GQ(λ) - and suggests new extensions - off-policy FPKF(λ), BRM(λ), gBRM(λ), GTD2(λ). We describe a comprehensive algorithmic derivation of all algorithms in a recursive and memory-efficent form, discuss their known convergence properties and illustrate their relative empirical behavior on Garnet problems. Our experiments suggest that the most standard algorithms on and off-policy LSTD(λ)/LSPE(λ) - and TD(λ) if the feature space dimension is too large for a least-squares approach - perform the best.
issued: 2026-05-11T13:09:05.759407
language: en
modified: 2026-05-11T13:09:05.759414
contact point: Organization
  fn: CCSD
keyword: eligibility-traces
keyword: infoeu-reposemanticsreport
keyword: infoinfo-aicomputer-science-csartificial-intelligence-csai
keyword: infoinfo-rocomputer-science-csoperations-research-mathoc
keyword: off-policy-learning
keyword: reinforcement-learning
keyword: reports
keyword: value-estimation
landing page: https://inria.hal.science/hal-00644516

Off-policy Learning with Eligibility Traces: A Survey
type: Distribution
format: HTML
issued: 2026-05-11T13:09:05.776494
modified: 2026-05-11T13:09:05.744190
access URL: https://inria.hal.science/hal-00644516

test_moissonnage_selune
type: Agent