Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

projection algorithm

The projection algorithm is an iterative apprenticeship learning method designed to learn an agent policy that matches the performance of an expert demonstrator in a Markov decision process without requiring prior knowledge of the underlying reward function. Operating under the assumption that the unknown reward is expressible as a linear combination of known state features, the algorithm works directly in the space of expected feature counts. In each iteration, it projects the expert feature expectations onto the convex set of feature expectations generated by previously evaluated policies to determine an orthogonal weight vector that serves as a candidate reward function. It then computes an optimal policy for that candidate reward via standard reinforcement learning, calculates the new policy feature expectations, and updates the projection. This cycle repeats until the distance between the expert feature expectations and the projected policy mixture drops below a specified threshold, guaranteeing that the resulting policy achieves performance close to that of the expert under the true reward.

1 item

Apprenticeship learning via inverse reinforcement learning

Apprenticeship learning via inverse reinforcement learning

Pieter Abbeel, Andrew Y. Ng

OrganizationsStanford University

Why you should read this

Develops an apprenticeship learning algorithm based on inverse reinforcement learning that guarantees performance matching an expert demonstrator on an unknown reward function by matching feature expectations, backed by rigorous iteration and sample complexity bounds.

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using “inverse reinforcement learning” to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert’s reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert’s unknown reward function.

Added

2026-09-11