Artin Kavoosi
← Back to all posts
Technical Note
September 2025MDP series · Part 2/2

On average reward criterion

Abstract

At first view, the average-reward Markov Decision Process (MDP) feels like the natural choice, especially next to discounted formulations that often need extra real-world justification. In practice, though, the average-criterion approach hasn’t become as popular: subtle technical assumptions, multichain behavior, and delicate limit arguments make it less straightforward than it appears.

Markov Decision Processes Average Reward ~1 min read
Contents

    At first view, the average-reward Markov Decision Process (MDP) feels like the natural choice, especially next to discounted formulations that often need extra real-world justification. In practice, though, the average-criterion approach hasn’t become as popular: subtle technical assumptions, multichain behavior, and delicate limit arguments make it less straightforward than it appears.

    Last year I took Stochastic Dynamic Programming with Prof. Mohammad Yazdi at Sharif U, where we worked through Professor Ross’s book. Prof. Yazdi’s mentorship was the backbone of the course; he pushed us to be precise, patiently untangled murky arguments, and consistently turned intuition into rigorous proofs. That course sharpened my appreciation for why working through the technical details is worth the effort.

    I’ve put together a short PDF here, where I collect some justifications from the literature on the average-criterion MDP. If you’re interested in MDPs, you might find it useful.

    Attachments

    Proof

    Cite this note

    Kavoosi, A. (2025). “On average reward criterion.” Research notes. https://akavosi.github.io/blog/posts/acmdp.html

    @misc{kavoosi2025acmdp,
  author = {Kavoosi, Artin},
  title  = {On average reward criterion},
  year   = {2025},
  note   = {Personal research notes, informal, not peer-reviewed},
  url    = {https://akavosi.github.io/blog/posts/acmdp.html}
}

    Informal research notes — not peer-reviewed.

    Continue reading

    Seminar Note · MDP series, Part 1/2 An introduction to MDP Research Note Stochastic Queens Elimination