[I am re-sending this to the class mailing list for the benefit of anyone who didn't get it on Monday.]
For the discussion on MDPs, you can start reading Russell and Norvig Chapter 17.1--17.3 in the 3rd edition.
(You might also be interested in the monograph
This is free to download if you are signed into ASU network)
Rao