Lecture Notes for CSE571 (F09)

1. Introduction
   • Audio of [Aug 24, 2009] Course overview. Contains a "review" of 471 topics that I expect students to know.
2. Pendulum Swings and current trends in AI
   • Audio of [Aug 26, 2009] Long discussion on current trends and pendulum swings in AI
3. Beyond Classical Search (Non-deterministic; Partially Observable)
   • Audio of [Aug 31, 2009] Search with non-deterministic actions and search in belief space.
   • Audio of [Sep 2, 2009] Belief-space search; propositional representations for belief states (CNF, DNF and BDD models) observation models; effect of observation actions on search and execution; state estimation problems (and how they become more interesting in the case of stochastic transition models).
4. Online Search (which ends beyond classical search); Belief-space planning
   • Audio of [Sep 9, 2009] Online Search--motivations, methods; model incompleteness considerations; need for ergodicity of the environment. Connections to Reinforcement Learning.
   • Audio of [Sep 11, 2009] Issues of Conformant and conditional planners searching in belief space.
5. Heuirstics for Belief Search
   • Audio of [Sep 14, 2009] Heuristics for belief-space search.
6. MDPs
   • Audio of [Sep 16, 2009] Markov Decision Processes
   • Audio of [Sep 27, 2009] MDPS continued
7. Efficient/Approximate approaches for MDP solving
   • Types of MDPs (for planning); RTDP
   • Probabilistic Planning Competition
   • FF-Hop (and FF-replan etc)
     • Audio of [Sep 29, 2009] Special cases of MDPs; Classes of efficient methods for MDPs
     • Audio of [Oct 2, 2009] LRTDP; FF-Hop
8. Heuristics for Stochastic Planning
9. Reinforcement Learning
   • Audio of [Oct 5, 2009] (part 1) Use of heuristics in Stochastic Planning (part 2) Reinforcement learning start (Montecarlo and Adaptive DP; Exploration/Exploitation)
   • Audio of [Oct 7, 2009] Planning--Acting--Learning cycle in the Reinforcement Learning terminology, the role of (and the difference between) simulator and model. Temporal difference learning; Generalizing TD to TD(k-step) and then to TD(\lambda) learning. Q-learning. Exploration policies for Q-learning.
   • Audio of [Oct 12, 2009] Revisiting TD(Lambda); Exploration strategies for Q-learning (that make less visited states look better); Spectrum of atomic RL strategies and their dimensions of variation (in terms of DP-based vs. Sample based and exhaustive vs. 1 step look ahead. Start of factored RL models--and the advantages of representing value and policy functions in a factored fashion. Basic idea of function-approximation techniques for RL.
   • Audio of [Oct 14, 2009]TD-learning and Q-learning with function approximation. Policy gradient search.
10. Decision Theory & Preference Handling
    • Audio of [Oct 19, 2009] Start of discussion of decision/utility theory (R&N Chap 16)
    • Audio of [Oct 21, 2009] Multi-attribute utility theory; discussion of Preference handling tutorial.
    • Audio of [Oct 26, 2009] Preference handling: Partial-ordering preferences; CP-nets; Preference compiliation.
11. Temporal Probabilistic Models
    • Audio of [Oct 28, 2009] Connecting Dynamic Bayes Networks chapter to "State Estimation" (from first part of the semester) and "Relational models" (from the part yet to come); Specifying DBNs; Types of queries on DBNs.
    • Audio of [Nov 2, 2009] Discussion of exact inference based on simultaneous roll-out and roll-up in dynamic bayes nets; motivation of kalman filters from the point of view of specifyingthe parameterized distribution for continuous variables.
    • Audio of [Nov 4, 2009] Discussion of particle filtering techniques for dynamic bayes networks; discussion of factored action representation methods for stochastic planning
12. Statistical Learning
    • Audio of [Nov 16, 2009] Foundations of statistical learning: Full bayesian; MAP and ML--and the tradeoffs. The importance of i.i.d. assumption, the importance of hypothesis prior.
    • Audio of [Nov 18, 2009] Density estimation; bias-variance tradeoff; generative vs. discriminative learning; taxonomy of learning tasks.
    • Audio of [Nov 23, 2009] ML estimation of parameters with complete data in bayes networks; understanding when and why parameter estimation decouples into separate problems. Incomplete data problem. The database example. The hidden variable problem--why would we focus on the hidden variable rather than learn from complete data (because we can reduce the number of parameters exponentially)
    • Audio of [Nov 25, 2009] Expectation Maximization--and why it works. Variants of EM. Connections between EM and other function optimization algorithms (such as gradient descent; newton-raphson)
13. Inference and Learning in Markov Nets (undirected graphical models)+ may be Markov Logic Nets
    • Audio of [Nov 30, 2009] Bayesian Learning for bayes nets. Conjugate priors and their use. Bayesian prediction and how that explains the rationale behind the laplacian correction.

      Start of Markov nets--undirected graphical models. How they differ from bayes nets--easier independence condition; more straightforward definition of bayes nets. Specification of markov nets in terms of clique potentials.

    • Audio of [Dec 2, 2009]Markov Networks: Expressiveness; Parameterization (product form; log-linear); Semantics; inference techniques; learning (generative case)--the need for gradient ascent; the need for inference in gradient computation
    • Audio of [Dec 7, 2009]Class discussion on markov logic networks that touches on topics such as (a) is relational learning useful if we still do ground level inference? (b) the fact that learning is always easier in MLNs than MNs--and that all the MLN ideas/challenges for learning are basically holdovers from MNs (c) the tradeoffs of lifted inference (and how planning has basically abandoned lifted planning--while probabilistic models are going that direction!) a

Classes to come

1. 12/14 (final): Project presentations