Seminar Series - Bayesian Reinforcement Learning With Censored Observations and Applications in Dynamic Pricing

Speaker: Sanmay Das, Department of Computer Science, Rensselaer Polytechnic Institute
Date: Tuesday, February 21, 2012
Time: 11:00am-12:00pm
Location: 1110 KnowledgeWorks II

In many real-world applications, a decision-maker receives censored information from the world. A censored observation is one where the value of a measurement is only partially known -- for example, whether or not a buyer values an item enough to purchase it at a particular price, or whether or not a particular drug dosage level is toxic. The decision-maker often controls the threshold at which censoring occurs (the seller sets a price; the drug designer sets a dosage level), setting up a dynamic decision problem with an exploration-exploitation dilemma: where to set the threshold in order to trade off the information gained and the actual value obtained from the interaction. Unfortunately, the censored nature of the observations typically makes exact inference intractable. I will describe a Bayesian reinforcement learning algorithm for such problems that uses moment-matching approximations to collapse a decision-maker's belief state into a tractable form. The algorithm can be applied in many domains, including dynamic pricing of digital goods and liquidity provision in prediction markets. I will present data from prediction market experiments with both human subjects and artificial trading agents showing that a Bayesian market maker based on the moment-matching algorithm outperforms standard cost-function based approaches. This talk is based on joint work with Aseem Brahma, Mithun Chakraborty, Meenal Chhabra, Allen Lavoie, Malik Magdon-Ismail, and Yonatan Naamad.

Sanmay Das is an Assistant Professor at Rensselaer Polytechnic Institute. His research interests are in machine learning and computational social science. He received an NSF CAREER award in 2010. He has served as program chair of AMMA and workshops chair of ACM EC, in addition to serving on the program committees of many conferences in artificial intelligence and machine learning. Sanmay received his Ph.D. in Computer Science from the Massachusetts Institute of Technology in 2006.