Introduction to Multi-Armed Bandits

This online textbook provides a comprehensive introduction of Multi-Armed Bandits (MAB), covering theoretical foundations, algorithm design, and advanced applications in sequential decision-making under uncertainty. Written by Dr. Fangli Ying (ECUST) for Teaching Fellow in a Summer Camp with Prof. Osman Yağan (CMU), it progresses from stochastic bandit fundamentals (UCB, ETC algorithms) to Bayesian methods (Thompson Sampling), structured bandits with hidden parameters, and adversarial settings, featuring rigorous mathematical analysis, regret bounds, and real-world case studies in recommendation systems, wireless communications, and healthcare.

• 1. Introduction to Multi-Armed Bandits
• 2.Stochastic Multi-Armed Bandits
• 3. Explore-then-Commit (ETC) & Upper-Confidence-Bound (UCB)*
• 4. Principles and Performance Comparison of ETC and UCB
• 5. Multi-Armed Bandits with Probing
• 6. Thompson Sampling
• 7. A Unified Approach to Translate Classic Bandit Algorithms to the Structured Bandit Setting
• 8. Full Feedback and Adversarial Costs & Adversarial Bandits