Publication: Bandidos estocásticos: introducción, algoritmos y experimentos
Loading...
Official URL
Full text at PDC
Publication Date
2022
Authors
Advisors (or tutors)
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Citations
Exportar
Abstract
Encuadrados dentro del aprendizaje por refuerzo, los bandidos constituyen una solución efectiva a las instancias más simples del dilema de la exploración-explotación. Un problema de bandidos consiste en un juego secuencial entre un agente y un entorno en el que en cada ronda el agente dispone de una serie de acciones disponibles y debe escoger una de ellas y recibir, como consecuencia, una cierta recompensa. Basándose en las recompensas recibidas anteriormente, el agente deberá ir perfeccionando sus decisiones de forma que consiga una recompensa acumulada lo mayor posible al final del juego. Para ello, tendrá que mantener un equilibrio entre la exploración de acciones poco probadas y la explotación de la mejor acción según la información de que dispone.
Desde el año 1933, en el que Thompson planteó la necesidad de aplicar un modelo como el de los bandidos a las pruebas clínicas, se han encontrado numerosas aplicaciones de los mismos. El A/B testing y los sistemas recomendadores son algunos de los campos en los que hoy en día el uso de bandidos resulta esencial, mientras que otros como el encaminamiento en redes o la inteligencia artificial en videojuegos son muy prometedores.
El modelo de los bandidos cuenta con numerosas y muy diversas variantes, pero aquí nos centraremos en los bandidos estocásticos. Son aquellos en los que cada acción disponible está asociada a una distribución de probabilidad, desconocida a priori por el agente, de la que se extrae la recompensa correspondiente cuando la acción es seleccionada. El agente tendrá, por tanto, que tratar de estimar la media de estas distribuciones a partir de las recompensas que obtenga de ellas.
En este trabajo se realiza un análisis de los algoritmos de bandidos, principalmente de aquellos pensados para bandidos estocásticos, utilizando una métrica conocida como remordimiento, que evalúa cómo de buenos son los resultados obtenidos en relación con cómo podrían ser. Se estudian, entre otros aspectos, los algoritmos de Explora- Primero y de la Cota Superior de Confianza (UCB), que obtienen buenos resultados al enfrentarse con bandidos estocásticos.
Framed in reinforcement learning, bandits are an effective solution to the most simple instances of the exploration-exploitation dilemma. A bandit problem consists in a sequential game between an agent and an environment in which, in every round, the agent has a set of available actions and must choose one of them and receive, as a result, a certain reward. Based on rewards received previously, the agent should improve its decisions in order to get a cumulative reward as big as possible at the end of the game. For this purpose, it will have to keep a balance between the exploration of actions that have been chosen few times and the exploitation of the best action according to the information it has. Since 1933, year in which Thompson proposed the need to apply a bandit-like model to clinical trials, there have been found a lot of applications for them. A/B testing and recommender systems are just some of the fields where the use of bandits is essential nowadays. Other uses, like routing in networks or artificial intelligence in videogames, are very promising. The bandit model has numerous and very diverse variants, but here we will focus on stochastic bandits. They are those in which every available action is associated to a probability distribution, unknown by the agent a priori, from which the corresponding reward is sampled when the action is selected. The agent must, then, try to estimate the mean of these distributions based on the rewards obtained from them. In this dissertation we make an analysis of bandit algoritms, especially those thought for stochastic bandits, using a metric known as regret, that evaluates how good the outcomes are in relation to how Good they could be. We study, among other aspects, the algorithms Explore-First and Upper Confidence Bound (UCB), which get good outcomes against stochastic bandits.
Framed in reinforcement learning, bandits are an effective solution to the most simple instances of the exploration-exploitation dilemma. A bandit problem consists in a sequential game between an agent and an environment in which, in every round, the agent has a set of available actions and must choose one of them and receive, as a result, a certain reward. Based on rewards received previously, the agent should improve its decisions in order to get a cumulative reward as big as possible at the end of the game. For this purpose, it will have to keep a balance between the exploration of actions that have been chosen few times and the exploitation of the best action according to the information it has. Since 1933, year in which Thompson proposed the need to apply a bandit-like model to clinical trials, there have been found a lot of applications for them. A/B testing and recommender systems are just some of the fields where the use of bandits is essential nowadays. Other uses, like routing in networks or artificial intelligence in videogames, are very promising. The bandit model has numerous and very diverse variants, but here we will focus on stochastic bandits. They are those in which every available action is associated to a probability distribution, unknown by the agent a priori, from which the corresponding reward is sampled when the action is selected. The agent must, then, try to estimate the mean of these distributions based on the rewards obtained from them. In this dissertation we make an analysis of bandit algoritms, especially those thought for stochastic bandits, using a metric known as regret, that evaluates how good the outcomes are in relation to how Good they could be. We study, among other aspects, the algorithms Explore-First and Upper Confidence Bound (UCB), which get good outcomes against stochastic bandits.
Description
Trabajo de Fin de Doble grado en Ingeniería informática y Matemáticas, Facultad de Informática UCM, Departamento de Sistemas Informáticos y Computación, Curso 2021/2022.