Publication: Comparativa de técnicas de análisis de sentimiento en contextos competitivos
Loading...
Official URL
Full text at PDC
Publication Date
2021-07
Authors
Advisors (or tutors)
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Citations
Exportar
Abstract
Cada vez es más común utilizar sistemas automáticos como IBM Watson para tareas tales como detectar la opinión en textos, el llamado análisis de sentimiento. Estos sistemas reemplazan a los modelos específicos en los que se seleccionaba un subconjunto de textos que se etiquetaban manualmente para luego generar un modelo para cada corpus concreto.
En este trabajo mostramos experimentalmente que, a pesar de los avances logrados por estos sistemas automáticos, la generación de modelos con entrenamiento manual sigue resultando útil en ciertas situaciones. Para ello hemos elegido textos de dos tipos de ejemplos: un primer tipo al que llamamos de contexto competitivo que incluye comentarios en redes sociales acerca de acontecimientos como procesos electorales, y un segundo tipo que incluye reseñas online acerca de productos o servicios. Nuestros resultados concluyen que incluso un método de clasificación sencillo basado en el entrenamiento manual puede superar la eficiencia de IBM Watson cuando se consideran ejemplos del primer tipo, mientras que IBM Watson sí supera a este método entrenado manualmente en los casos del segundo tipo.
Además, analizamos varios factores que pueden afectar a los resultados de estos experimentos, descartando la influencia en nuestros ejemplos de aspectos tales como la complejidad del texto o los errores ortográficos. También examinamos el papel de elementos específicos de las redes sociales y en particular de los tweets, tales como hashtags, menciones o enlaces externos, encontrando que estos elementos sirven para transmitir al menos parte del contexto que Watson no puede detectar.
Como contribución adicional que también merece ser comentada, presentamos una nueva métrica basada en la popular Kappa de Cohen a la que hemos llamado Kappa Penalizada. Aunque ya existen variantes de Kappa que buscan penalizar el resultado de Kappa teniendo en cuenta la distancia entre el valor predicho y el real, hemos encontrado que estas propuestas tienen un comportamiento anómalo al ser sus valores a menudo mayores que el valor de la Kappa original. Nuestra propuesta reduce significativamente el número de veces que esto sucede.
Today it is increasingly common to use automatic systems like IBM Watson in order to accomplish tasks like text opinion detection, also called sentiment analysis. These systems replace the specific models where we used to select a subset of texts that were manually labeled for the purpose of generating one model for each particular corpus. In this project we experimentally show that, despite the breakthroughs reached by these automatic systems, model generations with manual training are still useful in some situations. In order to achieve this goal we have chosen two types of examples: the first one that we called competitive context includes social networks commentaries about events like electoral processes, and the second one that includes online reviews about products or services. Our results conclude that even a simple classification method based on manual training can overcome the performance of IBM Watson when examples of the first type are considered, whereas IBM Watson outperforms this manual training method in the second type cases. In addition, we analyzed several factors that can affect the results of these experiments, rejecting the influence of features like text complexity or misspellings in our examples. We also examine the role of specific elements in social networks, especially in tweets, such as hashtags, mentions and external links, discovering that these elements are useful to transmit at least part of the context that Watson cannot detect. As an additional contribution that also deserves to be mentioned, we introduce a new metric based on the well-known metric Cohen’s Kappa named Penalized Kappa. Although there are some Kappa variations that already seek to penalize taking into account the distance between the predicted and the real value, we have found that these proposals can show an anomalous behaviour because their values often are higher than the value of the original Kappa. Our proposal reduces significantly the number of times that these situations happen.
Today it is increasingly common to use automatic systems like IBM Watson in order to accomplish tasks like text opinion detection, also called sentiment analysis. These systems replace the specific models where we used to select a subset of texts that were manually labeled for the purpose of generating one model for each particular corpus. In this project we experimentally show that, despite the breakthroughs reached by these automatic systems, model generations with manual training are still useful in some situations. In order to achieve this goal we have chosen two types of examples: the first one that we called competitive context includes social networks commentaries about events like electoral processes, and the second one that includes online reviews about products or services. Our results conclude that even a simple classification method based on manual training can overcome the performance of IBM Watson when examples of the first type are considered, whereas IBM Watson outperforms this manual training method in the second type cases. In addition, we analyzed several factors that can affect the results of these experiments, rejecting the influence of features like text complexity or misspellings in our examples. We also examine the role of specific elements in social networks, especially in tweets, such as hashtags, mentions and external links, discovering that these elements are useful to transmit at least part of the context that Watson cannot detect. As an additional contribution that also deserves to be mentioned, we introduce a new metric based on the well-known metric Cohen’s Kappa named Penalized Kappa. Although there are some Kappa variations that already seek to penalize taking into account the distance between the predicted and the real value, we have found that these proposals can show an anomalous behaviour because their values often are higher than the value of the original Kappa. Our proposal reduces significantly the number of times that these situations happen.
Description
Trabajo de Fin de Máster en Ingeniería Informática, Facultad de Informática UCM, Departamento de Sistemas Informáticos y Computación, Curso 2020/2021