Publication: Generadores escalables en Apache Flink: implementación y aplicaciones en testing y pruebas de carga
Loading...
Official URL
Full text at PDC
Publication Date
2020
Authors
Advisors (or tutors)
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Citations
Exportar
Abstract
Debido al auge del procesamiento masivo de datos y al surgimiento de motores capaces de realizar esta tarea, se ha visto la necesidad de cambiar la manera en la que se testean las aplicaciones que se usan para dichos motores. La comprobación mediante test unitarios es insuficiente para poder probar la funcionalidad de un programa que manipula millones de datos, sin tener que dedicar una gran cantidad de tiempo para implementar dichos test. Existe una metodología llamada Property-based testing (PBT), derivada del Random testing que genera entradas de datos de manera aleatoria, pudiendo así crear test genéricos. Actualmente, se podría considerar que algunos de los motores más usados son Apache Spark y Apache Flink, pudiendo ambos procesar tanto flujos continuos de datos, como datos en lotes. Apache Spark ya cuenta con aplicaciones basadas en PBT para ambos tipos de procesamiento, mientras que Apache Flink solo tiene bibliotecas para usar PBT en flujos de datos. Así pues, el objetivo principal de este proyecto es desarrollar una herramienta que permita usar PBT en grandes cantidades de datos procesadas en lotes, teniendo en cuenta que dicha herramienta debe de implementar algunas de las características principales de los motores de procesamiento masivo de datos (escalabilidad, distribución y tolerancia a fallos).
Due to the massive data processing boom and the rise of engines capable of executing this task, there is a need of changing how applications have been tested in these engines. Verification using unitary tests is insufficient to test the functionality of a program that deal with millions of data, without spending too much time in the process. There exists a methodology, born from Random testing, called Propertybased testing (PBT) which allows the programmer to create random input data, so that generic tests can be created. Nowadays, some of the most relevant engines are Apache Spark and Apache Flink, being both of them capable of doing stream data processing and batch data processing. Apache Spark has libraries based on PBT for both types of processing, whereas Flink only has libraries for stream processing. Therefore the main purpose of this project is to create a tool that allows the implementation of PBT in programs that process a large amount of data in batches, considering some of the main features of massive data processing engines (scalability, distribution, and fault-tolerance).
Due to the massive data processing boom and the rise of engines capable of executing this task, there is a need of changing how applications have been tested in these engines. Verification using unitary tests is insufficient to test the functionality of a program that deal with millions of data, without spending too much time in the process. There exists a methodology, born from Random testing, called Propertybased testing (PBT) which allows the programmer to create random input data, so that generic tests can be created. Nowadays, some of the most relevant engines are Apache Spark and Apache Flink, being both of them capable of doing stream data processing and batch data processing. Apache Spark has libraries based on PBT for both types of processing, whereas Flink only has libraries for stream processing. Therefore the main purpose of this project is to create a tool that allows the implementation of PBT in programs that process a large amount of data in batches, considering some of the main features of massive data processing engines (scalability, distribution, and fault-tolerance).
Description
Trabajo de Fin de Máster en Máster en Ingeniería Informática, Facultad de Informática UCM, Departamento de Sistemas Informáticos y Computación, Curso 2019/2020