Publication: Integración y visualización de datos abiertos medioambientales
Loading...
Official URL
Full text at PDC
Publication Date
2016
Authors
Advisors (or tutors)
Editors
Journal Title
Journal ISSN
Volume Title
Publisher
Citations
Exportar
Abstract
En la actualidad, muchos gobiernos están publicando (o tienen la intención de publicar en breve) miles de conjuntos de datos para que personas y organizaciones los puedan utilizar. Como consecuencia, la cantidad de aplicaciones basadas en Open Data está incrementándose. Sin embargo cada gobierno tiene sus propios procedimientos para publicar sus datos, y esto causa una variedad de formatos dado que no existe un estándar internacional para especificar los formatos de estos datos. El objetivo principal de este trabajo es un análisis comparativo de datos ambientales en bases de datos abiertas (Open Data) pertenecientes a distintos gobiernos. Debido a esta variedad de formatos, debemos construir un proceso de integración de datos que sea capaz de unir todos los tipos de formatos. El trabajo implica un pre-procesado, limpieza e integración de las diferentes fuentes de datos.
Existen muchas aplicaciones desarrolladas para dar soporte en el proceso de integración por ejemplo Data Tamer, Data Wrangler como se explica en este documento. El problema con estas aplicaciones es que necesitan la interacción del usuario como parte fundamental del proceso de integración. En este trabajo tratamos de evitar la supervisión humana aprovechando las similitudes de los datasets procedentes de igual área que en nuestro caso se aplica al área de medioambiente. De esta forma los procesos pueden ser automatizados con una programación adecuada.
Para conseguirlo, la idea principal de este trabajo es construir procesos ad hoc adaptados a las fuentes de cada gobierno para conseguir una integración automática. Concretamente este trabajo se enfoca en datos ambientales como lo son la temperatura, consumo de energía, calidad de aire, radiación solar, velocidad del viento, etc. Desde hace dos años el gobierno de Madrid está publicando sus datos relativos a indicadores ambientales en tiempo real. Del mismo modo, otros gobiernos han publicado conjuntos de datos Open Data relativos al medio ambiente (como Andalucía o Bilbao), pero todos estos datos tienen diferentes formatos. En este trabajo se presenta una solución capaz de integrar todas ellos que además permite al usuario visualizar y hacer análisis sobre los datos en tiempo real. Una vez que el proceso de integración está realizado, todos los datos de cada gobierno poseen el mismo formato y se pueden lanzar procesos de análisis de una manera más computacional.
Este trabajo tiene tres partes fundamentales: 1. Estudio de los entornos Open Data y la literatura al respecto; 2. Desarrollo de un proceso de integración y 3. Desarrollo de una Interface Gráfica y Analítica. Aunque en una primera fase se implementaron los procesos de integración mediante Java y Oracle y la Interface Gráfica con Java (jsp), en una fase posterior se realizó toda la implementación con lenguaje R y la interface gráfica mediante sus librerías, principalmente con Shiny. El resultado es una aplicación que provee de un conjunto de Datos Ambientales Integrados en Tiempo Real respecto a dos gobiernos muy diferentes en España, disponible para cualquier desarrollador que desee construir sus propias aplicaciones.
Today, many governments are publishing (or intend to publish soon) thousands of data sets for individuals and organizations can use them. As a result, the number of applications based on Open Data is increasing. However each government has its own procedures for publishing their data and this causes important problems while combining a great variety of formats from different sources due to there is still no an international normalization. The main objective of this paper is a comparative analysis of environmental data in open databases (Open Data) from different governments. Because of this variety of formats, we must build a data integration process that is capable of uniting all types of formats. The work involves a pre-processing, cleaning and integration of different data sources. There are many applications developed to support the integration process such Tamer Data, Data Wrangler as explained in this document. The problem with these applications is that they require user interaction as a fundamental part of the integration process. In this work we try to avoid human supervision taking advantage of the similarities of the datasets from the same area which in our case is applied to the area of environment. Thus the processes can be automated with appropriate programming. To achieve this, the main idea of this work is to build ad hoc processes adapted to each government sources for automatic integration. This paper specifically focuses on environmental data such as temperature, energy consumption, air quality, solar radiation, wind speed, etc. For two years the Madrid government is publishing its data on environmental indicators in real time. Similarly, other governments have published Open Data datasets relating to the environment (such as Andalusia and Bilbao), but all these data have different formats. This paper presents a solution able to integrate all of them and also that allows the user to view and perform analysis on data in real time. Once the integration process is completed, all data will have the same format and then analysis process can be run in a more computational way. This work has three main parts: 1. Study of Open Data environment and literature about that; 2. Development of an integration process and 3. Development of Graphical and Analytical Interface. Although integration processes through Java and Oracle and Java Graphical Interface (jsp) were implemented at a later stage in a first phase the entire implementation was done with R language and graphical interface through its libraries, mainly with Shiny. The result is an application that provides a set of Integrated Environmental Data in Real Time on two different governments in Spain, available to any developer who wants to build their own applications.
Today, many governments are publishing (or intend to publish soon) thousands of data sets for individuals and organizations can use them. As a result, the number of applications based on Open Data is increasing. However each government has its own procedures for publishing their data and this causes important problems while combining a great variety of formats from different sources due to there is still no an international normalization. The main objective of this paper is a comparative analysis of environmental data in open databases (Open Data) from different governments. Because of this variety of formats, we must build a data integration process that is capable of uniting all types of formats. The work involves a pre-processing, cleaning and integration of different data sources. There are many applications developed to support the integration process such Tamer Data, Data Wrangler as explained in this document. The problem with these applications is that they require user interaction as a fundamental part of the integration process. In this work we try to avoid human supervision taking advantage of the similarities of the datasets from the same area which in our case is applied to the area of environment. Thus the processes can be automated with appropriate programming. To achieve this, the main idea of this work is to build ad hoc processes adapted to each government sources for automatic integration. This paper specifically focuses on environmental data such as temperature, energy consumption, air quality, solar radiation, wind speed, etc. For two years the Madrid government is publishing its data on environmental indicators in real time. Similarly, other governments have published Open Data datasets relating to the environment (such as Andalusia and Bilbao), but all these data have different formats. This paper presents a solution able to integrate all of them and also that allows the user to view and perform analysis on data in real time. Once the integration process is completed, all data will have the same format and then analysis process can be run in a more computational way. This work has three main parts: 1. Study of Open Data environment and literature about that; 2. Development of an integration process and 3. Development of Graphical and Analytical Interface. Although integration processes through Java and Oracle and Java Graphical Interface (jsp) were implemented at a later stage in a first phase the entire implementation was done with R language and graphical interface through its libraries, mainly with Shiny. The result is an application that provides a set of Integrated Environmental Data in Real Time on two different governments in Spain, available to any developer who wants to build their own applications.
Description
Máster en Ingeniería Informática, Facultad de Informática, Departamento de Arquitectura de Computadores y Automática, curso 2015-2016