Procesamiento de archivos fuente de datos que provienen de las estaciones hidrometeorológicas automáticas

Abstract

This project aims to process data source files that come from automatic hydrometeorological stations of the National Institute of Meteorology and Hydrology of Ecuador. The source files transmitted by INHAMI's automatic stations are received on the server every minute. these files contain all the programmed information by the dataloggers that include on average 60 meteorological parameters. Therefore, this work carried out the development of a program to detect files with errors, based on the rules established by the institution. The main errors that were considered are empty files, with other formats, etc. After discarding the files with errors, the data is processed and stored in a DB. The methodology used for the development of the project was SCRUM. In this way, it was possible to meet the requirements requested by the end users of the program. The tools used for the development of the project include the use of free software such as Python with its Pandas and Numpy libraries, which facilitated the calculation and processing of data. Similarly, PostgreSQL was used for the database. Also through the use of multithreading the results that were obtained by working with it prove to be feasible for this project. According to the results, it was possible to reduce the frequency of arrival of the source files, increasing the reception time to five minutes, thus avoiding server congestion. In addition, it was possible to filter the (.rep) files, discarding files with errors. This makes it easy to clean the information before it is stored in the database.