A new perspective for longitudinal measurement and analysis of public education in Brazil based on open data and machine learning

In the mid-2000s, e-government began to be seen as a facilitator of transformation in the public sector, promoting transparency, reducing corruption, and increasing social engagement. One of the first steps towards this is providing access to information for the population. Additionally, the popularization of access to raw data from large-scale assessments has enabled analyses that promote public value in the education sector. Many Brazilian studies use large-scale assessments from high school to predict student or school performance, hindering the use of these analyses at the state and municipal levels. In this context, the objective of this study is to analyze the relationship between educational, economic, and social well-being characteristics of municipalities and their respective performance in IDEB for elementary education to support the decision-making process in the development of public polices for elementary education by public resource managers. For this, we used a method based on CRISP-DM. We began by understanding the business, collecting data from open data portals of the Brazilian government and the Human Development Atlas in Brazil. Among these data were HDI, educational investments, and educational indicators of municipalities. We then cleaned the data and proceeded to perform analyses using Spearman’s coefficient and prediction of IDEB-E and IDEB-L indicators using machine learning algorithms such as Linear Regression, Random Forest, and Artificial Neural Network. Finally, we conducted analyses of the important features identified by the most accurate model. As main results, we found a high positive correlation between IDEB and social well-being variables, as well as the level of education of teachers. The methodology and results of this paper may lead to a new perspective of analyzing elementary education at the municipality level by employing both statistical and machine learning models, supporting the public bodies in their decisions regarding investments.