Programa de Pós-Graduação em Administração Doutorado em Administração
Two essays on improving quality of primary
education in Brazil: the effect on school
performance from the projection of school
infrastructure and the impact of thePrograma
Mais Educação
Marcelo Victor Alves Bila Queiroz
Natal-RN Julho de 2019
in Brazil: the effect on school performance from the
projection of school infrastructure and the impact of
thePrograma Mais Educação
Defesa de Doutorado apresentada ao Pro-grama de Pós-Graduação em Administração do Departamento de Administração da Uni-versidade Federal do Rio Grande do Norte como requisito parcial para a obtenção do grau de Doutor em Administração.
Linha de pesquisa: Gestão Organizacional
Orientador
Dr. Luciano Menezes Bezerra Sampaio
PPGA – Programa de Pós-Graduação em Administração CCSA – Centro de Ciências Sociais Aplicadas UFRN – Universidade Federal do Rio Grande do Norte
Natal-RN Julho de 2019
cation in Brazil: the effect on school performance from the projection of school infrastructure and the impact of thePrograma Mais Educação apresentada por Marcelo Victor Alves Bila Queiroz e aceita pelo Programa de Pós-Graduação em Admin-istração da Universidade Federal do Rio Grande do Norte, sendo aprovada por todos os membros da banca examinadora abaixo especificada:
Dr. Luciano Menezes Bezerra Sampaio
Presidente
PPGA – Programa de Pós-Graduação em Administração UFRN – Universidade Federal do Rio Grande do Norte
Dra. Raquel Menezes Bezerra Sampaio
Coorientadora
PPGA – Programa de Pós-Graduação em Administração UFRN – Universidade Federal do Rio Grande do Norte
Aléssio Tony Cavalcanti de Almeida
Examinador externo
PPGE – Programa de Pós-Graduação em Economia UFPB – Universidade Federal da Paraíba
Ana Cláudia Annegues da Silva
Examinador externo
PPECO – Programa de Pós-Graduação em Economia UFRN – Universidade Federal do Rio Grande do Norte
PCE – Programa de Pós-graduação em Ciências Econômicas UEM – Universidade Estadual do Maringá
Anne Emília Costa Carvalho
Examinador interno
PPGA – Programa de Pós-Graduação em Administração UFRN – Universidade Federal do Rio Grande do Norte
I have no words to thank everyone on my journey until here. These will be a few words that represent only the tip of the iceberg of what I should be thanking. As this is an academic work, it is obvious that my mentors, Luciano Sampaio and Raquel Sampaio, are the main responsible for this. The positive and interesting part that you find in the thesis surely passed through their hands, the parts that need improvement are my responsibility because I am not always at the level of their insights. I thank Luciano for the friendship and guidance during all those 8 years in both academic and advices for life, I consider myself fortunate to have such a close friend in which I have shared countless moments. I consider Raquel to be THE model professor in all aspects. She elucidated my doubts and "unlocked" the progress of this work in each conversation, always with objective comments and attention in the smallest details. I could not get any further because of my legs, but their hands were always extended to me. I am also grateful to all the other true teachers that I have had throughout my life, since education is a cumulative process, especially Sérgio Trindade who was my teacher for 7 years straight.
To my friends of "Base 10" for sharing this entire academic process with me, because alone it is much more difficult, especially, Felipão and Anne for the support and always calling and making me laugh. And to my all old friends who are several to list.
I thank my parents for giving me the opportunity to be the first graduated family member. My father for being a tireless worker and my mother for having an exemplary mind and dedication. To my brother, thank you for your patience for enduring me in my ups and downs, I hope one day you will follow my example. Besides, I have two other mothers who are my grandmother and my aunt who have raised me from childhood until now, not everyone is so lucky like me. I thank my love Vivi who are my perfect woman, despite all the problems, and laugh at my dull jokes all these years.
I dedicate to my grandmother Jandira who unfortunately is no longer among us to see me this day, as well my cousin Gustavo and my friend Leonardo who have left us in the last four years. I miss you everyday.
Although this may seem a paradox, all exact science is dominated by the idea of approximation. When a man tells you that he knows the exact truth about anything, you are safe in inferring that he is an inexact man. Every careful measurement in science is always given with the probable error ... every observer admits that he is likely wrong, and knows about how much wrong he is likely to be.
primário no Brasil: o efeito sobre o desempenho escolar
a partir da projeção da infraestrutura escolar e o
impacto do Programa Mais Educação
Autor: Marcelo Victor Alves Bila Queiroz Orientadores: Dr. Luciano sampaio e Dra. Raquel Sampaio
Resumo
A qualidade do ensino público brasileiro está aproximadamente no mesmo patamar desde os anos 2000. A estagnação faz com que a lacuna educacional entre o Brasil e os países de-senvolvidos, e até mesmo outros países da América Latina, torne-se cada vez maior. Além disso, o país convive com grandes disparidades socioeconômicas regionais e no provimento de recursos educacionais, isto, por sua vez, pode estar associado à diferença existente na qualidade educacional. Assim, o Estado vem buscando diversas formas de melhorar o de-sempenho estudantil, com destaque nos últimos anos para políticas de educação integral .Tendo em vista este cenário, esta tese está dividida em duas partes que tem como objeto o ensino fundamental público brasileiro, a primeira consiste em diagnosticar o nível de efi-ciência por meio de um modelo dinâmico, levando em consideração as diferenças escolares nos níveis socioeconômicos de seus alunos. A modelagem dinâmica é possibilitada pela inclusão de um índice de infraestrutura como variável que faz a ligação entre os períodos. Foram elaboradas três especificações do modelo que diferem no tratamento da variável socioeconômica. Os resultados mostraram que as distribuições de eficiência desses mode-los não são estatisticamente iguais, mas as conclusões gerais são convergentes. Não houve quase nenhuma evolução na eficiência escolar entre 2007 e 2015, mas indicam uma pos-sível melhoria de eficiência pelo investimento em infraestrutura escolar. A segunda parte da tese avalia o Programa Mais Educação (PME) criado pelo Governo federal com o intu-ito de melhorar a educação do nível fundamental. Esta política busca efetivar o modelo de escola em tempo integral, com no mínimo sete horas diárias, por meio da implementação de atividades socioeducativas extras no contraturno escolar das escolas públicas. O obje-tivo é estimar impacto do Programa Mais Educação no desempenho acadêmico médio em
beneficiadas. Para isto, propõe-se a utilização do método propense score ponderado o qual procura estabelecer um grupo de controle adequado para se comparar com o grupo das escolas participantes do programa, visto que o programa é destinado, sobretudo, às escolas mais vulneráveis. Em seguida, sugere-se a estimação do impacto da atuação do programa entre os anos de 2007 e 2017 por meio do método diferenças em diferenças. As evidências obtidas indicam efeitos negativo entre 2009 e 2011, sem significância estatística em 2013 e efeitos positivo em 2015 e 2017. O custo-efetividade do programa é em média 0.2 pontos na Prova Brasil em português e matemática e 0.4%-0.8% na taxa de aprovação para cada dez mil reais investidos na escola. Por fim, o programa possui efeito positivo na eficiência escolar em todos os modelos específicados.
Palavras-chave: Eficiência escolar, Infraestrutura escolar, Avaliação de Impacto, Programa Mais Educação, Escola de Educação Integral
in Brazil: the effect on school performance from the
projection of school infrastructure and the impact of
thePrograma Mais Educação
Author: Marcelo Victor Alves Bila Queiroz Supervisors: Dr. Luciano Sampaio and Dr. Raquel Sampaio
Abstract
The quality of Brazilian public education has been around the same level since the 2000s. This stagnation means that the educational gap between Brazil and the developed coun-tries, and even other Latin American councoun-tries, is increasing. In addition, the country coexists with large regional socioeconomic disparities and provision of educational re-sources, which in turn may be associated with the difference in educational quality. Thus, the Brazilian government has been seeking different ways to improve student performance, with emphasis, in recent years, in policies of full time education. Given this scenario, this thesis is divided in two parts that aims at Brazilian public primary education. The first is to assess the level of efficiency, using a dynamic model, taking into account the differences in the socioeconomic levels of its students. This model includes an infrastructure index as a variable that links each period of time. We used three model’s specifications that differ in the usage of the socioeconomic variable. The results showed that the efficiency distributions of these models are not statistically equal, but the general conclusions are convergent. There was almost no evolution in school efficiency between 2007 and 2015, but our model indicate a possible way to improvement the efficiency by investing in school in-frastructure. The second part of the thesis evaluates the Programa Mais Educação (PME) created by the Brazilian Federal Government with the aim of improving education at the fundamental level. This policy seeks to implement the model of school full-time, with at least seven hours a day, through the implementation of extra socio-educational activities in the public school second-shift. Our objective is to estimate the impact of the program on the average academic performance in Portuguese, mathematics and approval rate, in
propensity score method, which seeks to establish an adequate control group to compare with the group of treated schools, since the program is aimed mainly at the most vulner-able schools. Then, we estimated the impact of the program’s performance between 2007 and 2017 through the differences-in-differences method. The evidence obtained points to negative effects between 2009 and 2011, no statistical significance in 2013 and positive effects in 2015 and 2017. The cost-effectiveness of the program is on average 0.2 points in Portuguese and Mathematics and 0.4 % - 0.8 % in the approval rate for every ten thousand reais invested in the school. Finally, the program has a positive effect on school efficiency in all models.
Keywords: School Efficiency, School infrastructure, Impact Assessment, Programa Mais Educação, Full time School
1 Categorization of the different ways of using time in school culminating
in learning time. . . p. 54
2 Impact of Programa Mais Educação . . . p. 68
3 Propense score bias reduction for 2007-2009 period. . . p. 108
4 Propense score bias reduction for 2007-2011 period. . . p. 109
5 Propense score bias reduction for 2007-2013 period. . . p. 109
6 Propense score bias reduction for 2007-2015 period. . . p. 110
7 Propense score bias reduction for 2007-2017 period. . . p. 110
8 Common support between treatment and control groups for 2007-2009
period. . . p. 111
9 Common support between treatment and control groups for 2007-2011
period. . . p. 111
10 Common support between treatment and control groups for 2007-2013
period. . . p. 112
11 Common support between treatment and control groups for 2007-2015
period. . . p. 112
12 Common support between treatment and control groups for 2007-2017
1 Variables used in the DEA models . . . p. 32
2 Population and sample comparison by year . . . p. 32
3 DEA variables means and standard error for the 6025 sample schools . p. 33
4 Variable differences between Prova Brasil and Sample Schools . . . p. 35
5 Average efficiency models . . . p. 36
6 Anova - Pairwise comparisons of means with equal variances . . . p. 36
7 School efficiency by socioeconomic level and model . . . p. 37
8 Efficiency and socioeconomic differences by region . . . p. 38
9 ANOVA and Bartlett tests by region and socioeconomic level . . . p. 38
10 Definition of variables used in the second stage analysis. . . p. 39
11 Bootstrap Truncated Regression on efficiency scores by DEA model. . . p. 41
12 Comparing number of observed schools and their average indices of infras-tructure in the “advanced” infrasinfras-tructure category with those obtained
by the DEA projection. . . p. 42
13 Carroll’s (1963) variables description . . . p. 51
14 Summary of previous evaluations of Programa Mais Educação . . . p. 65
15 Alignment of the objectives with the methodologies. . . p. 67
16 Propensity score variables . . . p. 81
17 Estimates for the propensity scores for participation in the program. . . p. 84
18 Balance between treatment and control groups means using propensity
score. . . p. 85
21 Impact of PME’s previous evaluations. . . p. 87
22 Average PME’s cost-effectiveness for each outcome variable in 2017. . . p. 89
23 Impact of the Programa Mais Educação on non-discretionary efficiency
scores using panel data with fixed effects. . . p. 91
24 Impact of the PME on two-stage efficiency score using panel data with
fixed effects. . . p. 92 25 Year 2007 . . . p. 103 26 Year 2009 . . . p. 104 27 Year 2011 . . . p. 105 28 Year 2013 . . . p. 106 29 Year 2015 . . . p. 107 30 Balacing for 2007-2009 . . . p. 114 31 Balacing for 2007-2011 . . . p. 115 32 Balacing for 2007-2013 . . . p. 116 33 Balacing for 2007-2015 . . . p. 117 34 Balacing for 2007-2017 . . . p. 118 35 AIPW results 2007-2009. . . p. 119 36 AIPW results 2007-2011. . . p. 120 37 AIPW results 2007-2013. . . p. 121 38 AIPW results 2007-2015. . . p. 122 39 AIPW results 2007-2017. . . p. 123
1 Introduction p. 17
2 Essay 1 - Dynamic efficiency of primary education in Brazil:
socioe-conomic and infrastructure influence on school performance. p. 20
2.1 Introduction . . . p. 20
2.2 Literature review . . . p. 23
2.3 Methodology . . . p. 26
2.3.1 Construction of socioeconomic and infrastructure indices through
Item Response Theory (IRT) . . . p. 26
2.3.2 Dynamic non-discretionary DEA model . . . p. 27
2.4 Database and Results . . . p. 30
2.4.1 Global and yearly efficiencies . . . p. 34
2.4.2 Efficiency by socioeconomic level and region . . . p. 37
2.4.3 Explaining efficiency scores . . . p. 38
2.4.4 Projection of the infrastructure index . . . p. 42
2.5 Concluding remarks . . . p. 43
3 Essay 2 - Does more time at school mean better education? The effects of the Programa Mais Educação on academic performance
and school violence. p. 45
3.1 Introduction . . . p. 45
3.1.1 General objective . . . p. 47
3.2.1 Full-time education . . . p. 49
3.2.1.1 Full time schooling (integral education) theoretical
per-spective . . . p. 50
3.2.2 Empirical evidence of full time education . . . p. 57
3.2.3 School Violence . . . p. 59
3.2.4 Programa Mais Educação . . . p. 61
3.2.5 Previous Empirical Evidence of PME . . . p. 63
3.3 Methodology . . . p. 67 3.3.1 Theory of Change . . . p. 67 3.3.2 Treatment effects . . . p. 69 3.3.3 Propensity Score . . . p. 70 3.3.3.1 Balance diagnosis . . . p. 76 3.3.4 Differences in differences . . . p. 77
3.3.5 Item Response Theory . . . p. 78
3.4 Data and Variables . . . p. 80
3.5 Empirical approach . . . p. 81
3.6 Results . . . p. 83
3.6.1 Balance with propensity score . . . p. 83
3.6.2 Balance with AIPW . . . p. 84
3.6.3 Impact of Programa Mais Educação . . . p. 85
3.6.4 Cost-effectiveness of Programa Mais Educação . . . p. 88
3.6.5 Effect of Programa Mais Educação in the school efficiency . . . p. 89
4 Final remarks p. 93
Appendix B -- Essay 2 p. 108
B.1 Balancing using Propense Score . . . p. 108
B.2 Common support . . . p. 108
B.3 AIPW balancing among treated and control group . . . p. 114
1
Introduction
The Human Capital Theory proposes that the educational difference between coun-tries is one of the most important factors to explain the different levels of economic development and well-being of their populations (SCHULTZ, 1961). This is because the accumulated educational level of the country is responsible for the development of new technologies and adoption of existing ones, even more so in the present-day dynamic and volatile technological context (BENHABIB; SPIEGEL, 1994). Investment in education, which provides for the increase of human capital in the individual, in turn, results in increased productivity, income and more opportunities in jobs that use more technology (BECKER, 1995). However, its effect is not restricted to the individual with more years of study, the family and society are also benefited indirectly, for example: in the family, the edu-cational habits of the parents influence the behavior and eduedu-cational investment of the children; in society, among the effects are the reduction of crime, and, in a still inconclu-sive way, increase of the pair’s wages (ACEMOGLU; ANGRIST, 2000; CURRIE; MORETTI, 2003;LOCHNER; MORETTI, 2004; PSACHAROPOULOS; *, 2004).
In 1960, Brazil had educational levels similar to East Asian countries, such as Ko-rea and Singapore. Despite expanding coverage by 32 percent by the 1990s the country educational quality was lower compared to the same East Asian countries. Among the explanations for this fact are: crisis in the economy, low return on education for the poor-est, low technology due to the focus on internal industrialization, higher birth rate while investment remained the same over the years, market price controls private, among oth-ers. This led to a delay in the educational quality of the period, an increase in income inequality and a lower rate of economic growth (Inter-American Development Bank, 1996).
Moreover, Brazil was caught in a vicious cycle: high economic inequality meant that there was little pressure and investment of basic education by the State, further exacer-bating income inequality. The lack of demand for quality education among the poorest occurs because they are most likely not to know the educational returns; there is a high cost of opportunity when putting the child in school, because many children work to help
in the family income; they do not have the resources to pay for their children school sup-plies; and given the socioeconomic relationship of the student’s family to the quality of the local school - the schools available to the most needy are generally the lowest quality (Inter-American Development Bank, 1996, p.11).
Above arguments are just a few to justify the provision and investment in education by the Government, especially for the most economically vulnerable populations. Likewise, Brazil establishes in 6th Article of the 1988 Constitution education as a fundamental social right (Brasil, 1988).
Among the options for improving the educational quality of schools are the improve-ment of school efficiency, the better use of available resources, based on a conceptual model of the educational process; the creation of incentive systems that strengthen in-stitutional changes in school (HANUSHEK, 1986, p.1142); and increased school resources (HANUSHEK, 2003, p.66). Within this categorization, the Programa Mais Educação PME can be characterized as a public policy of adding resources, whose main school input is the additional time of the student in the school that provides more classes and activities.
Hanushek (2003, p.84) warns about the fragility of the policies of adding resources this maybe is correlated to the better socioeconomic conditions of the family (i.e. additional school hours permits the parents to work more time to have a better income) and this is what would be impacting on student performance. Another consideration is because the amount of resources allocated to students increases over time to compensate the less prepared and motivated students to increase their chances to go to school. In any case, this does not mean that resource growth policies work, although the author points that these policies are more probable to have effect in developing countries.
Hanushek and Woessmann (HANUSHEK; WOESSMANN, 2007, p.77) also point out that school resource allocation policies have little impact on student performance. However, they indicate that to be more efficient the school needs time to adapt and to use incen-tives. For this it is necessary more autonomy and accountability which, in turn, produces different management decisions and competition between schools. The authors emphasize the importance of policies that relate incentives to institutional changes rather the ones that just transfers resources, so the evaluation of these policies is necessary in order to know whether it is worthwhile to maintain and evolve the policy or to discard it com-pletely. Thus, policies to increase school resources, although they are the most abundant, do not translate into greater student learning.
schooling. However, school day extension policies have divergent impacts on student learn-ing and studies on other indicators are scarce, but when they are targeted at the most vulnerable students and when there is planning for better use of the additional time there is greater probability of success and positive effects (PATALL; COOPER; ALLEN, 2010).
The object of this thesis is the Brazilian primary schools. We approach the issue of low education quality in Brazilian’s primary schools in two ways, (1) from the the school efficiency perspective and (2) from economic education perspective though a evaluation of a public policy that increases the school time.
Firstly, we assess the level of efficiency, using a dynamic model (2007-2015), taking into account the differences in the socioeconomic levels of its students. This model includes an infrastructure index as a variable that links each period of time. We used three model’s specifications that differ in the usage of the socioeconomic variable.
Secondly, we evaluate the Programa Mais Educação created by the Brazilian Federal Government with the aim of improving education at the fundamental level. This policy seeks to implement the model of school full-time, with at least seven hours a day, through the implementation of extra socio-educational activities in the public school second-shift. Our objective is to estimate the impact of the program on the average academic perfor-mance in Portuguese, mathematics and approval rate, in addition to the new violence index from 2009 to 2017.
Although this work does not intend to solve the problem of the low educational quality, we think that at least it brings new perspectives and empirical evidences to approach this theme.
2
Essay 1 - Dynamic efficiency of
primary education in Brazil:
socioeconomic and infrastructure
influence on school performance.
*Paper submitted to journal Socio-Economic Planning Sciences.
2.1
Introduction
Primary education reached almost all 6 to 14-year-old children in Brazil: 98.2% were enrolled in 2011. However, quality has not kept pace with this expansion. PISA’s results showed little improvement or even stabilization of Brazilian scores 1, which have been
among the lowest of all countries. In PISA 2015 Brazil was placed 63 among 70 countries (OECD, 2016).
These PISA mean scores disguise regional, state, municipality and school variations in a heterogeneous country like Brazil. Low socioeconomic level limits student performance but, even in less developed regions, like the Northeastern one, some schools have good results and their experience can serve as benchmarks for other schools.
The difference in performance on science tests of a student from a high socioeconomic level and that of a low level is equivalent to one school year in Brazil. OECD considers as resilient a student that overcomes the difficulties of his lower socioeconomic level and has a good performance. In Brazil only 9% of students are resilient, while the OCDE average is 29% and Shanghai has 48% of resilient students (OECD, 2016).
1Source: Programme for International Student Assessment (PISA) is a comparative evaluation applied to 15-year-old students which is organized by the Organization for Economic Co-operation and Devel-opment (OECD). It has been applied every three years, since 2003, and evaluates reading, mathematics and science.
This raises two questions: 1) which school resources are mostly related to student performance in standard tests; and 2) which schools make the best use of their resources. The first question, in general, is the focus of the “economic education” literature, while the second question is the object of the “efficiency in education” literature (WITTE; López-Torres, 2017), that constitute the theoretical framework of this paper.
Moreover, part of school’s resources may be applied in long-term capital investment whose return may not occur in the same period, such as school physical structure. The importance of incorporating capital or quasi-fixed inputs to the analysis was highlighted by (Fallah-Fini; TRIANTIS; JOHNSON, 2014). The inclusion of long-term investments naturally calls for a dynamic model. Besides, such issue might be especially relevant in developing countries, such as Brazil, which present large heterogeneity in school’s infrastructure with some of its schools still lacking basic physical attributes.
Our main goal is to analyze the efficiency of Brazilian primary schools, with both discretionary and non-discretionary variables of school management using the dynamic model developed by Tone e Tsutsui (2010). To this aim, we elaborated two indices based on the Item Response Theory (IRT): one socioeconomic index that may be included in the model as a discretionary or as a non-discretionary variable and one school infrastructure index which corresponds to a carry-over variable between time periods of the model.
Our favorite specification is a dynamic DEA model where the socioeconomic index is treated as a non-discretionary input. For robustness checks, we also introduced two other models: (1) a dynamic DEA model where the socioeconomic index enters as a discretionary input; and (2) a two-stage DEA model that discounts the effect of the socioeconomic index with a regression analysis as suggested by Ray (1991), but using the procedure developed in Simar e Wilson (2007).
In addition, the global efficiency scores of each one of the three dynamic DEA models was regressed on environmental and other management variables to reveal their potential contribution to school efficiency.
Our work contributes to two literature gaps pointed by Johnes, Portela e Thanassoulis (2017): the analysis of educational problems in a dynamic context; and the comparison of different evaluation methods using the same database to identify convergent results. To the best of our knowledge, this is the first study to use a dynamic DEA model for primary education based on Tone e Tsutsui (2010)2 and to apply the Item Response Theory to
2A search on Science Direct, Google Scholar and Wiley Online Library crawlers did not reveal any Dynamic DEA application for primary education using Tone e Tsutsui (2010). Also, no such study was
construct socioeconomic and infrastructure school indexes.
The dynamic DEA model of Tone e Tsutsui (2010) focus on the temporal dimension of inputs and outputs in an efficiency analysis combining several time periods. In education, this is especially important given the cumulative character of education and the opportu-nity of identifying schools that are efficient on a longer run, thus reducing random effects associated to specific years or student’s classes. In this dynamic model, a school must be efficient in all periods to be globally efficient, which reduces the number of reference schools in comparison to static models.
Results indicate that both discretionary and non-discretionary one stage dynamic models have similar average efficiency scores, but they rank schools in a different way. The two-stage model results in lower average efficiency scores and lower number of efficient schools. The average efficiency is almost constant along the years in all three models.
As expected, different Brazilian regions presented different average efficiency levels, with the South and Southeast regions displaying the highest number of globally efficient schools. However, the model identified efficient schools in all regions, including the North-east region. Moreover, two-stage regressions show that, once we control for other envi-ronmental school characteristics, the order of regional average efficiencies changes leading the South to be the least efficient one, whereas the Southeast remained the region with the highest school efficiency on average.
One way to foster school performances might be by upgrading their infrastructure, a relevant issue for developing countries. Indeed, the projection of the infrastructure index reveals that there is room for improvement in the dynamic efficiency of schools. Our second stage analysis corroborates this result by showing the importance of infrastructure to efficiency even when we control for environmental variables and the socioeconomic level.
Besides this Introduction, the second section contains a literature review with the general panorama of efficiency evaluation of primary education using DEA. The third section presents the methodological procedures – the dynamic DEA models and the Item Response Theory. The fourth section describes the database, compares the efficiency scores of the three models and analyzes its relation to the schools’ educational environment. The last section concludes this work.
cited on the Emrouznejad e Yang () list of 10300 major articles along the 40 years of DEA model use (1978-2016) and in Witte e López-Torres (2017).
2.2
Literature review
Efficiency in education is mainly measured by the relative ability of schools to generate educational products using the minimum levels of inputs, including the innate ability of students as one of the school resources (JOHNES; PORTELA; THANASSOULIS, 2017). This measure of efficiency, named technical efficiency, is associated to a possible reduction of inputs while maintaining outputs or to a possible increase in outputs while maintaining inputs. (COOPER; SEIFORD; TONE, 2007).
The study “Equality of educational opportunity”, known as the Coleman report, was the first to separate the school effect from exogenous factors, such as student family back-ground and socioeconomic level, which are important factors determining educational results. Differences in school and teacher quality also have a significant, although small, impact on students results Coleman (1966). Later, Hanushek (1986) structured a produc-tion funcproduc-tion for educaproduc-tion considering family background as inputs, including socioeco-nomic conditions, innate abilities and other external influences, such as management and institutions. In this function, the product is measured by the value added on standard proficiency test scores over time.
One problem in the production function form is due to the several inputs and edu-cational products. In the end of the 1970s, the United Sates Department of Education created an educational program, called “Follow Through”, to evaluate programs and less endowed students in the country. Despite the large number of observations and several input and product variables, evaluation results were not satisfactory using econometric models. Then, to solve this problem, Charnes, Cooper e Rhodes (1978), based on the sem-inal ideas of Farrell (1957), developed the CCR initial model, named Data Envelopment Analysis (DEA).
Since then, a long list of empirical literature and models were developed to fit the specific processes of education (JOHNES; JOHNES, 2007), and DEA is one of the most used methodologies, among other reasons because of its flexibility and absence of specification of functional forms.
The DEA literature have introduced various ways to deal with exogenous factors. As noted by Cordero-Ferrera, Pedraja-Chaparro e Salinas-Jiménez (2008) and Huguenin (2015), there is not a consensus about the best model to use; the literature on efficiency in education deals with several approaches to incorporate non-discretionary variables into the analysis. The main options are divided in two groups: DEA models with only one
stage and models that use DEA together with other methodologies. These last models analyze efficiency in education when there are a number of variables that compose the socioeconomic profile of the student, such as income and parent education level. Witte e López-Torres (2017) provides a complete literature review of DEA in education, including multistage studies and an overview of non-discretionary variables.
For instance, Banker e Morey (1986) developed the non-discretionary DEA model (DEA-NDSC), a one-stage model which allowed the inclusion of variables that influence the performance of units under analysis but are not controlled by the unit. To compare the performance of schools, in a particular way, the model is able to incorporate student socioeconomic conditions not as traditional inputs but as variables that affect results without being directly chosen by the schools.
Wanke, Blackburn e Barros (2016) was the first study to use a second-stage network analysis for education, which allowed them to consider both cost efficiency and learning efficiency in a DEA network model. By studying how cost efficiency translate into human resource decisions and then on learning efficiency, their model better describes the educa-tional process and provides useful guideline for policymakers. They considered 14 inputs, 10 intermediate inputs/outputs and 25 outputs, from to 2008 to 2010, for 1,400 primary schools and, separately, for 387 secondary Australian schools. They also used neural net-works to point out, among 59 contextual variables as cost and learning efficiency drives, the most important environmental variables and suggest recommendations for different groups of schools, each one with its efficiency profile.
A recent DEA application to the Brazilian primary education is Lauro, dos Santos Figueiredo e Wanke (2016) two-stage model. In the first stage, they compared 465 schools of Rio de Janeiro state based on four inputs (number of rooms, number of staff members, number of teachers and number of computers) and three outputs (approval rate, language and mathematics test scores). In the second stage, they used environmental factors such as: the socioeconomic level of the school, the education support infrastructure, student behavior, characteristics of the director and characteristics of the school management. Two of the main results are: higher student socioeconomic level does not correlate with higher efficiency and longer school day contributes to reducing efficiency.
Brennan, Haelermans e Ruggiero (2014) used an extension to the traditional Malmquist index decomposition that include an environmental component for the public sector. They applied their model for six years to assess the Dutch secondary school efficiency. As in-puts they used school personnel full time equivalent (management, teacher and staff) and
school material expenses; as outputs, the average student national examination grades per school, the average student achievement and the total number of students enrolled. Their second stage results showed the importance of socioeconomic background, measured by categories of the proportion of less endowed students in the school, on school efficiency, even in a developed country context.
Yahia, Essid et al. (2019) measured efficiency of 113 Tunisian secondary schools based on PISA 2015 data set. The DEA model considered the number of students and the number of teachers full time equivalent as inputs and science, mathematics and language PISA tests as outputs. In the second stage analysis, they identified a positive relationship between efficiency score and parent socioeconomic background and a negative relationship of efficiency score and class size.
Comparing schools of 30 OECD countries, Agasisti e Zoido (2018) evaluated 8,500 schools using PISA 2012 through a two-stage bootstrap DEA model. In the first stage the inputs were the socioeconomic level of the student, the student teacher ratio and the proportion of computers per student; while the products were the PISA score in languages and mathematics. Second stage variables included school practices and school and student characteristics. The results indicate greater heterogeneity of efficiency within countries than among them; and lower efficiency in schools with a higher proportion of students with low socioeconomic status.
Analyzing the same database, Aparicio et al. (2018) used a non-radial DEA application along with a second stage analysis. The non-radial approach sought to identify the different types of inefficiency in each one of the student’s tests (Language and Mathematics). They found that there is a large potential to improve the Language test score, because this learning process is more efficient, although, most countries focus on the mathematics learning process since it is a better predictor of future earnings.
Masci, Witte e Agasisti (2018), using a student-level approach, analyzed the principal factors of the trainee’s and school’s traits and practices that influence the performance (efficiency) of 6572 Italian students. The inputs were the socioeconomic background and the students’ grades in previous periods and the product was the student’s grade in the current school year. The second stage identified that language performance is related to the socioeconomic makeup of peers, and learning mathematics is more related to practices adopted by principals and teachers.
With a methodological approach that combined an exogenous allocation of 422 teach-ers, DEA per class and second stage regression with the main characteristics of teachers
and school, Santín e Sicilia (2018) explored the main characteristics of teachers that influ-ence the efficiency of students in primary schools in Spain. They found that the quality of teachers influences student performance, since the mean difference in grade between the student with the best teacher and the worst teacher was approximately 0.45 standard de-viation in languages and mathematics. Also, the experience or the training of the teacher alone does not impact the efficiency and smaller classes positively impact the efficiency of the teacher despite being a costly policy.
Based on the above arguments and applications review, this article tries to fill the literature gap using a non-discretionary dynamic DEA model to compare efficiency of public schools in a developing country context.
2.3
Methodology
2.3.1
Construction of socioeconomic and infrastructure indices
through Item Response Theory (IRT)
One important issue with DEA models is how to incorporate a large group of vari-ables that are interrelated into the analysis. If the number of varivari-ables is too high, it is impossible to use all the variables directly, especially dichotomous variables. This paper dealt with this issue by using the Item Response Theory to compute a socioeconomic and an infrastructure index.
Among the advantages of IRT is its possibility to use observations even when data are lacking; in the case of our article, both the infrastructure and the student questionnaires had many missing observations. Another reason for the choice of this method is that it is the same one used by the Ministry of Education to construct an official socioeconomic index for the 2015 year. Following this, we created a socioeconomic index for the years 2007 to 2015 to use as nondiscretionary input in our dynamic DEA model.
The construction of the infrastructure index was done using the IRT model of two parameters (2PL), based on the data from Censo Escolar (Ministry of Education, ) of years 2007, 2009, 2011, 2013 and 2015. The index followed the procedure delineated by the article of Neto et al. (2013). It is computed based on 20 variables that are indicators of the existence of: electric energy, sewer, directory room, teacher’s room, information laboratory, science laboratory, special attendance room, sports court, kitchen, library, playground, restrooms, restroom for people with special needs, rooms for people with
special needs, TV, DVD, copying machine, printer, computers and internet. Variables re-lated to early childhood education and water availability were omitted because the first ones are outside the focus of primary school analysis and the last one has low discrimina-tory power (less than 0.5), since more than 99% of the schools have access to water. These are all dichotomous variables that take value 1 if the school has an item and 0 otherwise. The index was calculated simultaneously for all years, so that we can track if there was infrastructure improvement for the same school over the years.
The Ministry of Education’s Socioeconomic Index of 2015 was based on Alves, Soares e Xavier (2014) which aggregated several databases from different years: Basic Education Assessment System (Sistema de Avaliação da Educação Básica - 2001, 2003 and 2005); Prova Brasil (2005, 2007 and 2009) and National High School Exam (Exame Nacional do Ensino Médio - 2007, 2008 and 2009) to compute a time invariant index.
We preferred to construct an index using only Prova Brasil data (Ministry of Education, ) (years 2007, 2009, 2011, 2013 and 2015) which allows to calculate an index for each year separately, following the same logic we used for the infrastructure index, so that the results of each school may vary along the periods. Besides the 12 questions from Prova Brasil data used in the Ministery’s Index, we added two new questions in our model: mother and father alphabetization, both dichotomous, in order to increase the discrimination power of the latent variable for the lower socioeconomic levels.
The variables used in the estimation of the socioeconomic index were: TV, radio, DVD, refrigerator, freezer, washing machine, number of cars, computer, restroom, maid service, mother educational level and father educational level. The correlation between our index and the official one is 0.84, indicating a strong correlation and a good external validity of our index. The indices were linearly transformed, in the same way as was done by Alves, Soares e Xavier (2014), Neto et al. (2013) and the Ministry of Education, so that they had positive values to be used in the DEA model. Thus, their values ranged from 0 to 10, with an average of 5 and standard deviation of 1.5.
2.3.2
Dynamic non-discretionary DEA model
Data Envelopment Analysis (DEA) is the non-parametric, non-statistical mostly used model to analyze efficiency. Charnes, Cooper e Rhodes (1978) developed this method based on linear programming to estimate a production frontier and then to evaluate the relative efficiency of each Decision Making Unit (DMU) based on this frontier.
The frontier constructed by the efficient DMUs may not necessarily form a “production frontier” but may represent a “best practices frontier”. There are three ways to transform a non-efficient unit into an efficient one: maintaining a certain level of production and reducing the inputs (input oriented); maintaining a certain level of inputs and increasing production (product oriented); and altering simultaneously inputs and production (non-oriented or additive model).
Among models that deal with efficiency changes over time, the dynamic slacks-based model developed by Tone e Tsutsui (2010) is characterized by: (1) allowing for a global analysis alongside a time period analysis; (2) being a non-radial model, so that changes in input and output projections need not to be proportional, thus eliminating slacks inefficiencies; (3) weighing each input, output and carry-over variable with their own factor of efficiency index (FEI), differently from the radial model that assumes the same factor for all variables; (4) supporting the three usual orientation types (input, product and non-oriented); and (5) introducing carry-overs to link activities between different time periods.
These carry-over variables are categorized into four types: good, bad, free and fixed. A good carry-over is treated as an output which cannot be reduced, that is, its value cannot be lower than that observed in the DMU. A bad carry-over differs from the good one because it is treated as an input and its excess implies inefficiency. Once again, its value cannot be reduced by a model projection. A discretionary (free) carry-over assumes that the DMU controls this variable and its value can increase or decrease, thus imposing no restrictions on its projection. A non-discretionary (fixed) carry-over indicates that the DMU has no control over this variable and, therefore, it is not possible to look for projections in this dimension. The last two types indirectly influence efficiency according to the excess or scarcity of the variable, acting directly on the objective function (TONE; TSUTSUI, 2010).
Since the school infrastructure is a necessary input, lasts for a long time period and can be improved or worsened, it was modeled as a discretionary good carry-over, so that projections are always above observed values for each school.
The three models used in this article differ by their treatment of the socioeconomic index in the DEA model: discretionary dynamic, non-discretionary dynamic and two-stage according to Ray (1991). All models use variable returns to scale, product orientation and equal weights for the five years (0.2).
Equation 2 is the condition that guarantees the continuity of link flows between periods. Equations 3 to 8 are the model restrictions. This model considers the socioeconomic index as non-discretionary (xf ix): 1 τ0∗ = max 1 t T X t=1 wt h 1 + 1 s + ngood Ps i=1w + i s + it yiot + ngood X i=1 sgoodit ziotgood)] (2.1) n X j=1 zijtgoodλtj = n X j=1 zijtgoodλt+1j (2.2) xiot = n X j=i xijtλtj + s − it; (i = 1, ..., m; t = 1, ..., T ) (2.3) xf ixiot = n X j=i xijtλtj; (i = 1, ..., p; t = 1, ..., T ) (2.4) yiot = n X j=i yijtλtj − s + it; (i = 1, ..., s; t = 1, ..., T ) (2.5) ziotgood = n X j=i zijtλtj − s good it ; (i = 1, ..., ngood; t = 1, ..., T ) (2.6) n X j=i λtj = 1(t = 1, ..., T ) (2.7) λti = 0; s−it >= 0; s+it >= 0; sgoodit >= 0 (2.8)
Where: j is a DMU index, and N is the total number of DMUs; x is the discretionary inputs; xf ix is the non-discretionary inputs; y is the discretionary products; t is a time
index and T is the total number of time periods; i is an index of time periods with discretionary inputs; m (i = 1, ..., m) is the total number of periods of discretionary inputs; p (i = 1, ..., p) is the number of periods of non-discretionary inputs; s (i = 1, ..., s) is the number of periods of discretionary products; wt is the weight of period t; ngood
is the number of good carry-overs, which are the only type of carry-overs considered in this work; s−it, s+it, sgoodit are slacks variables denoting, respectively, input excess, output shortfall and link shortfall.
The only difference between the discretionary and non-discretionary dynamic models is that the socioeconomic index becomes a non-discretionary input and, consequently, the DEA model includes a restriction to the projection of this input. This change is important because in the dynamic model with output orientation the socioeconomic index
may decrease to increase efficiency.
From both theoretical and managerial points of view, it makes no sense to decrease the average socioeconomic level of a school to improve its efficiency – it is neither controlled nor desirable for the school. Thus, between these two models, we prefer the non-discretionary one.
The two-stage model consists in calculating the DEA efficiency score without the socioeconomic variable (first stage) and then regressing the estimated efficiency by covari-ates that represent environmental variables (second stage). Simar e Wilson (2007) pointed out that it is not ideal to use ordinary least squares (OLS) to estimate the second stage, given the fact that DEA relative efficiency scores are serially correlated and truncated be-tween 0 and 1. To solve these problems, their procedure incorporated bootstrap algorithms into the regression to correct the estimation standard errors and the unknown correlation among the observations. Therefore, we adopted Simar e Wilson (2007)’s method to better estimate the efficiency score in the second stage.
According to Ray (1991), the second stage procedure to obtain the new efficiency score requires that the environmental variable be statistically significant, and if this is the case, the residue of each observation is calculated. The regression to the global score is based on the average socioeconomic index of the five periods. This means that only one DMU will be efficient: that with the lowest proportion of its score explained by the environmental variable. When this is done, the efficiency score is adjusted by the following equation: score = residue + (1-maximum residue value in the sample).
2.4
Database and Results
One of the first tasks in constructing a DEA model is the choice of variables. They must fit the theoretical model and the data available for analysis. Our input and output variables were chosen to capture dynamic characteristics of the educational process. This work uses federal databases (Censo Escolar - the educational census - and Prova Brasil - the national standardized test score) which are consistent among the years and have a reliable data collection.
The Censo Escolar is the most complete survey on primary education in Brazil, con-ducted every year by the Federal Government together with state and municipal secretari-ats and compiled by the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). It includes general characteristics of the teacher staff, school structure,
enrolment numbers, etc.
The Prova Brasil is a Portuguese and Mathematics standardized proficiency test ap-plied to all students from urban public schools with more than 20 students every other year. Therefore, our database comprises the years of 2007, 2009, 2011, 2013 and 2015. Be-sides the test scores, the database includes information on students, teachers, managers and school characteristics.
The educational outputs were assessed by schools’ average rates at proficiency tests in Portuguese and Mathematics. Prova Brasil data was also used to calculate the socioe-conomic index based on the student’s questionnaires.
Four input type variables were used: average number of students per class; average number of students per teacher; proportion of teachers with university degree; and the socioeconomic index (SEI).
Brazil has the lowest student to teacher ratio among all countries submitted to PISA: on average 29 students for each professor but some states have up to 45:1 (OECD, 2016). Krueger (1999), with data from the STAR experiment, estimated a positive impact of the reduction of student numbers on students’ performance, more so when the students have a low socioeconomic level.
Teachers’ formal training also seems to be an important factor determining student performances. The random experiment of Barbara Nye, Spyros Konstantopoulos e Larry V. Hedges (2004) brings positive evidences – similar to those of non-experimental studies – especially for mathematics and low socioeconomic level students.
In order to compute student to teacher ratio variables and a measure of teachers’ formal training, the total number of enrolled students, classes, teachers and teachers with university degrees, were collected data from the questionnaires of the Censo Escolar.
The model also includes the infrastructure index as a carry-over variable between the years and four outputs: average Portuguese and Mathematics scores in the fifth and ninth grade. As previously discussed, our models differ from previews literature on the treatment given to the socioeconomic index.
Table 1 summarizes characteristics and sources of input and product data used in the DEA model.
Table 1: Variables used in the DEA models
Variable Description Use
Student per class Number of students divided by number of classes Input Student per teacher Number of students divided by number of teachers Input Teachers with degree Number of teacher with university degree divided by total teachers Input Infrastructure index School index based on 20 variables Carry-over good Socioeconomic index School average based on 14 student variables Non-disc. input Language score Average school score on Prova Brasil Product Mathematics score Average school score on Prova Brasil Product
Some observations were lost due to lack of data in some questionnaires and while merging databases of different years. The main factor contributing to the reduction of the sample was the restriction that the same school should have results for Prova Brasil in all studied years. Small schools may be present in some years, but not others, due to the 20 students’ threshold. Also, the more inputs, outputs and periods included in the model the smaller tend to be the sample. Since our outputs include proficiency scores for both 5th and 9th grades, it only keeps schools that offer both grades.
After constructing all variables, merging of all databases and excluding all observations with missing variables, the sample comprised 6025 schools. Table 2 illustrates the total number of schools and students in each year and their proportions to the whole sample, while Table 3 displays some descriptive statistics of the chosen DEA variables.
Table 2: Population and sample comparison by year
Year
Schools
Students
Prova Brasil
Sample
Prova Brasil
Sample
2007
48.001
6.025 (12.6%)
4.082.879
773.428 (18.9%)
2009
57.916
6.025 (10.4%)
4.506.008
780.123 (17.3%)
2011
56.222
6.025 (10.7%)
4.248.778
780.848 (18.4%)
2013
59.251
6.025 (10.2%)
4.178.018
721.869 (17.3%)
2015
57.744
6.025 (10.4%)
3.970.684
707.944 (17.8%)
Table 3: DEA variables means and standard error for the 6025 sample schools
Variable
2007
2009
2011
2013
2015
Student per class
29.096
27.849
26.719
26.008
25.271
(0.063)
(0.060)
(0.061)
(0.060)
(0.059)
Students per teacher
6.915
5.567
5.351
5.108
4.833
(0.052)
(0.025)
(0.023)
(0.020)
(0.021)
Teachers with university degree
0.866
0.869
0.901
0.909
0.867
(0.002)
(0.002)
(0.002)
(0.002)
(0.002)
Infrastructure index
5.151
5.254
5.165
5.017
4.974
(0.020)
(0.020)
(0.020)
(0.020)
(0.020)
Socioeconomic Index
4.857
6.328
5.138
5.065
4.976
(0.005)
(0.010)
(0.009)
(0.009)
(0.008)
Language score 5th grade
172.178
182.982
189.830
193.863
210.925
(0.231)
(0.257)
(0.263)
(0.298)
(0.243)
Mathematics score 5th grade
189.626
202.904
208.515
209.378
222.529
(0.249)
(0.302)
(0.304)
(0.331)
(0.259)
Language score 9th grade
228.690
240.454
241.368
240.971
254.718
(0.227)
(0.245)
(0.246)
(0.248)
(0.199)
Mathematics score 9th grade
240.588
243.443
248.119
245.548
257.435
(0.250)
(0.263)
(0.279)
(0.268)
(0.212)
Table 4 shows average values of inputs and outputs for the population and the final sample3. There are statistical differences between sample and population averages for
all variables, mostly due to the requirement that a school needs to present Prova Brasil results in Portuguese and Mathematics in all years to be included in the final sample. Thus, this probably excludes new schools and small schools that may not have at least 20 students in at least one year to participate in the Prova Brasil. The fact that 10% of schools have about 18% of the students seems to corroborate this idea.
2.4.1
Global and yearly efficiencies
Table 5 shows the average efficiency, with standard deviation in parenthesis, of each model for each year and the global efficiency for all years. Considering only average efficien-cies, the dynamic discretionary and the dynamic non-discretionary models are virtually identical in all aspects, while the two-stage model always resulted in lower efficiencies. This is expected because in this model only one DMU is in the frontier and the efficiency in general is lower due to the adjustment in efficiency that is made in the second stage.
Only 16 out of 6025 schools were efficient in the non-discretionary model; 12 of these 16 were also efficient in the discretionary model, and one school was efficient on all models. This school is located in Bahia state (Northeast Region) and it has a socioeconomic index of 4.42 (level 3 in the SEI rating), which is the second lowest SEI index among all 16 efficient schools of the non-discretionary model. On the other hand, the school with the lowest SEI index among these 16 ones (3.89, level 2 in the SEI rating), is efficient in both discretionary and non-discretionary models and it is located in Minas Gerais state (Southeast Region).
Table 4: Variable differences between Prova Brasil and Sample Schools
(1) (2) T-test
Prova Brasil Sample Difference
Variable N Mean/SE N Mean/SE (1)-(2)
Student per class 30819 26.405 (0.030)
6025 26.989 (0.051)
-0.584***
Students per teacher 28092 8.769 (0.035)
6025 5.555 (0.022)
3.214***
Teacher with degree 27061 0.838 (0.001) 6025 0.883 (0.002) -0.044*** Infrastructure index 26934 5.133 (0.006) 5338 5.132 (0.013) 0.001 Socioeconomic index 31274 5.289 (0.004) 6025 5.273 (0.008) 0.016* Language 5th grade 19806 184.591 (0.133) 6025 184.713 (0.234) -0.122 Mathematics 5th grade 19806 203.017 (0.154) 6025 202.606 (0.265) 0.411 Language 9th grade 14821 236.961 (0.126) 6025 237.871 (0.210) -0.909*** Mathematics 9th grade 14822 243.766 (0.143) 6025 244.424 (0.238) -0.658**
Notes: The value displayed for t-tests are the differences in the means across the groups. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.
Table 5: Average efficiency models
Model
2007
2009
2011
2013
2015
Global
Dynamic
0.837
0.8
0.8533
0.829
0.832
0.828
(0.059)
(0.07)
(0.06)
(0.067)
(0.064)
(0.055)
Dynamic non-discretionary
0.837
0.801
0.8533
0.829
0.833
0.829
(0.06)
(0.07)
(0.06)
(0.068)
(0.064)
(0.055)
Two Stage
0.703
0.693
0.708
0.713
0.733
0.753
(0.048)
(0.059)
(0.056)
(0.059)
(0.048)
(0.043)
Table 5: Note: Standard errors shown in parentheses.
Both the discretionary and the non-discretionary models average efficiencies present only small variations over time. Indeed, the maximum yearly efficiency of 0.85 occurred in 2011, decreasing slowly to 0.83 in 2013 and 2015. In the two-stage model the situation is slightly different but also without a large evolution over time, since its average varies from 0.70 in 2007 to 0.73 in 2015. Thus, the models show that schools can increase on average their tests score by 17% to 25% while using the existing inputs.
Table 6: Anova - Pairwise comparisons of means with equal variances
Model
Contrast
Std. Err.
t
P > t
Discretionary vs Non-Discretionary
-.0005296
.0009293
-0.57
0.836
Two Stage vs Non-Discretionary
-.0756599
.0009293
-81.42
0.000
Two Stage vs Discretionary
-.0751303
.0009293
-80.85
0.000
Considering the standard deviations, there is also no major change in dispersion of efficiency along the period of analysis on all models, although looking for the ANOVA results in Table 6 shows that there is a statistical difference between both one stage models and Ray’s two-stage model. The similarity of the results between the discretionary and the non-discretionary models does not mean that their distribution and ranking are the same. In fact, a non-parametric Wilcoxon pairing test for dependent samples rejects the null hypothesis that distributions are equal with p-values close to zero. Samples are dependent, because scores in the three models come from the same DMU. Even the ranking of the two dynamic models differ despite the resemblance on their mean and variance. This can influence the efficiency frontier, the efficiency score and the benchmark of inefficient DMUs, even if by a small amount.
Based on the Wilcoxon pairing test, the ranking of schools is not equivalent between any pair of models with statistical significance of 0.01%. Huguenin (2015) reached similar
results when comparing the efficiency score of four different non-discretionary DEA models of 90 primary schools in Switzerland.
2.4.2
Efficiency by socioeconomic level and region
Table 7 presents mean efficiency scores and the number of efficient schools by so-cioeconomic level intervals. The ranking differences in models show up in the number of efficient DMUs in the non-discretionary model in comparison to the discretionary model: there are four more efficient schools in the non-discretionary model and they all belong to the “Medium high” level.
There seems to be a U -shaped relationship between the SEI and efficiency of schools, since levels 2 (Low), 5 (Medium high) and 6 (High) have average efficiency scores higher than those of levels 3 (Medium low) and 4 (Medium). In other words, the smallest and largest socioeconomic level intervals have efficiency averages higher than the median level intervals. This effect is observed in all three models. Besides, even when using models that deal with socioeconomic variables there remains a strong relation between efficiency score and the socioeconomic level.
Table 7: School efficiency by socioeconomic level and model
Two Stage Discretionary Non-Discretionary # Efficient # Efficient # Efficient Level Schools Average SEI Score Schools Score Schools Score Schools Low (2 ≤ SEI < 3) 107 3.8 0.784*** 0 0.844*** 1 0.844*** 1 Medium low (3 ≤ SEI < 4) 1814 4.5 0.754*** 1 0.804*** 2 0.804*** 2 Medium (4 ≤ SEI < 5) 3436 5.4 0.750*** 0 0.832*** 3 0.832*** 3 Medium high (5 ≤ SEI < 6) 661 6.2 0.767*** 0 0.882*** 5 0.882*** 9 High (6 ≤ SEI < 7) 7 7.1 0.795*** 0 0.974*** 2 0.951*** 2
Table 7: Note: Anova p value *p < .05. **p < .01. ***p < .001.
Another possible analysis is to compare school efficiency by regions of the country, because students’ socioeconomic backgrounds diverge a lot from one region to the other. Since the model already considers SEI heterogeneity, if one region is identified as being more successful than the others in fostering efficiency improvements over time, that would motivate a deeper analysis of primary public policies adopted by that region.
Table 8 shows statistical difference between average score of each region in all models. The average socioeconomic index of the schools by region is a good representation of the socioeconomic status of the country as a whole, the southern regions having better performance than the northern poorer ones. However, within regions there is still a lot of heterogeneity, and some schools located in the poorest regions may be highly efficient even
Table 8: Efficiency and socioeconomic differences by region
Regions
Schools
Two Stage
Disc.
Non-Disc.
Socio Index
North
549
0.738***
0.801***
0.801***
5.07***
(0.38)
(0.47)
(0.48)
(0.02)
Northeast
1462
0.748***
0.791***
0.792***
4.55***
(0.45)
(0.50)
(0.52)
(0.01)
Southeast
2756
0.758***
0.840***
0.842***
5.52***
(0.44)
(0.50)
(0.50)
(0.00)
South
506
0.765***
0.867***
0.869***
5.97***
(0.39)
(0.45)
(0.46)
(0.01)
Center-West
752
0.749***
0.828***
0.829***
5.43***
(0.35)
(0.43)
(0.42)
(0.01)
Table 8: Note: Standard errors shown in parentheses. Anova p value *p < .05. **p < .01. ***p < .001.
when compared to schools of richest regions. Indeed, as previously mentioned, the only school that was globally efficient in all three models is located in Bahia, a state ranked in the 20th position out of 27 states in terms of gdp per capita.
An analysis of variance (ANOVA) in Table 9 provides statistical evidence of hetero-geneity in efficiency scores by SEI categories confirming the distinction of mean efficiency scores, while a Bartlett test indicates a difference in the variances of those scores by socioeconomic groups and regions at the 1% significance level.
Table 9: ANOVA and Bartlett tests by region and socioeconomic level
Model Region Socioeconomic level
ANOVA (F test) Bartlett (chi-square) ANOVA (F test) Bartlett (chi-square)
Two-stage 44.84 84.3 39.8 43.02
Disc. 335.3 45.9 302.8 106.7
Non-disc. 336.3 41.12 317.8 100.8
2.4.3
Explaining efficiency scores
A second stage analysis helps explaining the global efficiency scores in terms of en-vironment and other management variables. We are particularly interested in the effect of infrastructure and socioeconomic indices on global efficiency levels, but we also con-trolled for other environmental variables based on Wanke, Blackburn e Barros (2016) yet
limited by their availability in the Brazilian database. A complete list and definitions of the explanatory variables and their source is presented on Table 10.
Table 10: Definition of variables used in the second stage analysis.
Variable Description Source
Socioeconomic index Socioeconomic index Prova Brasil
Infrastructure index Infrastructure index Censo Escolar
City population Municipal population of the school’s city location IBGE Percentage of approved students School average of approved students to the next school year Prova Brasil Number of staff per 100 students Total school staff divided by total students multiplied by 100 Prova Brasil Number of teachers per 100 students Total school teachers divided by total students multiplied by 100 Prova Brasil
Attendance School average year attendance Prova Brasil
Urban Dummy for urban or rural school Prova Brasil
Municipal Dummy for municipal or state school level administration Prova Brasil
Region Brazilian region that the school is located Prova Brasil
While it would be extremely interesting to add cost variables à la Wanke, Blackburn e Barros (2016) to the analysis in a dynamic network model, Brazilian official public databases do not include financial and costs variables for each school. However, in Brazil, the municipal secretaries of education manage most of the primary education and the federal government complements the budget through specific public policies. This funding system leaves little space for schools to choose their bundle of intermediate goods. For example, the municipal secretary of education set a uniform salary policy for all schools. To control for common municipal factors, we have also added municipal dummies to the second stage analysis.
Table 11 shows the estimates of the second stage regression for efficiency scores of the three models previously discussed both with and without municipal dummies. The list of municipal dummies was omitted from columns 2, 4 and 6 due to space limitations. Results are broadly similar for all models.
there is evidence that an increase in the proportion of teachers or of non-academic staff are associated with an increase in the average efficiency score. In addition, the percentage of approved students is positively associated to school efficiency. Students attendance is only significant in models that do not control for municipal dummies, and in this case it also presents positive results as expected.
In contrast to the results obtained in Table 8, once we control for additional factors via a two-stage regression, the is no longer a clear relation between region gdp and av-erage efficiency. In our favorite specification without municipal dummies (column 5), the Southeast stands out as the region associated with highest average scores, followed by the Center-west, the North, the Northeast and the South in this order. Therefore, it must be that some of the environmental variables added to the two-stage regression do capture
important school characteristics which are associated with the affluence of a region.
As expected from the analysis on Table 7, the socioeconomic index has a non-linear relationship with efficiency, as shown by the positive and significant estimate of the square of the socioeconomic index. By jointly looking at the estimates of the coefficients of the socioeconomic index and its square for our favorite specification (column 6), we observe a positive partial effect for schools with an index of at least 4.89, which represent around 73% of schools in the sample. So, for most schools, an increase in the socioeconomic index is associated to average increase in efficiency, even when this variable is included in the DEA, as it is the case of discretionary and non-discretionary models. This result is convergent with Brennan, Haelermans e Ruggiero (2014), Yahia, Essid et al. (2019) and Agasisti e Zoido (2018).
Regarding the infrastructure index, we have evidence of a positive relation with the efficiency in all models, suggesting that an improvement in infrastructure conditions of schools might lead to an increase in efficiency. The next subsection further explores this idea.
