English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
Introduction
In Latin America, the evaluative function was put at the service of state reform during the 1980s and 1990s 1. This decade saw the emergence of a demand for the evaluation of public programs and actions, as well as the need to develop new policies as a strategic response to a period marked by deep economic crisis which had increased unemployment and impoverished much of the population. These realities required greater rationality during decision-making, and one which was based on evaluative processes in order to maximise actions aimed at minimising problems 2) (3.
According to Solarte, evaluative processes have been based on two paradigms: postpositivism and constructivism. In the first paradigm, the need to approach reality from different perspectives is presented, using modified experimental methods, complemented by qualitative methods. The constructivism paradigm, on the other hand, questions the difference between subject and object and the distinction between the internal and external aspects of the individual. Constructivists use dialectical and hermeneutic approaches in which individual construction is accomplished through the interaction of the researcher with the research subjects, seeking consensus in the construction of knowledge 2.
Cavalcanti makes a classification of social program evaluation models from the analysis of some contemporary authors, proposing the following groups: 1) classical analytical or quantitative; 2) global alternative or qualitative; and 3) mixed (global analytic, classical alternative) 4. The first group brings together models that preference information of a quantitative nature and that are based on explanatory epistemological paradigms with objectivist positions. The models included in the global alternative or qualitative group were developed in order to understand, interpret and give meaning to the actions and activities that are carried out in social programs. These models can also be considered naturalistic and one of their main characteristics is the emphasis on the role of the beneficiaries or participants as fundamental agents for valuing what is being evaluated. The third group encompasses the mixed models (global-analytic, classical-alternative), and mixes the characteristics of the two previous groups. It combines information of a quantitative and qualitative nature, as well as giving a central role to the users of the programs 4.
Based on the above, we find that in the public domain the evaluation models mainly used are those classified as classical, analytical or quantitative. This evaluation is intended to make assessments on specific aspects of a policy, program or project, and based on the conclusions, seeks to make recommendations to contribute to its improvement. To do this, it uses a set of ordered and related information according to a methodological framework 5.
An analysis of methods for evaluating large-scale nutrition programs in Latin America found that most evaluations focus on program impact and results, using quantitative studies and biological indicators 6) (7. Other studies have established that this type of evaluation emphasises the technical, economic and financial aspects of the programs, tending mainly to look at the cost-benefit ratio and compare the profitability of the actions 7) (8) (9) (10) (11. According to the United Nations Economic Commission for Latin America and the Caribbean (ECLAC), in Latin America, the most common type of social program evaluation used, is impact evaluation. This type of evaluation aims to measure the efficiency and effectiveness of the program’s actions, thus adopting a model of results-based public administration 9) (10) (11) (12.
This trend has led to serious criticisms based on the absence of sociopolitical and cultural dimensions in evaluative processes - aspects that are hard to translate into quantifiable terms 8. Some authors argue that the evaluation and monitoring tools and approaches that are applied are often ineffective and ignore information that can help to understand the dynamics related to social change 13.
Nutritional Programs in Latin America Today
In Latin America it is increasingly common to find social programs, sectoral policies and reforms that are designed under the so-called ‘social investment’ approach 14, which is based on human capital development, with emphasis on maternal and child education, and health and nutritional status 15.
The programs that seek human capital development construct a particular notion of law and citizenship, since, as mentioned by Serrano “social policy not only operates as a dispenser of social services or repairer of the damages generated by the market, but also an integration, citizenship and national community system builder” 16. Cecchini and Martínez call this tendency “social protection understood as a citizen guarantee”, as it ultimately seeks to fulfil the economic, social and cultural rights of the population 17.
The objective of this study was to examine whether evaluations of nutrition programs included the following sociopolitical categories: (a) the right to food; (b) citizenship building; (c) civil participation in public policies; (d) women’s empowerment; and (e) territoriality in policy planning. These categories were selected for the following two reasons: firstly, as mentioned above, most of the food and nutrition programs developed in Latin America claim to be committed to the promotion of rights, human development and the fight against inequality; and secondly, we believe that these programs have a direct effect on building relationships between the state and citizens, with a particular impact on the lives of women, who are often beneficiaries and/or are co-organisers of the programs. These sociopolitical categories are part of a model proposed by us to evaluate nutrition programs on a large scale. A broader definition of the model as a whole and each category are described below 18.
The Right to Food
The right to food is defined as “the human right to consume food in a dignified manner”. It means that all members of society should have constant access to food that is sufficient, not only in terms of quantity, but also with regard to adequate nutritional quality. Food should also be safe and culturally acceptable for the person who consumes it. “It is equally crucial that access to food be stable and ecological and socially sustainable” 19) (20.
Citizenship Building
Citizenship is the result of relationships between the state and people in a specific territory. It constitutes a set of rights, privileges and duties, creating a system that determines the nature of the relations between individuals and government systems 21. Although other entities are part of social protection systems and participate in social programs, it is clear that the state is the main actor, acting as leader and executor 16. Thus, among the ultimate objectives of any social policy is the creation and preservation of a social space belonging to the sociopolitical community 22.
Civil Participation in Public Policies
Civil participation refers to the involvement of civil society entities in public issues to transform power relations as well as material, social and political conditions 23. People participate to obtain goods and services, and participation in and of itself is a tool which can democratise social ties and influence state structures and public policies 23) (24.
Women’s Empowerment
The empowerment of women is defined as “a personal and political process whereby what is personal, intimate, and corporeal cannot be separated from the political. This process challenges family power relations, where subjects live on a daily basis, that are part of the social order” 25. Empowerment is the most important strategy for women to independently gain status either individually or collectively, through participatory and transformative actions 26, and as other authors mention, refers to, “raising awareness of their own rights, with the aim of participating in decision-making from a more solid position as well as influencing the decisions that are made” 18) (27.
Territoriality in Policy Planning
The territorial policy approach seeks to conceive and construct social development in a different way. It seeks to operate in communities taking into account their range of needs rather than just traditional sectoral planning. Through its multidimensional nature, “the recognition of this characteristic should lead to strategies to overcome purely sectoral goals and to address the interrelatedness of the multiple variables that make up a more participative society and one that jointly organises its own development” 28.
As mentioned above, the objective of this study was to identify whether the sociopolitical categories described were included in the evaluations of food and nutrition programs in Latin America. The study also examined how these categories are conceived by the different actors in the evaluation, such as the institutions that develop the programs, the academics who carry them out, and the beneficiaries of the programs.
This article is part of a larger study titled “Development of an Evaluation Model for Food and Nutrition Social Programs Through the Combination of Methods” [Elaboración de un modelo de evaluación de programas sociales de alimentación y nutrición mediante la combinación de métodos, in Spanish], which aims to put forward a model that overcomes a weakness found in the most used evaluation methods of nutrition programs 18.
Methodology
An integrative literature review was carried out based on articles from academic journals and reports from governmental and non-governmental institutions. Integrative reviews are a methodology that allows the inclusion of research using different methodological approaches, including experimental and non-experimental, as well as practical experiences and policy analysis. Its goal is to develop theory and provide information for practice 29) (30. The methods for conducting integrative reviews are varied, however, some patterns must be followed 31. In this study, the following stages were developed:
Identification of the Research Question
The question that guided this work was: do the evaluations of food and nutrition programs developed in Latin America include the sociopolitical categories: right to food, citizenship building; civil participation in public policies; women's empowerment; and territoriality in policy planning? In the case of a positive response: how are these categories conceived for the different actors in the evaluation? Evaluation actors were defined as the institutions that carry out the programs, the academics conducting the evaluations, and the beneficiaries of the programs.
Definition of the Selection and Search Criteria
The databases used for the search of the articles were: Scielo, ScienceDirect, PubMed and BIREME. The following key words were used in both Spanish and English: “evaluation and public policies”, “evaluation and social policies”, “evaluation and social programs”, “evaluation and food security” and “conditional cash transfer and evaluation”.
Articles published between January 2005 and December 2013, written in the English, Spanish or Portuguese, and that were available in full text, were selected for inclusion. This search returned 719 items. Articles with program evaluations from countries outside of Latin America were excluded, as were those which did not evaluate programs for populations but were rather programs with small groups or groups with specific health conditions, such as clinical studies. Finally, evaluations that did not focus on food and/or nutrition, and any repeated articles, were also excluded. This process resulted in 52 articles being selected (Figure 1).
In order to search for reports of governmental and non-governmental institutions on the subject, the websites of different organisations available to the public were reviewed using an intentional search strategy. Evaluations of social programs were selected based on having a nutrition component and being within the established period of time. In addition, a Google search was performed using the exact words “evaluación de programas sociales” [social program evaluation] with the specific time interval according to the inclusion criteria and in Spanish. Three hundred and thirty-two documents were found. The search was carried out in Google Scholar with the same words, restricted to the established period of time, in Spanish, and excluding results of citations and patents. Seven hundred and fifty-one entries were obtained giving a total of 1,083 documents. In both cases, the exclusion criteria set forth in the previous paragraph were applied which meant that 82 reports were obtained from institutional and Google web pages.
Evaluation of Studies Included in the Review
Once the above criteria were applied, no other conditions were applied to the evaluations analysed, nor were there specific methodological requirements.
Defining the Information to be Extracted from the Selected Studies
After selecting the articles and reports, we proceeded to carry out a first level analysis with a descriptive approach in which these general criteria were identified: name of the program, country, type of program, level of implementation (national, regional, local), year of publication, evaluation objective, name of the institution, evaluation method, evaluation approach, methodological design and timing of the evaluation (ex-ante, ex-post). It is important to note that although the above criteria were applied to all of the documents, in some cases it wasn’t possible to identify some analysis components.
A second level analysis searched for the central concepts prioritised in our study: the right to food; construction of citizenship; civil participation in public policies; women’s empowerment; and territoriality in policy planning.
To identify this information, the coding of all texts was done through the program ATLAS.ti 6.0, which allows for text to be coded and for subsequent retrieval of coded key citations. Categories and subcategories were created for analysis and a network of relationships between codes, categories and text extracts was established. In addition, the quantitative information found in the analysed texts was exported to Microsoft Excel for analysis.
Finally, the codes and citations identified in the texts were classified according to their origin in the following groups: a) characteristics of the program; b) evaluators' theoretical basis; c) evaluators’ findings; and d) evaluators’ opinions or recommendations.
The research protocol of this study was approved by the Bioethics Committee of the University Research Headquarters of the Universidad de Antioquia, Colombia, approval record number 13-11-546.
Results
Characteristics of the Evaluations Analysed
A total of 52 articles and 82 reports were analysed. Reports are evaluations published in documents found on the web, more precisely on the pages of governmental and non-governmental organisations. It was found that the evaluations published in articles of academic journals were carried out by universities (80%), by governmental organisations (12%) and lastly, by expert organisations (8%). On the other hand, evaluations published in the reports are more heterogeneous: most were carried out by expert organisations (58%), by universities (25%), governmental organisations (7%) and lastly, by international bodies such as the Inter-American Development Bank, the World Bank and the Food and Agriculture Organization of the United Nations (10%).
Characteristics of the Evaluated Programs
According to evaluations completed by type of program, it was found that 66% of evaluations corresponded to conditional cash transfer programs, mostly belonging to the Bolsa Família program from Brazil and the Oportunidades program from Mexico. The other programs evaluated included maternal and child nutrition, and food security programs (10% each), supplementation and support programs (6%), and food and nutrition education programs, programs for workers, and other programs (less than 5%).
Sociopolitical Categories Analysed in the Evaluations
The following are the sociopolitical categories evaluated in the food and nutrition programs in Latin America that we analysed.
Right to Food
The right to food as a category included in the evaluation of nutrition programs was found only in two cases. One of them evaluated the Asignación Universal por Hijo [Universal Child Allowance] program in Argentina and the other was an evaluation carried out by the Food and Agriculture Organization of the United Nations (FAO). Firstly, it should be noted that for some evaluators the right to food must be integrated with other rights in order to make them all more effectively attained. For example, an eight-country FAO evaluation of school feeding programs states the need to progressively integrate the role of food services at schools as an important food and nutrition security policy towards the fulfilment of the human right to food, as well as other rights of children, such as education and health.
We also found the debate about the role of conditionalities. That is, some authors argue that these are contrary to the guarantee of human rights and to the universalisation of programs and policies.
“From a rights-based approach, conditionalities present limits that are important to consider, particularly when such limits become punitive so that their non-fulfilment predetermines the comprehensive perception of the benefit or becomes a cause of exclusion” - P11AR evaluators’ theoretical basis.
Hence, a proposal is put forward to evaluate rights, from the point of view of the progress that a particular policy or program has towards universalisation as a right without restrictions or conditionalities.
“It is time to review conditionalities and strengthen mechanisms so that the state guarantees effective access to universal and unconditional social policies for every person, whether they are children, older adults, disabled people, formal workers or informal workers” - P11AR evaluators’ opinions or recommendations.
The Construction of Citizenship
In the integrative review carried out, we found that in general the most repetitive modality is welfare assistance as the programs conceive their actions to be directed towards “beneficiaries”, sometimes called “recipients”. These programs do not start from the notion that the participants are citizens, on the contrary, they conceive them as individuals with particular conditions that disadvantage them and hence the need to intervene in a focused way, as expressed by one of the evaluations analysed:
"The beneficiaries of this social protection system are people living in structural poverty, below the poverty line, and belonging to special needs groups." - P150BRA evaluators’ findings.
When paying attention to the notion of citizenship that the evaluators have, it was identified that they consider the programs and efforts carried out as welfare strategies directed at the most vulnerable and poorest populations, following a line similar to that stated in the foundations of the programs. For this reason, the evaluators focus their research on the registering and targeting processes of the programs, as presented in an evaluation of a program in Brazil:
"Problems of lack of supervision in the transfer of benefits were evidenced in this work, since families outside of the profile required by the program received benefits. Nevertheless, it seems that the Bolsa Família program serves, in fact, the neediest families. - P30BRA evaluators’ findings.
This notion of aid is also what the beneficiaries of the programs have:
"There is another similar element here, the beneficiaries of both the ‘Familias en Acción’, and the ‘Bolsa Família’ programs do not believe that the benefit they receive is a right. When asked whether they would classify the program as a right or as an aid, the beneficiaries considered it to be an ‘aid provided by the Government’, or even ‘fallen from Heaven’” - P34BRACOL evaluators' findings.
Civil Participation in Public Policies
The reviewed evaluations identified that participation and social protection activities are those where the beneficiaries take part in the verification of service delivery of the programs. In evaluations carried out in Mexico on the Oportunidades program we found that:
"In all the communities studied there are committees composed of members of the community, almost always women, responsible for reporting, monitoring and supervising the adequate distribution of supplements, as well as other benefits granted by the program” - P86MEX evaluators' findings.
The conception of the programs shows that citizen participation is conceived as a responsibility that falls largely on the people themselves since:
“The proper functioning of the Social Comptrollership Committees depends on the willingness and motivation of the beneficiaries to participate. An area of opportunity is in the dissemination of the importance of the social comptroller to the beneficiaries in the Madres Trabajadoras y Padres Solos [Working Mothers and Single Parents] modality” - P132ME evaluators' findings.
There are also more democratic perspectives of participation:
“Government management consists of orienting, conceptually and programmatically, social policy around social integration (strengthening social fabric, promoting solidarity) and social inclusion (building ramps of inclusion) with the aim of fulfilling and exercising rights through the real participation of the social group. From this perspective, it is possible to advance the building of citizenship and the consolidation of democracy” - P37AR program characteristics.
In some evaluations social participation in municipalities or territories is examined via assessing if the people belong to, or at least are aware of, participation entities:
"Knowledge of the participation entities is assessed by: (a) Knowledge of the existence of the Municipal Health Council; b) participation in the Health Council; c) participation in a discussion forum on human health problems;" - P36BRA evaluators’ theoretical basis.
We also found that some programs attempt, with the aim of strengthening their objectives, to carry out work and to network with various actors from the public and private sectors, civil society organisations, educational institutions and universities, as well as with international organisations.
“Crecer was a national social development strategy created in 2007 through cooperation and partnership between public entities (national, regional and local governments) and private entities (NGOs, international cooperation and civil society) that worked together on poverty reduction and social exclusion issues. The Crecer strategy coordinated many institutions, ministries and programs in the field of social development” - P151PE program characteristics.
Women’s Empowerment
Analysis of this category in the evaluations found some discussion on the role of women in program strategies and on the ways in which gender equity is assessed. The role of women in programs is generally seen as part of the empowerment of women and is presented as progress and recognition. However, this was also questioned as, according to some evaluators, this central role requires that women carry out domestic activities and care for children, limiting their ability to venture into other areas such as employment and education.
“There is a gender bias tendency in public policies that attempts to strengthen the value placed on the traditional reproductive role of women. However, in the design of each program and in official documents, women are given preference as beneficiaries of the policies, with the aim of expanding their power (empowerment) and autonomy” - P34BRACOL evaluators' findings.
Some evaluations show signs of progress on gender equality issues but also the existence of certain problems women face. In Mexico, a study on the processes of the Centros de Cuidado Infantil [Childcare Centres] program showed the following:
“The role of women in the workplace has changed dramatically in recent decades. In the 1950s, this sector's share of economic activity in Mexico was 13.5%. In 1995 it was 36.8%, and in 2009 women's participation reached 40.8%. The greater participation of women in economic activities does not mean however that men are acquiring more care and domestic activities and this has led to the problem known as the ‘care crisis’ - whereby paid work and unpaid domestic work are not balanced within families” - P132ME evaluators' findings.
With regard to how women's empowerment is evaluated, it was found that women are expected to gain empowerment as a result of the actions of social programs and their proper running. This effect should be considered as an outcome of the family and social conditions in which they live. However, the results may vary depending on the conditions of the country and the programs themselves. In Haiti, for example, the following was expected in a strategy based on conditional cash transfer programs:
“This measure refers to the principle of “anti-exploitation”, which refers to living a life free of all economic dependence that makes women vulnerable to the misuse of their bodies by others” - P154CE program characteristics.
In Brazil however, based on the activities and running of the Bolsa Família program, it can be observed that it had the following effects:
“A greater transparency of the beneficiaries as consumers because the benefit gave them greater purchasing power: the affirmation of the authority of these women in the domestic space, derived to a large extent from the purchasing power obtained by the benefit, which necessarily causes changes in the traditional gender relations; and the changes in the beneficiaries’ perceptions of themselves as citizens, mainly due to the regulation of their documents, which was a prerequisite to participating in the program” - P14BRA evaluators’ findings.
Territoriality in Policy Planning
With this category, we found that the evaluations first inquire about the physical infrastructure and human resources that the programs have, so they can respond to the conditionalities of education and health that make up part of the conditional cash transfer programs. They then evaluate the coordination that programs and policies have achieved in order to provide better care to families. Finally, research is done on the coverage of programs on populations in a given geographical area.
The social services established in the territories that are most commonly evaluated in the reviewed research are health and education. One possible explanation for this situation is that these two concepts are the basis for achieving adequate human capital development in the future. This is demonstrated by the following statement from an evaluation of the Asignación Universal por Hijo program in Argentina:
“It is also essential to discuss the institutional competence of the health and education sectors - both at the level of their national leadership and in the provincial jurisdictions where provision is their responsibility - in terms of the damage suffered and the lack of continuous investment that has affected coverage, quality and social inclusion” - P11AR evaluators’ opinions or recommendations.
In relation to the coordination of policies and programs within a territory, we found that this trend has been developed with the aim of providing better possibilities to the population, ensuring that these activities are more assertive and comprehensive.
“Coordination between the Bolsa Família program and other public policies reinforces the idea of poverty as a complex phenomenon that is not restricted to insufficient income. Therefore, battling it must not be limited to the transfer of financial resources. In regard to this, the coordination of this program with complementary activities and services is essential as it develops the families’ capabilities and expands their options, contributing significantly to their social inclusion” - P130CEP program characteristics.
Finally, in the area of territoriality, we found that some research evaluates the coverage of the programs by geographical area, as well as the mechanisms used for the selection of the territories in which the programs are carried out. For the selection of territories, we found that in Mexico information from the programs implemented is used to develop new strategies to reach populations that do not have social interventions:
“The program operates in localities that are not part of the Oportunidades program nor the Diconsa program. For this section, the Coordination Unit of the Oportunidades program sends the Administrative Unit Responsible for Program Operation a list of the places where it will not be conducted. In the first year that Programa Alimentario en Zonas Marginadas [Food in Marginalised Zones Program] operated, the Oportunidades program reported that it would not take action in 90,492 localities” - P94MEX program characteristics.
Discussion
This study analysed the sociopolitical categories: the right to food; citizenship building; civil participation in public policy; women's empowerment; and territoriality in policy planning and found that although they are not prioritised in evaluations of nutrition programs in Latin America, in most cases they are the intended basis of these programs. It seems that institutions that promote program evaluations, such as international financial institutions and government agencies in each country, have not developed evaluation models to incorporate these new program objectives within the dimensions to be assessed. However, this study also found that the contributions made by some evaluators can be considered as a first step towards clearer ways of evaluating these categories.
In relation to the right to food, the main contribution of some authors is that conditionalities such as those established in the conditional cash transfer programs, and even inclusion criteria such as the poverty of the households or the socioeconomic conditions of the neighbourhoods, constitute a denial of this human right. Changes in those requirements to achieve universal coverage may be an indicator for evaluating this category.
It is important to emphasise that the "right to food" has developed both from a theoretical and a practical point of view and can be re-examined to develop new evaluation tools 20) (32. For example, the agreements on social and economic rights signed by most Latin American states have defined that social rights are characterised by being universally guaranteed and that their guarantee must be progressive, which means that they cannot weaken the quality of goods and services provided 33.
For the construction of citizenship category, in most evaluations analysed for this study, beneficiaries are considered passive recipients of government aid due to particular conditions that make them unable to support themselves. According to some authors, this type or state-beneficiary relationship is called “bienestarismo” [welfarism]; it is defined as a social protection strategy created by European colonial countries 16) (34) (35 that have been transformed but whose foundations remain the same. In this sense, our findings coincide with the description made by Benelli and Costa-Rosa: according to these authors, in some Latin American countries there is now a renewed welfare, which they identified as “social aid as a state policy”, which purports to “promote the care of individual social rights as a duty of the state and society to eradicate poverty” 36. However, “it continues to be permeated by a naturalising vision of social problems and particularly of the phenomena of poverty”. In other words, in this new vision the structural causes of unequal opportunities and poverty are not intended to be eliminated by social policies.
With regard to the category of civil participation in public policies, it was found that beneficiaries do not participate at the higher levels where decisions are made about nutrition programs, nor do they control the allocation of financial and physical resources. On the contrary, they are responsible for operational tasks to ensure the proper functioning of the programs at the lower levels. As previously mentioned, the sociopolitical categories: right to food; construction of citizenship; and civil participation in public policies are strongly related to each other. Due to welfare and low participation in strategic decisions, the idea of charity rather than rights is reinforced in the program beneficiaries.
The evaluation of women’s empowerment deserves special attention because women are the beneficiaries of most programs and usually contribute to the tasks related to the provision of services and goods. They are also responsible for demonstrating fulfilment of household requirements for improving human capital, such as children’s school attendance and health check-up programs. Whilst most programs insist on improving the social and private role of women, it is clear that no indicators have been established to measure these achievements, and discussions continue on the effects that these programs have on women's emancipation, since some can reinforce cultural stereotypes that circumscribe women to the private sphere and/or overwhelm them with other domestic duties. Regarding the presence of this debate in the evaluations, a study aimed at exploring gender relations extracted from the Oportunidades program in Mexico found that, “gender relations are not limited to household institutions such as family and spousal relationships, they are reproduced through other patterns of social interaction, in this case through the fulfilment of the requirements of the Oportunidades program” 37.
Further research on the same theme stresses that social programs have to balance their goals in terms of children's well-being, with women's needs and expectations: “above all, women need a reliable source of income and sustainable paths out of poverty, which are at the same time more realistic and imaginative than the maternalistic options that are currently in place. The limits of programs such as Oportunidades are evident, not only in their selective approach to addressing social needs, but in their narrow vision of how to overcome poverty” 38.
Our findings are in part similar to those found in a study that explored the relationship between rights and empowerment in social protection programs that fight hunger in India, Kenya and Ethiopia. The author concluded that advances in this relationship are not always connected, and that some programs have a greater impact in terms of social networks and beneficiaries participating in the objectives of the program. The author explains this in part by the particular social, political and cultural contexts and historical processes found in each country analysed 39. In conclusion, nutrition programs of social protection are demanding new goals and efforts. Therefore, it is necessary to apply evaluation criteria to account for these new underpinnings in order to establish consistency between government institutions’ discourse, and the reality of their actions.
