ARTÍCULO ORIGINAL / ORIGINAL ARTICLE / ARTIGO ORIGINAL

Evaluation of formal educational processes for healthcare professionals

Evaluación de los procesos educativos formales para los profesionales sanitarios

Avaliação de processos educativos formais para profissionais da saúde

Eloá Otrenti¹; Vera Lúcia Mira²; Sarah Marília Bucchi³; Jairo Eduardo Borges-Andrade⁴

¹RN, Master. Escola de Enfermagem, Universidade de São Paulo – USP. São Paulo (SP), Brasil. email: eloaotrenti@usp.br.

²RN, Ph.D. Professor, USP. São Paulo (SP), Brasil. email: vlmirag@usp.br.

³RN, Ph.D candidate. USP. São Paulo (SP), Brasil. email: smbucchi@uol.com.br.

⁴Psychologist, Ph.D. Professor, Universidade de Brasília. Brasilia (DF), Brasil. email: jairo@unb.br.

Receipt date: Feb 18, 2013. Approval date: Oct 7, 2013.

Article linked to research: ''Avaliação de processos educativos formais para profissionais da saúde''.

Subventions: none.

Conflicts of interest: none.

How to cite this article: Otrenti E, Mira VL, Bucchi SM, Borges-Andrade JE. Evaluation of formal educational processes for healthcare professionals. Invest Educ Enferm. 2014;32(1): 103-111

ABSTRACT

Objective. Analyzing the scientific literature on the evaluation of formal educational processes for healthcare professionals. Methodology. Integrative literature review in which were reviewed the following databases: VHL, Pubmed and Cochrane. The final sample was composed of 19 articles in Portuguese, English and Spanish published from 2000 to July 2010. The subject of study was the evaluation of formal educational processes for health professionals, which had at least the abstract available online. Results. There is no use of a systematic methodology to evaluate the formal educational processes in this study group. The evaluation focus mainly on the learning of participants, with little attention to the teaching process. There are no evaluations on the impact caused by this type of training in institutions and users of the health system, which can incur the risk of reducing the value of formal education processes. Conclusion. A full evaluation of the formal educational processes for professionals during a longer time is important to assess the impact of these processes and provide information about the necessities of continuing education of this population.

Key words: review literature as topic; inservice training; health manpower.

RESUMEN

Objetivo. Analizar la producción científica sobre evaluación de los procesos educativos formales para los profesionales sanitarios. Metodología. Revisión integrativa de literatura, en la que fueron revisadas las bases de datos BVS, Pubmed y Cochrane, con una muestra final de 19 artículos en portugués, inglés y español, publicados desde 2000 a julio de 2010, cuyo tema de estudio fue la evaluación de los procesos educativos formales para los profesionales del área de la salud, y que, al menos, los resúmenes estuvieron disponibles online. Resultados. No se utiliza una metodología sistematizada para evaluar los procesos educativos formales en este grupo de estudio. El foco de evaluación es principalmente el aprendizaje del participante, con poca atención al proceso de enseñanza. Hay ausencia de evaluaciones sobre el impacto de esta formación en las instituciones y en el usuario del sistema de salud, con lo que se puede incurrir en el riesgo de reducir el valor de los procesos educativos formales. Conclusión. Es importante que se evalúen integralmente los procesos educativos formales en los profesionales por un tiempo más prolongado, con el fin de analizar el impacto de los mismos, así como disponer de información sobre las necesidades de educación continua en esta población.

Palabras clave: literatura de revisión como asunto; capacitación en servicio; recursos humanos en salud.

RESUMO

Objetivo. Analisar a produção científica sobre avaliação dos processos educativos formais para os profissionais sanitários. Metodologia. Revisão integrativa de literatura, na que foram revisadas as bases de dados BVS, Pubmed e Cochrane, com uma mostra final de 19 artigos em português, inglês e espanhol, publicados desde 2000 a julho de 2010, cujo tema de estudo foi a avaliação dos processos educativos formais para os profissionais da área da saúde e que estiveram disponíveis online ao menos o resumo. Resultados. Não se utiliza uma metodologia sistematizada para avaliar os processos educativos formais neste grupo de estudo. O foco de avaliação é principalmente a aprendizagem do participante, com pouco atendimento ao processo de ensino. Há ausência de avaliações sobre o impacto desta formação nas instituições e no usuário do sistema de saúde, com o que se pode incorrer no risco de reduzir o valor dos processos educativos formais. Conclusão. É importante que se avaliem integralmente os processos educativos formais nos profissionais por um tempo mais prolongado, com o fim de avaliar o impacto destes processos e de dispor de informação de sobre as necessidades de educação continue nessa povoação.

Palavras chave: literatura de revisão como assunto; capacitação em serviço; recursos humanos de saúde.

INTRODUCTION

Increased investments in formal educational activities for health professionals have emphasized the importance and necessity of implementing evaluation strategies that are able to show the necessary information to feedback programs and therefore improve the healthcare practice, which is the main purpose of the educational activities in health. We found a wide range of concepts when naming the formal educational processes for professionals, which is why in this study we preferred to use the term formal educational processes, that comprises actions covering: Situational Diagnosis (survey and analysis of needs), Planning (preparation, approval, implementation) and Evaluation. This last step should be a way to measure the quality of education, and due to that reason, it must be demystified to become an important development tool.¹

The evaluation of training can be a systematic process of gathering information that can enhance formal educational activities for professionals. Thus, the main objectives of training evaluation are: controlling the process, feeding back the system, making decisions about training as well as making it able to promote changes in the environment.² In literature many models of training evaluation are found. The more traditional types only assess the results of training, while newer models also consider environmental variables and the characteristics of trainees, among others.³ In 1976 Kirkpatrick proposed the first structured evaluation model consisting of four levels called reaction, learning, behavior and results, and suggested that they should be sequential, linear and strongly correlated with each other. However, researches ^4-6 question this relationship.³ Hamblin⁷ added a level to the Kirkpatrick model but the author points out that sometimes this division may not be possible due to not showing clear boundaries. The Model of Integrated and Summative Evaluation – MISE is the most used in the national scene, consisting of five items, namely: inputs, procedures, processes, results and environments.⁸ The last item – environments - subdivides in necessities, support, dissemination and long-term effects.

The nursing team has a large number of professionals with different levels of training performing several work activities that vary according to the employer institution. These workers generally participate in various educational activities often not directly related to their needs or interests. By recognizing the importance of assessing formal educational activities and considering its undeniable complexity, we understand that the first step to propose an efficient evaluation strategy is to know previous productions in this sense. Thus, the objectives of this study were to analyze the published scientific literature on evaluation of formal educational processes of professionals in the health area and analyzing the characteristics of the instruments for evaluation of formal educational processes of health professionals found in the literature review.

METHODOLOGY

An Integrative Literature Review was carried out to achieve the proposed objectives. In this method, the primary research data are analyzed for compiling a summary of the knowledge produced about a topic. It is useful when there is little research on a given topic.⁹ When locating and integrating the studies produced by different authors about certain topic, the integrative review helps to identify at what stage is the knowledge produced or even clarify unresolved issues. Among the surveyed authors^10-16 a relative diversity of phases was found to develop an integrative literature review. In the present study were established seven stages, which were based on the diversity of references herein cited: i ) Problem identification and definition of the research question of the review , ii ) Definition of inclusion and exclusion criteria of the articles ; iii ) Bibliographic review , iv ) Definition of information to be extracted from the reviewed studies, v) Analysis of the studies included in the integrative review vi ) Discussion and interpretation of results; vii ) Synthesis of knowledge.

In this integrative review was formulated the following question: How are the formal educational activities for health professionals evaluated? The problem that triggered this question was the scarcity of material produced on the subject, especially in healthcare and due to the need to discuss this topic. The inclusion criteria for the selected articles in this integrative review were: Language (Portuguese, English and Spanish); Publication period (from the year 2000 until July/2010); Object of study (evaluation of formal educational activities for health professionals); Availability (online abstract with free access). The exclusion criteria were: Publications in the form of dissertations, thesis, monographs, books and reports; approaches of educational actions for academics or users of health services; reviews of any style, bibliographic, integrative, meta-analysis and meta-synthesis by making it impossible to fill out the instrument of data collection.

Aiming at selecting the articles that met the inclusion criteria, was conducted a thorough reading of the title and abstract of each study in order to verify that the research question was answered. Then, the included articles were accessed in full for completing the instrument and the exclusion criteria were used at this stage. The period of collection of articles was between May and July/2010 in the Virtual Health Library - VHL. This library has several databases, including the Medical Literature Analysis and Retrieval System online - MEDLINE, the Latin American and Caribbean Literature on Health Sciences - LILACS and the Cochrane Library, which were used in this integrative review.

The full articles were accessed through the Portal Services SIBI-USP, that provides electronic titles for the USP (Universidade de São Paulo) community in the existing equipment at the University. Search strategies were adapted for each base, considering the peculiarities of each and always keeping in mind the research question of the integrative review as well as the inclusion and exclusion criteria. Therefore, it was possible to maintain the coherence of the selected material. To search for articles in the selected databases were used the following Descriptors in Health Sciences – DeCS: in-service training, continuing education, training, development, evaluation studies as topic, patient care staff, educational evaluation and evaluation of programs.

For the search in the Cochrane Library the strategies had to be adapted. To this end the used terms were staff development, continuing education and program evaluation. A search was also carried out in Pubmed, a virtual library that adds MEDLINE, life science journals and books. As this was the last base to be searched no article was maintained, as described below. Despite the fact that the term continuing education is not considered a descriptor, a search was still carried out with it due to its significant use in the national literature. This was the only used term that was not indexed in the DeCS. At the end of the search and selection were included 19 articles in total, which were analyzed in detail for this integrative review. After reading the articles and then defining which were included, the instrument for data collection of the selected articles was filled out, which includes the database of origin, identification of the study and the authors, bibliographic reference, problem/hypothesis or research question, goals, method, results, conclusions, recommendations, tool for evaluating the educational process used in the included study.

The analysis of the included studies - as in primary research - must be critical and seek for an answer to the research question supported by a theoretical framework. In some researches, the type of study adopted was not clear, so it was necessary to use a single bibliographic reference to name the methodological designs found in the review. ¹⁷ To classify the objectives of evaluation of the studies by an assessment level recognized in the literature, we chose to use the Hamblin⁷ referential. At this point it is important to define each level and what is evaluated in each of them. The level 'Reaction' regards the opinion of the trainees about educational intervention. At this stage are assessed the performance of the facilitator and the student, the infrastructure of the place, the relationship between teachers and students. Among the trainees is evaluated the interest in learning, among many other variables.

On the second level, called 'Learning', the focus is on the expansion or change in knowledge after the training. In general, cognitive tests are used for that, to measure what was learned after the intervention. Regarding the level 'Behavior in the Position', it seeks to ascertain whether there were changes in the work routine after the training. The level called 'Organization and Final Value' seeks to measure more structural effects at the institution where workers were trained, and the 'Final Value' is concerned mainly with the relation between costs and benefits of the educational action.

Although this is an old model, it is the most used internationally. Other variables shall be considered when evaluating a formal educational process for professionals. For that, there is the Model of Integrated and Summative Evaluation - MISE which proposes that the six components, Necessities, Inputs, Procedures, Processes, Dissemination and Support predict two other components, Immediate Results and Long-Term Results.⁸ As this is a literature review which does not involve humans, the study has not been approved by a Research Ethics Committee. It is also important to mention that there was no conflict of interest among the authors.

RESULTS

Regarding the year of publication, no concentration in any particular period was found. Regarding the formation of the first author, few journals have provided this information, and only in eight articles (42.1%) it was possible to identify the professional field of the first author. Among them, four were teachers and one was also a nursing supervisor. We also searched for the institution of the first author and this information was available in all items included in the integrative review. Nine articles (42.1%) were carried out by university professionals and seven (36.8%) by professionals working at hospitals. The other three studies were carried out by professionals working in health programs, management centers and health departments. The problem or research question of the study and hypothesis were explicitly found in only four studies (21%).

Regarding the methodological characteristic, eight (42.1%) of the 19 evaluated studies had the study design explicitly described. The other 11 studies (57.9%) were classified by the reviewer according to the chosen reference.¹⁷ Among the selected articles 16 (84.2%) had quantitative methodological approach and three others (15.8%) used the quali-quantitative approach. Nine studies (52.6%) with non-experimental design were found, eight (42.1%) of experimental character and two (10.5%) quasi-experimental.

In relation to the place where the study was carried out, there was predominance of hospitals (47.4%), followed by basic health units (15.8%) and health services such as medical centers (15.8%). We also found studies in home care, pre-hospital and mental health care, nursing homes for the elderly, all cited just once (5.2%). Among the 19 selected articles, six (31.6%) were carried out in the United States. Brazil, Australia and Canada had two (10.5%) studies each. Azerbaijan, Holland, Ireland, United Kingdom, Pakistan, Taiwan and Uganda had one study each. The objectives of evaluations of educational actions described in the studies were grouped according to the evaluation standards of Hamblin⁷ as shown in table 1.

Table 1. Categorization of evaluation objectives of the reviewed studies according to evaluation levels of Hamblin

Table 1.

Only one study assessed the impact of the educational process by measuring the change in care delivery, which is a primary aim of educational activities in health. Impact evaluation is the transfer of the contents learned in training to the work practice.¹⁸ The efficacy measurement seeks to understand if the educational process has contributed to increase the knowledge of professionals.

The 19 included articles were grouped into two themes according to the focus of the evaluation: education process and the final outcome. Only two turned part of the attention to the education process, yet efficacy was also assessed. With respect to the evaluation instrument, studies were initially grouped into three categories: Eleven described the instrument (57.9%), six did not describe the instrument (31.6%) and two showed the attached instrument (10.5%). The described information about the instrument referred to the number and contents of questions; type of questions, open or closed; whether identification of the trainee was necessary and at what time was the instrument delivered.

Regarding the type of question, the multiple-choice kind was used in six studies. Six studies used essay questions and multiple-choice. Only one study used interviews with training participants. The Likert scale was mentioned in eight studies; in two of them this was the only strategy used and in three this scale was combined with multiple-choice questions. In two other studies it was combined with essay questions. Finally, a study carried out a systematic observation added to the scale. As for training methods and types of intervention, in the present review all the articles showed group educational actions. As for the strategies, whenever identification was possible, we found lectures, reading and discussion of texts, workshops, role-playing and exercises.

DISCUSSION

Although the present study did not aim to know specific formal educational actions for nursing managers, we believe that these professionals can benefit from the results found to guide the planning and implementation of educational activities for the nursing staff. The first level of evaluation, 'Reaction', was conducted in seven studies. At this level were evaluated the instructor, content, care to individual and collective training needs, the physical structure and the participation of the subject. In all these studies, this level was combined with another; six combined the evaluation of 'Reaction' with 'Learning'. The acquisition of cognitive knowledge, i.e. learning, was measured in 14 studies. The change in behavior after the educational action was assessed in eleven studies. Two studies showed strategies to evaluate the effect of training on the organizational context.

The use of learning and satisfaction evaluation provides important results, however, this is a restricted use of evaluation tools for educational actions, and insufficient to subsidize changes that meet the needs of individuals, the organization and the users. It is not enough to look just for the satisfaction of the trainees and the acquisition of knowledge. It is necessary to investigate the application of this new knowledge in the workplace, and its impacts on institutional outcomes and on the assistance provided to the users of the health system.

The evaluations of pre-training knowledge could be used to diagnose the training needs. However, this practice is not common, or at least not reported in the researched literature. Evaluation is recommended at all levels, advancing to the evaluation of satisfaction and learning because this way it would be possible to measure the effects of educational actions for professionals. It is important to consider the economic analysis aspect in the evaluation of educational programs.

Regarding the terms adopted by the authors of the selected articles that referred to the formal educational actions for professionals, we found a great diversity of options. In none of the studies was possible to find the use of a particular word to define a specific type of educational activities for professionals. We believe it is important to adopt a theoretical model to guide the formal educational actions for healthcare professionals and urgently demystify some concepts and strategies adopted. The type of evaluation used also depends on that.

When observing the trained public we found seven studies that carried out educational actions for two or more professional categories. The conduction of training in multidisciplinary teams is a key feature of Continuing Education in Health (Educação Permanente em Saúde - PNEPS) that addresses 'the team and the group as a structure of interaction, avoiding disciplinary fragmentation'.¹⁹ It is important to consider the teaching process adopted for adults. The reference of andragogy reinforces the differences of the teaching-learning process of adults; differentiated strategies that consider prior knowledge and enable strong correlation with daily work should be used. Adults learn more when assigning concrete meaning to the subject taught. The PNEPS proposes the use of a meaningful and problem-based learning methodology that promotes changes in the health relationships and practices.¹⁹

The effective implementation of strategies for assessing formal educational actions for healthcare professionals faces some barriers, namely: work overload; uncertainty regarding the evaluation as a possible form of punishment; and negative evaluation of the institution. Some organizational characteristics may also affect the implementation of the skills and knowledge acquired in training. Organizational changes may also be necessary for the application of knowledge by professionals. These aspects are intervening variables in the evaluation process.

Adopting more rigorous research designs such as the experimental may help to elucidate the variables involved in the formal educational process for professionals and in the identification of predictors of learning and impact. Thus it would be possible to understand the determinants of success of an educational action. The adoption of a theoretical and methodological framework is essential to guide the development and validation of instruments and techniques of evaluation, making it possible to compare results from different educational processes in order to analyze and pursue the best practices.

With the analysis of the results it was possible to realize the importance of monitoring professionals that have been trained for a long time, not just in the sense of capturing knowledge retention, but also providing support in case of any doubts arising in the work routine. Thus, it is possible to enhance knowledge and provide continued educational actions that meet the needs of professionals. It is important to carry out evaluations of formal educational processes for professionals periodically and systematically because without this there is the risk of reducing the value of training, perpetuating misleading and isolated educational actions.

REFERENCES

1. Castro LC. Percepção dos enfermeiros sobre o processo de avaliação da aprendizagem nos treinamentos desenvolvidos num hospital da cidade de São Paulo [Dissertation]. São Paulo: Universidade de São Paulo. Escola de Enfermagem; 2006.         [ Links ]

2. Goldstein IL. Training in work organizations. In: Dunnette MV, Hough LM. Handbook of industrial and organizational psychology. California: Consulting Psychology Press; 1991. P. 507-619.         [ Links ]

3. Meneses P, Zerbini T, Abbad G. Manual de treinamento organizacional. Porto Alegre: Bookman artmed; 2010.         [ Links ]

4. Alliger G, Janak E. Kirkpatrick's levels of training criteria: Thirty years later. Pers Psychol. 1989; 42(3):331-42.         [ Links ]

5. Tannenbaum SI, Yukl G. Training and development in work organizations. In: Annual review of psychology. California: Annual Reviews; 1992.         [ Links ]

6. Abbad GS, Borges-Andrade JE, Gama ALG. Treinamento: Análise do relacionamento da avaliação nos níveis de reação, aprendizagem e impacto no trabalho. Rev Adm Contemp. 2000; 4(3):25-45.         [ Links ]

7. Hamblin AC. Avaliação e controle de treinamento. São Paulo: McGraw Hill do Brasil; 1978.         [ Links ]

8. Borges-Andrade JE. Avaliação somativa de sistemas instrucionais: integração de três propostas. Tecnol Educ. 1982; 11(46):29-39.         [ Links ]

9. Ursi ES. Prevenção de lesões de pele no perioperatório: revisão integrativa de literatura [Dissertação]. Ribeirão Preto: Universidade de São Paulo. Escola de Enfermagem; 2005.         [ Links ]

10. Cooper HM. The integrative research review: a systematic approach. Beverly Hills: Sage Pulications; 1984.         [ Links ]

11. Campos RG. Burnout: uma revisão integrativa na enfermagem oncológica [Dissertação]. Ribeirão Preto: Universidade de São Paulo. Escola de Enfermagem; 2005. 158p.         [ Links ]

12. Mendes KDS. Transplante de fígado: evidências para o cuidado de enfermagem [Dissertation]. Ribeirão Preto: Universidade de São Paulo. Escola de Enfermagem; 2006.         [ Links ]

13. Ganong LH. Integrative reviews of nursing research. Res Nurs Health. 1987; 10(1):1-11.         [ Links ]

14. Beyea SC, Nicoll LH. Writing an integrative review. AORN J. 1998; 67(4):877-80.         [ Links ]

15. Broome ME. Integrative literature reviews in the development of concepts. Em: Rodgers BL, Knafl KA, coordenadores. Concept development in nursing: foundations, techniques and applications. Philadelphia: W.B. Saunders; 1993. p. 231-50.         [ Links ]

16. Stetler CB, Morsi D, Rucki S, Broughton S, Corrigan B, Fitsgerald J, et al. Utilization-focused integrative reviews in nursing service. Appl Nurs Res. 1998; 11(4):195-206.         [ Links ]

17. Polit DF, Beck CT, Hungler BP. Fundamentos de pesquisa em enfermagem: métodos, avaliação e utilização. 5th ed. Porto Alegre: Artmed; 2004.         [ Links ]

18. Mira VL. Avaliação de programas de treinamento e desenvolvimento da equipe de enfermagem de dois hospitais do município de São Paulo. São Paulo: Universidade de São Paulo. Escola de Enfermagem; 2010.         [ Links ]

19. Brasil. Ministério da Saúde. Política Nacional de Educação Permanente em Saúde como estratégia Pacto de gestão: garantindo saúde para todos. Portaria n. 198/GM/MS de 13 de fevereiro de 2004. Brasília: Editora do Ministério da Saúde; 2004.         [ Links ]

20. Otrenti E. Avaliação de processos educativos formais para profissionais da área da saúde: revisão integrativa de literatura [Dissertation]. São Paulo: Universidade de São Paulo. Escola de Enfermagem; 2011.         [ Links ]