English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
Introduction
The cold chain of immunobiological conservation is a strategic activity of the National Immunization Program (NIP) of the Brazilian Health System to maintain the quality of these inputs. Despite its undeniable relevance, research studies carried out in several countries detected shortcomings such as: storage temperatures outside the recommended, lack of devices for monitoring this temperature, absence of electrical generators, inadequate reception of immunobiologicals at various levels of the cold chain, lack of conformity in the ambiance of the coils of reusable ice, putting immunobiologicals at risk of exposure to freezing temperatures, among others, which endanger the effectiveness of immunobiologicals and burden the NIP 1-8.
This strategic activity of the nip needs to be monitored because its process directly affects the quality of immunization since immunobiologicals are organic products, sensitive to heat and cold, that require to be kept within a narrow range of temperature 9. In this sense, the improvement of the maintenance of the cold chain runs through the implementation of the management process, in vaccination room, which includes planning, organization, supervision, and monitoring/evaluation to keep the cold chain for the conservation of immunobiologicals 10.
To ensure the execution of this process of management of the maintenance of the cold chain, it needs to be orchestrated by a guide aiming to coordinate the team in the vaccination room in order to preserve the quality of immunobiologicals provided to the population. According to international literature, there is no current measuring tool available to evaluate the compliance of technical standards for maintenance of the cold chain for the conservation of immunobiologicals.
The development of this study is justified by the need for a valid measuring instrument to assess the cold chain for the conservation of immunobiologicals to be used by the nurse line manager of the immunization room, and to subsidize supervision and excellence by improving the conservation activities of immunobiologicals. Besides, this instrument may also be applied to scientific investigations. The purpose of this evaluation instrument is to examine whether the technical standards recommended by immunization programs are in accordance with the decision-making process. Furthermore, the population will also be benefited, as the use of this instrument in the supervision of activities of conservation of immunobiologicals may ensure the maintenance of the immunogenic quality of the offered products. In this sense, the objective of this study was to develop and to test the content and layout validity of a multidimensional tool to evaluate the maintenance of the cold chain of immunobiological conservation.
Method
This is a methodological study that led to the elaboration and validation of the content of a measurement instrument to evaluate the maintenance of the cold chain for the conservation of immunobiologicals in the structure and process dimensions 11. This study was carried out in three stages: integrative review; elaboration of the logical model of cold chain of conservation of immunobiologicals; elaboration and validation of content and appearance from the Delphi technique. The integrative review 3 was performed based on Brazilian and international literature on the study issue and had the purpose to identify the critical events of the maintenance of the cold chain for the conservation of immunobiologicals as well as the existence of measuring instruments to evaluate the maintenance of this chain.
The results of the integrative review, along with the analysis of NIP documents (ordinance, handbooks, technical standards, reports), subsidized the creation of the logical model of the cold chain of immunobiological conservation 12, thus constituting the second phase of this research. The logical models are tools that outline the basic aspects of an intervention, from available resources, and the activities carried out until possible outcomes, in addition to clarifying the relations of assumptions linking these elements and subsidizing the identification of evaluative questions that compose a measurement instrument 12. Modeling is a necessary phase to plan an assessment, which includes, among other aspects, the creation and validation of instruments such as scales and matrixes of analysis and judgment 12.
The logical model of a cold chain of immunobiological conservation shown in Figure 1 was structured contemplating three components: transportation and reception, storage and handling, and supervision and permanent education. For each component described, available resources (structure), desired activities (process), expected impacts (outcomes), and causal connections presumed are presented 11.
Source: Developed by the authors based on the integrative review 3.
Figure 1 Logical model of the cold chain for immunobiological conservation, Brazil, 2018
The structure presents the physical, human and organizational conditions in which care occurs, and the process refers to the care dynamic to achieve the expected results 11. The questions related to the structure dimension include facilities, equipment and inputs, human resources, and normative (handbook of norms and procedures in the vaccination room, cold chain handbook) 1,2,6,13,14. Since it is a modeling at local level (vaccination rooms), the structure dimension did not include the financial resources used for the maintenance of the cold chain.
The questions related to the process dimension of the components of transportation and reception, storage and handling, and supervision and permanent education are presented in a logical model. When executed with the available resources, they favor short-, mid- and long-term effects, keeping the appropriate conservation of immunobiologicals; ensure that all immunobiologicals keep their initial characteristics to confer immunity; reduce unnecessary loss of immunobiologicals due to change in conservation; minimize risks of adverse events following immunization (AEFI) resulting from failures in conservation; implement permanent education in the routine of vaccination rooms; keep updated knowledge about immunization; and ensure the provision of safe and quality service to the population. In the long term, it is expected to keep the cold chain of immunobiological conservation 7,15,16.
This phase was essential to identify the evaluative questions that composed the measurement instrument validated in this study. The measurement instrument followed the scheme proposed for modeling, thus consisting of two dimensions (structure and process) and three components (transportation and reception, storage and handling, and supervision and permanent education). In this way, the logical model initially revealed 34 evaluative questions (10 of structure dimension and 24 of process dimension), with 3 questions about transportation/reception, 17 about handling/ storage, and 4 relating to supervision/permanent education, arranged in a structured questionnaire whose content and layout would be tested.
For the elaboration of the questions, the authors considered the criteria proposed by other studies 17, as well as the relevance, objectivity, and clarity of content. In the construction of the answers to the questions, the options were presented as in a Likert scale (never, rarely, often, always). The Likert scale is a nominal instruments that proposes a series of statements, with each clearly positive or negative in relation to the topic under study 18. For this purpose, scores were assigned to the items, which, when totaled, constituted the scores of the dimensions (structure and process) of the instrument. The score met the following model: never: 0; rarely: 1; often: 2; always: 3. For items with inverted answers, scores were also inverted: never: 3; rarely: 2; often: 1; always: 0.
To perform the test of the validity of content and layout, the Delphi technique was used in the third phase of this study, as described lines below. The advantages of this technique are the flexibility in the number of steps to reach the desired level of agreement about the questions and the diversity of the respondents 19,20.
Selection of judges
The corps of judges was made up of 22 Ph.D., 8 M.Sc., and 17 specialists, for a total of 47 participants. The criteria for the selection of the judges to participate in the test of validity of content and layout of the evaluative questions followed recommendations found in the literature 18: master's, doctor's degree or professor in the vast area of concentration of health sciences, acting in the graduate education sector in Brazil, regardless of the state and municipality, present activities and/or scientific production in the area of immunization during the last 5 years; being a specialist active in the area.
For the selection of judges, an advanced search by subject was performed at the Lattes Platform databases, which gathers the resumes of researchers, using the following descriptors: vaccines, conservation of vaccine, cold chain. After using the descriptors, resumes were selected by means of criteria such as professional performance, publication in scientific events in the area, and the subject-related M.Sc. or Ph.D. held by incumbents. The resumes revealed the vast experience of the selected judges, confirmed by their technical-scientific productions and professional experience at renowned Brazilian institutions.
Data collection
A prior contact by e-mail was established with the 47 researchers selected in order to show them the study goals and methodology, and to request their participation in the research by means of a link to access a form drawn on eSurv software. The collection of data from the committee of judges by e-mail or internet is a recent procedure that, however, has some advantages, such as a rapid collection and processing of data, considering that questionnaires are easily distributed. Besides, for the respondents is easy to answer the questions when appropriate and without the need for an in-person meeting 21.
In the sequence, an e-mail with the link to access the online questionnaire was sent to participants. The access to the questionnaire required reading the informed consent form and accepting to participate in the study, indicating the specific field authorizing the research. Hereinafter, the judges had access to questions relating to the first cycle of Delphi with a view to obtaining consensus through the evaluated aspects. This process occurred from March to May 2017. The judges who did not answer the first cycle were excluded from the sample.
For content validity, each judge was requested to assess each question according to the relevance, objectivity and clarity criteria, classifying them on an opinion scale, as follows: "1" Not Representative; "2" Representative, but in need of review; "3" Representative item 22.
The test of the layout validity has fundamental importance, since it aims to verify that all items are comprehensible to the target-population of the instrument, in addition to checking whether the items are clear to respondents, thus ensuring the layout validity of item 23. The analysis of the test of the layout validity observed the content presentation, clarity and ease in reading and the adequacy of the item to dimension it appropriately represents 23. Depending on the results of the degree of agreement of judges and to keep the methodological rigor, there could be other cycles to discuss the items assessed between the judges. Furthermore, the recommendations by judges were accepted, which included minimal information about the semantics of two questions. Finally, an email to thank the collaboration in the test of content and layout validity of the questions was sent to the judges.
Data analysis
Data analysis used the CVI to identify the judges' degree of agreement. This index measures the percentage of judges who are in agreement about certain aspects of the tool and its items and allows for analyzing each item individually and the instrument as a whole. The convergence of the answers or the level of consensus expected for this study was 75.0% 16.
The CVI was calculated from the sum of responses "3" (representative) of each judge in each question of the questionnaire, divided by the total number of responses: CVI = 100 x (number of responses "3"/total number of responses). The mean CVI of criteria for each item was also calculated, adding each percentage obtained in each criterion and dividing by 3, i.e., by the three criteria used.
The CVR was also used 22. While CVI measures the proportion of judges with answers "3" (representative), the CVR compares this proportion to the expected number if the judges were responding randomly [CVR = ne - (N/2) / (N/2)], where "ne" is the number of judges who rated each item as "3" and "N" is the total number of respondent judges.
The CVI varies between 0 and 1 and, the closer to 1, the better the performance of the item according to the judges. The CVR varies between -1 and 1, and a good item is expected to have a CVR value, at least, positive. According to the literature 22, the minimum value of the CVR to discard the hypothesis that judges are responding randomly depends on the number of judges. Considering 20 judges who participated in the second cycle, the minimum value of the CVR should be equal to 0.4 22.
This study was approved by the Human Research Ethics Committee of the Universidade Federal de São João del-Rei, under Opinion n. 1.231.140 and CAAE 47997115.2.0000.5545.
Results
There were two cycles in the Delphi technique, which constituted an important technique of consensus to guide relevant adjustments regarding the content and the layout of the questions.
Of the total number of judges identified (N=47), 27 responded to the questionnaires in the first cycle. Of the total number of respondents (n=27), more than half hold a Ph.D. degree (n=16; 59.3%), 3 a M.Sc. (11.1%), 6 are specialist (22.2%), and 2 are city line managers (7.4%). In the second cycle, 20 judges responded: 10 Ph.D. (50%), 2 M.Sc. (10%), 6 specialists (30%) and 2 nursing professionals (10%).
In the first cycle, the cvi was 79.1% in the set of assessed questions. In the structure dimension, the CVI was 76% and, in the process dimension, this index reached 85.5% in the transportation and reception component, 81.2% in the storage and handling component and 75.5% in the supervision and permanent education component. The latter presented a greater number of questions with a cvi lower than 75%.
The questions that received a score lower than 75% in any of the evaluation criteria were re-evaluated according to the suggestions by judges, being some of them rewritten or deleted.
In relation to the structure dimension, three questions were excluded and only one question from the process dimension due to mean cvi smaller than 75%. Twelve questions of the process dimension were rewritten. The suggestions made by judges referred mainly to the semantics, such as appropriate direct phrases, and questions not leading to answers or affirmative or negative questions. The restructured tool with 30 questions (7 structure questions and 23 process questions) was re-sent to the judges for a new assessment.
The total CVI of the tool in the second cycle was 87.4%. The mean cvi in relation to the dimensions was 85.7% in structure and 89% in process, in which the CVI of components transportation and reception, storage and handling, and supervision and permanent education were 88.9, 88.9 and 89.2 %, respectively. The total CVR was 0.8, with values of 0.7 and 0.8 in the structure and process dimensions, respectively.
In this cycle, two questions in the storage and handling component were excluded from the process dimension due to redundancy. The judges recommended the junction of two equivalent questions, changing to "Is the refrigeration equipment cleaned in accordance with the regulations of the National Immunization Program?".
Tables 1 and 2 present the results of the cvi and the CVR of the questions according to the structure and process dimensions of the instrument.
Table 1 Judges' agreement according to structure dimension, Brazil, 2018
Source: Developed by the authors.
Table 2 Judges' agreement according to components of the process dimension, Brazil, 2018
Source: Developed by the authors.
To proceed with the analysis of the test of layout validity, two questions with a CVI of 70% in the clarity criterion remained, since they are fundamental requirements to ensure the quality of the immunobiological conservation (relevance of 90 and 95%, respectively) and record a CVI of 0.4. The first question refers to the structure dimension (is there an electric generator or no-break or battery available and in conditions of use in the event of a power failure?) and the second, to the storage and handling component of the process dimension (in this unit, is the opening/dilution of the multidose vaccine bottles identified?)
Chart 1 presents the final instrument called "Scale of Assessment of Immunobiological Conservation" (SAIC), with 27 questions validated. The instrument was divided into two sections. The first refers to questions about structure (N=7) and the second to questions about process, subdivided into three distinct components: transportation/ reception (N=3); storage/handling (N=13); supervision/permanent education (N=4).
Discussion
The monitoring of immunobiological conservation is part of ensuring the maintenance of the immunogenic power of these inputs, conferred by the producing laboratory, and must be ensured by means of appropriate structure and procedures, in order to ensure effective results in the practice of immunization 13,14,24.
Studies indicate that supervision in the vaccination room is an important step to guarantee quality services in immunization 3,10,25,26. A case-control study conducted in Odisha, India, identified improvements in management practices in the cold chain in the districts that received the intervention of support supervision 27. Notably, an effective supervision requires planning this activity, which can be organized and structured from the adoption of an instrument to coordinate activities 10,28.
The multidimensional SAIC instrument, developed and tested regarding content and layout validity to evaluate the immunobiological conservation, had a positive evaluation in relation to the items of relevance, objectivity and clarity of the analyzed questions, with CVI and CVR exceeding the established cuts. The cut-off point for obtaining consensus varies in the literature from 50 to 80%, being recommended, in nursing researches, a percentage not smaller than 75% 16.
This study also included the analysis of documents and scientific articles that presented coherent and consistent information, which subsidized the modeling of maintenance of the cold chain for immunobiological conservation. The elaboration of the logical model, in the perspective of the literature 12, was essential to understand the theoretical premises on which the maintenance of the cold chain is based, to define exactly what should be measured (resources employed and activities) and their contribution to the observed outcomes. This strategy affects the internal validity of the instrument, which is a fundamental aspect for an assessment.
Two questions in the validated instrument with CVI of 70% in the clarity criterion remained. The first refers to the use of an electric generator or no-break in the structure dimension. Power failures are frequent in several vaccination rooms and can happen outside business hours. A study conducted in Zaragoza (Spain) identified that an electrical power system would prevent economic loss for the NIP resulting from the interruption in the cold chain by cutting off the electrical power 8. Besides, the absence of an alternative energy supply source has been associated with exposure of immunobiologicals to temperatures outside the limits 4,24. Therefore, a no-break or a generator is required to compensate the shortcomings of the electrical network 9.
The other questions refers to the identification of opening/dilution of multidose vaccine bottles and corresponds to one of the process components. Many immunobiological products are presented as multidose bottles with expiration time defined by the producing laboratory 13,14. Good practices of administration of immunobiological are important not to endanger the quality and safety of the immunization process.
Importantly, there is the Protocol to assess nursing safe care with vaccines in primary care29, whose goal is to assess the nursing safe care with immunobiologicals in primary care. Nevertheless, the existing instrument differs from the SAIC concerning its constructs, being the SAIC responsible for measuring the maintenance of the cold chain of immunobiological conservation in immunization rooms, which defines it as the first instrument with such goal.
Regarding the use of surveys, this technology is highly valued in conducting scientific researches. However, there were some problems regarding the equivalence of eSurv operating software during the cycles of the Delphi technique. The impossibility of opening the questionnaire more than once on the online system favored the loss of respondents. The delay of the judges' committee to accept and return the questionnaire completed was also considered an obstacle, demanding new calls to stimulate their participation in the research, and thus avoid losses. Although the technology has helped positively the completion of surveys, it can also be a source of problems that lead to increased rates of non-respondents or partial responses 30.
Another limitation observed refers to the unaware-ness about classifications of maintenance of cold chain of immunobiological conservation. This practice is not yet institutionalized a t health services, remaining only normative evaluations using standardized forms already outdated.
Methodological studies have been seem with great importance by the scientific community and, for this reason, the strong point of the present study is the addition of new theoretical and methodological frameworks in the development and test of validity of measurement instruments. For nursing, solely responsible for the activities carried out in vaccination rooms, using a validated instrument favors the supervision in the vaccination room, reducing unnecessary immunobiological losses, minimizing risks of AEFI, updating knowledge about immunization, thus providing safe and quality service to the population.
Conclusion
The study provides a scale that is considered as valid by the test of validity of content and layout to measure the maintenance of the cold chain for immunobiological conservation. This can contribute to the organization of actions for monitoring and evaluation in vaccination, as a supervision instrument, promoting the formation of the nursing team and, consequently, the maintenance of the immunobiological conservation.
The Scale of Assessment of Immunobiological Conservation (SAIC), Brazilian version, is expected to be disclosed amidst the scientific community to subsidize new researches in the area and among health services to support the supervision in the vaccination room.
Complementary investigations of validity are being developed to ratify a good adjustment of the scale. Moreover, its application in various contexts will allow for a generalization of results and will certify the viability of using the instrument for supervision in the immunization room.
