English (pdf)
Article in xml format
Article references
Automatic translation
Send this article by e-mail
Cited by SciELO 
Cited by Google 
Similars in
SciELO 
Similars in Google 
Permalink
Introduction
Information and communication technology (ICT) can expand and enrich educational opportunities for learners in diverse settings, according to UNESCO 1. In Colombia, the incorporation and use of ICT in classrooms has been promoted by the National Government 2,3 through different programs 4 that focus on the provision of devices to support teaching-learning processes. Consequently, the Ten-Year National Education Plan, 2016-2026 5 fosters the comprehensive development of citizens and highlights the potential that new technologies have in the teaching-learning, knowledge-building, research, and innovation processes. Mobile Learning Systems (MLS) thus emerged, focused on autonomous and constructivist learning, through an environment of participation and collaboration. Mobile learning, according to 6, is the ability of any person to use mobile network technology to access relevant information or store new information, regardless of their physical location, enabling students to take advantage of the benefits of mobile technology in teaching-learning processes. Context-sensitive systems 7 consider such aspects of the student's learning context as (i) profile, (ii) educational resources, (iii) learning objectives and (iv) pedagogical strategy, in addition to aspects of the technological context related to the mobile device as a means of information delivery 8. It is worth noting that in referring to teaching-learning processes, it is essential to address the assessment process as a mechanism that provides suitable feedback, ensuring achievement of the learning objectives. Mobile learning linked to context-based formative assessment processes enhances teacher-student communication and encourages synchronous and asynchronous communication, favoring dynamic interaction and facilitating feedback independent of location, device and physical environment 7. According to 9, the most important component of the assessment process is feedback, considering it as the only and powerful means to achieve and/or improve academic achievements. Suitable feedback must be continuous, constructive and meaningful 10, i.e. it must inform both student and teacher about progress and difficulties with the objective of making timely decisions about the instructional plan. Different studies focused on the field of assessment have suggested promoting greater efforts and strategies that demonstrate the effectiveness of learning in continuous assessment/formative assessment processes supported by technology 7-11. The literature contains proposals in specific domains 12-16. These do not consider contextual entities that allow capturing the characteristics or situations of each student, limiting possible adjustments or improvements to the assessment conditions and educational practices established by the teacher. Likewise, the effectiveness of existing feedback mechanisms is subject to the events that each student may trigger, whether due to their performance, level of understanding, motivation, skills, change of location or the characteristics of the device they are using 17,18. Considering the above and the importance of the integrity and dynamism of the assessment process, this research asks the research question of How to support formative assessment processes in mobile learning systems that allow providing effective feedback according to contextual information?
Methodology
The methodology for the creation of the Formative Assessment Model in Mobile Learning Systems (FAM-MLS) was based on the adaptation of the qualitative method for the creation of conceptual frameworks 19. FAM-MLS is a conceptual model that provides researchers and developers with an overview of the essential elements that must be considered in the development of formative assessment processes in mobile learning systems. To obtain the results, the steps described below were followed:
Stage 1: Literature review and analysis of data from various institutional access sources such as Scopus, Science Direct, IEEE Xplore and ACM Digital Library. The refined search string corresponded to the scope of the research and the most relevant terms close to the studies that were intended to be investigated: (Assessment OR Formative Assessment OR Feedback) AND (Model OR System) AND (Mobile Learning OR mlearning) AND (Context-Aware OR Context awareness). The study was guided by specific research questions such as: (i) What are the research works carried out on context-sensitive Mobile Learning Systems?, (ii) What are the context-sensitive Mobile Learning Systems that take account of assessment processes?, (iii) What context entities are used in the systems found that include assessment?, (iv) What are the reasoning techniques considered for feedback purposes in the systems that include assessment? And (v) What are the modeling components in systems that include assessment?
Stage 2: Definition of the FAM-MLS model is based on the qualitative method for the creation of conceptual frameworks 19, which includes (i) information source analysis; (ii) selection, grouping and integration of concepts (new and existing) and (iii) construction of the model and its formalization.
Stage 3: Adaptation and implementation of a software prototype based on the concepts defined in the FAM-MLS model, using the Google Classroom platform, an Android mobile application and a cloud database service. Development of the prototype was carried out using SCRUM 20 considering three events: (i) planning, (ii) sprint execution and (iii) review and retrospective.
Stage 4: Assessment of the proposed model in terms of the effectiveness of the feedback, through a case study following the methodological scheme of Runeson 21 and Yin 22. To respond to the questions posed in the study, the multimodal analysis supported by the ATLAS.ti tool 23 and the parametric tests statistical method 24 is presented.
Results
Characterization of modeling entities, techniques and components
At the characterization level, the selected primary studies revealed the existence of a significant set of context-sensitive mobile learning systems, within the framework of collaborative, meaningful and inverted learning. However, very few of them were focused on the formative assessment process to provide feedback and support to students and teachers, in order to improve their level of performance, achieve achievements and make decisions about learning paths and educational materials. Regarding context information, the entities considered by different researchers independently corresponded to: (i) time, (ii) location; (iii) device; (iv) physical characteristics and (v) student profile, aspects used to create and deliver learning materials. Regarding reasoning techniques and measurement objectives, the literature reports rule-based systems, item-response theoretical models, fuzzy logic, performance factor-based algorithms, computerized adaptive tests, prioritizing level of knowledge as the measurement objective in most studies. Independent modules considered in each of the systems were identified, such as: delivery and adaptation of content, creation and updating of the student profile, capture and analysis of context information, storage and adaptation of assessment questions and delivery of feedback in real time.
Definition of the FAM-MLS Assessment Model
The creation of the model was based on the characterization previously described. The model is based on the concept of offering feedback during the assessment process. In this sense, the works reviewed in the state of the art reflected little use of feedback when considering summative strategies 7, where feedback is only provided at the end of the educational process. The proposed assessment model is based on a training strategy 7, which encourages continuous feedback through the identification of sources, formats, types and moments of delivery of feedback. The phases addressed for the conception of the proposed model are described below.
Analysis of Information Sources: in this first phase, groupings were made of the elements and characteristics related to: sensitive context entities, reasoning techniques and MLS modeling components that incorporate assessment processes. The analysis made it possible to show that the elements “context entities” and “reasoning techniques” are components generally related to the information capture and analysis, and knowledge-performance modules, respectively. Likewise, the individual exercise of characterizing the “context entities” made it possible to identify the most common entities, such as Device, User, Physical, Time and Location; and the set of “reasoning techniques”, used mostly to measure the level of knowledge and level of performance of the students.
Selection and grouping of concepts
In this phase, a grouping of the components was carried out, synthesizing them into general concepts (See Figure 1). The entity component of the context was defined as “Sensitive Context” grouping the concepts used as: Device, Physical and Time and excluding the entity “User” because its representation, relevance and meaning was represented with the “Student” concept. The measurement of the level of knowledge and the level of performance is given by the use of Reasoning Techniques, this component being termed “Inference”. For the common modeling components, these were grouped according to their functionality in the concepts Student, Inference, Sensitive Context, Assessment and Knowledge Domain. Table 1 describes the grouping and synthesis of concepts.
Integration of new concepts
To generate new concepts, two strategies were carried out which corresponded to: (i) focus group and (ii) expansion of the feedback concept. The objective of the focus group was to identify the concepts related to the study topics, including Pedagogical Model, Learning Environments, ICT, Assessment, Formative Assessment and Feedback, as well as corroborate the similarity, discordance and/or absence with respect to to the concepts found in the initial characterization process. For the focus group, a guide protocol was defined based on 21,22 and five experts were selected, among them a Doctorate in Educational Sciences, a Master's degree in Pedagogy and Educational Technologies with a Bachelor's degree and/or basic training in the Engineering field. For the analysis of the collected data, the ATLAS.ti tool 23 was used. Figure 1 shows the network of concepts created around the ideas and arguments expressed by the interviewees. The process began with labeling the experts' responses with the objective of identifying concepts around the topic of study. Once the labeling process was completed, the concepts were processed to analyze the level of correspondence between them and the number of occurrences. According to the processed concepts, the respective associations were made taking into account the correspondence between concepts and the scope of the problem. The hierarchically highest concept corresponded to Learning Environment, from which the Learning System emerges, and this in turn is related to the concepts: Assessment, Teacher, Pedagogical Model, Sensitive Context, Student and Knowledge Domain, which coincide with those found in the characterization process in Table 1, with the exception of the Inference concept. Likewise, it was observed that feedback is a fundamental part associated with time of occurrence, type of feedback and source of supply. Additionally, a co-occurrence matrix was generated from a set of concepts with high correspondence with the problem. Two new concepts appear in this stage, which correspond to Teacher and Pedagogical Approach, where the number of occurrences compared to the concepts of Student and Assessment respectively, are significantly high, which is why they were considered in the creation of the model.
In relation to the expansion of the concept of feedback, the dimensions identified in 25 were taken into account, where three fundamental axes are considered for the delivery of feedback based on (i) occurrences, (ii) type of response and (iii) means of presentation. Additionally, the "Feedback Sources" feature was included, associated with: i) teacher (personalized delivery), ii) virtual assistant iii) classmate and (iv) own feedback.
Construction of the FAM-MLS Model
The FAM-MLS model was represented using an object-oriented approach 26, which is one of the most common, after ontologies, allowing the parts of the problem to be encapsulated in components that consider name, stereotype, attributes and associations among the concepts modeled (see Figure 2).
Formalizing the model
During the formalization phase, each of the concepts was described in detail:
Student: refers to the educational profile, which includes characteristics such as learning style, level of knowledge and level of performance.
Teacher: focuses on the tutor or advisor of an area of knowledge, whose function focuses on content preparation and generation of the question bank. He acts as a source of feedback and has responsibility in the decision-making process.
Inference: describes the mechanism used to analyze information during the assessment process and make a decision to provide feedback based on information from the learning context. In addition, this concept is linked to the use of reasoning techniques through advanced data mining algorithms and learning analytics, among others.
Pedagogical approach: allows the teacher to be guided on educational practices and describes systematic ways to apply ideas and activities. It particularly emphasizes the formative assessment process that ensures the understanding of the topics and the fulfillment of the learning objectives.
Knowledge domain: represents the areas of knowledge that are intended to be addressed in the teaching-learning process.
Sensitive context: describes the context entities that are considered around the evaluative process. The context information is represented by entities and their respective characteristics. Together with the concept of inference, they allow managing the most appropriate feedback deliveries according to the data captured by the sensors of the mobile device. The sensitive context entities considered correspond to Device (network connectivity, communication cost and broadband), Physical (noise level, lighting and temperature) and Time (date, duration of the activity).
Assessment: the concept of assessment represents the contents to be evaluated, the question bank and the mechanisms to provide feedback according to the student's learning process. It is managed by the teacher through the mobile learning system and is directly related to domain knowledge and feedback.
Feedback: this is a concept closely related to the rules of inference and in turn to the sensitive context, in order to support the assessment process by providing effective feedback to the student. Feedback is the most representative concept of the FAM-MLS model, through which it is intended to provide support to the student based on the dimensions proposed by 23, which correspond to: level of response (contingent response, contingent topic, of correct knowledge, until it is correct and of response knowledge), level of presentation (audio, text, image or video), level of occurrence (cumulative or immediate) and sources of feedback. This last dimension is a new dimension that considers elements such as tutor or teacher (personalized delivery), virtual assistant, classmate and own feedback.
Adaptation and implementation of the software prototype
The scientific literature revealed the lack of mobile tools to support feedback processes in real time incorporating context elements. Of the software tools identified, the majority were discarded by the present study, given their complexity in carrying out adaptation processes and the type of licensing. The Classroom learning platform was thus chosen, which has a web and mobile version to support teaching-learning assessment processes, facilitating the mapping process of the greatest number of concepts of the FAM-MLS model. Table 2 describes each of the concepts of the model and their respective component in Classroom.
Table 2 Concepts and components related to Google Classroom.
| FAM-MLS concept | Classroom component | Observation |
|---|---|---|
| Pedagogical approach | Blended learning, mobile learning, flipped learning, among others. | Allows deployment of different learning approaches, including Mobile Learning by having an application for mobile devices. |
| Knowledge domain | Varied or diverse. | Different learning materials or content can be configured independently of the knowledge domain. |
| Teacher | Creator and manager | Allows the teacher to have control of the educational space by means of the creation, editing and deletion of materials. |
| Student | Assistant and participant | Allows the student to interact with educational materials and put forward evidence. |
| Assessment | Questionnaires | Allows linking assessment questionnaires with different types of questions and feedback only at the end of the questionnaire. As such, it can be considered that the assessment is not formative. |
| Sensitive context | No correspondence | Does not allow reading data through sensors for assessment purposes. |
| Inference | No correspondence | Does not consider basic inference rules that allow taking advantage of context information |
| Feedback | QuizFeedback | Allows configuring feedback in terms of text and URL for each question. But it does not consider immediate feedback, only at the end of the questionnaire. |
Since no correspondence was found with the concepts of “Assessment, Sensitive Context, Inference and Feedback” (see Table 2), two macro tasks were established at the requirements level: i) development of an assessment form that considered the characteristics of feedback and inference rules according to the proposed model and ii) development of a mechanism for collecting contextual information in real time using light and environment sensors. The sensors were used in order to capture the light intensity and noise of the environment to determine what type of multimedia materials were delivered in the feedback. For multimedia delivery, conditional inference rules were used in accordance with the W3C specification 27 that allowed defining a set of ranges (20<=L<100: text; 100<=L<150: image; 150<=L<250: video; L>=250: audio), likewise the environment or noise sensor ranges (R<35: environment suitable for audio; R>=35: environment not suitable for audio). Due to the above, and using the SCRUM framework 18, software development tasks were carried out that made it possible to replace the concepts not present in the Classroom. There was thus an application for capturing and storing sensory information from mobile devices (EvaluaSense-own development) and a web form that considered real-time feedback aspects.
Case study to evaluate the FAM-MLS model
Case study design
The design of the case study corresponded to a single holistic type design 28, which related important aspects, among them study objective, research questions, units of analysis, constructs and operational definitions, and theoretical propositions. A methodological scheme is defined following the guidelines set forth by 21,22 to evaluate the effectiveness of the feedback in terms of the concepts defined in the FAM-MLS model. The software prototype developed was used, seeking to respond to the questions: R.Q.1. What was the type of feedback provided according to the students' experience?; R.Q.2. How was the delivery of feedback considered, immediate or delayed?; R.Q.3. What were the most common presentation formats during feedback? and R.Q.4. What was the level of learning achieved in terms of the knowledge obtained with the use of the training model? The constructors who guided the study mainly focused on observing the level of learning acquired through the formative assessment process and knowing the level of student satisfaction with the delivery of feedback received during the assessment process. The units of analysis corresponded to middle school students, ninth grade, in an urban area of the Municipality of Popayan. Table 3 presents a description of the population sample.
Table 3 Description of the population sample
| Item | Description |
|---|---|
| Name | Institución Educativa Antonio García Paredes |
| Sector and EE zone | Official and Urban |
| Timetable | Morning and Afternoon |
| Levels of study | Preschool, Transition, Primary, Secondary, Middle |
| Location | Calle 17 #12-40, La Ladera (Antigua Casona), Popayán, Cauca |
| Socio-economic stratum | 1 and 2 |
| Grade and number of students | Ninth grade (9th). 16 students in total |
| Number of boys and girls | 7 boys and 9 girls |
Running the case study
With the aim of responding to the questions posed, qualitative and quantitative information collection instruments were selected, such as online questionnaires integrated into the mobile platform to capture data provided by the population sample, in terms of correct and incorrect responses about the assessment process and structured satisfaction survey that allowed information to be collected on student perceptions, based on experience and recognized feedback elements. A virtual course was established using Classroom, for the provision of learning materials, assessment forms and satisfaction survey. A user manual was created to guide the implementation of the assessment process, taking into account situations of non-presence caused by COVID-19. For the information collection process, the APK was used to install the software on mobile devices so that the sensory information from cell phones and tablets could be accessed. Applying the proposed model, there was a repository of materials classified into text, image, audio and video to support the feedback process, according to the characteristics of the students. To verify the level of learning achieved with the proposed model, tests were carried out with and without feedback, analyzing the data set following the T-Student test method 24, performing statistical calculations of arithmetic mean, standard deviation (data dispersion) and coefficient of variation (relative degree of variability).
Analysis of results
Once the data collection and information processing had been carried out, the respective calculations (quantitative) and approximations (qualitative) were carried out to respond to the questions defined in the case study. Table 4 presents the set of concepts about the students' opinions, where the aspects associated with satisfaction in the assessment are distinguished by being the conceptual elements of the proposed model. The concept of "correction in learning" is associated with how assessment with feedback permitted the student to realize errors and try not to make them later. The aspects related to dissatisfaction in the assessment correspond to requirements such as: (i) presenting feedback for questions responded to correctly, ii) delay in the delivery of feedback as a result of unstable connections and iii) very little content delivered in multimedia format caused by incorrect performance of mobile devices, unsuitable learning environment and unstable internet connections.
Table 4 Concepts related to the students' opinions.
| Training feedback | |
|---|---|
| Satisfaction in the assessment | Detailed feedback |
| Correction in learning | |
| Immediate occurrence | |
| Presentation formats | |
| No satisfaction in the assessment | Feedback on correct responses |
| Delayed occurrence | |
| More presentation times | |
Regarding the response to research question R.Q.1., it can be seen that 69% of the students considered that the type of feedback received according to the model was “Topic-contingent feedback”, that is, the correct response and the solution to find the response were indicated. The remaining 31% found another type of feedback (correct response without additional information, progress was not allowed until the response was correct, etc.). For question R.Q.2., it was found that 88% of the students found an immediate response during the feedback process. The remaining percentage indicated that they had not received feedback or encountered delays in responding. Regarding the formats of presentation of the contents, response related to question R.Q.3., 59% of the students received content in plain text, 23% video format, 14% image and 4% audio. When investigating other factors that affected the delivery of content in different formats, aspects such as: (i) place of assessment, 7% of students had connectivity difficulties, (ii) time of the session, 11% expressed fatigue and high levels of noise in the place, (iii) performance, 5% of students had problems with their mobile devices, (iv) others, 5% expressed other conditions. In relation to question R.Q.4., the starting point is the conception that the formative feedback process positively influences the assessment and that students who are exposed to this type of process can improve and achieve learning objectives or goals. The tests showed that the average number of correct responses per student corresponded to 5/8 in the pre-test, while in the post-test it was 6/8. The Pearson Correlation Coefficient (PCC) percentage of the students with respect to the number of questions responded correctly shows that the arithmetic mean of the first and second tests are representative of the paired samples, with 35.88% (See Table 5). This means that the data is within the normal framework of homogeneity. The Alpha error rate for the analysis corresponds to 5%. According to the results presented, it can be observed that the statistical value of the t test (2.09) exceeds the critical value (1.75) and it could be stated that the feedback given to the students managed to improve the level of learning with a confidence factor of 95%.
Discussion
The research findings clearly show that no proposal in the literature related information about the student's context in formative assessment processes, specifically associated with feedback. Additionally, running the focus group allowed us to identify new elements relevant to the definition of the model. The consideration of different views for the proposed model enriched its definition and expanded the understanding of the concepts. The statistical analyzes applied made it possible to show that the mechanism based on formative feedback with the use of sensitive context of the FAM-MLS model significantly influenced the final results of the students in terms of the effectiveness of the feedback, fulfilling the objective of providing support in real time, making use of contextual information and significantly improving their learning. However, it is necessary to carry out more assessments with experts and expand the population sample to corroborate the findings and generalize the results.
Conclusions
Assessment is a very important aspect in the teaching-learning process. Offering students formative assessment processes, adapted to the needs of the context regardless of time and place, enhances teacher-student communication, facilitates feedback and increases student motivation and confidence. Therefore, the proposal of the FAM-MLS model as a response to the research question posed, the development of the prototype as an accessible web resource integrated into the GoogleClassroom LMS platform allowing the delivery of feedback in real time and the verification of the level of learning achieved with a significant increase, points to the relevance of research focused on supporting teaching-learning processes with the use of technology and based on rigorous conceptual guidelines. As future work, it is necessary to explore the relationship between contextual information, routes and learning materials. Likewise, the definition of reasoning and inference techniques typical of modeling is required in order to support adaptation and personalization in real time and measure the level of knowledge and performance of students.
