Conceptualization of Language Assessment Literacy

The term LAL has been conceptualized by many scholars in the past two decades (e.g., ^{Fulcher, 2012}; ^{Inbar-Lourie, 2008}; ^{Pill & Harding, 2013}; ^{Taylor, 2013}). LAL was defined by ^{Inbar-Lourie (2008)} as “having the capacity to ask and answer critical questions about the purpose for assessment, about the fitness of the tool being used, about testing conditions, and about what is going to happen on the basis of the test results” (p. 389). ^{Fulcher (2012)} defined LAL as “the knowledge, skills and abilities required to design, develop, maintain or evaluate, large-scale standardized and/or classroom-based tests, familiarity with test processes, and awareness of principles and concepts that guide and underpin practice, including ethics and codes of practice” (p. 125).

^{Taylor (2013)} put forward a model of assessment competency and expertise for different stakeholder groups. Placing language teachers at an intermediary position between the measurement specialist and the general public, Taylor argued that LAL is best defined in terms of the particular needs of each stakeholder group. Likewise, ^{Pill and Harding (2013)} rejected a dichotomous view of “literacy” or “illiteracy,” arguing for viewing LAL in terms of a continuum from “illiteracy” to “multidimensional literacy.” They contended that non-practitioners do not require assessment literacy at the “multidimensional level” or the “procedural level;” rather, it would be desirable for policy makers and other non-practitioners to gain “functional level” of assessment literacy in order to deal with language tests.

Research on Language Assessment Literacy

The last two decades have witnessed an increasing number of studies investigating teachers’ self‐described levels of assessment literacy (e.g., ^{Lan & Fan, 2019}; ^{Vogt & Tsagari, 2014}; ^{Xu & Brown, 2017}), approaches to assessment (e.g., ^{DeLuca et al., 2018}; ^{DeLuca et al., 2019}; ^{Tajeddin et al., 2018}), perceptions about language assessment (e.g., ^{Tsagari & Vogt, 2017}), and assessment use confidence (e.g., ^{Berry et al., 2019}). For instance, Vogt and ^{Tsagari (2014)} explored the current level of language testing and assessment (LTA) literacy of foreign language teachers from seven European countries through questionnaires and teacher interviews. They found that the LTA literacy of teachers was not very well-developed. ^{Xu and Brown (2017)}, utilizing an adapted version of the Teacher Assessment Literacy Questionnaire developed by ^{Plake et al. (1993)}, investigated 891 Chinese university English teachers’ assessment literacy levels and the effects of their demographic characteristics on assessment literacy performance. The findings of the study revealed that the vast majority of the teachers had very basic to minimally acceptable competencies in certain dimensions of assessment literacy. ^{Tajeddin et al. (2018)} aimed at exploring 26 novice and experienced language teachers’ knowledge and practices with regard to speaking assessment purposes, criteria, and methods. The researchers concluded that, although divergence between novice and experienced teachers’ knowledge and practice of assessment purpose was moderate, the data revealed more consistency in the experienced teachers’ assessment literacy for speaking.

^{DeLuca et al. (2018)} sought to explore 404 Canadian and American teachers’ perceived skills in classroom assessment across their career stage (i.e., teaching experience). The researchers observed that more experienced teachers, as opposed to less experienced teachers, reported greater skill in monitoring, analyzing, and communicating assessment results as well as assessment design, implementation, and feedback.

More recently, ^{DeLuca et al. (2019)} looked into 453 novice teachers’ classroom assessment approaches using five assessment scenarios. The researchers observed that teachers were quite consistent regarding their learning principles. However, they showed some difference in their actual classroom practice, indicating the situated nature of classroom assessment practice. In another study, ^{Lan and Fan (2019)} explored 344 in-service Chinese EFL teachers’ status of LAL. They observed that teachers’ classroom-based LAL was at the functional level, namely “sound understanding of basic terms and concepts” (^{Pill & Harding, 2013}, p. 383). The researchers concluded that teacher education courses should acquaint teachers with the necessary knowledge and skills for conducting classroom-based assessment literacy.

Considering the variety of conceptualizations of the term LAL and its intricacies, more studies must be carried out in local contexts (^{Inbar-Lourie, 2017}) with a focus on language teachers’ perspectives (^{Lee & Butler, 2020}) in order to help the field come to grips with the dynamics of the issue. Also, as the above literature review shows, we still have a limited understanding of language teachers’ classroom-based assessment literacy. Against this backdrop, the main purpose of this study is twofold: developing and validating a new CBLAL scale to assess language teachers’ perceived classroom-based assessment knowledge and practice, and looking into Iranian EFL teachers’ perceived classroom-based language assessment knowledge and practice. The following are the questions that guided this study:

Participants

The participants of the study for the initial piloting of the scale were 54 Iranian EFL teachers, including 23 male (42.6%) and 31 female (57.4%) teachers. It should be noted that the pilot study participants, albeit identical to those in the main study, did not take part in the later study. A total of 346 EFL teachers, including 143 men (41.3%) and 203 women (58.7%), participated in the development and validation of the scale as well as the investigation of the classroom-based assessment literacy of EFL teachers. The teachers were all teaching general English (i.e., integrated four language skills) to various levels and age groups in private language schools in the Iranian context. In these language schools, teachers are required to follow a fixed syllabus using well-known communication-oriented international textbook series (^{Sadeghi & Richards, 2015}). The private school supervisors commonly use written examinations and interviews to recruit qualified teachers and regularly observe their performance for promotional and career growth purposes (^{Sadeghi & Richards, 2015}).

The participants’ ages ranged between 18 and 67, with the average age of 32. All teachers, based on a convenience sampling procedure, voluntarily took part in the study. More than half of the teachers had majored in teaching EFL. Moreover, almost half of the teachers had taken a language testing/assessment course at university (see Table 1).

Table 1 Demographic Information of the Participants 

Scale Development

The process of developing the CBLAL scale began with a review of previously validated assessment literacy scales in the literature (e.g., ^{Fulcher, 2012}; ^{Mertler & Campbell, 2005}; ^{Plake et al., 1993}; ^{Zhang & Burry-Stock, 1994}). It was observed that most previous studies exploring teachers’ assessment literacy used the Teacher Assessment Literacy Questionnaire (^{Plake et al., 1993}), or its adapted version-the Assessment Literacy Inventory (^{Mertler & Campbell, 2005}). Both scales included 35 items assessing teachers’ understanding of general concepts about testing and assessment, which were tightly aligned to the seven Standards for Teacher Competence in Educational Assessment for Students (^{AFT et al., 1990}).

To address the recent conceptualizations of classroom-based assessment needs of language teachers (^{Brookhart, 2011}), we set out to develop a CBLAL scale based on ^{Xu and Brown’s (2016)} six-component interrelated framework of teacher assessment literacy in practice (TALiP). Assessment knowledge base, constituting the basis of the framework, was used for developing the CBLAL scale in this study. According to Xu and Brown, teacher assessment knowledge base refers to “a core body of formal, systematic, and codified principles concerning good assessment practice” (p. 155). The key domains of teacher assessment knowledge base are briefly defined next:

It should be noted that as the present study sought to develop a language assessment-specific scale to be used for pinpointing language teachers’ classroom-based assessment knowledge and practice, the first domain of the assessment knowledge base (i.e., disciplinary knowledge and pedagogical content knowledge) was not taken into account in the scale development process. It was reasoned that an adequate understanding of the disciplinary content and the general principles regarding how it is taught or learned is a pre-requisite for all teachers regardless of the content area they are teaching (^{Brookhart, 2011}; ^{Firoozi et al., 2019}).

After existing assessment literacy scales were reviewed by the researchers, 25 assessment knowledge items (i.e., 62.5%) which corresponded to the subcomponents of the Xu and Brown’s framework (i.e., assessment literacy knowledge base) were identified. Next, they were borrowed and reworded in order to measure teachers’ (a) perceived classroom-based assessment knowledge, and (b) perceived classroom-based assessment practice. According to ^{Dörnyei (2003}, p. 52), “borrowing questions from established questionnaires” is one of the sources that successful item designers mostly rely on. Then, 15 assessment knowledge items (i.e., 37.5%) were originally developed by the authors to ensure an acceptable number of items for each subcomponent. Later, 40 assessment practice items were developed corresponding to the 40 assessment knowledge items. The final draft of the CBLAL scale comprised a demographics part and two other sections. The demographics part consisted of both open-ended and close-ended items inquiring into the participants’ demographic information. The first of the other two sections aimed at exploring language teachers’ knowledge of classroom-based assessment. A pool of 40 items was generated in line with the six components of Xu and Brown’s framework. The items asked teachers to evaluate their own knowledge of assessment on a Likert scale ranging from 1 (strongly disagree), 2 (slightly disagree), 3 (moderately agree), to 4 (strongly agree). The items in the second part of the last two sections of the scale corresponded to the preceding one in terms of the targeted construct (i.e., the six components of Xu and Brown’s framework). However, they were modified to probe into the teachers’ perceptions of their classroom-based assessment practices. There was a total of 40 items on a Likert scale ranging from 1 (never), 2 (rarely), 3 (sometimes), 4 (very often), to 5 (always) in the last section of the scale.

Prior to subjecting the scale to psychometric analysis, it was filled out by six teachers to check the intelligibility of the items. Having resolved the ambiguities and unintelligible items based on the teachers’ feedback, the researchers pilot-tested the scale on 54 teachers to assure its reliability using Cronbach’s alpha. As for the validity of the scale, a panel of four instructors doing their PhD in applied linguistics was consulted to review the content validity of the scale. Also, exploratory factor analysis (EFA) was performed in the main phase of the study to extract major factors and item loadings of the scale. To do so, the scale was distributed among a large pool of language teachers through both online media (e.g., email) and personal contacts. Overall, a total of 346 teachers filled out the scale and their results were subjected to EFA. Regarding the status of language teachers’ classroom-based assessment knowledge and practice, descriptive statistics were run based on the teachers’ responses to the CBLAL scale.

Exploratory Factor Analysis

The initial 80 items of teachers’ CBLAL, including 40 classroom-based assessment knowledge items on a 4-point Likert scale and 40 classroom-based assessment practice items on a 5-point Likert scale, were subjected to EFA, namely principal axis factoring (PAF) with direct Oblimin rotation. The suitability of data for factor analysis was investigated prior to performing PAF. First, the normality of the distribution of the data was checked by considering the skewness and kurtosis measures of the items. It was found that all items’ statistics ranged between -2 and +2, satisfying the assumption of normality (^{Tabachnick & Fidell, 2013}). Second, the Kaiser-Meyer-Olkin (KMO) measure was used to estimate the sampling adequacy for the analysis. As can be seen in Table 2, the KMO value was .91 for assessment knowledge items and .92 for assessment practice items, exceeding the recommended minimum value of .6 (^{Pallant, 2016}; ^{Tabachnick & Fidell, 2013}). Further, as shown in Table 2, Bartlett’s test of sphericity reached statistical significance for both measures, which indicated that correlations between items were sufficiently large for PAF.

After running PAF on the assessment knowledge items, an initial 7-factor solution emerged with eigenvalues exceeding 1, explaining 40.4%, 5.8%, 4.7%, 4.2%, 3.1%, 2.8%, and 2.6% of the variance, respectively. However, an inspection of the scree plot and parallel analysis showed only four factors with eigenvalues exceeding the corresponding criterion values for a randomly generated data matrix of the same size (40 variables × 346 respondents; ^{Pallant, 2016}). The final 4-factor solution of assessment knowledge measure explained a total of 55.3% of the variance. The internal consistency of the assessment knowledge scale as a whole was estimated and its Cronbach’s Alpha was found to be .95.

Regarding assessment practice items, an initial 7-factor solution emerged, with eigenvalues exceeding 1, explaining 40.6%, 7.2%, 4.5%, 4.0%, 3.3%, 2.9%, and 2.5% of the variance, respectively. The inspection of the scree plot and parallel analysis, however, yielded a 4-factor solution for the assessment practice scale, which explained a total of 56.5% of the variance. The internal consistency of the assessment practice scale as a whole was estimated and its Cronbach’s Alpha was found to be .94.

To aid in the interpretation of the extracted factors, Oblimin rotation was performed. Also, only variables with loadings of .4 and above were interpreted, as suggested by ^{Field (2013)}. It should be noted that Items 25 and 26 were omitted from the assessment knowledge scale due to their low coefficients (see Table 3). Moreover, Item 21 was omitted from the assessment knowledge scale due to cross-loadings. Regarding the assessment practice scale pattern matrix (Table 4), Items 14, 15, 16, and 40 were suppressed by SPSS from the factor solution because of their low coefficients.

The assessment knowledge items that clustered around the same factors (bolded items) in the pattern matrix presented in Table 3 suggested that factor one, containing 15 items, represented “Knowledge of Assessment Use and Grading.” The items elicit teachers’ familiarity with the purpose of classroom tests and how to choose appropriate classroom tests to fit intended course objectives. Factor one also probes into teachers’ knowledge of grading students’ classroom test performance and how to get assistance from experienced colleagues in this regard. Factor two comprised 8 items which represented “Knowledge of Assessment Ethics.” It elicits teachers’ knowledge of how to observe assessment fairness and to avoid assessment bias in classroom tests. Factor three comprised 6 items which tapped on “Knowledge of Student Involvement in Assessment.” The items examine teachers’ knowledge of strategies to encourage students to assess their own and their peers’ classroom test performance. Finally, factor four consisted of 8 items which represented “Knowledge of Feedback and Assessment Interpretation.” The items inquire into teachers’ knowledge of providing students with regular feedback (i.e., practical suggestions) to help them improve their test performance. The items also elicit teachers’ familiarity with reporting and communicating students’ classroom test performance to both parents and/or school managers.

As for assessment practice items, those that clustered around the same factors (bolded items) in the pattern matrix presented in Table 4 suggested that factor one, comprising 13 items, represented “Assessment Purpose and Grading.” It elicits the frequency of the use of classroom tests to track students’ progress during the course, to plan future instruction, and to assign grades to students. Factor two, consisting of 9 items, represented “Assessment Ethics.” Factor two items inquire into observing assessment fairness and avoiding assessment bias in classroom tests. Factor three, comprising 6 items, implies “Student Involvement in Assessment.” It explores the frequency of assisting students to learn how to grade their own and their peers’ classroom test performance. Finally, factor four, with 8 items, refers to “Feedback and Assessment Interpretation and Communication.” The items elicit the frequency of reminding students of their strengths and weaknesses in their classroom test performance and of using classroom test results for decision-making purposes.

Overall, the classroom-based assessment knowledge and practice items loaded on four thematic areas with 37 items falling under classroom-based assessment knowledge and 36 items relating to classroom practice.

Teachers’ Classroom-Based Assessment Knowledge and Practice

The newly-developed CBLAL scale was used to probe into Iranian EFL teachers’ knowledge and practice of four factors of these teachers’ classroom-based assessment knowledge base (^{Xu & Brown, 2016}). Table 5 presents the percentages of the teachers’ responses on classroom-based assessment knowledge factors.

As can be seen in Table 5, the teachers in the present study reported to be knowledgeable about classroom assessment by moderately or strongly agreeing with the items. More specifically, around 76% of the teachers self-reported to be knowledgeable about the uses of assessment and grading procedures in language classrooms. Regarding knowledge of assessment ethics, around 82% of the teachers believed that they were knowledgeable about ethical considerations in the classroom. As for knowledge of student involvement in assessment, 71% of the teachers believed they were knowledgeable about how to encourage students to assess their own and their peers’ classroom performance. Finally, regarding knowledge of feedback and assessment interpretation, 88% of the teachers were of the belief that they knew how to provide accurate feedback as well as report on students’ class performances.

As for the teachers’ classroom-based assessment practice, Table 6 presents the percentages of their responses on a 5-point Likert scale.

Table 6 shows that around 61% of the teachers stated that they use classroom assessment for the purpose of grading students’ performance as well as informing future instruction. Also, around 71% of the teachers were of the belief that they very often consider ethical issues in their everyday classroom assessment tasks. As for student involvement in assessment, more than half (51%) of the teachers reported that they help students learn how to grade their own classroom test performance. Finally, as for feedback and assessment interpretation and communication, around 72% of the teachers were of the belief that they practice constructive feedback in their classes to help students set goals for their future success.