Abstract

HERNANDEZ-URENA, Olman  and  MONTERO-ROJAS, Eiliana. Validation of a standard¡zed performance test for selection of Architecture students with the Many-Facet Rasch Measurement Model. Rev. Arquit. (Bogotá) [online]. 2023, vol.25, n.1, pp.3-11.  Epub June 01, 2023. ISSN 1657-0308.  https://doi.org/10.14718/revarq.2023.25.4040.

A performance assessment involves examinees creating a product or developing a process, which is evaluated by several raters. The Multi-faceted Rasch Measurement Model (MFRM), an extension of the Rasch Model, allows quantifying diverse attributes associated with measurement quality in this type of assessments, including the degree of inter-rater agreement (inter-rater reliability), which is an essential requirement for validity. Data from a performance test, currently applied for selection purposes in the undergraduate program of the School of Architecture at the University of Costa Rica (UCR), were analyzed with MFRM. Four data sets were used, from 2015 to 2018 test administrations, each one having between 600 and 800 applicants. Each examinee's product was evaluated by three raters. The rater teams had between 12 and 15 members. The first three years showed a high degree of variability between raters' severities, extending over 2 logits on the Rasch Scale. Modifications were introduced in the 2018 application, aiming to improve inter-rater reliability. The corresponding analyses showed a relevant decrease in the dispersions of raters' severities, with a range of 1.09 logits. The study illustrates the benefits of the MFRM Model for analyzing rater data and improving the technical quality of a high-stakes performance assessment.

Keywords : architecture; aptitude tests; calibration; evaluation; measurement methods; Rasch measurement; performance appraisal.

        · abstract in Spanish     · text in English     · English ( pdf )