Evaluation of the CIPP Model in a Project-Based Basic Electronics Course: A Discriminant Validity and Common Method Bias Analysis

Le résumé fourni par la source

Project-based instruction has become increasingly common in introductory electronics courses within physics education. However, evaluations of these courses rarely assess the quality of their instruments with the same rigor applied to student learning outcomes. This study evaluated the Context, Input, Process, and Product components of a project-based Basic Electronics course in an Indonesian undergraduate Physics Education program. A concurrent mixed-methods design was used. Data were obtained from curriculum documents, questionnaires completed by all 59 enrolled students, and interviews with two course lecturers and a subset of students. Context and Input were rated very good, Process was rated sufficiently, and Product showed mixed results. Skills development was rated highly, whereas product quality and functionality received the lowest rating at 58.6%. Four complementary diagnostics were applied: sub-indicator-level reliability analysis, cross-instrument association testing, Harman’s single-factor test, and achievement-confound testing. Across 294 pairwise correlations with false discovery rate correction, no significant association remained between the CIPP Process and Product indicators and the broader course-experience scores. This result indicated limited detectable alignment between the observed scores but was not treated as complete evidence of discriminant validity. The first unrotated component of the supplementary battery explained 74.5% of the total variance, exceeding the conventional threshold for flagging possible common method bias. In addition, 49% of respondents provided identical ratings across all items in at least one instrument, indicating invariant response patterns that could affect score quality and inflate internal consistency estimates. No statistically significant association or group difference involving prior academic achievement was detected across the 34 indicators examined. The study provides a replicable diagnostic workflow for evaluating routinely collected CIPP data prior to programmatic decisions. The findings indicate that CIPP component ratings should be interpreted alongside reliability, response quality, cross-instrument, and potential confound diagnostics.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.