Co-Designing Visual Novels in Literary Education Classes: Instruments for Measuring Learner Engagement

This study develops and validates instruments for measuring secondary school students' engagement when co-designing educational Visual Novels, a story-driven genre of video game, within Literary Education. Motivated by a persistent gap in research on rigorously built, subject-specific tools for assessing the affective dimension of digital game-based learning in the Humanities, the authors created two questionnaires (pre- and post-intervention) and a class observation grid. The instruments draw on established scales, including the Four-Dimensional Scale of Student Engagement at School, the Flow State Scale, and the EGameFlow scale, and are framed around cognitive, affective, behavioural, and agentive dimensions of engagement. The work is positioned as preparatory to a doctoral case study with Portuguese students aged 15 to 17 who would create Visual Novels inspired by Eça de Queirós's "Os Maias."

Validation proceeded in stages. Four experts in Literary Education, Educational Psychology, and Educational Technology reviewed the instruments qualitatively, and their suggestions were incorporated. The first questionnaire was then pretested with 18 eleventh-grade students at a private school in Portugal. All respondents reported the items were easy to understand, and one suggestion, clarifying the term "narrative video game," was adopted. Cronbach's alpha was 0.876 for the engagement section and 0.925 for the section on perceptions of educational video games, both above the conventional 0.7 threshold. The post-intervention questionnaire was not pretested but was presented as structurally identical, so its validity was inferred from the first.

The authors acknowledge clear limitations: the pretest sample was small, and the context was a single private school in Portugal, so generalisability is limited. They suggest the instruments may be adapted to other Humanities subjects and narrative game genres, and that replication with larger and more varied samples is needed.