Exploring the Challenges in Designing and Implementing an Assessment Plan for a Virtual Engineering Materials Lab

This article examines the design, implementation, and assessment of MATERIALS-ISLE, a game-based virtual materials laboratory developed for engineering students at a large public university in the southwestern United States. The simulation is intended to replicate the learning outcomes of a traditional hands-on Mechanical Engineering materials lab, both for students who cannot access the physical course and as a supplementary practice tool for those who can. The authors' central aim is to describe the process of building a quasi-experimental assessment plan for the simulation and to document the practical challenges encountered, offering recommendations for others pursuing similar game-based educational projects.

The assessment framework follows a five-step sequence: writing learning outcomes drawn directly from the existing lab syllabus, measuring prior knowledge through pre-tests and attitude surveys, directly assessing outcome achievement through in-game transaction logs, quizzes delivered by a virtual lab technician, and student-created poster presentations, and comparing pre-, mid-, and post-assessment data to gauge learning gains. Students were randomly assigned to a "game-based simulation" group and a "no game-based simulation" group, with access to the simulation withheld from the latter until post-testing was complete. The authors report that defining learning outcomes posed little difficulty, whereas coordinating assessment timing across instructors, managing individualized access credentials for more than 100 students, and ensuring the accuracy of Blackboard-administered instruments proved considerably harder.

Among the lessons the team highlights are the value of weekly cross-disciplinary meetings, the recommendation that non-research instructors attend several briefings to understand the logistical stakes, and the switch to SurveyMonkey and SPSS to reduce manual data entry errors and free class time for instruction. The authors also acknowledge that revising survey items mid-study complicated longitudinal comparisons, and they recommend supplementing their quantitative tests and surveys with student focus groups and faculty interviews in future work. The article is primarily a methodological account rather than a report of outcome results, and the authors state their expectation that learning goals were met without presenting definitive comparative data to support that claim.