Evaluating Cybersecurity Awareness in Employees Using Gameplay: Data and Machine Learning Models

This study introduces CyberEmployee, a serious digital card game designed to assess and improve employee cybersecurity awareness through simulated attacker-defender scenarios, and it evaluates whether gameplay data can be used to classify employee skill levels with machine learning. Gameplay records from an IT organisation, collected through the game's scoreboard and exported as CSV, were encoded, split 80/20 for training and testing, and used to train Random Forest, Support Vector Machine, XGBoost, K-Nearest Neighbors, and Logistic Regression classifiers. Performance was assessed with accuracy, precision, recall, F1-score, and ROC AUC.

Random Forest and XGBoost achieved perfect results across all metrics, while SVM reached 95% accuracy, KNN 93%, and Logistic Regression 86%, with the lowest F1-score at 75%. The analysis identified defender score, gameplay duration, and attacker score as the most influential predictors, while the game outcome and player identifier contributed little, which the authors interpret as evidence that the models rely on behavioral signals rather than outcomes or identity.

The study is narrow in scope: it draws on a single IT organisation, a small participant pool, and one game, and the authors acknowledge limited generalizability. The perfect scores of the tree-based ensemble models warrant caution given the dataset size and the classification task, which assigns employees to tiers defined by the game's own scoring. Survey responses, in which over 80% of participants agreed or strongly agreed that the game improved their awareness, provide supporting but self-reported evidence. Overall, the work suggests that serious games combined with machine learning could support data-driven cybersecurity training, though validation across larger and more diverse populations remains necessary.