AI-powered play assessment approach using video language models: A feasibility study

Social-behavioral observation-based assessments are integral to clinical practice and research in health and psychology. However, these methods are often time-intensive and prone to human error, bias, and inconsistency. Deep neural networks (DNNs), a class of machine learning models, offer distinct advantages in healthcare assessments due to their advanced ability to process complex data with greater accuracy than traditional approaches, such as tree-based models. We developed innovative AI-powered software that integrates DNNs with computer vision techniques to analyze parent–child joint play interactions. Our objective was to utilize Video Large Language Models (Video LLMs) to automatically score a validated parent–child play scale (PC-SCP). Using convenience sampling, we recruited 37 mother–child dyads, including both neurotypical and neurodiverse children aged 1–6 years. Following eligibility screening and consent procedures, data collection involved recording 10–15-minute videos of parent–child interactions at home, which were then manually scored using the Parent/Caregiver Support of Children’s Playfulness criteria. We trained and evaluated several DNN models, including Qwen2.5VL, to identify and track parental behaviors in video frames and automatically score the interactions based on the PC-SCP guidelines. Among the models evaluated, our fine-tuned Qwen2.5VL achieved an accuracy of 38.2% and a best-5 accuracy of 61.3%, demonstrating promising potential for automating the scoring of social-behavioral assessments. This novel application of AI represents a significant advancement toward more efficient, objective, and consistent behavioral assessments in both clinical and research settings.