The EngageAI Institute team aims to examine the patterns of students’ engagement in collaborative problem solving (CPS) activities in the Institute’s narrative-centered learning environments by analyzing students’ collaborative dialogue during small group interactions. To analyze students’ collaborative problem solving, researchers from Indiana University have annotated students’ collaborative discourses during their small group activities within the EcoJourneys learning environment through the lens of the CPS framework by Liu et al. (2016) and von Davier et al. (2017). Drawing upon a discursive perspective, the project leveraged the CPS framework, which identifies crucial components of CPS practice. These components include sharing ideas, negotiating ideas, regulating problem-solving, and maintaining communication, particularly in the domain of science. The coding scheme allows for the examination of the discursive aspect of collaboration, enabling the researchers to convert collective discourses into observable CPS practices.
Students’ conversations during collaborative learning are often quite complex and therefore substantial effort is required to analyze the details of these conversations. Intending to assist educators in analyzing students’ group conversations during narrative-centered learning experiences, the EngageAI Institute team is investigating language model-based methods for automatic identification of domain-specific dialogue acts related to narrative-centered learning and collaborative problem solving.
- Supervised Fine-Tuned T5-Based Approach for Classroom Dialogue Analysis: Automatically tagging and analyzing student dialogues can offer valuable insights into how knowledge is shared and negotiated during collaborative learning sessions. Furthermore, understanding epistemic stance and collaborative problem-solving strategies enables educators to grasp a more holistic view of student learning experiences. In response to this need, we devised language model-based approaches for automatic dialogue act recognition. Specifically, we applied a dual contrastive fine-tuning approach with label-aware data augmentation to enhance a T5 model’s ability to distinguish between closely related dialogue utterances produced by students during collaborative narrative-centered learning in EcoJourneys. This method maximizes the correlation between input samples and their corresponding labels while minimizing irrelevant associations. Our findings indicate that this approach outperforms T5 models without augmentation and other BERT-based methods. Nonetheless, the proposed approach still faces challenges with class labels that exhibit significant semantic and contextual overlap.
- Few & Zero-Shot Learning Approaches for Classroom Dialogue Analysis: Motivated by the success of the above-mentioned approach, we have been working on using large language models like Llama-7B fine-tuned on this dataset. Our results are promising with these models outperforming the T5-based model, but we have identified several technical challenges. The first challenge is the limited amount of labeled data. The team is exploring few/zero-shot approaches to address this challenge. The second challenge is that the dialogue act annotations are very domain specific and have complex definitions. It is non-trivial to automatically identify them in a few/zero shot manner. The team is exploring a tiered prompting strategy to address this challenge. The third challenge is label imbalance. Some of the labels are very diverse and have fewer training instances. The team has begun exploring active-learning based approaches to identify good instances to use in few-shot learning.
Upcoming Plans: The team will continue to collect and annotate students’ collaborative dialogues during narrative-centered learning with EcoJourneys. Using this data, the team will continue to design and refine LLM-based systems that can effectively analyze students’ conversations during collaborative narrative-centered learning. Additionally, the team plans to investigate the integration of dialogue act recognition models into the run-time EcoJourneys narrative-centered learning environment to drive adaptive scaffolding that supports students’ collaborative learning processes.