Authors: Jeremy Roschelle, Gabrielle Lue and Gautam Biswas
On July 17, 2024, Professor Gautam Biswas presented the “Test of Time” keynote at the Educational Data Mining conference in Atlanta, Georgia. Prof. Biswas, a co-PI of the NSF EngageAI Institute, was honored for his 2011 paper “Identifying Learning Behaviors by Contextualizing Differential Sequence Mining with Action Features & Performance Evolution,” which establishes a basis for a strand of multimodal analytics research that remains important to learning scientists and learning engineers today. This paper was recognized for standing up to the “test of time.” As discussed below, he connected this older paper to a strand of research that began in the 1990s and that EngageAI is now advancing into the future.
The issue addressed in this research lies at the heart of the learning sciences, a field that looks for ways to develop students’ ability to do meaningful, extensive, independent reasoning in science and other STEM fields. To develop such skills, educators give students tasks that are open-ended and thought-provoking. These tasks engage students over longer periods of time compared to typical problems on worksheets. Also, students typically create more extended responses, not just a multiple-choice or fill-in-the-blank answer. The issue is—how can educators analyze students’ progress, reasoning, and responses?
In today’s AI-driven age, this issue becomes the “paradox of personalization.” AI offers adaptive capabilities, but the data it can access about students is typically extremely limited (especially compared to what a human teacher would notice), and, therefore, adaptivity ends up also being very limited. Indeed, most adaptive educational products on the market only personalize based on the correctness of the students’ last few short-answer responses. No teacher would only look at a few recent items to decide how best to support a student.
In his talk, Prof. Biswas described how his teams’ 2011 paper described how to detect “phases” of learning in a longer, open-ended task and then how to understand “sequences” of learning behaviors in those phases. His techniques advanced based on earlier work by other investigators in the 1990s (see references below). Using the more advanced techniques he developed, student actions and the sequences of their actions can be automatically tagged as productive or non-productive. Then an analysis technique identifies patterns across many students and many sequences—looking for patterns that lead to more or less learning of the science content. A benefit of their approach is that teachers and educational curriculum developers can understand the output: the more and less positive sequences can be described in terms that educators readily understand (such as take-a-quiz, read relevant material, then work on making knowledge links). Then, decisions to adapt the curriculum resources and supports to students can be based on this knowledge of how students learn.
Today, Prof. Biswas is advancing these sorts of analysis capabilities by building a multimodal analysis pipeline for EngageAI and other applications in the field. “Multimodal” refers to the ability to collect, synchronize, and analyze parallel streams of information about what the students are doing as they engage in a long-term, open-ended science task. The modalities can include voice, eye gaze, gesture, logs from the curricular software, and even aspects of student affect and their collaborative behaviors with other students. The team is developing this pipeline to work with EngageAI’s ambitious science curriculum units, which aim to develop students’ collaborative reasoning about science. By being able to analyze multiple streams of information, the team can better understand what students do over time, what’s more or less productive, and how to support students to succeed in these science reasoning tasks.
Prof. Biswas acknowledges the challenging ethical and privacy considerations that must be addressed and are addressed in EngageAI research through Institutional Review Board processes. The eventual benefits will be (a) analysis methods that align with today’s educational goals, such as supporting students’ development of sophisticated reasoning skills and (b) using broad information—like what a teacher could observe, and not just limited to simple, short answer student responses and (c) greater ability to adapt learning supports to students needs and experiences than is possible when automated analyses are limited to simple data or one-dimension streams of information.
The resources below include links to Prof. Biswas’s recorded keynote, his presentation deck, and key references: