Using LMS Data to Provide Early Alerts to Struggling Students

Donald F. Hayes; Wonjoon Hong; MATTHEW L BERNACKI; Nicholas Voorhees

Download Paper | Permalink

Conference: 2017 FYEE Conference
Location: Daytona Beach, Florida
Publication Date: August 6, 2017
Start Date: August 6, 2017
End Date: August 8, 2017
Conference Session: Student Success & Development - Focus on Academic Support
Tagged Topics: Diversity and FYEE Division - Paper Submission
Page Count: 8
DOI: 10.18260/1-2--29442
Permanent URL: https://peer.asee.org/29442
Download Count: 389

Request a correction

Paper Authors

biography

Donald F. Hayes PE University of Nevada, Las Vegas

visit author page

Dr. Hayes is currently the Chair of Civil and Environmental Engineering and Construction at UNLV. He has over 25 years of academic experience and 12 years of industrial experience. He has been teaching a First Year Engineering Experience course since 2014.

visit author page

author page

Wonjoon Hong University of Nevada, Las Vegas

biography

MATTHEW L BERNACKI University of Nevada, Las Vegas

visit author page

Matthew Bernacki is an Assistant Professor of Educational Psychology. He earned his Ph.D. in educational psychology in 2010 from Temple University in Philadelphia and also holds master’s degrees in Experimental Psychology from Saint Joseph’s University and Social Work from Temple University. Prior to arriving at UNLV, Matt worked at the Learning Research & Development Center (University of Pittsburgh) and the Human Computer Interaction Institute (Carnegie Mellon University) as a postdoctoral researcher at LearnLab.
Matt’s research focuses on (1) the roles that motivations and metacognitive processes play when learners use technologies like hypertext, intelligent tutoring systems, and learning management systems, (2) the development of interventions and software to promote effective learning strategies, and motivation to learn, and (3) the development of learning materials and environments that personalize learning to students’ interests.

visit author page

author page

Nicholas Voorhees

Download Paper | Permalink

Abstract

The academic demands of college curricula often expose poor time management and study skills of freshmen students. Interventional advising can help get struggling students on track. Mid-term grades traditionally provide the first opportunity to identify those students; unfortunately, by the time mid-term grades are posted and interventional advising can occur, it is difficult for struggling students to significantly change their final grades. This project maximizes the utility of a Learning Management System (LMS) to improve the efficacy of early warnings by providing timely alerts to struggling students before they accrue poor performances.

An LMS can serve as a comprehensive platform for delivering rich multimedia content to learners, managing discussions, organizing collaborative and problem-based learning activities, and conducting assessments. In many engineering courses, the LMS is a barren space that provides only a syllabus, a few handouts, and - maybe - an online gradebook. We used the LMS to provide students with a rich digital environment for learning by creating and hosting all course materials within the LMS. A data management and visualization tool called Splunk was used to model usage of LMS resources and provide a timely picture of students’ learning progress. In an initial correlational study, students’ usage of course resources were found to correlate to performance in the first year engineering course (rLMSevents = .44, rfolders_accessed = .42, rLectureNoteDownloads = .39).

The traditional model of having mid-semester grades prompt meetings with an advisor is inherently flawed. First, it comes after mid-semester (i.e., week 9), which limits the time remaining for students who receive support to get themselves on track. Second, it relies on early poor performance, which once achieved diminish the potential to recover from early failure. Early, behavior-based prediction modeling and intervention avoids both these weaknesses.

A second study utilized educational data mining methods to produce a prediction algorithm based on digital course material usage. A logistic regression model was estimated using LMS behavioral data from the first five weeks of the course to predict student performance: whether a student obtained a C or better, or a D or worse. The cross-validated prediction model accurately classified 79% of students as C or Better vs. D or worse learners (Kappa = .57) based upon LMS access patterns. The model identified learners likely to perform poorly well before mid-semester grades. It accurately identified 54 of the 79 of students who ultimately failed to obtain a C or Better during the training and testing phase of prediction model development. This degree of specificity (68.39%) provided sufficient accuracy that the prediction algorithm was programmed back into Splunk to provide real time predictions of students’ success projections. An initial intervention study is ongoing to 1) identify students likely to struggle in the course, and 2) alert these students and provide them additional learning resources. The full paper will include a detailed account of prediction model features and additional results on the number of students identified, the percent of whom responded to alerts and learning supports, and the effects of the intervention.

Citation
Format

Hayes, D. F., & Hong, W., & BERNACKI, M. L., & Voorhees, N. (2017, August), Using LMS Data to Provide Early Alerts to Struggling Students Paper presented at 2017 FYEE Conference, Daytona Beach, Florida. 10.18260/1-2--29442

TY - CPAPER
AB - The academic demands of college curricula often expose poor time management and study skills of freshmen students. Interventional advising can help get struggling students on track. Mid-term grades traditionally provide the first opportunity to identify those students; unfortunately, by the time mid-term grades are posted and interventional advising can occur, it is difficult for struggling students to significantly change their final grades. This project maximizes the utility of a Learning Management System (LMS) to improve the efficacy of early warnings by providing timely alerts to struggling students before they accrue poor performances.

An LMS can serve as a comprehensive platform for delivering rich multimedia content to learners, managing discussions, organizing collaborative and problem-based learning activities, and conducting assessments. In many engineering courses, the LMS is a barren space that provides only a syllabus, a few handouts, and - maybe - an online gradebook. We used the LMS to provide students with a rich digital environment for learning by creating and hosting all course materials within the LMS. A data management and visualization tool called Splunk was used to model usage of LMS resources and provide a timely picture of students’ learning progress. In an initial correlational study, students’ usage of course resources were found to correlate to performance in the first year engineering course (rLMSevents = .44, rfolders_accessed = .42, rLectureNoteDownloads = .39).

The traditional model of having mid-semester grades prompt meetings with an advisor is inherently flawed. First, it comes after mid-semester (i.e., week 9), which limits the time remaining for students who receive support to get themselves on track. Second, it relies on early poor performance, which once achieved diminish the potential to recover from early failure. Early, behavior-based prediction modeling and intervention avoids both these weaknesses.

A second study utilized educational data mining methods to produce a prediction algorithm based on digital course material usage. A logistic regression model was estimated using LMS behavioral data from the first five weeks of the course to predict student performance: whether a student obtained a C or better, or a D or worse. The cross-validated prediction model accurately classified 79% of students as C or Better vs. D or worse learners (Kappa = .57) based upon LMS access patterns. The model identified learners likely to perform poorly well before mid-semester grades. It accurately identified 54 of the 79 of students who ultimately failed to obtain a C or Better during the training and testing phase of prediction model development. This degree of specificity (68.39%) provided sufficient accuracy that the prediction algorithm was programmed back into Splunk to provide real time predictions of students’ success projections. An initial intervention study is ongoing to 1) identify students likely to struggle in the course, and 2) alert these students and provide them additional learning resources. The full paper will include a detailed account of prediction model features and additional results on the number of students identified, the percent of whom responded to alerts and learning supports, and the effects of the intervention.

AU - Donald F. Hayes PE
AU - Wonjoon Hong
AU - MATTHEW L BERNACKI
AU - Nicholas Voorhees
CY - Daytona Beach, Florida
DA - 2017/08/06
PB - ASEE Conferences
TI - Using LMS Data to Provide Early Alerts to Struggling Students
UR - https://peer.asee.org/29442
DO - 10.18260/1-2--29442
ER -