Tracking and Predicting Student Performance Across Different Semesters with Matched Action-State Orientation Surveys and Interventions

Ismail Uysal; Mehmet Bugrahan Ayanoglu; Rania Sherif Elashmawy; Rifatul Islam; Paul E. Spector; Chris S. Ferekides

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: Community Building and Student Engagement
Tagged Division: Electrical and Computer Engineering Division (ECE)
Permanent URL: https://peer.asee.org/48169

Request a correction

Paper Authors

biography

Ismail Uysal University of South Florida

visit author page

Dr. Ismail Uysal has a Ph.D. in Electrical and Computer Engineering from the University of Florida. He is an Associate Professor and the Undergraduate Director at the University of South Florida's Electrical Engineering Department. His research focuses on theory and applications of machine learning in education, healthcare and smart agriculture including machine intelligence for sensor applications.

visit author page

author page

Mehmet Bugrahan Ayanoglu University of South Florida

biography

Rania Sherif Elashmawy University of South Florida orcid.org/0000-0003-1691-8353

visit author page

Dr. Rania Elashmawy has a Ph. D in Electrical Engineering from the University of South Florida, USA. Her research interests include smart agriculture, precision agriculture, and time-series data.

visit author page

author page

Abstract

This paper presents the second year results of the work supported by the National Science Foundation’s Revolutionizing Engineering Departments (IUSE/PFE: RED) Program under the project titled "IUSE/PFE:RED: Breaking Boundaries: An Organized Revolution for the Professional Formation of Electrical Engineers." Specifically, this part of the study looks at action-state orientation and its impacts on student success. The first-year results were presented at the 2023 ASEE Conference in Baltimore, MD with the academic paper titled "to be included after blind review." for further reference (Reference after blind review, 2023). The objective of the first phase of the study was to find out how survey responses could be used to predict whether a student could be considered at-risk for failing academically. The objective of the second phase discussed in this article is to analyze and quantify the effects of in-class interventions on student study habits and, ultimately, their academic performance using action-state orientation surveys as engineering students progress in their respective curriculum.

While these surveys are anonymous, it is crucial to be able to track changes in academic performance for individual students across multiple semesters as they go through the various stages of their academic program (in this case, Electrical Engineering). As part of the second phase, we developed a powerful matching method that can automate the demographic information matching in the background with Python libraries to ensure sustainable analysis as the data collected from both new and ongoing students naturally grew larger over the past several years. Ultimately, we were able to match a total of 840 unique students based on their self-provided information such as gender, month of birth, ethnicity, and high school names across 2148 unique survey responses collected in 5 different academic semesters: Spring 2021, Fall 2021, Spring 2022, Fall 2022, and Spring 2023. Beyond the scale of data, which is unprecedented for this kind of survey, we were able to significantly boost the prediction performance of our machine learning algorithms from 74.4% reported in the previous study for a simpler question (i.e., is this student's GPA less than 2.0? - a more apparent anomaly) to 82.6% for a more challenging question (i.e., is this student's GPA higher or lower than 3.33? - a more subtle distinction).

To accomplish this, we leveraged sophisticated machine learning classifiers before settling on the random forest classifier with feature elimination, thanks to the increasing size of the collected data from the newly added surveys. The students in the dataset were split into two groups based on their GPA such that the method can learn from survey responses to correctly identify the category (high or low GPA) where k-fold (10) cross-validation was used to ensure robust and repeatable accuracy metrics were obtained. The dataset was further split into two partitions by classifying survey responses as pre-intervention and post-intervention, where 921 unique responses were classified as post-intervention (POST), and 1227 unique responses were classified as pre-intervention (PRE). Using this information, a new predictor was trained using only the PRE dataset and was tested on both the PRE and the POST datasets. The hypothesis was that the new empirical model would perform worse with more false-positives on the POST dataset due to newly acquired and hopefully improved study habits after the interventions. Our results show a 35% increase in prediction error when the same algorithm is tested on the POST student population and more importantly a corresponding 24% increase in the false-positive rate which indicates that the interventions are working at the population level where students adopt study habits that outperform their current academic performance as likely indicators.

Citation
Format

Uysal, I., & Ayanoglu, M. B., & Elashmawy, R. S., & Islam, R., & Spector, P. E., & Ferekides, C. S. (2024, June), Tracking and Predicting Student Performance Across Different Semesters with Matched Action-State Orientation Surveys and Interventions Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/48169

TY  - CPAPER
AB  - This paper presents the second year results of the work supported by the National Science Foundation’s Revolutionizing Engineering Departments (IUSE/PFE: RED) Program under the project titled &quot;IUSE/PFE:RED: Breaking Boundaries: An Organized Revolution for the Professional Formation of Electrical Engineers.&quot; Specifically, this part of the study looks at action-state orientation and its impacts on student success. The first-year results were presented at the 2023 ASEE Conference in Baltimore, MD with the academic paper titled &quot;to be included after blind review.&quot; for further reference (Reference after blind review, 2023). The objective of the first phase of the study was to find out how survey responses could be used to predict whether a student could be considered at-risk for failing academically.  The objective of the second phase discussed in this article is to analyze and quantify the effects of in-class interventions on student study habits and, ultimately, their academic performance using action-state orientation surveys as engineering students progress in their respective curriculum. 

While these surveys are anonymous, it is crucial to be able to track changes in academic performance for individual students across multiple semesters as they go through the various stages of their academic program (in this case, Electrical Engineering). As part of the second phase, we developed a powerful matching method that can automate the demographic information matching in the background with Python libraries to ensure sustainable analysis as the data collected from both new and ongoing students naturally grew larger over the past several years. Ultimately, we were able to match a total of 840 unique students based on their self-provided information such as gender, month of birth, ethnicity, and high school names across 2148 unique survey responses collected in 5 different academic semesters: Spring 2021, Fall 2021, Spring 2022, Fall 2022, and Spring 2023. Beyond the scale of data, which is unprecedented for this kind of survey, we were able to significantly boost the prediction performance of our machine learning algorithms from 74.4% reported in the previous study for a simpler question (i.e., is this student&#39;s GPA less than 2.0? - a more apparent anomaly) to 82.6% for a more challenging question (i.e., is this student&#39;s GPA higher or lower than 3.33? - a more subtle distinction). 

To accomplish this, we leveraged sophisticated machine learning classifiers before settling on the random forest classifier with feature elimination, thanks to the increasing size of the collected data from the newly added surveys. The students in the dataset were split into two groups based on their GPA such that the method can learn from survey responses to correctly identify the category (high or low GPA) where k-fold (10) cross-validation was used to ensure robust and repeatable accuracy metrics were obtained. The dataset was further split into two partitions by classifying survey responses as pre-intervention and post-intervention, where 921 unique responses were classified as post-intervention (POST), and 1227 unique responses were classified as pre-intervention (PRE). Using this information, a new predictor was trained using only the PRE dataset and was tested on both the PRE and the POST datasets. The hypothesis was that the new empirical model would perform worse with more false-positives on the POST dataset due to newly acquired and hopefully improved study habits after the interventions. Our results show a 35% increase in prediction error when the same algorithm is tested on the POST student population and more importantly a corresponding 24% increase in the false-positive rate which indicates that the interventions are working at the population level where students adopt study habits that outperform their current academic performance as likely indicators.
AU  - Ismail Uysal
AU  - Mehmet Bugrahan Ayanoglu
AU  - Rania Sherif Elashmawy
AU  - Rifatul Islam
AU  - Paul E. Spector
AU  - Chris S. Ferekides
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - Tracking and Predicting Student Performance Across Different Semesters with Matched Action-State Orientation Surveys and Interventions
UR  - https://peer.asee.org/48169
ER  -