A Cognitive Model for Automatic Student Assessment: Classification of Errors in Engineering Dynamics

Jeffrey A. Davis; Shelley Lorimer

Download Paper | Permalink

Conference: 2014 ASEE Annual Conference & Exposition
Location: Indianapolis, Indiana
Publication Date: June 15, 2014
Start Date: June 15, 2014
End Date: June 18, 2014
ISSN: 2153-5965
Conference Session: Student Learning, Problem Solving, and Critical Thinking 1
Tagged Division: Educational Research and Methods
Page Count: 10
Page Numbers: 24.26.1 - 24.26.10
DOI: 10.18260/1-2--19918
Permanent URL: https://strategy.asee.org/19918
Download Count: 354

Request a correction

Paper Authors

biography

Jeffrey A. Davis Grant MacEwan University

visit author page

With degrees in both civil and mechanical engineering, Jeff went on to obtain a Ph.D. from the Institute of Energy Technology at ETH Zurich in 2004. His past research includes dispersion of pollutants in rivers, and turbulent and multiphase flow modeling from a numerical perspective. Currently, Jeff is a first-year engineering instructor at MacEwan University. With a passion for teaching, his focus on research has turned to understanding and automating student assessment techniques as well as looking at the socioeconomic sustainability of educational institutions.

visit author page

biography

Shelley Lorimer Grant MacEwan University

visit author page

Dr. Shelley Lorimer, P.Eng., is chair of the bachelor of science in engineering transfer Program (BSEN) at Grant MacEwan University in Edmonton, Alberta. She teaches undergraduate courses in statics and dynamics, as well as courses in engineering professionalism. She currently is participating in a research project with Alberta Innovates – Technology Futures in the oil sands and hydrocarbon recovery group doing reservoir simulation of enhanced oil-recovery processes. She has a Ph.D. in numerical modeling from the University of Alberta, also in Edmonton.

visit author page

Download Paper | Permalink

Abstract

Development of a Cognitive Model for Automatic Student Assessment: Classification of Student Errors in Engineering Statics and DynamicsThe taxonomy of human errors is critical for professions dealing with public safety such asengineering and healthcare. Once errors are identified, preventative and corrective measures canbe sought. From a pedagogical perspective, detection of error is also important. When studentsmake an error it is important to determine why the error was made and then begin the process ofcorrecting it by providing them with individualized feedback and assignments. For anexperienced educator, student assessment can be a fairly straightforward task. However, during atypical course, when a student is in the learning stage of development, assessment of thestudent's work is typically performed by a marker. This undergraduate or graduate student oftenhas teaching skills that are not fully developed and therefore may have difficulty properlyassessing the students' work. Guidelines or rules can be provided to the marker in the form of arubric but can be either too general or assignment specific. In addition, with decreasing educationbudgets and increasing popularity of distance learning in the form of massive open onlinecourses, automatic assessment methods are becoming increasingly important.This paper summarizes the errors that students made in two first year engineering final exams:engineering statics and dynamics. A cognitive model is then developed based on an errorclassification scheme rooted in Action Theory, which serves to classify errors as either mistakesor slips in logic. Simple rules are then developed into an algorithm for a computer to categorizeerrors based on a priori concepts such as: student experience level and priority level given on thecourse concept inventory as well as a posteriori concepts such as the frequency of the error. Theassessment algorithm is then “trained” using two final exams. The coefficients in the algorithmare optimized by comparing the results from experienced educators and that of the algorithm.The optimized algorithm is then tested on a third independent final exam and the resulting marksare compared. The cognitive model used is found to be general and easily extendible for use inother STEM courses. Results of the study show that the model performs well for probabilityregions where there is a clear distinction between slips and mistakes, but becomes moredependent on the model coefficients when the probability of the mistake, P(M)=0.5. Extension ofthis model is shown to be useful for automatically assessing a students' work as well as providingtimely and meaningful feedback.

Citation
Format

Davis, J. A., & Lorimer, S. (2014, June), A Cognitive Model for Automatic Student Assessment: Classification of Errors in Engineering Dynamics Paper presented at 2014 ASEE Annual Conference & Exposition, Indianapolis, Indiana. 10.18260/1-2--19918

TY - CPAPER
AB - Development of a Cognitive Model for Automatic Student Assessment: Classification of Student Errors in Engineering Statics and DynamicsThe taxonomy of human errors is critical for professions dealing with public safety such asengineering and healthcare. Once errors are identified, preventative and corrective measures canbe sought. From a pedagogical perspective, detection of error is also important. When studentsmake an error it is important to determine why the error was made and then begin the process ofcorrecting it by providing them with individualized feedback and assignments. For anexperienced educator, student assessment can be a fairly straightforward task. However, during atypical course, when a student is in the learning stage of development, assessment of thestudent&#39;s work is typically performed by a marker. This undergraduate or graduate student oftenhas teaching skills that are not fully developed and therefore may have difficulty properlyassessing the students&#39; work. Guidelines or rules can be provided to the marker in the form of arubric but can be either too general or assignment specific. In addition, with decreasing educationbudgets and increasing popularity of distance learning in the form of massive open onlinecourses, automatic assessment methods are becoming increasingly important.This paper summarizes the errors that students made in two first year engineering final exams:engineering statics and dynamics. A cognitive model is then developed based on an errorclassification scheme rooted in Action Theory, which serves to classify errors as either mistakesor slips in logic. Simple rules are then developed into an algorithm for a computer to categorizeerrors based on a priori concepts such as: student experience level and priority level given on thecourse concept inventory as well as a posteriori concepts such as the frequency of the error. Theassessment algorithm is then “trained” using two final exams. The coefficients in the algorithmare optimized by comparing the results from experienced educators and that of the algorithm.The optimized algorithm is then tested on a third independent final exam and the resulting marksare compared. The cognitive model used is found to be general and easily extendible for use inother STEM courses. Results of the study show that the model performs well for probabilityregions where there is a clear distinction between slips and mistakes, but becomes moredependent on the model coefficients when the probability of the mistake, P(M)=0.5. Extension ofthis model is shown to be useful for automatically assessing a students&#39; work as well as providingtimely and meaningful feedback.
AU - Jeffrey A. Davis
AU - Shelley Lorimer
CY - Indianapolis, Indiana
DA - 2014/06/15
PB - ASEE Conferences
TI - A Cognitive Model for Automatic Student Assessment: Classification of Errors in Engineering Dynamics
UR - https://strategy.asee.org/19918
DO - 10.18260/1-2--19918
ER -