Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol

Carolyn Plumb; Rose M. Marra; Douglas J. Hacker; John Dunlosky

Download Paper | Permalink

Conference: 2018 ASEE Annual Conference & Exposition
Location: Salt Lake City, Utah
Publication Date: June 23, 2018
Start Date: June 23, 2018
End Date: July 27, 2018
Conference Session: Problem Solving, Adaptive Expertise, and Social Engagement
Tagged Division: Educational Research and Methods
Page Count: 23
DOI: 10.18260/1-2--30796
Permanent URL: https://strategy.asee.org/30796
Download Count: 631

Request a correction

Paper Authors

biography

Carolyn Plumb Montana State University

visit author page

Carolyn Plumb is the recently retired Director of Educational Innovation and Strategic Projects in the College of Engineering at Montana State University (MSU). Plumb has been involved in engineering education and program evaluation for 30 years, and she continues to work on externally funded projects relating to engineering education.

visit author page

biography

Rose M. Marra University of Missouri

visit author page

Rose M. Marra is a Professor of Learning Technologies at the University of Missouri. She is PI of the NSF-funded Supporting Collaboration in Engineering Education, and has studied and published on engineering education, women and minorities in STEM, online learning and assessment. Marra holds a PhD. in Educational Leadership and Innovation and worked as a software engineer before entering academe.

visit author page

biography

Douglas J. Hacker University of Utah

visit author page

Dr. Hacker is a full professor in the Department of Educational Psychology and participates in both the Learning Sciences Program and the Reading and Literacy Program. Prior to receiving his Ph. D. in educational psychology from the University of Washington in 1994, Dr. Hacker worked as a high school science and math teacher and then as a school counselor. From 1994 to 1999, Dr. Hacker was an assistant/associate professor in the Department of Counseling, Educational Psychology and Research at The University of Memphis. During those years, he worked in the areas of reading and writing processes, metacognition, self-regulated learning, teacher education, and school and program evaluation. Dr. Hacker moved to the University of Utah in 1999 and has continued his research in the previous areas and has added to them research in the area of the detection of deception. Also at the University of Utah, he served as chair of the Teaching and Learning Department. His publications have appeared in the Journal of Educational Psychology, Contemporary Educational Psychology, Journal of Experimental Psychology: Applied, and Journal of Experimental Education. At both universities, Dr. Hacker has maintained a strong commitment to work in elementary and middle schools, working directly with teachers by providing professional development in reading and writing instruction. Since 1994, Dr. Hacker has been either the principal investigator or co-principal investigator on grants totaling $2,548,960. He has served as an editorial board member for the Journal of Educational Psychology, Metacognition and Learning, and Frontiers of Educational Psychology. He is a former Associate Editor for the Journal of Educational Psychology.

visit author page

author page

John Dunlosky

Download Paper | Permalink

Abstract

Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol

This evidence-based practice paper describes one method of measuring engineering students’ use of metacognition while problem solving. The capacity for life-long learning is critical for engineering practice and according to the Accreditation Board for Engineering and Technology (ABET) is a required outcome for engineering accreditation. Metacognition, defined as cognition of one’s cognitions, is a higher-order thinking skill and provides the key to developing life-long learning skills necessary for ABET and for an effective work career, yet it is rarely integrated into engineering education. At Iron Range Engineering (IRE), an innovative, problem-based-learning (PBL) engineering program in Virginia, Minnesota, students learn about and apply metacognitive skills. In our NSF-funded project, we are (1) identifying and understanding the metacognitive skills students develop and use during their preparation as engineers in a PBL program and (2) examining whether the preparation of students in the PBL program (particularly in the area of metacognition) gives them a “leg up” in their transition to the engineering workforce. Our mixed-methods study consists of student interview data supplemented with interviews of recent IRE graduates employed as engineers and of employers of the IRE graduates to better understand the effect of this unique undergraduate program on student preparedness for the engineering workforce. In addition, we are conducting think-aloud protocols during student solving of an open-ended design problem.

In this paper, we focus on the think-aloud data we have gathered during this project. We describe our process of gathering think-aloud data from students, as well as our coding structure, which categorizes utterances as metacognitive or not, and, if metacognitive, whether the utterance is a type of metacognitive monitoring or metacognitive control. In addition, we detail our coding process and measures of inter-rater reliability. Finally, we share results to date, including comparisons of the think-aloud data from students about to graduate from the program with students entering the program. The think-alouds from entering students and graduating students are showing marked differences. Our intent is to demonstrate the presence of critical metacognitive processes during problem solving as students progress through their engineering education.

Citation
Format

Plumb, C., & Marra, R. M., & Hacker, D. J., & Dunlosky, J. (2018, June), Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30796

TY - CPAPER
AB - Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol

This evidence-based practice paper describes one method of measuring engineering students’ use of metacognition while problem solving. The capacity for life-long learning is critical for engineering practice and according to the Accreditation Board for Engineering and Technology (ABET) is a required outcome for engineering accreditation. Metacognition, defined as cognition of one’s cognitions, is a higher-order thinking skill and provides the key to developing life-long learning skills necessary for ABET and for an effective work career, yet it is rarely integrated into engineering education. At Iron Range Engineering (IRE), an innovative, problem-based-learning (PBL) engineering program in Virginia, Minnesota, students learn about and apply metacognitive skills. In our NSF-funded project, we are (1) identifying and understanding the metacognitive skills students develop and use during their preparation as engineers in a PBL program and (2) examining whether the preparation of students in the PBL program (particularly in the area of metacognition) gives them a “leg up” in their transition to the engineering workforce. Our mixed-methods study consists of student interview data supplemented with interviews of recent IRE graduates employed as engineers and of employers of the IRE graduates to better understand the effect of this unique undergraduate program on student preparedness for the engineering workforce. In addition, we are conducting think-aloud protocols during student solving of an open-ended design problem.

In this paper, we focus on the think-aloud data we have gathered during this project. We describe our process of gathering think-aloud data from students, as well as our coding structure, which categorizes utterances as metacognitive or not, and, if metacognitive, whether the utterance is a type of metacognitive monitoring or metacognitive control. In addition, we detail our coding process and measures of inter-rater reliability. Finally, we share results to date, including comparisons of the think-aloud data from students about to graduate from the program with students entering the program. The think-alouds from entering students and graduating students are showing marked differences. Our intent is to demonstrate the presence of critical metacognitive processes during problem solving as students progress through their engineering education.

AU - Carolyn Plumb
AU - Rose M. Marra
AU - Douglas J. Hacker
AU - John Dunlosky
CY - Salt Lake City, Utah
DA - 2018/06/23
PB - ASEE Conferences
TI - Measuring Engineering Students’ Metacognition with a Think-Aloud Protocol
UR - https://strategy.asee.org/30796
DO - 10.18260/1-2--30796
ER -