Examining Learner-driven Constructs in Co-curricular Engineering Environments: The Role of Student Reflection in Assessment Development

Lori C. Bland; Stephanie Marie Kusano; Xingya Xu; Aditya Johri

Download Paper | Permalink

Conference: 2017 ASEE Annual Conference & Exposition
Location: Columbus, Ohio
Publication Date: June 24, 2017
Start Date: June 24, 2017
End Date: June 28, 2017
Conference Session: Assessment of Student Work
Tagged Division: Educational Research and Methods
Page Count: 8
DOI: 10.18260/1-2--28315
Permanent URL: https://strategy.asee.org/28315
Download Count: 426

Request a correction

Paper Authors

biography

Lori C. Bland George Mason University orcid.org/0000-0003-4411-634X

visit author page

Lori C. Bland, Ph.D., is an associate professor at George Mason University. She teaches courses in educational assessment, program evaluation, and data-driven decision-making. Bland received her Ph.D. in Educational Psychology from the University of Virginia. Her current research focuses on identifying, examining, and assessing learning and professional outcomes in formal and informal learning environments in K-12, higher education, and the workforce; how data is used from assessments to inform decision-making; and the application of assessment or evaluation methods to solve educational problems.

visit author page

biography

Stephanie Marie Kusano University of Michigan

visit author page

Stephanie Kusano is an assessment and evaluation postdoctoral research associate at the Center for Research on Learning and Teaching at University of Michigan. She has a Ph.D. in Engineering Education, M.S. in Biomedical Engineering, and B.S. in Mechanical Engineering, all from Virginia Tech. Her research interests include engaged learning and high impact practices, assessment, and design education. Her teaching experience has primarily been with first-year engineering.

visit author page

biography

Xingya Xu George Mason University

visit author page

Xingya Xu is a Ph.D candidate of the Department of Educational Psychology in the College of Education and Human Development at George Mason University. She has a M.S. in Psychological Science at Western Kentucky University. Her research interests include metacognition, epistemic cognition and self-regulated learning.

visit author page

biography

Aditya Johri George Mason University orcid.org/0000-0001-9018-7574

visit author page

Aditya Johri is Associate Professor in the department of Information Sciences & Technology. Dr. Johri studies the use of information and communication technologies (ICT) for learning and knowledge sharing, with a focus on cognition in informal environments. He also examine the role of ICT in supporting distributed work among globally dispersed workers and in furthering social development in emerging economies. He received the U.S. National Science Foundation’s Early Career Award in 2009. He is co-editor of the Cambridge Handbook of Engineering Education Research (CHEER) published by Cambridge University Press, New York, NY. Dr. Johri earned his Ph.D. in Learning Sciences and Technology Design at Stanford University and a B.Eng. in Mechanical Engineering at Delhi College of Engineering.

visit author page

Download Paper | Permalink

Abstract

Motivation and Background Co-curricular engineering experiences range from very informal, such as face-to-face social networking within an engineering club to service learning experiences that approximate the workplace. Such experiences situate learning within an environment that may foster integration of knowledge and skills to solve problems (Pierrakos, Borrego, & Lo, 2007). These informal learning environments represent degrees of complexity. Therefore, applying design or problem solving within such environments may also lead to other types of latent outcomes, such as innovative ideas or adaptive expertise, or navigational flexibility (Author, 2014a; Pellegrino, DiBello, & Brophy, 2014; Sawyer & Greeno, 2008; Stevens et al., 2008). However, research about how to assess outcomes attained via participation in informal learning endeavors is nascent. We applied situated learning theory as a theoretical framework (Author, 2014b). We empirically examined student outcomes across six different informal or co-curricular experiences, as aligned with the 11 ABET (2013) student outcomes, outcomes that are critical to students learning to become an engineer (Balascio, 2014). While we report out the results based on the ABET outcomes, the main purposes for this paper were to: (a) examine the nature of the reflective discourse that revealed student self-directed learning of non-cognitive outcomes; (b) investigate the interactive nature of the technical and professional skills outcomes; and (c) explain the methods used to map the reflective discourse to develop assessments of student outcomes.

Methods We used qualitative methods to collect and analyze the data. Then, we applied content and response process validation methods (AERA, APA, NCME, 2014; Messick, 1995). This study took place in two large public universities. Both with multiple engineering majors. Data were collected from undergraduate engineering students participating in several different informal environments, including a engineering competitions, service learning, and a robotics club. Approximately 60 students participated across the events. We conducted focus groups and individual interviews with students participating in the challenge. Focus groups provided insight into the collective learning experiences (Ryan, Gandha, Culbertson, & Carlson, 2014). Sample protocol included open-ended questions, such as: “How would you describe your process?”

Results Preliminary Results indicated the incidence of learning outcomes associated with each of the ABET (2013) student technical skills. In addition, outcomes were inter-related across the ABET criteria. For example, students’ reflections often integrated their learning processes about design, experimentation, and problem-solving. Importantly, students also demonstrated adaptability throughout the competition. Illustrative examples of the mapping of response processes and test content are provided.

Implications This study provided evidence of student learning across the ABET technical and professional skills across a range of informal and co-curricular engineering experiences adding to the growing body of research examining learning in informal environments. More importantly, students engaged adaptive expertise (Walker, Cordray, King, & Brophy, 2006) understanding the need for flexibility and self-evaluation to solve a problem (Zimmerman, 2002). In addition, the study provided examined response processes as a validation method to establish test content.

Citation
Format

Bland, L. C., & Kusano, S. M., & Xu, X., & Johri, A. (2017, June), Examining Learner-driven Constructs in Co-curricular Engineering Environments: The Role of Student Reflection in Assessment Development Paper presented at 2017 ASEE Annual Conference & Exposition, Columbus, Ohio. 10.18260/1-2--28315

TY - CPAPER
AB - Motivation and Background
Co-curricular engineering experiences range from very informal, such as face-to-face social networking within an engineering club to service learning experiences that approximate the workplace. Such experiences situate learning within an environment that may foster integration of knowledge and skills to solve problems (Pierrakos, Borrego, &amp; Lo, 2007). These informal learning environments represent degrees of complexity. Therefore, applying design or problem solving within such environments may also lead to other types of latent outcomes, such as innovative ideas or adaptive expertise, or navigational flexibility (Author, 2014a; Pellegrino, DiBello, &amp; Brophy, 2014; Sawyer &amp; Greeno, 2008; Stevens et al., 2008). However, research about how to assess outcomes attained via participation in informal learning endeavors is nascent. We applied situated learning theory as a theoretical framework (Author, 2014b). We empirically examined student outcomes across six different informal or co-curricular experiences, as aligned with the 11 ABET (2013) student outcomes, outcomes that are critical to students learning to become an engineer (Balascio, 2014). While we report out the results based on the ABET outcomes, the main purposes for this paper were to: (a) examine the nature of the reflective discourse that revealed student self-directed learning of non-cognitive outcomes; (b) investigate the interactive nature of the technical and professional skills outcomes; and (c) explain the methods used to map the reflective discourse to develop assessments of student outcomes.

Methods
We used qualitative methods to collect and analyze the data. Then, we applied content and response process validation methods (AERA, APA, NCME, 2014; Messick, 1995). This study took place in two large public universities. Both with multiple engineering majors. Data were collected from undergraduate engineering students participating in several different informal environments, including a engineering competitions, service learning, and a robotics club. Approximately 60 students participated across the events. We conducted focus groups and individual interviews with students participating in the challenge. Focus groups provided insight into the collective learning experiences (Ryan, Gandha, Culbertson, &amp; Carlson, 2014). Sample protocol included open-ended questions, such as: “How would you describe your process?”

Results
Preliminary Results indicated the incidence of learning outcomes associated with each of the ABET (2013) student technical skills. In addition, outcomes were inter-related across the ABET criteria. For example, students’ reflections often integrated their learning processes about design, experimentation, and problem-solving. Importantly, students also demonstrated adaptability throughout the competition. Illustrative examples of the mapping of response processes and test content are provided.

Implications
This study provided evidence of student learning across the ABET technical and professional skills across a range of informal and co-curricular engineering experiences adding to the growing body of research examining learning in informal environments. More importantly, students engaged adaptive expertise (Walker, Cordray, King, &amp; Brophy, 2006) understanding the need for flexibility and self-evaluation to solve a problem (Zimmerman, 2002). In addition, the study provided examined response processes as a validation method to establish test content.

AU - Lori C. Bland
AU - Stephanie Marie Kusano
AU - Xingya Xu
AU - Aditya Johri
CY - Columbus, Ohio
DA - 2017/06/24
PB - ASEE Conferences
TI - Examining Learner-driven Constructs in Co-curricular Engineering Environments: The Role of Student Reflection in Assessment Development
UR - https://strategy.asee.org/28315
DO - 10.18260/1-2--28315
ER -