Investigating the Impact of College Students’ Personal Characteristics on Peer Assessment: A Multilevel Linear Modeling Approach

Xiaping Li; Robin Fowler; Mark Mills

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: Educational Research and Methods Division (ERM) Technical Session 10
Tagged Division: Educational Research and Methods Division (ERM)
Tagged Topic: Diversity
Permanent URL: https://peer.asee.org/47697

Request a correction

Paper Authors

biography

Xiaping Li University of Michigan orcid.org/0000-0002-2620-1690

visit author page

Xiaping Li is a Ph.D. candidate in Engineering Education Research at the University of Michigan. Her research interests encompass faculty development and change, neurodiverse college student learning experiences and outcomes, international students in engineering, and cognitive sciences. She holds a B.S. in Hydrology and Water Resources Engineering and an M.S. in Geological Sciences.

visit author page

biography

Robin Fowler University of Michigan orcid.org/0000-0001-6161-0986

visit author page

Robin Fowler is a Technical Communication lecturer and a Engineering Education researcher at the University of Michigan. Her teaching is primarily in team-based engineering courses, and her research focuses on equity in communication and collaboration as well as in group design decision making (judgment) under uncertainty. She is especially interested in how power relationships and rhetorical strategies affect group judgment in engineering design; one goal of this work is to to understand factors that inhibit full participation of students who identify with historically marginalized groups and investigate evidence-based strategies for mitigating these inequities. In addition, she is interested in technology and how specific affordances can change the ways we collaborate, learn, read, and write. Teaching engineering communication allows her to apply this work as she coaches students through collaboration, design thinking, and design communication. She is part of a team of faculty innovators who originated Tandem (tandem.ai.umich.edu), a tool designed to help facilitate equitable and inclusive teamwork environments.

visit author page

biography

Mark Mills University of Michigan orcid.org/0000-0002-1145-9592

visit author page

Mark Mills (he/him) is a Data Scientist on the Research & Analytics team at University of Michigan’s Center for Academic Innovation. He directs and supports analytics across CAI’s portfolio of educational technologies. His experience is in prediction and classification of longitudinal and hierarchically cross-classified data structures such as students in courses measured over time.

visit author page

Download Paper | Permalink

Abstract

Work in Progress: Investigating the Impact of College Students’ Personal Characteristics on Peer Assessment: A Multilevel Linear Modeling Approach

Keywords: Peer Assessment; Multilevel Linear Modeling; Personal Characteristics

Abstract: Teamwork is an essential skill expected of college students, and it has been frequently integrated into courses by instructors, making the assessment of these activities an important component of university curricula. Peer assessment serves as a crucial assessment method, offering valuable feedback and enhancing student learning outcomes. It is commonly used in college courses that employ team-based learning, and there is a growing focus on the impact of peer assessment design on student learning outcomes. Previous research highlights correlations between peer ratings and factors related to personal characteristics, including gender, academic performance, personality traits, and group preferences; however, a significant gap remains in our understanding of how these factors specifically influence peer ratings in the context of college course teamwork. This knowledge gap hinders our ability to provide instructors, students, and researchers with evidence-based insights into the effectiveness of team-based pedagogy and into the interpretation of peer assessments. To address this gap, our study employs multilevel linear modeling to investigate the relationships between these factors and peer rating scores in the context of college course teamwork.

We collected peer rating data from 5,322 college students at a Midwest research university spanning the period from 2019 to 2023. The data were collected using Tandem, a digital instructional tool designed to foster equitable teamwork. We employed a four-level linear model where responses nested in the crossing of students and items, which in turn are nested in teams within courses. Peer ratings measured on a 9-Likert scale serve as the dependent variable, and the main factors include gender, cumulative GPA, personality traits (Extraversion and Task Control), and group preference. Our analysis revealed statistically significant influences of these factors. Students with higher cumulative GPAs earned greater average peer rating values compared to their peers. Students who rated themselves higher in extraversion and preference for task control received lower ratings, on average. In addition, on average, students who indicated a preference for working in groups received lower peer assessment scores than those who preferred individual work or collaboration.

Our results emphasize the importance of considering the effects of gender, group preference, and personality traits in the design of peer assessment for evaluating team-based learning outcomes. Moreover, when conducting research aimed at examining team-based learning outcomes and related factors (e.g., designed interventions), we should also account for the effects of personal factors that are not usually considered, such as gender, academic performance, personality traits, and group preferences.

Citation
Format

Li, X., & Fowler, R., & Mills, M. (2024, June), Investigating the Impact of College Students’ Personal Characteristics on Peer Assessment: A Multilevel Linear Modeling Approach Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/47697

TY  - CPAPER
AB  - Work in Progress: Investigating the Impact of College Students’ Personal Characteristics on Peer Assessment: A Multilevel Linear Modeling Approach

Keywords: Peer Assessment; Multilevel Linear Modeling; Personal Characteristics

Abstract: Teamwork is an essential skill expected of college students, and it has been frequently integrated into courses by instructors, making the assessment of these activities an important component of university curricula. Peer assessment serves as a crucial assessment method, offering valuable feedback and enhancing student learning outcomes. It is commonly used in college courses that employ team-based learning, and there is a growing focus on the impact of peer assessment design on student learning outcomes. Previous research highlights correlations between peer ratings and factors related to personal characteristics, including gender, academic performance, personality traits, and group preferences; however, a significant gap remains in our understanding of how these factors specifically influence peer ratings in the context of college course teamwork. This knowledge gap hinders our ability to provide instructors, students, and researchers with evidence-based insights into the effectiveness of team-based pedagogy and into the interpretation of peer assessments. To address this gap, our study employs multilevel linear modeling to investigate the relationships between these factors and peer rating scores in the context of college course teamwork. 

We collected peer rating data from 5,322 college students at a Midwest research university spanning the period from 2019 to 2023. The data were collected using Tandem, a digital instructional tool designed to foster equitable teamwork. We employed a four-level linear model where responses nested in the crossing of students and items, which in turn are nested in teams within courses. Peer ratings measured on a 9-Likert scale serve as the dependent variable, and the main factors include gender, cumulative GPA, personality traits (Extraversion and Task Control), and group preference. Our analysis revealed statistically significant influences of these factors. Students with higher cumulative GPAs earned greater average peer rating values compared to their peers. Students who rated themselves higher in extraversion and preference for task control received lower ratings, on average. In addition, on average, students who indicated a preference for working in groups received lower peer assessment scores than those who preferred individual work or collaboration.

Our results emphasize the importance of considering the effects of gender, group preference, and personality traits in the design of peer assessment for evaluating team-based learning outcomes. Moreover, when conducting research aimed at examining team-based learning outcomes and related factors (e.g., designed interventions), we should also account for the effects of personal factors that are not usually considered, such as gender, academic performance, personality traits, and group preferences.
AU  - Xiaping Li
AU  - Robin Fowler
AU  - Mark Mills
CY  - Portland, Oregon
DA  - 2024/06/23
PB  - ASEE Conferences
TI  - Investigating the Impact of College Students’ Personal Characteristics on Peer Assessment: A Multilevel Linear Modeling Approach
UR  - https://peer.asee.org/47697
ER  -