Developing a Writing Rubric to Answer Research Questions (not for Grading!)

John William Lynch; Sheryl A. Sorby; Betsy M. Aller; Teri J Murphy

Download Paper | Permalink

Conference: 2024 ASEE North Central Section Conference
Location: Kalamazoo, Michigan
Publication Date: March 22, 2024
Start Date: March 22, 2024
End Date: March 23, 2024
Tagged Topic: Diversity
Page Count: 10
DOI: 10.18260/1-2--45609
Permanent URL: https://peer.asee.org/45609
Download Count: 21

Request a correction

Paper Authors

biography

John William Lynch University of Cincinnati orcid.org/0000-0001-5580-7387

visit author page

I am an engineering and computing education PhD student at the University of Cincinnati. My interests are in retention of computer science students and improving Computer Science education for undergraduates by leveraging current technology. I am also interested in exploring the links between spatial skills and computer science, particularly how they can contribute to success in computer science. My overarching goal is to increase the retention rate for studying Computer Science at all education levels and make the field accessible for more populations.

visit author page

biography

Sheryl A. Sorby University of Cincinnati

visit author page

Dr. Sheryl Sorby is currently a Professor of STEM Education at the University of Cincinnati and was recently a Fulbright Scholar at the Dublin Institute of Technology in Dublin, Ireland. She is a professor emerita of Mechanical Engineering-Engineering Mec

visit author page

biography

Betsy M. Aller Western Michigan University

visit author page

Betsy M. Aller is Associate Professor Emerita in Engineering Design, Manufacturing, and Management Systems. At Western Michigan University, she coordinated and taught capstone design courses for 20 years, and developed courses in sustainability and project management. Her focus was on students' professional development and support for underrepresented groups in engineering.

visit author page

biography

Teri J Murphy University of Cincinnati

visit author page

Dr. Murphy is a professor in the Department of Engineering Education at the University of Cincinnati.

visit author page

Download Paper | Permalink

Abstract

Industry leaders perennially emphasize that engineering students' technical communication and writing skills should be improved. Despite various institutional efforts, which typically include separate technical communication courses or communication tasks interwoven in design projects, there has seemingly been a lack of progress in this area. As part of a larger effort aimed at identifying potential factors that contribute to poor technical communication skills, first-year engineering students in their second semester at a large R1 Midwestern university were tasked with creating a written report responding to a set of questions that asked about their team-based engineering design project completed in their first semester. Because the writing task occurred several months after their design project completion, student writing samples became a reflection on their experience from the previous semester. Further, because the writing task was not linked to a course grade, the stakes were low for students. We believe that for this reason, the writing task mimicked a more authentic writing experience similar to many writing tasks students might encounter in industry.

In order to assess the quality of student writing for this research project, it was necessary to first create a suitable rubric. Most rubrics in use by faculty for evaluating the written work of students are designed with grade assignment in mind. Designing a rubric to answer specific research questions (rather than assigning grade points) requires a different mindset. A representative sample of student responses was examined collaboratively by an interdisciplinary team in order to create a draft rubric through an iterative process. This rubric was then distributed to a team of evaluators composed of project PIs, individuals in industry, and engineering faculty to determine its suitability for use in the project. An inter-rater reliability analysis was conducted to examine levels of agreement between evaluators and the rubric was refined as needed to obtain an acceptable level of agreement across evaluators.

This paper will include details on the development process for a rubric that examines students’ technical communication and writing skills to answer research questions rather than to assign grades. Examining how to design a rubric with a narrower focus can provide further insights into evaluating the communication and writing competencies of engineering students. Through presentation of the methodology and evaluation of rubric development, it is hoped that others can apply this process to their own rubric development needs.

Citation
Format

Lynch, J. W., & Sorby, S. A., & Aller, B. M., & Murphy, T. J. (2024, March), Developing a Writing Rubric to Answer Research Questions (not for Grading!) Paper presented at 2024 ASEE North Central Section Conference, Kalamazoo, Michigan. 10.18260/1-2--45609

TY - CPAPER
AB - Industry leaders perennially emphasize that engineering students&#39; technical communication and writing skills should be improved. Despite various institutional efforts, which typically include separate technical communication courses or communication tasks interwoven in design projects, there has seemingly been a lack of progress in this area. As part of a larger effort aimed at identifying potential factors that contribute to poor technical communication skills, first-year engineering students in their second semester at a large R1 Midwestern university were tasked with creating a written report responding to a set of questions that asked about their team-based engineering design project completed in their first semester. Because the writing task occurred several months after their design project completion, student writing samples became a reflection on their experience from the previous semester. Further, because the writing task was not linked to a course grade, the stakes were low for students. We believe that for this reason, the writing task mimicked a more authentic writing experience similar to many writing tasks students might encounter in industry.

In order to assess the quality of student writing for this research project, it was necessary to first create a suitable rubric. Most rubrics in use by faculty for evaluating the written work of students are designed with grade assignment in mind. Designing a rubric to answer specific research questions (rather than assigning grade points) requires a different mindset. A representative sample of student responses was examined collaboratively by an interdisciplinary team in order to create a draft rubric through an iterative process. This rubric was then distributed to a team of evaluators composed of project PIs, individuals in industry, and engineering faculty to determine its suitability for use in the project. An inter-rater reliability analysis was conducted to examine levels of agreement between evaluators and the rubric was refined as needed to obtain an acceptable level of agreement across evaluators.

This paper will include details on the development process for a rubric that examines students’ technical communication and writing skills to answer research questions rather than to assign grades. Examining how to design a rubric with a narrower focus can provide further insights into evaluating the communication and writing competencies of engineering students. Through presentation of the methodology and evaluation of rubric development, it is hoped that others can apply this process to their own rubric development needs.
AU - John William Lynch
AU - Sheryl A. Sorby
AU - Betsy M. Aller
AU - Teri J Murphy
CY - Kalamazoo, Michigan
DA - 2024/03/22
PB - ASEE Conferences
TI - Developing a Writing Rubric to Answer Research Questions (not for Grading!)
UR - https://peer.asee.org/45609
DO - 10.18260/1-2--45609
ER -