Validating a Measure of Problem-Framing Ability to Support Evidence-Based Teaching Practice
- Conference
- Location
-
Tampa, Florida
- Publication Date
-
June 15, 2019
- Start Date
-
June 15, 2019
- End Date
-
June 19, 2019
- Conference Session
- Tagged Division
-
Educational Research and Methods
- Page Count
-
18
- DOI
-
10.18260/1-2--33528
- Permanent URL
-
https://peer.asee.org/33528
- Download Count
-
502
Paper Authors
Lindsey D White University of New Mexico
Lindsey is a PhD student in the Organization, Information, & Learning Science program at the University of New Mexico. Her research interests include instructional design, technology, and adult learning as they relate to clinical educators and clinical education in healthcare. Lindsey, a Nebraska native, completed her Bachelor of Science in Education at the University of Nebraska-Omaha. She earned her Master of Science in Exercise Science from the University of Georgia. Prior to beginning her PhD, she worked for almost 7 years at Stanford University as a Certified Athletic Trainer.
Vanessa Svihla
University of New Mexico
orcid.org/0000-0003-4342-6178
Dr. Vanessa Svihla is a learning scientist and associate professor at the University of New Mexico in the Organization, Information & Learning Sciences program and in the Chemical & Biological Engineering Department. She served as Co-PI on an NSF RET Grant and a USDA NIFA grant, and is currently co-PI on three NSF-funded projects in engineering and computer science education, including a Revolutionizing Engineering Departments project. She was selected as a National Academy of Education / Spencer Postdoctoral Fellow and a 2018 NSF CAREER awardee in engineering education research. Dr. Svihla studies learning in authentic, real world conditions; this includes a two-strand research program focused on (1) authentic assessment, often aided by interactive technology, and (2) design learning, in which she studies engineers designing devices, scientists designing investigations, teachers designing learning experiences and students designing to learn.
Yan Chen
University of New Mexico
orcid.org/0000-0002-9479-7377
Yan Chen is a Postdoctoral Fellow in the Departments of Chemical & Biological Engineering at the University of New Mexico. Her research interests focus on computer supported collaborative learning, learning sciences, online learning and teaching, and educational equity for multicultural/multiethnic education.
Todd Hynson University of New Mexico
With 16 years of experience working in the registrar/student services environment, Todd Hynson, who initially began in front-line customer service, now serves as the registrar for the University of New Mexico Health Sciences Center. He currently is pursing a PhD in the OILS program at the University of New Mexico.
Ian A. Drackert University of New Mexico
I am currently a Sr. Academic Advisor for Liberal Arts and Integrative Studies at the University of New Mexico. I am also an OILS graduate student working with a team designing an instructional training program to help increase the consistency of Design Skills Test coders for better data analysis as it pertains to FACETS research.
Jordan Orion James University of New Mexico
Jordan O. James is a Native American Ph.D. student in the Organization, Information, and Learning Sciences (OILS) program as well as a lecturer at the University of New Mexico’s School of Architecture and Planning in the Community & Regional Planning program. He has served as a graduate research assistant on an NSF-funded project, Revolutionizing Engineering Departments, and has been recognized as a Graduate Studies student spotlight recipient and teaching scholar. Jordan studies learning in authentic, real-world conditions utilizing Design-Based Research methodologies to investigate design learning and social engineering, in which he studies urban planners who design real-world interventions for communities and students who use design to learn. A member of the Grand Portage Band of the Lake Superior Chippewa Jordan obtained both his Masters of Community & Regional Planning and Bachelor of Media Arts from the University of New Mexico in Albuquerque where he lives with his wife and three daughters.
Claire Yvonne Saul
Austin C. Megli University of New Mexico
Austin C. Megli is an Organization, Information, and Learning Sciences PhD Student at the University of New Mexico. He also holds a Juris Doctor and Masters of Business Administration from the University of New Mexico. His research interests are in distance learning and interaction analysis.
Abstract
Purpose. In this evidence-based teaching practice paper, we report on our efforts to design a means to assess problem framing ability. More faculty than ever are incorporating design into core engineering courses; in doing so, they may be concerned that adding design means sacrificing content. In the absence of a measure of problem framing ability, instructors tend to rely on existing assessments that typically focus on conceptual knowledge. We report on the development, implementation, and validation of the Design Skills Test (DST), an assessment of design problem framing ability.
Methodology. The DST includes an authentic design scenario and a coding scheme to characterize 1) factual and conceptual information used to frame the problem in terms of needs/constraints; 2) design practices used (e.g., generating ideas, considering multiple stakeholders, remaining tentative); and 3) stylistic choices (e.g., organizing their response, depicting context in representations). We developed three DST scenarios and tested them in a chemical engineering program that began threading design challenges throughout all core courses. We collected data over a three-year period (n=580). In the first and senior years, students completed the same DST twice as a pre/post measure. In the sophomore and junior years, they additionally completed a mid-year version. Students were given 15 minutes to work on the problem during class; instructors explained that there was no single right answer and that it would take a team many months to develop a solution, but that we were interested in how they start such a problem. To make data analysis feasible, two undergraduate peer-learning facilitators analyzed each DST independently (14 PLFs contributed), following minimal training.
Results. Using a validity-as-argument approach (Linn, 1994), we argue that the DST provides valid information about design problem-framing ability, provided the information is used for course improvement purposes. Inter-rater reliability for factual/conceptual codes was 65% to 83%; for practice codes 52% to 77%; and for stylistic codes 68% to 80%).
Conclusions. Our findings indicate that the DST sheds light on students’ design problem framing ability and provides valid evidence to help faculty evaluate the impact of incorporating design challenges, as not all design challenges support students to learn how to design. Given that professional engineering design practice relies on knowing how to frame problems, it is important for students to have opportunities to develop problem framing ability.
Implications. While reliability with minimal training was lower than would be acceptable for research purposes, we argue that for instructional purposes, this represents a significant reduction of faculty time. To enhance reliability, we worked with instructional designers to develop an online, self-paced training. Future studies will explore the impact of this training on reliability and explore the extent to which DSTs are predictive of later design behaviors.
White, L. D., & Svihla, V., & Chen, Y., & Hynson, T., & Drackert, I. A., & James, J. O., & Saul, C. Y., & Megli, A. C. (2019, June), Validating a Measure of Problem-Framing Ability to Support Evidence-Based Teaching Practice Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--33528
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015