Design and Testing of a Quantitative Instrument to Evaluate Engineering Research Center Participation
- Conference
- Location
-
Baltimore , Maryland
- Publication Date
-
June 25, 2023
- Start Date
-
June 25, 2023
- End Date
-
June 28, 2023
- Conference Session
- Tagged Division
-
Educational Research and Methods Division (ERM)
- Tagged Topic
-
Diversity
- Page Count
-
13
- DOI
-
10.18260/1-2--42935
- Permanent URL
-
https://strategy.asee.org/42935
- Download Count
-
126
Paper Authors
Marcus Vinicius Melo de Lyra Arizona State University
Marcus is a second-year Ph.D. student in the Engineering Education Systems and Design (EESD) Program at Arizona State University. His research interests include teaching faculty development and early-career faculty experiences. Before joining the EESD program, Marcus earned his BS in Civil Engineering at the Federal University of Rio Grande do Norte and his MS in Civil and Environmental Engineering at the Federal University of Campina Grande, both in Brazil.
Adam R. Carberry
Arizona State University
orcid.org/0000-0003-0041-7060
Dr. Adam Carberry is an associate professor at Arizona State University in the Fulton Schools of Engineering, The Polytechnic School. He earned a B.S. in Materials Science Engineering from Alfred University, and received his M.S. and Ph.D., both from Tufts University, in Chemistry and Engineering Education respectively. His research investigates the development of new classroom innovations, assessment techniques, and identifying new ways to empirically understand how engineering students and educators learn. He currently serves as the Graduate Program Chair for the Engineering Education Systems and Design Ph.D. program. He is also the immediate past chair of the Research in Engineering Education Network (REEN) and a deputy editor for the Journal of Engineering Education (JEE). Prior to joining ASU he was a graduate research assistant at the Tufts’ Center for Engineering Education and Outreach.
Jean S. Larson
Arizona State University
orcid.org/0000-0003-4898-2149
Jean Larson, Ph.D., is the Educational Director for the NSF-funded Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG), and Associate Research Professor in both the School of Sustainable Engineering and the Built Environment and the Division of Educational Leadership and Innovation at Arizona State University. She has a Ph.D. in Educational Technology, postgraduate training in Computer Systems Engineering, and many years of experience teaching and developing curriculum in various learning environments. She has taught technology integration and teacher training to undergraduate and graduate students at Arizona State University, students at the K-12 level locally and abroad, and various workshops and modules in business and industry. Dr. Larson is experienced in the application of instructional design, delivery, evaluation, and specializes in eLearning technologies for training and development. Her research focuses on the efficient and effective transfer of knowledge and learning techniques, innovative and interdisciplinary collaboration, and strengthening the bridge between K-12 learning and higher education in terms of engineering content.
Zhen Zhao Massachusetts Institute of Technology
Zhen Zhao is a Postdoctoral Associate at the Massachusetts Institute of Technology. His research interests include engineering student mentorship and leadership development, engineering research center education and diversity impact evaluation, and engine
Allison Godwin
Purdue University at West Lafayette (COE)
orcid.org/0000-0002-0741-3356
Allison Godwin, Ph.D. is an associate professor in the Robert Frederick Smith School of Chemical and Biomolecular Engineering at Cornell University. She is also the Engineering Workforce Development Director for CISTAR, the Center for Innovative and Strategic Transformation of Alkane Resources, a National Science Foundation Engineering Research Center. Her research focuses on how identity, among other affective factors, influences diverse students to choose engineering and persist in engineering. She also studies how different experiences within the practice and culture of engineering foster or hinder belonging and identity development. Dr. Godwin graduated from Clemson University with a B.S. in Chemical Engineering and Ph.D. in Engineering and Science Education. Her research earned her a National Science Foundation CAREER Award focused on characterizing latent diversity, which includes diverse attitudes, mindsets, and approaches to learning to understand engineering students’ identity development. She has won several awards for her research including the 2021 Journal of Civil Engineering Education Best Technical Paper, the 2021 Chemical Engineering Education William H. Corcoran Award, and the 2022 American Educational Research Association Education in the Professions (Division I) 2021-2022 Outstanding Research Publication Award.
Wilhelmina C. Savenye Arizona State University
Dr. Wilhelmina "Willi" C. Savenye is a Professor Emeritus of Learning, Design and Technologies / Educational Technology at Arizona State University. She is a former Education Director, and currently serves as Senior Education Advisor, for the NSF Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG). She previously taught at the University of Texas at Austin and San Diego State University. She earned her M.Ed. and Ph.D. in Educational Technology from ASU, and B.A/ in Anthropology from the University of Washington. Dr. Savenye focuses on instructional design and evaluation of technology-based and online learning systems, employing both quantitative and qualitative research methodologies. She has published over 70 articles and book chapters; made over 140 conference presentations and workshops in the U.S., Europe and Asia; been awarded numerous grants, and has produced many digital learning programs. She is Editor Emeritus of the Journal of Applied Instructional Design. She has served on the editorial boards of journals including Educational Technology: Research and Development and the Quarterly Review of Distance Education, and reviews for additional journals. She served on the editorial board for the Encyclopedia of Educational Technology and has held elected leadership positions.
Dr. Savenye’s instructional design and evaluation work has been conducted in such diverse settings as engineering education, school districts, museums, botanical gardens, zoos, universities, corporations, and Army tank maintenance training.
Christopher Barr
Rice University Office of Assessment and Evaluation of STEM Programs led by Dr. Barr is the Director of Assessment and Evaluation of STEM Programs at Rice University. He has been an evaluator and psychometric expert on several federally funded projects in
Abstract
This research paper reports the in-progress validation of a quantitative instrument designed to assess the perceived impact of participating in a National Science Foundation (NSF)-funded Engineering Research Center (ERC). A multi-institutional consortium composed of ERC education directors, researchers, and evaluators from six NSF-funded ERCs designed easily accessible evaluation instruments and tools that specifically help measure anticipated outcomes for ERC participants for all ERCs. The total effort underway by the consortium includes creating a suite of qualitative and quantitative instruments, an evaluator toolkit, and a user-friendly online platform to host the inventory materials. This paper focuses on the quantitative instrument created to evaluate the experiences of those who engage with a center. It consists of Likert-type questions assessing the impact of the ERC on participants' self-reported: 1) understanding of the ERC, 2) research and communication skills, 3) climate of inclusion, 4) mentorship experiences, and 5) program satisfaction. The instrument also included additional demographic questions and questions to capture STEM-related future plans. The instrument was designed using multiple rounds of design iterations and pilot tests. Separate surveys used by individual ERCs were compiled and categorized to ensure all requirements from the National Science Foundation were met. The web-based survey was administered to six ERCs during the Summer of 2021, Fall of 2021, and Spring of 2022. A total of 549 responses were collected; 535 were used following data cleaning procedures. Sample sizes for each component of the survey varied because some ERCs chose to only use some parts of the new instrument. Exploratory Factor Analyses (EFA) were performed to identify latent factors and items that needed further revision. The following factors emerged from our analyses: 1) ERC general understanding; 2) development of research skills; 3) development of professional skills; 4) experience in the ERC; 5) feelings toward the ERC; 6) Beliefs about the ERC, 7) mentors performance; and 8) mentorship experience. The results provide preliminary evidence that the survey can be used across ERCs. This effort is the first that has been undertaken to develop a shared ERC instrument. The data collected was used to continue in-progress validation. The collaborative nature of this effort can provide ways for ERCs to benchmark impacts of their efforts and share effective practices across ERCs and other similarly structured STEM centers going forward.
Melo de Lyra, M. V., & Carberry, A. R., & Larson, J. S., & Zhao, Z., & Godwin, A., & Savenye, W. C., & Barr, C. (2023, June), Design and Testing of a Quantitative Instrument to Evaluate Engineering Research Center Participation Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--42935
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2023 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015