Asee peer logo

Design and Development: NSF Engineering Research Centers Unite: Developing and Testing a Suite of Instruments to Enhance Overall Education Program Evaluation

Download Paper |

Conference

2021 ASEE Virtual Annual Conference Content Access

Location

Virtual Conference

Publication Date

July 26, 2021

Start Date

July 26, 2021

End Date

July 19, 2022

Conference Session

NSF Grantees Poster Session

Tagged Topic

NSF Grantees Poster Session

Page Count

14

DOI

10.18260/1-2--36906

Permanent URL

https://strategy.asee.org/36906

Download Count

418

Request a correction

Paper Authors

biography

Zhen Zhao Arizona State University, Polytechnic campus

visit author page

Zhen Zhao is a Ph.D. student in The Polytechnic School at Arizona State University. His research interests include engineering student mentorship and leadership development, engineering research center education and diversity impact evaluation, and engineering graduate student attrition. Zhen earned a B.S. in Computer Science and an M.S. in Software Engineering, both from Xi'an Jiaotong University in China. He also received an M.S.E in Industrial Engineering from Arizona State University. Zhen also has over five years of collegiate teaching experience. Zhen is passionate and dedicated to better preparing the future engineering workforce.

visit author page

biography

Adam R. Carberry Arizona State University Orcid 16x16 orcid.org/0000-0003-0041-7060

visit author page

Dr. Adam Carberry is an associate professor at Arizona State University in the Fulton Schools of Engineering, The Polytechnic School. He earned a B.S. in Materials Science Engineering from Alfred University, and received his M.S. and Ph.D., both from Tufts University, in Chemistry and Engineering Education respectively. His research investigates the development of new classroom innovations, assessment techniques, and identifying new ways to empirically understand how engineering students and educators learn. He currently serves as the Graduate Program Chair for the Engineering Education Systems and Design Ph.D. program. He is also the immediate past chair of the Research in Engineering Education Network (REEN) and an associate editor for the Journal of Engineering Education (JEE). Prior to joining ASU he was a graduate student research assistant at the Tufts’ Center for Engineering Education and Outreach.

visit author page

biography

Jean S. Larson Arizona State University Orcid 16x16 orcid.org/0000-0003-4898-2149

visit author page

Jean Larson, Ph.D., is the Educational Director for the NSF-funded Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG), and Assistant Research Professor in both the School of Sustainable Engineering and the Built Environment and the Division of Educational Leadership and Innovation at Arizona State University. She has a Ph.D. in Educational Technology, postgraduate training in Computer Systems Engineering, and many years of experience teaching and developing curriculum in various learning environments. She has taught technology integration and teacher training to undergraduate and graduate students at Arizona State University, students at the K-12 level locally and abroad, and various workshops and modules in business and industry. Dr. Larson is experienced in the application of instructional design, delivery, evaluation, and specializes in eLearning technologies for training and development. Her research focuses on the efficient and effective transfer of knowledge and learning techniques, innovative and interdisciplinary collaboration, and strengthening the bridge between K-12 learning and higher education in terms of engineering content.

visit author page

biography

Michelle Jordan Arizona State University

visit author page

Michelle Jordan is as associate professor in the Mary Lou Fulton Teachers College at Arizona State University. She also serves as the Education Director for the QESST Engineering Research Center. Michelle’s program of research focuses on social interactions in collaborative learning contexts. She is particularly interested in how students navigate communication challenges as they negotiate complex engineering design projects. Her scholarship is grounded in notions of learning as a social process, influenced by complexity theories, sociocultural theories, sociolinguistics, and the learning sciences.

visit author page

biography

Wilhelmina C. Savenye Arizona State University

visit author page

Dr. Wilhelmina "Willi" C. Savenye is a Professor Emeritus of Learning, Design and Technologies / Educational Technology at Arizona State University. She is a former Education Director, and currently serves as Senior Education Advisor, for the NSF Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG). She previously taught at the University of Texas at Austin and San Diego State University. She earned her M.Ed. and Ph.D. in Educational Technology from ASU, and B.A/ in Anthropology from the University of Washington. Dr. Savenye focuses on instructional design and evaluation of technology-based and online learning systems, employing both quantitative and qualitative research methodologies. She has published over 70 articles and book chapters; made over 140 conference presentations and workshops in the U.S., Europe and Asia; been awarded numerous grants, and has produced many digital learning programs. She is Editor Emeritus of the Journal of Applied Instructional Design. She has served on the editorial boards of journals including Educational Technology: Research and Development and the Quarterly Review of Distance Education, and reviews for additional journals. She served on the editorial board for the Encyclopedia of Educational Technology and has held elected leadership positions.
Dr. Savenye’s instructional design and evaluation work has been conducted in such diverse settings as engineering education, school districts, museums, botanical gardens, zoos, universities, corporations, and Army tank maintenance training.

visit author page

author page

Kristi L. Eustice Arizona State University

biography

Allison Godwin Purdue University at West Lafayette Orcid 16x16 orcid.org/0000-0002-0741-3356

visit author page

Allison Godwin, Ph.D. is an Associate Professor of Engineering Education and Chemical Engineering at Purdue University. Her research focuses what factors influence diverse students to choose engineering and stay in engineering through their careers and how different experiences within the practice and culture of engineering foster or hinder belongingness and identity development. Dr. Godwin graduated from Clemson University with a B.S. in Chemical Engineering and Ph.D. in Engineering and Science Education. Her research earned her a National Science Foundation CAREER Award focused on characterizing latent diversity, which includes diverse attitudes, mindsets, and approaches to learning, to understand engineering students’ identity development. She has won several awards for her research including the 2016 American Society of Engineering Education Educational Research and Methods Division Best Paper Award and the 2018 Benjamin J. Dasher Best Paper Award for the IEEE Frontiers in Education Conference. She has also been recognized for the synergy of research and teaching as an invited participant of the 2016 National Academy of Engineering Frontiers of Engineering Education Symposium and the Purdue University 2018 recipient of School of Engineering Education Award for Excellence in Undergraduate Teaching and the 2018 College of Engineering Exceptional Early Career Teaching Award.

visit author page

biography

Gillian Roehrig University of Minnesota - Twin Cities Orcid 16x16 orcid.org/0000-0002-6943-7820

visit author page

Dr. Roehrig is a professor of STEM Education at the University of Minnesota. Her research explores issues of professional development for K-12 science teachers, with a focus on beginning teachers and implementation of integrated STEM learning environments. She has received over $30 million in federal and state grants and published over 80 peer-reviewed journal articles and book chapters. She is a former board member of the National Association of Research in Science Teaching and past president of the Association for Science Teacher Education.

visit author page

biography

Christopher Barr Rice University

visit author page

Rice University Office of Assessment and Evaluation of STEM Programs led by Dr. Barr is the Director of Assessment and Evaluation of STEM Programs at Rice University. He has been an evaluator and psychometric expert on several federally funded projects in education, natural science, and engineering. His focus is to conduct rigorous quantitative and qualitative measurement and program evaluation utilizing validated assessment tools with published psychometric properties, qualitative rubrics with reliable scoring procedures, and developing and validating assessments in-line with the recommendations of the Standards for Educational and Psychological Testing.

visit author page

biography

Kimberly Farnsworth Arizona State University Orcid 16x16 orcid.org/0000-0003-4132-0281

visit author page

Kimberly Farnsworth is Educational Technologist at the Department of Defense Education Activity (DoDEA). Previously, Kimberly served as Education Coordinator at the Center for Bio-mediated and Bio-inspired Geotechnics (CBBG) a National Science Foundation Engineering Research Center (ERC). She is currently a doctoral candidate in Instructional Systems Technology at Indiana University and has a M.Ed. from Arizona State University. Kimberly has over 25 years of experience in the fields of education and technology. Her research focus is on authentic learning environments in the sciences.

visit author page

Download Paper |

Abstract

National Science Foundation (NSF) funded Engineering Research Centers (ERC) must complement their technical research with various education and outreach opportunities to: 1) improve and promote engineering education, both within the center and to the local community; 2) encourage and include the underrepresented populations to participate in Engineering activities; and 3) advocate communication and collaboration between industry and academia. ERCs ought to perform an adequate evaluation of their educational and outreach programs to ensure that beneficial goals are met. Each ERC has complete autonomy in conducting and reporting such evaluation. Evaluation tools used by individual ERCs are quite similar, but each ERC has designed their evaluation processes in isolation, including evaluation tools such as survey instruments, interview protocols, focus group protocols, and/or observation protocols. These isolated efforts resulted in redundant resources spent and lacking outcome comparability across ERCs. Leaders from three different ERCs led and initiated a collaborative effort to address the above issue by building a suite of common evaluation instruments that all current and future ERCs can use. This leading group consists of education directors and external evaluators from all three partners ERCs and engineering education researchers, who have worked together for two years. The project intends to address the four ERC program clusters: Broadening Participation in Engineering, Centers and Networks, Engineering Education, and Engineering Workforce Development. The instruments developed will pay attention to culture of inclusion, outreach activities, mentoring experience, and sustained interest in engineering. The project will deliver best practices in education program evaluation, which will not only support existing ERCs, but will also serve as immediate tools for brand new ERCs and similar large-scale research centers. Expanding the research beyond TEEC and sharing the developed instruments with NSF as well as other ERCs will also promote and encourage continual cross-ERC collaboration and research. Further, the joint evaluation will increase the evaluation consistency across all ERC education programs. Embedded instrumental feedback loops will lead to continual improvement to ERC education performance and support the growth of an inclusive and innovative engineering workforce. Four major deliveries are planned. First, develop a common quantitative assessment instrument, named Multi-ERC Instrument Inventory (MERCII). Second, develop a set of qualitative instruments to complement MERCII. Third, create a web-based evaluation platform for MERCII. Fourth, update the NSF ERC education program evaluation best practice manual. These deliveries together will become part of and supplemented by an ERC evaluator toolbox. This project strives to significantly impact how ERCs evaluate their educational and outreach programs. Single ERC based studies lack the sample size to truly test the validity of any evaluation instruments or measures. A common suite of instruments across ERCs would provide an opportunity for a large scale assessment study. The online platform will further provide an easy-to-use tool for all ERCs to facilitate evaluation, share data, and reporting impacts.

Zhao, Z., & Carberry, A. R., & Larson, J. S., & Jordan, M., & Savenye, W. C., & Eustice, K. L., & Godwin, A., & Roehrig, G., & Barr, C., & Farnsworth, K. (2021, July), Design and Development: NSF Engineering Research Centers Unite: Developing and Testing a Suite of Instruments to Enhance Overall Education Program Evaluation Paper presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference. 10.18260/1-2--36906

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2021 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015