To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development

Lori C. Bland; Jill K. Nelson; Margret Hjalmarson; Anastasia P. Samaras

Download Paper | Permalink

Conference: 2018 ASEE Annual Conference & Exposition
Location: Salt Lake City, Utah
Publication Date: June 23, 2018
Start Date: June 23, 2018
End Date: July 27, 2018
Conference Session: Career Decisions and Faculty Development
Tagged Division: Educational Research and Methods
Page Count: 16
DOI: 10.18260/1-2--31145
Permanent URL: https://strategy.asee.org/31145
Download Count: 528

Request a correction

Paper Authors

biography

Lori C. Bland George Mason University orcid.org/0000-0003-4411-634X

visit author page

Lori C. Bland, Ph.D. teaches courses in educational assessment, program evaluation, and data-driven decision-making. Bland received her Ph.D. in Educational Psychology from the University of Virginia. Her current work focuses on evaluating programs in higher education, STEM education, and gifted education, assessing learning and professional outcomes in formal and informal learning environments in higher education and the workforce; with a focus on project- and problem-based learning.

visit author page

biography

Jill K. Nelson George Mason University

visit author page

Jill Nelson is an associate professor in the Department of Electrical and Computer Engineering at George Mason University. She earned a BS in Electrical Engineering and a BA in Economics from Rice University in 1998. She attended the University of Illinois at Urbana-Champaign for graduate study, earning an MS and PhD in Electrical Engineering in 2001 and 2005, respectively. Dr. Nelson's research focus is in statistical signal processing, specifically detection and estimation for applications in target tracking and physical layer communications. Her work on target detection and tracking is funded by the Office of Naval Research. Dr. Nelson is a 2010 recipient of the NSF CAREER Award. She is a member of Phi Beta Kappa, Tau Beta Pi, Eta Kappa Nu, and the IEEE Signal Processing, Communications, and Education Societies.

visit author page

biography

Margret Hjalmarson George Mason University

visit author page

Margret Hjalmarson is a Professor in the Graduate School of Education at George Mason University. Her research interests include engineering education, mathematics education, faculty development and mathematics teacher leadership.

visit author page

biography

Anastasia P. Samaras George Mason University

visit author page

ANASTASIA P. SAMARAS is Professor of Education in the College of Education and Human Development at George Mason University, USA. She is an educational researcher and pedagogical scholar with signature work in self-study research methodology including co-editor of Polyvocal Professional Learning through Self-Study Research (2015) and author of Self-Study Teacher Research (2011) and lead editor of Learning Communities In Practice (2008). She is recipient of the Dissertation Research Award, University of Virginia, the Outstanding Scholar Award, University of Maryland, a Fulbright Scholar, and a Visiting Self-study Scholar. She served as chair of S-STEP from 2013-2015 and is a current Co-PI of two National Science Foundation (NSF) funded grants: Designing Teaching: Scaling up the SIMPLE Design Framework for Interactive Teaching Development and a research initiation grant: Student-directed differentiated learning in college-level engineering education. Her research centers on facilitating and studying her role in faculty development self-study collaboratives.

visit author page

Download Paper | Permalink

Abstract

Motivation and Background: The purpose for this study was to compare two methods for evaluating the design, implementation, and outcomes of an NSF-funded faculty development grant via conjecture mapping1,2 and a logic model3. One grant goal was to support faculty in their learning about and using evidence-based interactive teaching strategies. A second goal was to implement the project across multiple STEM programs. A third goal was to support faculty research about their own teaching. The hypothesis for the grant was that a learning community can support initiation of teaching change and faculty research about the changes. The purpose for using both methods was to determine which method best captures the dynamism in an organically evolving faculty development along a continuum of learning structure (semi-formal and informal).

Methods: Two methods were used to examine the design, implementation, and outcomes of the grant: (a) conjecture mapping1,2, a tool in design-based research4,5; and (b) a logic model3, a tool used to develop program theory and evaluate implementation and outcomes6. The design, implementation, and outcomes were analyzed using both tools. Both tools were also adapted to account for the situated nature of learning within community7 and informally8.

Specifically, the initial design decisions were examined9,10,11 to identify the high-level conjectures for the conjecture map and the situation, external factors, and assumptions for the logic model. The materials and tools, structures for tasks and participant engagement, and discursive practices were examined for the conjecture map12,13. The logic model was used to identify inputs, activities, and strategies to support faculty learning12,13. Mediating process1,2 or outputs3 to map the evolution of project implementation were examined14. The evolution of project implementation and outcomes were also examined.

Results: Initial results revealed that the conjecture mapping revealed multiple stages of learning and enactment of interactive teaching and research about teaching that were not revealed in the development of the logic model. However, the logic model was more useful in describing the initial situation and underlying assumptions related to program development. Conjecture mapping was better able to capture processes and incremental outcomes due to the informal and organic nature of faculty learning.

Implications: Logic models have been historically used to help develop and evaluate implementation of program theory. However, they have limitations when a program evolves organically. Conjecture mapping allows for an examination of organic changes. The advantages and disadvantages of conjecture mapping and logic models to examine faculty development are situated along three continua which can help the STEM faculty development program evaluator identify which model is most useful for research and evaluation: a) a formal-informal design continue; (b) a static-dynamic implementation continua; and c) a process-outcome-impact continua.

References

1 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39, 213-223, DOI:10.1207/s15326985ep3904_3

2 Sandoval, W. A. (2014) Conjecture mapping: An approach to systematic educational design Research. Journal of the Learning Sciences, 23,1, 18-36. doi:10.1080/10508406.2013.778204

3 Taylor-Powell, E., & Henert, E. Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin-Extension. Retrieved from https://fyi.uwex.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf

4 Kelly, A. E. (2014). Design-based research in engineering education: Current state and next steps. In A. Johri & B. M. Olds (Eds.), Cambridge handbook of engineering education research (pp. 497–418). New York: Cambridge University Press.

5 Kelly, A. E., Lesh, R. A., & Baek, J. Y. (Eds.). (2008). Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching. NY: Routledge.

6 Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.

7 Johri, A., Olds, B. M., & O’Connor, K. (2014). The social nature of representational engineering knowledge. In A. Johri and B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research, (pp. 47-66). NY: Cambridge University Press.

8 Bland, L. C. (in press). Assessing Learner-driven Constructs in Informal Learning Environments: Synergies Created by the Nexus of Psychometrics, Learning Analytics, and Educational Data Mining. In: R. Lissitz & J. Hong (Eds.). MARCES Book Series: Data Analytics and Psychometrics: Informing Assessment Practices. Charlotte, NC: Information Age Publishing.

9 Hjalmarson, M. A., & Nelson, J. K. (2014). Creating small interactive teaching groups. In Proceedings of the 121st ASEE Annual Conference. Indianapolis, IN.

10 Nelson, J. K., & Hjalmarson, M. A. (2015). Faculty Development Groups for Interactive Teaching. In Proceedings of the 122nd ASEE Annual Conference. Seattle, WA.

11 Samaras, A. P. (2011). Self-study teacher research: Improving your practice through collaborative inquiry. Thousand Oaks, CA: Sage. [Translated into Korean] http://www.sagepub.com/booksProdDesc.nav?prodId=Book233400&#tabview=title

12 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

13 Hjalmarson, M., Nelson, J., Gerasimova, D., Bland, L., & Samaras, A. (2016, April). Faculty Professional Development through Teaching Development Groups: Principles in Action. Paper presented at the Annual Meeting of American Educational Research Association, Washington, DC.

14 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

Citation
Format

Bland, L. C., & Nelson, J. K., & Hjalmarson, M., & Samaras, A. P. (2018, June), To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--31145

TY - CPAPER
AB - Motivation and Background: The purpose for this study was to compare two methods for evaluating the design, implementation, and outcomes of an NSF-funded faculty development grant via conjecture mapping1,2 and a logic model3. One grant goal was to support faculty in their learning about and using evidence-based interactive teaching strategies. A second goal was to implement the project across multiple STEM programs. A third goal was to support faculty research about their own teaching. The hypothesis for the grant was that a learning community can support initiation of teaching change and faculty research about the changes. The purpose for using both methods was to determine which method best captures the dynamism in an organically evolving faculty development along a continuum of learning structure (semi-formal and informal).

Methods: Two methods were used to examine the design, implementation, and outcomes of the grant: (a) conjecture mapping1,2, a tool in design-based research4,5; and (b) a logic model3, a tool used to develop program theory and evaluate implementation and outcomes6. The design, implementation, and outcomes were analyzed using both tools. Both tools were also adapted to account for the situated nature of learning within community7 and informally8.

Specifically, the initial design decisions were examined9,10,11 to identify the high-level conjectures for the conjecture map and the situation, external factors, and assumptions for the logic model. The materials and tools, structures for tasks and participant engagement, and discursive practices were examined for the conjecture map12,13. The logic model was used to identify inputs, activities, and strategies to support faculty learning12,13. Mediating process1,2 or outputs3 to map the evolution of project implementation were examined14. The evolution of project implementation and outcomes were also examined.

Results: Initial results revealed that the conjecture mapping revealed multiple stages of learning and enactment of interactive teaching and research about teaching that were not revealed in the development of the logic model. However, the logic model was more useful in describing the initial situation and underlying assumptions related to program development. Conjecture mapping was better able to capture processes and incremental outcomes due to the informal and organic nature of faculty learning.

Implications: Logic models have been historically used to help develop and evaluate implementation of program theory. However, they have limitations when a program evolves organically. Conjecture mapping allows for an examination of organic changes. The advantages and disadvantages of conjecture mapping and logic models to examine faculty development are situated along three continua which can help the STEM faculty development program evaluator identify which model is most useful for research and evaluation: a) a formal-informal design continue; (b) a static-dynamic implementation continua; and c) a process-outcome-impact continua.

References

1 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39, 213-223, DOI:10.1207/s15326985ep3904_3

2 Sandoval, W. A. (2014) Conjecture mapping: An approach to systematic educational design Research. Journal of the Learning Sciences, 23,1, 18-36. doi:10.1080/10508406.2013.778204

3 Taylor-Powell, E., &amp; Henert, E. Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin-Extension. Retrieved from https://fyi.uwex.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf

4 Kelly, A. E. (2014). Design-based research in engineering education: Current state and next steps. In A. Johri &amp; B. M. Olds (Eds.), Cambridge handbook of engineering education research (pp. 497–418). New York: Cambridge University Press.

5 Kelly, A. E., Lesh, R. A., &amp; Baek, J. Y. (Eds.). (2008). Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching. NY: Routledge.

6 Funnell, S. C., &amp; Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.

7 Johri, A., Olds, B. M., &amp; O’Connor, K. (2014). The social nature of representational engineering knowledge. In A. Johri and B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research, (pp. 47-66). NY: Cambridge University Press.

8 Bland, L. C. (in press). Assessing Learner-driven Constructs in Informal Learning Environments: Synergies Created by the Nexus of Psychometrics, Learning Analytics, and Educational Data Mining. In: R. Lissitz &amp; J. Hong (Eds.). MARCES Book Series: Data Analytics and Psychometrics: Informing Assessment Practices. Charlotte, NC: Information Age Publishing.

9 Hjalmarson, M. A., &amp; Nelson, J. K. (2014). Creating small interactive teaching groups. In Proceedings of the 121st ASEE Annual Conference. Indianapolis, IN.

10 Nelson, J. K., &amp; Hjalmarson, M. A. (2015). Faculty Development Groups for Interactive Teaching. In Proceedings of the 122nd ASEE Annual Conference. Seattle, WA.

11 Samaras, A. P. (2011). Self-study teacher research: Improving your practice through
collaborative inquiry. Thousand Oaks, CA: Sage. [Translated into Korean]
http://www.sagepub.com/booksProdDesc.nav?prodId=Book233400&amp;#tabview=title

12 Nelson, J.K., Hjalmarson, M.A., Bland, L., &amp; Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

13 Hjalmarson, M., Nelson, J., Gerasimova, D., Bland, L., &amp; Samaras, A. (2016, April). Faculty Professional Development through Teaching Development Groups: Principles in Action. Paper presented at the Annual Meeting of American Educational Research Association, Washington, DC.

14 Nelson, J.K., Hjalmarson, M.A., Bland, L., &amp; Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

AU - Lori C. Bland
AU - Jill K. Nelson
AU - Margret Hjalmarson
AU - Anastasia P. Samaras
CY - Salt Lake City, Utah
DA - 2018/06/23
PB - ASEE Conferences
TI - To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development
UR - https://strategy.asee.org/31145
DO - 10.18260/1-2--31145
ER -