Work in Progress: Using Machine Learning to Map Student Narratives of Understanding and Promoting Linguistic Justice

Harpreet Auby; Milo Koretsky

Download Paper | Permalink

Conference: 2023 ASEE Annual Conference & Exposition
Location: Baltimore , Maryland
Publication Date: June 25, 2023
Start Date: June 25, 2023
End Date: June 28, 2023
Conference Session: Work-in-Progress Session: Emergent Methods for Engineering Education Research
Tagged Division: Educational Research and Methods Division (ERM)
Tagged Topic: Diversity
Page Count: 9
DOI: 10.18260/1-2--44389
Permanent URL: https://strategy.asee.org/44389
Download Count: 102

Request a correction

Paper Authors

biography

Harpreet Auby Tufts University

visit author page

Harpreet is a graduate student in Chemical Engineering and STEM Education. He works with Dr. Milo Koretsky and helps study the role of learning assistants in the classroom as well as machine learning applications within educational research and evaluation. He is also involved in projects studying the uptake of the Concept Warehouse. His research interests include chemical engineering education, learning sciences, and social justice.

visit author page

biography

Milo Koretsky Tufts University

visit author page

Milo Koretsky is the McDonnell Family Bridge Professor in the Department of Chemical and Biological Engineering and in the Department of Education at Tufts University. He received his B.S. and M.S. degrees from UC San Diego and his Ph.D. from UC Berkeley,

visit author page

Download Paper | Permalink

Abstract

This work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias.

In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019).

In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives.

References

Authors. (2022, June). [Title omitted for blind review].

Burstein, J., Horbach, A., Kochmar, K., Laarmann-Quante, R., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. & Zesch,T.. (2021) Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics

Burstein,J., Kochmar, K., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. & Zesch,T.. (2020). Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics.

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Chang, K.-W., Prabhakaran, V., and Ordonez, V. (2019). Bias and Fairness in Natural Language Processing [Paper presentation]. In Proceedings of the 2019 Conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP): Tutorial abstracts. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (China), Hong Kong, China. Association for Computational Linguistics.

Koretsky, M. D., Falconer, J. L., Brooks, B. J., Gilbuena, D. M., Silverstein, D. L., Smith, C., & Miletic, M. (2014). The AIChE Concept Warehouse: A web-based tool to promote concept-based instruction. Advances in Engineering Education, 4(1), 7:1-27.

Mayfield, E., Madaio, M., Prabhumoye, S., Gerritsen, D., McLaughlin, B., Dixon-Román, E., and Black, A. W. (2019, August). Equity beyond bias in language technologies for education [Paper presentation]. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (Italy), Florence, Italy (pp. 444–460). Association for Computational Linguistics.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.

Citation
Format

Auby, H., & Koretsky, M. (2023, June), Work in Progress: Using Machine Learning to Map Student Narratives of Understanding and Promoting Linguistic Justice Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--44389

TY - CPAPER
AB - This work-in-progress paper expands on a collaboration between engineering education researchers and machine learning researchers to automate the analysis of written responses to conceptually challenging questions in statics and dynamics courses (Authors, 2022). Using the Concept Warehouse (Koretsky et al., 2014), written justifications of ConcepTests (CTs) were gathered from statics and dynamics courses in a diverse set of two- and four-year institutions. Written justifications for CTs have been used to support active learning pedagogies which makes them important to investigate how students put together their problem-solving narratives of understanding. However, despite the large benefit that analysis of student written responses may provide to instructors and researchers, manual review of responses is cumbersome, limits analysis, and can be prone to human bias.

In efforts to improve the analysis of student written responses, machine learning has been used in various educational contexts to analyze short and long texts (Burstein et al., 2020; Burstein et al., 2021). Natural Language Processing (NLP) uses transformer-based machine learning models (Brown et al., 2020; Raffel et al., 2019) which can be used through fine-tuning or in-context learning methods. NLP can be used to train algorithms that can automate the coding of written responses. Only a few studies for educational applications have leveraged transformer-based machine learning models further prompting an investigation into its use in STEM education. However, work in NLP has been criticized for heightening the possibility to perpetuate and even amplify harmful stereotypes and implicit biases (Chang et al., 2019; Mayfield et al., 2019).

In this study, we detail the aim to use NLP for linguistic justice. Using methods like text summary, topic modeling, and text classification, we identify key aspects of student narratives of understanding in written responses to mechanics and statics CTs. Through this process, we seek to use machine learning to identify different ways students talk about a problem and their understanding at any point in their narrative formation process. Thus, we hope to help reduce human bias in the classroom and through technology by giving instructors and researchers a diverse set of narratives that include insight into their students’ histories, identities, and understanding. These can then be used towards connecting technological knowledge to students’ everyday lives.

References

Authors. (2022, June). [Title omitted for blind review].

Burstein, J., Horbach, A., Kochmar, K., Laarmann-Quante, R., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. &amp;amp; Zesch,T.. (2021) Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics

Burstein,J., Kochmar, K., Leacock, C., Madnani, Nitin., Pilan, I., Yannakoudakis, H. &amp;amp; Zesch,T.. (2020). Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, Association for Computational Linguistics.

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Chang, K.-W., Prabhakaran, V., and Ordonez, V. (2019). Bias and Fairness in Natural Language Processing [Paper presentation]. In Proceedings of the 2019 Conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP): Tutorial abstracts. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (China), Hong Kong, China. Association for Computational Linguistics.

Koretsky, M. D., Falconer, J. L., Brooks, B. J., Gilbuena, D. M., Silverstein, D. L., Smith, C., &amp;amp; Miletic, M. (2014). The AIChE Concept Warehouse: A web-based tool to promote concept-based instruction. Advances in Engineering Education, 4(1), 7:1-27.

Mayfield, E., Madaio, M., Prabhumoye, S., Gerritsen, D., McLaughlin, B., Dixon-Román, E., and Black, A. W. (2019, August). Equity beyond bias in language technologies for education [Paper presentation]. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (Italy), Florence, Italy (pp. 444–460). Association for Computational Linguistics.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... &amp;amp; Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.

AU - Harpreet Auby
AU - Milo Koretsky
CY - Baltimore , Maryland
DA - 2023/06/25
PB - ASEE Conferences
TI - Work in Progress: Using Machine Learning to Map Student Narratives of Understanding and Promoting Linguistic Justice
UR - https://strategy.asee.org/44389
DO - 10.18260/1-2--44389
ER -