Asee peer logo

A Statistical Study of Concept Mapping Metrics

Download Paper |

Conference

2013 ASEE Annual Conference & Exposition

Location

Atlanta, Georgia

Publication Date

June 23, 2013

Start Date

June 23, 2013

End Date

June 26, 2013

ISSN

2153-5965

Conference Session

ERM Potpourri

Tagged Division

Educational Research and Methods

Page Count

14

Page Numbers

23.105.1 - 23.105.14

DOI

10.18260/1-2--19119

Permanent URL

https://peer.asee.org/19119

Download Count

538

Paper Authors

biography

Kathryn W. Jablokow Pennsylvania State University

visit author page

Dr. Kathryn Jablokow is an Associate Professor of Mechanical Engineering and Engineering Design at Penn State University. A graduate of Ohio State University (Ph.D., Electrical Engineering), Dr. Jablokow's teaching and research interests include problem solving, invention, and creativity in science and engineering, as well as robotics and computational dynamics. In addition to her membership in ASEE, she is a Senior Member of IEEE and a Fellow of ASME. Dr. Jablokow is the architect of a unique 4-course module focused on creativity and problem solving leadership and is currently developing a new methodology for cognition-based design. She is one of three instructors for Penn State’s Massive Open Online Course (MOOC) on Creativity, Innovation, and Change, and she is the founding director of the Problem Solving Research Group, whose 50+ collaborating members include faculty and students from several universities, as well as industrial representatives, military leaders, and corporate consultants.

visit author page

biography

Joanna F. DeFranco Pennsylvania State University, Great Valley

visit author page

Joanna F. DeFranco is Assistant Professor of Software Engineering and a member of the Graduate Faculty at The Pennsylvania State University. Prior to joining Penn State, she held faculty positions at Cabrini College and the New Jersey Institute of Technology. She also held a number of positions in industry and government including an Electronics Engineer for the Naval Air Development Center in Warminster, PA and a Software Engineer at Motorola in Horsham, PA.
Dr. DeFranco received her B.S. in Electrical Engineering from Penn State University, M.S. in Computer Engineering from Villanova University, and Ph.D. in Computer and Information Science from the New Jersey Institute of Technology. She is a member of ASEE and has had numerous publications in journals and conference proceedings. She is also on the curriculum advisory board for a local technical high school.

visit author page

author page

Sally Sue Richmond Penn State Great Valley School of Graduate Professional Studies

Download Paper |

Abstract

A Statistical Study of Concept Mapping MetricsBackground: The use of concept maps in engineering education research is growing, with applications inthe assessment of knowledge mastery and integration within courses, programs, and across multipledisciplines. Concept maps are also being used in the early stages of engineering problem solving anddesign to help teams gain a shared understanding of group tasks. Within these applications, a variety ofmetrics have been developed for assessing concept maps, including both “traditional” and “holistic”approaches to measuring the accuracy, breadth, and depth of students’ understanding. In general,traditional metrics rely on counting elements of a concept map (e.g., concepts, links, hierarchies) or thecomputation of map descriptors (e.g., map density, map complexity) as functions of these elements. Dueto their dependence on relatively clear-cut features, traditional metrics are generally considered to bequite objective (i.e., different evaluators are likely to derive the same results); nevertheless, holisticmetrics that focus on a more subjective “quality of understanding” represented in a concept map (ratherthan the “quantity” of its elements) have also emerged. These holistic scoring methods include structuralcomplexity approaches that assess the dominant structural patterns of concept maps (e.g., hub/spoke, tree,network), as well as integrated rubrics based on the organization, comprehensiveness, and correctness ofmap content.Motivation: A wide variety of concept mapping metrics exists, but very few studies have examined therelationships between them in detail. To address this need, we performed an exploratory statisticalanalysis to determine if and how the predominant traditional and holistic concept mapping metrics arecorrelated, with the future aim of identifying sets of metrics that are most effective for the evaluation ofstudents’ understanding in specific situations.Research Methods: Our samples included 73 undergraduate engineering students (first-year engineeringdesign) and 52 graduate engineering students (master’s-level systems engineering) at a large, publicuniversity. Each student completed at least one technical concept map of a course-related topic; thesemaps were assessed by two trained evaluators using eleven traditional and seven holistic mappingmetrics. Traditional metrics included number of concepts, number of links, map density, map complexity,and number of hierarchies, among others; holistic metrics included comprehensiveness, organization, andcompleteness, as well as dominant structural patterns (e.g., tree, network). Statistical analyses wereperformed to determine if and how these metrics were correlated, both among and between the traditionaland holistic metric subsets.Results: Our analyses revealed a range of statistically significant correlations among and between thetraditional and holistic map metrics. Some of the strongest correlations were found betweencomprehensiveness (a holistic metric) and numbers of concepts and links (traditional metrics),respectively, as well as between similarity and matching links (both traditional metrics), using an expertmap for reference. Equally interesting was the lack of significant correlations between metrics that appearto be related from a theoretical perspective, including (for example) number of hierarchies (traditional)and organization (holistic).Conclusions and Significance: While the small sample sizes used in this study limit our conclusions, theresults are encouraging and suggest further investigation. Understanding the relationships betweendifferent concept mapping metrics marks the first step in helping educators make informed choices aboutwhich metrics they use to assess student outcomes effectively.

Jablokow, K. W., & DeFranco, J. F., & Richmond, S. S. (2013, June), A Statistical Study of Concept Mapping Metrics Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--19119

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015