Ohland, M. W., & Layton, R. A., & Ferguson, D. M., & Loughry, M. L., & Woehr, D. J., & Pomeranz, H. R. (2011, June), SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork  Paper presented at 2011 ASEE Annual Conference & Exposition, Vancouver, BC. 10.18260/1-2--18978

\bibitem{asee_peer_18978}
  Ohland, M. W., \& Layton, R. A., \& Ferguson, D. M., \& Loughry, M. L., \& Woehr, D. J., \& Pomeranz, H. R. (2011, June), \emph{SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork}
  Paper presented at 2011 ASEE Annual Conference \& Exposition, Vancouver, BC.
  10.18260/1-2--18978

Matthew W. Ohland, Richard A. Layton, Daniel Michael Ferguson, Misty L. Loughry, David J. Woehr, and Hal R. Pomeranz.  "SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork".  2011 ASEE Annual Conference & Exposition, Vancouver, BC, 2011, June.  ASEE Conferences, 2011.  https://strategy.asee.org/18978 Internet.  22 Jul, 2024

\bibitem{asee_peer_18978}
  Matthew W. Ohland, Richard A. Layton, Daniel Michael Ferguson, Misty L. Loughry, David J. Woehr, and Hal R. Pomeranz.
  "SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork".
  \emph{2011 ASEE Annual Conference \& Exposition, Vancouver, BC, 2011, June}.
  ASEE Conferences, 2011.
  https://strategy.asee.org/18978 Internet.  22 Jul, 2024

@INPROCEEDINGS{asee_peer_18978
author = "Matthew W. Ohland, Richard A. Layton, Daniel Michael Ferguson, Misty L. Loughry, David J. Woehr, and Hal R. Pomeranz"
title = "SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork"
booktitle = "2011 ASEE Annual Conference \& Exposition"
year = "2011"
month = "June"
address = "Vancouver, BC"
publisher = "ASEE Conferences"
note = {https://strategy.asee.org/18978}
number = {10.18260/1-2--18978}
}

TY  - CPAPER
AB  -          SMARTER Teamwork: System for Management, Assessment,        Research, Training, Education, and Remediation for TeamworkAbstractThe rapid adoption of Team-Maker and the Comprehensive Assessment of Team MemberEffectiveness (CATME), tools for team formation and peer evaluation, make it possible toextend their success to have a significant impact on the development of team skills in highereducation. The web-based systems are used by over 700 faculty at over 200 institutionsinternationally.This paper and its accompanying poster will describe strategies for broadening the scope of thosetools into a complete system for the management of teamwork in undergraduate education. TheSystem for the Management, Assessment, Research, Training, Education, and Remediation ofTeamwork (SMARTER Teamwork) has three specific goals: 1) to equip students to work inteams by providing them with training and feedback, 2) to equip faculty to manage student teamsby providing them with information and tools to facilitate best practices, and 3) to equipresearchers to understand teams by broadening the system’s capabilities to collect additionaltypes of data so that a wider range of research questions can be studied through a secureresearcher interface. The three goals of the project support each other in hierarchical fashion:research informs faculty practice, faculty determine the students’ experience, which, if wellmanaged based on research findings, equips students to work in teams. Our strategies forachieving these goals are based on a well-accepted training model that has five elements:information, demonstration, practice, feedback, and remediation.Different outcomes are expected for each group of people. For the students, both individualoutcomes, such as student learning, and team outcomes, such as the development of sharedmental models, are expected. For the faculty, individual outcomes such as faculty learning andfaculty satisfaction are expected. The outcomes for researchers will be community outcomes,that is, benefits for stakeholders outside the research team, such as generating new knowledge forteaming theory and disseminating best practices. Measuring these outcomes is the basis for theproject’s evaluation plan.Research Overview. The broad and deep scope of the proposed SMARTER Teamwork researchis summarized in Figure 1. The figure addresses the project’s three broad research goals, peopleimpacted, strategies for achieving the goals, and measureable outcomes.    Goals. The proposed work has three goals: 1), equip students to work in teams; 2), equipfaculty to manage teams; and 3), equip this research team to understand student teams. Thesegoals support each other in hierarchical fashion: research informs faculty practice, facultydetermine the students’ experience, which, if well managed based on research findings, shouldequip students to work in teams.    People. People are the groups that will use the proposed system: students, faculty, andresearchers. The hierarchy of people reflects the hierarchy of goals: the work of the researchteam supports the work of faculty, which in turn supports the work of students and their teams.         GOALS                      OUTCOMES                      PEOPLE                STRATEGIES                                                                             INFORMATION                                                                              1. Why teams are important                                                                              2. What makes a good team member                                                                              3. Overview of SMARTER Teamwork toolkit                            INDIVIDUAL OUTCOMES                              DEMONSTRATION                             1. Student learning                              1. Frame of reference examples                             2. Student satisfaction                          2. Video-based modeling of team skills                             3. Course grade                                  3. Sample vignettes with expert ratings                             4. Individual grade for team work                                                                             PRACTICE      Equip students                                                          1. Vignettes for rater calibration     to work in teams                                             Students                                                                              2. Students participate in teams                            TEAM OUTCOMES                             1. Shared mental models                                                                             FEEDBACK                             2. Collective efficacy (potency)                                                                              1. Peer evaluation ratings                             3. Cohesiveness                                                                              2. Team formation results                             4. Climate                                                                              3. Exceptional conditions from all systems                             5. Viability                                                                              4. Scenario rating accuracy                             6. Conflict                                                                              5. Performance assessment                                                                             REMEDIATION                                                                              1. Context-specific remediation                                                                              2. Redirection to relevant simulation exercise                                 determines the                                                                             INFORMATION                                environment for                               1. Best practices in team management                                                                              2. References to research base                                                                              3. Overview of SMARTER Teamwork toolkit                                                                             DEMONSTRATION                                                                              1. Sample feedback / system emails /                                                                                      tutorials / viewing student interface                                                                              2. Video-based modeling of team                                                                                      management                            INDIVIDUAL OUTCOMES                               3. Sample vignettes of team management      Equip faculty to       1. Faculty learning                                      scenarios                             2. Faculty satisfaction              Faculty      manage teams                                                           PRACTICE                             3. Student reactions                                                                              1. Vignettes for practicing team management                                                                              2. Faculty manage student teams                                                                             FEEDBACK                                                                              1. Peer evaluation ratings                                                                              2. Exceptional conditions from all systems                                                                              3. Scenario decision accuracy                                                                              4. Student team performance                                                                              5. Student reactions                                                                             REMEDIATION                                                                              1. Context-specific remediation                                          informs                             2. Redirection to relevant simulation exercise                                                                             INFORMATION                                                                              1. Using the system for research                                                                              2. Theoretical foundations / training model                                                                              3. Overview of SMARTER Teamwork toolkit                                                                             DEMONSTRATION                                                                              1. Sample research interface / tutorial                            COMMUNITY OUTCOMES                             1. Teaming theory                                2. Testing new user interfaces                                                                              3. Viewing student and faculty interface           Equip             2. Best practices                                                                 Research                             3. Faculty reactions       researchers to        4. Number of users                    team      PRACTICE      understand teams       5. Publications                                  1. Researchers design / implement studies                             6. Improved SMARTER toolkit                                                                             FEEDBACK                                                                              1. Data from research interface                                                                              2. Research results and peer review                                                                              3. Faculty reactions                                                                             REMEDIATION                                                                              1. Revising best practices                                                                              2. Revising the SMARTER toolkitFigure 1. System for the Management, Assessment, Research, Training, Education, and Remediation forTeamwork    Strategies. For each group of people — students, faculty, and researchers — we developedstrategies for achieving our goals based on a well-accepted training model that has five elements:information, demonstration, practice, feedback, and remediation. By following this model, wewill enable the people affected by the system to become proficient in teamwork (all users), managingteamwork (faculty and researchers) and creating new knowledge about teamwork (researchers).    Outcomes. Different outcomes are expected for each group of people. For the students, bothindividual outcomes, such as student learning, and team outcomes, such as shared mental models, areexpected. For the faculty, individual outcomes such as faculty learning and faculty satisfaction areexpected. The outcomes for the research team will be community outcomes, that is, benefits forstakeholders outside the research team, such as generating new knowledge for teaming theory anddisseminating best practices. Measuring these outcomes is the basis for the project’s evaluation plan.HIGHLIGHTS OF THE PAST YEAR    •   Continued growth of the CATME and Team-Maker user base;    •   System improvements, including repairs addressing usability concerns;    •   Progress toward development of the SMARTER system;    •   Development of material for training vignettes, including selection of video clips for        training using video-based modeling and video vignettes, permissions for using the video        clips has been granted;    •   Further progress on databases of literature on team formation and (separately) peer        evaluation;    •   Multiple workshops promoting the system were conducted, with more scheduled.CONTINUED GROWTH OF THE CATME AND TEAM-MAKER SYSTEMS    The growth in users of CATME and Team-Maker system has been substantial. Since October2005, 1042 instructors have registered to use the system at 296 different institutions to collectratings from 46,252 unique student users. As shown, system use has grown dramatically.                    1100                                    46,252 unique                     1000                                    students have used                      900            the system since 2005                     800                                         Number of faculty and                                                                  staff using CATME                      700                                         Team Tools                                                                                                           Sep             Number 600                                                                                   2010             of users 500                     400                     300                     200        Oct 05                     100                                                                  No. of institutions  using                                                                                           CATME Team Tools                       0                            0        6     12      18       24      30     36     42      48     54       60      66                                                Months since the rollout of the CATME  System    Figure 2. Growth in the Number of Faculty and Institutions using CATME Team Tools.The most recent growth in system use has introduced an interesting complication—as the userbase expands, it extends beyond “early adopters,” who are comfortable manipulating theinterface with little guidance. Rather, the most recent users are more likely to seek help gettingstarted, which can be quite time-consuming. Rather than divert resources to technical support, ausability study of the interface (scheduled as part of this project) has revealed opportunities tomake the interface more accessible to a broader audience.DEVELOPMENT OF MATERIAL FOR TRAINING VIGNETTESThe use of critical incident analysisThe development of training vignettes is a central strategy for this project. Our plan was to use acritical incident methodology to identify a wide variety of team behavior to include in thevignettes. Originally developed by Flanagan (1954), the critical incident technique gathersspecific, behaviorally focused descriptions of work or other activities. Bownas & Bernardin(1988) assert that “a good critical incident has four characteristics: it is specific, focuses onobservable behaviors exhibited on the job, describes the context in which the behavior occurred,and indicates the consequences of the behavior.” Thus, a good critical incident describesbehaviors, rather than traits or judgmental inferences. Normally, critical incident data arecollected by asking subject matter experts to describe particularly effective or ineffectivebehaviors from their experience, a content analysis identifies underlying dimensions ofperformance, and the critical incidents are rewritten to highlight the underlying dimensions thatwere found. In this work, a critical incident was used to develop the behaviorally anchored ratingscale for the CATME instrument as well as the sample vignette developed earlier. In this stage ofthe research, it is important to develop additional vignettes, but subject matter experts close tothis work were struggling to identify enough critical incidents to support the development of alarge pool of behaviors aligned with the dimensions of the CATME instrument.Identifying behavioral descriptions from student commentsThe research team has identified another source of behavioral descriptions that can be used forvignettes—from student comments about their teammates. Large numbers of peer evaluationshave been conducted, and the research team has access to a large volume of comments studentshave made about their teammates. These comments are a rich source of behavioral descriptions.A large volume of student comments has been processed by two undergraduate researchers todistill those comments down to essential behaviors. This task is ongoing and has been taken overby a graduate assistant at Purdue. This process requires:    • Deleting non-behavioral comments (e.g., “Nice guy!” and “nothing to say, really.”);    • Eliminating redundant phrasing to isolate a superset of unique behavioral descriptions;    • Reducing all comments to the most basic elements representing a single behavior; and    • Removing all names and pronouns.Building vignettes from individual behavioral commentsAs the comments are processed, graduate students at the University of Central Florida willconvert those behavioral elements into phrases that remain gender neutral, but are completesentences. Calibration ratings for each behavioral phrase will be determined by subject matterexperts. Where there is significant disagreement about the category to which a behavior isassigned or the rating level, behaviors will be deleted as ambiguous. In preliminary work, thesoftware developer has designed a system that will piece together a collection of behavioralphrases into a comprehensive vignette that spans all the behavioral dimensions measured byCATME.PROGRESS ON DATABASES OF LITERATUREDatabases of literature on both team formation and peer evaluation are being developed. Whilethese resources are being developed, these are for internal use only. As they near completion,they will be released and faculty who use the database will have the opportunity to proposeadditions. The team has concerns that such a literature database will quickly grow stale, as newwork emerges that must be added. The team discussed the ideal solution to this problem wouldbe an automated system that is trained to perform certain search tasks regularly to dynamicallyupdate the database. The development of such a system would be well beyond the scope of thisgrant, so the team will look for opportunities to leverage this work.PUBLICATIONS (Journal and Conference)•   Ohland, Matthew W., Richard A. Layton, Misty L Loughry, Hal R. Pomeranz, Eduardo    Salas, David J Woehr, “SMARTER Teamwork: System for Management, Assessment,    Research, Training, Education, and Remediation for Teamwork,” American Society for    Engineering Education 2010 Annual Conference.•   Ohland, Matthew W., Lisa G. Bullard, Richard M. Felder, Cynthia J. Finelli, Richard A.    Layton, Misty L. Loughry, Hal R. Pomeranz, Douglas G. Schmucker, David J. Woehr, “The    Comprehensive Assessment of Team Member Effectiveness: Development of a Behaviorally    Anchored Rating Scale for Self and Peer Evaluation,” Academy of Management 2010    Annual Meeting.•   Ohland, M.W., M.L. Loughry, D.J. Woehr, C.J. Finelli, L.G. Bullard, R.M. Felder, R.A.    Layton, H.R. Pomeranz, and D.G. Schmucker, “The Comprehensive Assessment of Team    Member Effectiveness: Development of a Behaviorally Anchored Rating Scale for Self and    Peer Evaluation.” In revision for Academy of Management: Learning & Education, March    26, 2010, Manuscript ID: AMLE-RR-2010-0056.•   Layton, R.A., M.L. Loughry, M.W. Ohland, and G.D. Ricco, “Design and Validation of a    Web-Based System for Assigning Members to Teams Using Instructor-Specified Criteria,”    Advances in Engineering Education, 2(1), Spring 2010, pp. 1-28.•   Zhang, B., and M.W. Ohland, “How to Assign Individualized Scores on a Group Project: an    Empirical Evaluation,” Applied Measurement in Education, 22(3), 2009.•   Layton, R.A., M.L. Loughry, and M.W. Ohland, “Design and Validation of a Web-Based    System for Assigning Members to Teams Using Instructor-Specified Criteria,” accepted with    revisions to Advances in Engineering Education, September 10, 2008, MS AAE-09-078.•   Meyers, K., S. Silliman, M. Ohland, “Comparison of Two Peer Evaluation Instruments for    Project Teams,” Proceedings of the American Society of Engineering Education Annual    Conference, Pittsburgh, PA, June 2008.PRESENTATIONS•   Ohland, M., “Managing Teams,” Project-Centered Learning Symposium, Cambridge -MIT    Institute, http://web.mit.edu/cmi/ue/workshop2008/, March 18, 2008.•   Layton, R.A., M.L. Loughry, M.W. Ohland, and H.R. Pomeranz, Assigning Students to    Teams: Scholarship, Practice, and the Team-Maker Software System, ASEE/IEEE Frontiers    in Education Saratoga, NY, October 22, 2008.•   Pomeranz, H.R., Managing Student Teams Scholarship, Practice, and the Team-    Maker/CATME Applications, Faculty Brown Bag Lunch Series, Oregon State University,    February 20, 2009.•   Layton, R.A., M.L. Loughry, M.W. Ohland, H.R. Pomeranz, “Resources for Student Teams:    The Team-Maker and CATME systems (and why they work),” Academy of Process    Educators Conference, Gaston College, July 9, 2009.•   Ohland, M.W., “Tools for Teams,” invited workshop, Wichita State University, October 30,    2009.•   Ohland, M.W., “Teams: creating a community of learning through peer accountability,”    invited talk, November 20, 2009, Clemson University Environmental Engineering and    Environmental Science.•   Layton, R.A., M.L. Loughry, M.W. Ohland, “The Effective Management of Student Teams    Using the CATME/Team-Maker System: Practice Informed by Research,” invited to    Capstone Design Conference 2010, June 7-9, 2010: Boulder, CO.•   Layton, R.A., M.L. Loughry, M.W. Ohland, “Research Into Practice: Tools for Effective    Management of Student Teams,” workshop at American Society for Engineering Education    2010 Annual Conference.•   Layton, R.A., M.L. Loughry, M.W. Ohland, H.R. Pomeranz, “Workshop on the Effective    Management of Student Teams Using the CATME/Team-Maker System,” submitted to    INGRoup (Interdisciplinary Network for Group Research) Conference, Arlington, VA, July    22-24, 2010.OTHER DISSEMINATION•   Team-Maker / CATME flyers distributed at Mudd Design Workshop, May 2009, Claremont,    CA, and the INGRoup Interdisicplinary Network for Group Research conference in Colorado    Springs, CO, in July 2009.•   Richard Layton will champion the development of presentation resources so that other    members of the team can effectively promote the use of the system. Further, our “power    users” – those who use the system frequently and who are very excited about using it – might    be able to give presentations on behalf of the team (particularly to smaller groups of    faculty.at their own institution).•   Hal Pomeranz will investigate the possibility of user group conference at San Francisco State    University. Depending on the success of such an event, there are active user communities at    University of Southern Maine, Rose-Hulman Institute of Technology, Georgia Southern, and    other sites. A multi-site EPICS conference might be possible.PUBLICATIONS PLANNEDA paper validating the results of the Team-Maker algorithm is nearing acceptance forpublication—we have heard that we may expect favorable word shortly. Because that paperfocused on the validation of the system, only a short section was devoted to which criteria aremost useful in team formation and why. A manuscript is under development that focusesexplicitly on this subject—a broad review of the literature on what criteria are used to formedteams, why, and in what contexts.In addition to this manuscript on best practices in team formation, papers are also plannedreviewing best practices in peer evaluation and separately on best practices in managing teams.This last paper will benefit from a 2006 meta-analysis on which Co-PI Salas was a co-author(Burke et. al 2006).Dave Woehr (Co-PI) has nearly completed a significant work validating the CATMEbehaviorally anchored rating scale. This is a three-study paper that provides firm evidence for theinstrument. Abbreviated results from these papers are provided in the Findings section. Datafrom a time-series experimental design at Rose-Hulman will be analyzed to determine the effectsof rater training. Data from Rose-Hulman and other schools at which students use CATME inmultiple courses over multiple semesters provide an opportunity to study the longitudinal effectsof using the CATME system.A manuscript is also planned publishing and analyzing the details of the Team-Maker algorithmdesign. A diverse literature has been published on strategies for team formation, but few of thosestrategies have achieved widespread implementation. This makes it possible to author a paperthat addresses not only the theoretical design of a team-formation algorithm, but also studies theresults of using that algorithm.REFERENCESBownas, D. & Bernardin, H. (1988). Critical Incident Technique. In Gael, S. (Hg.). The jobanalysis handbook for business, industry and government, 2. New York: Wiley, 1120-1137.Flanagan, J.C. (1954). The Critical Incident Technique. Psychological Bulletin, 51, 4, 327-358.Burke, C. S., Stagl, K. C., Klein, C., Goodwin, G. F., Salas, E., & Halpin, S. M. (2006). Whattypes of leadership behaviors are functional in teams? Leadership Quarterly, 17, 288–307.
AU  - Matthew W. Ohland
AU  - Richard A. Layton
AU  - Daniel Michael Ferguson
AU  - Misty L. Loughry
AU  - David J. Woehr
AU  - Hal R. Pomeranz
CY  - Vancouver, BC
DA  - 2011/06/26
PB  - ASEE Conferences
TI  - SMARTER Teamwork: System for Management, Assessment, Research, Training, Education, and Remediation for Teamwork
UR  - https://strategy.asee.org/18978
DO  - 10.18260/1-2--18978
ER  -