Theoretical and methodological considerations in assessing collaborative problem solving competence Results of a pilot study

Abstract

The great relevance of the development of collaborative problem solving (ColPS) competence as a 21st-century skill is widely recognized. However, the creation of a reliable, valid measurement to follow improvement of this competence seems extremely challenging from a number of theoretical and methodological aspects. The aim of this paper is twofold: (1) to provide a review of problematic issues in the measurement of ColPS competence and of assessment trends to date; and (2) to present the first results of a trial of a new ColPS assessment tool. The pilot study whose results we report investigated the psychometric properties of our online test and the attitudes toward its new, unusual testing context. 71 bachelor’s students completed our test in 17 three-, four-and five-member groups via the eDia (Electronic Diagnostic Assessment) platform. The test consisted of four analytical, content-general, decision-making problems with 11 items in all. To make their joint decision, group members discussed their ideas via chat. After the problem solving part of the test, the students were asked for their opinions on the test via five-point Likert scale items and a qualitative question. The reliability index of the eleven-item test was Cronbach’s α=.79. We found significant positive correlations between the first and the second (r=.66; p<0.01), the first and the third (r=.41; p<0.01), and the second and the third problem (r=0.53; p<0.01); the fourth problem, however, showed only a weak correlation with the first (r=.29; p<0.05) and the second problem(r=.27; p<0.05) but did not correlate with the third one. 92.5% of the participants reported partly enjoying, enjoying or very much enjoying this collaborative effort. 94% of the answers expressing general attitudes toward the test clearly suggested a positive opinion. 60% of the comments referring directly to the chat function found this mode of communication modern, interesting and great, while 40% described it as less effective, disturbing or frustrating. Based on our results, we consider our new, third-generation test reliable and acceptable in general. Nevertheless, our findings also provided some very useful conclusions about the necessity of reconstructing the fourth problem and/or reconsidering some core conditions like the content or the way problems are presented in our assessment tool with regard to our future research aims.