Measuring complex skills Dilemmas and approaches

Abstract

Measuring complex skills is a very important area of pedagogy raising several open questions. Many empirical data collections and student performance measurements lack a proper measurement theoretical foundation, and statistical methods are often used which would only be permissible for interval scales, but the conditions for this are not met. The dilemmas can be traced back to two partly interrelated basic questions: one is „what do we measure” and the other is „how do we measure”. This study reviews the difficulties associated with the second question and some of their possible solutions, also taking into account the results of multidimensional item response theory. Since development means ordering, the developmental state of a multidimensional skill should be characterized by a single number so that the procedure corresponds to the representational theory of measurement, and at least an ordinal scale is created. The proposed novel approach can help solve some measurement theory problems arising from the multidimensional nature of complex skills. The study seeks a solution in an alternative direction from the international mainstream, but retains its probabilistic view. It presents a task-oriented model as opposed to the usual dimensional approach. Skill is characterized by the tasks it allows to be solved. Central to the model is the concept of skill volume, which characterizes a given skill state with the amount of usual learning/practice time required to achieve it, based on the data of a given student reference group. The skill states measured in this way can be sorted; moreover, the effort-based skill volume becomes measurable at ratio scale level for a finite number of task classes, without the need to explore the latent dimensional structure of the complex skill. The skill volume can provide pedagogically important and well-usable information on the scope of a complex skill represented by a set of task classes. The procedure does not try to capture the complex skill itself in its completeness and qualitative diversity, but highlights one of its pedagogically relevant attributes, thus providing an opportunity for well-founded statistical analysis based on higher level scales.