A research project ranking method based on independent reviews by using the principle of the distance to the perfectly assessed project

Abstract

The paper discusses the problem of ranking research projects based on the assessment obtained from $n\geq 1$ independent blinded reviewers. Each reviewer assesses several project features, and the total score is defined as the weighted arithmetic mean, where the weights of features are determined according to the well-known AHP method. In this way, it is possible to identify each project by a point in $n$-dimensional space.
The ranking is performed on the basis of the distance of each project to the perfectly assessed project. Thereby the application of different metric functions is analyzed. We believe it is inappropriate to use a larger number of decimal places if two projects are almost equidistant (according to some distance function) to the perfectly assessed project. In that case, it would be more appropriate to give priority to the project that has received more uniform ratings. This can be achieved by combining different distance functions.
The method is illustrated by several simple examples and applied by ranking internal research projects at Josip Juraj Strossmayer University of Osijek, Croatia.