Measuring the Stability of University Rankings in the Field of Education
Keywords:
education, university ranking, innovation, research, QS, URAP, ranking quality, knowledge, CIDIAbstract
The stability of ranking entities is one of the majorly discussed topics when concerning the ranking methodology. It is an important property which increases rankings reliability and credibility. Ranking of universities is, among others, an often examined topic in the terms of its stability, making the researchers worldwide agree that a statistically solid and firm ranking of universities is needed. This paper provides the stability analysis of the particular subject of the QS (Quacquarelli Symonds) World University Rankings compared to the University Ranking by Academic Performance (URAP). Namely, institutions that rank universities have turned the alertness from global academic rankings to more particular rankings in given scientific fields. In this paper, we particularly chose the field of Education, as one of the major factors that contribute to general well-being. We conducted the uncertainty and sensitivity analysis on the QS and URAP rankings in the field of Education to analyze the effects of their weighting schemes. The main goal of this paper is to compare the stability of QS and URAP ranking methodologies. Furthermore, we propose a Composite I-distance Indicator(CIDI) methodology as a tool for implementing the distinguishing innovations into the ranking methodology, in order to provide a more stable and solid ranking lists.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
References
Alaşehir, O., Çakir, M.P., Acartürk, C., Baykal, N. and Akbulut, U. (2014), “URAP-TR: a national ranking for Turkish universities based on academic performance”, Scientometrics, Vol. 101 No. 1, pp. 159–178.
Blossfeld, H.-P., Roßbach, H.-G. and von Maurice, J. (2011), "Education as a Lifelong Process", Springer, New York.
Cheng, Y. (2015), “Academic Ranking of World Universities (ARWU) by Subjects and Fields: Methodologies, Challenges and Future Plans”, Subject and Discipline Related Rankings - a More Inclusive Approach to University Performance (IREG 2015), IREG Observatory on Academic Ranking and Excellence.
Cohen, A. and Saisana, M. (2014), “Quantifying the Qualitative: Eliciting Expert Input to Develop the Multidimensional Poverty Assessment Tool”, The Journal of Development Studies, Vol. 50 No. 1, pp. 35–50.
Dobrota, M., Benkovic, S. (2014), “Comparing ’Ex-Cathedra’ and IT-Supported Teaching Methods and Techniques: Policy of Teaching Practice”, Croatian Journal of Education, Vol. 16 No. Sp.Ed.3, pp. 91–108.
Dobrota, M., Bulajic, M., Bornmann, L., Jeremic, V. (2016), “A new approach to the QS university ranking using the composite I-distance indicator: Uncertainty and sensitivity analyses”, Journal of the Association for Information Science and Technology, Vol. 67 No. 1, pp. 200–211.
Dobrota, M., Dobrota, M. (2016), “ARWU ranking uncertainty and sensitivity: What if the award factor was Excluded?”, Journal of the Association for Information Science and Technology, Vol. 67 No. 2, pp. 480–482.
Dobrota, M., Jeremic, V. (2016), “Shedding the Light on the Stability of University Rankings in the ICT Field”, IETE Technical Review, pp. 1–8.
Dobrota, M., Savic, G. Bulajic, M. (2015), “A New Approach to the Evaluation of Countries’ Educational Structure and Development: The European Study”, European Review, Vol. 23 No. 04, pp. 553–565.
Federkeil, G. (2015), “Doing Field-based Rankings: Lessons Learned from U-Multirank and CHE-rankings”, Subject and Discipline Related Rankings - a More Inclusive Approach to University Performance (IREG 2015), IREG Observatory on Academic Ranking and Excellence.
Fleming, T. (2011), "Models of Lifelong Learning: An Overview", Oxford University Press, available at: http://doi.org/10.1093/oxfordhb/9780195390483.013.0018 (5/5/2016)
Hazelkorn, E. (2013), “How Rankings are Reshaping Higher Education”, in Climent, V., Michavila, F. and Ripolles, M. (Eds.), Los Rankings Universitarios: Mitos Y Readidades, Tecnos, Madrid.
Hoskins, B., Saisana, M., Villalba, C.M.H. (2015), “Civic Competence of Youth in Europe: Measuring Cross National Variation Through the Creation of a Composite Indicator”, Social Indicators Research, Vol. 123 No. 2, pp. 431–457.
Nguyen, T.T.H., Walker, M. (2016), “Sustainable assessment for lifelong learning”, Assessment & Evaluation in Higher Education, Vol. 41 No. 1, pp. 97–111.
Paruolo, P., Saisana, M., Saltelli, A. (2013), “Ratings and rankings: voodoo or science?”, Journal of the Royal Statistical Society: Series A - Statistics in Society, Vol. 176 No. 3, pp. 609–634.
QS Subject. (2016), “QS World University Rankings by Subject”, available at: http://www.topuniversities.com/subject-rankings/2015 (accessed 15 November 2015).
Roche, S. (2015), “Of integration and disintegration: The ‘hows’ and ‘whys’ of lifelong learning in a rapidly changing world”, International Review of Education, Vol. 61 No. 5, pp. 577–583.
Saisana, M., D’Hombres, B. (2008), "Higher Education Rankings: Robustness Issues and Critical Assessment. How Much Confidence Can We Have in Higher Education Rankings?", EUR23487 ed., Joint Research Centre, Publications Office of the European Union, Italy, available at: http://doi.org/10.2788/92295 (5/5/2016)
Saisana, M., D’Hombres, B., Saltelli, A. (2011), “Rickety numbers: Volatility of university rankings and policy implications”, Research Policy, Elsevier B.V., Vol. 40 No. 1, pp. 165–177.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., et al. (2008), "Global Sensitivity Analysis: The Primer", John Wiley and Sons, New York, NY, NY, available at: http://doi.org/10.1002/9780470725184 (5/5/2016)
Sowter, B. (2015), “Rewarding and Revealing: Reflections upon Four Years into the QS World University Rankings by Subject”, Subject and Discipline Related Rankings - a More Inclusive Approach to University Performance (IREG 2015), IREG Observatory on Academic Ranking and Excellence.
Uddin, A., Singh, V.K. (2014), “Mapping the Computer Science Research in SAARC Countries”, IETE Technical Review, Vol. 31 No. 4, pp. 287–296.
URAP. (2016), “University Ranking by Academic Performance”, available at: http://www.urapcenter.org/2014/index.php (7/8/2015)
Volles, N. (2016), “Lifelong learning in the EU: changing conceptualisations, actors, and policies”, Studies in Higher Education, Vol. 41 No. 2, pp. 343–363.
