Application of Data Envelopment Analysis (DEA) in Information and Communication Technologies

: The consistent improvements and fast - growing trend of information and communication technology (ICT) have impacted all areas of society and the economy. In 2020, with the sudden pandemic of COVID -19 , businesses worldwide faced great challenges and had to transform and become digital - native enterprises by using data analytics, digital business platforms and personalized customer approaches. In such a digital era, innovation, entrepreneurial dynamism and ICT are the key ingredients for business' success and sustainability. Furthermore, it has become very clear that the influence of the ICT industry on economic growth is immense. Therefore, it should not come as a surprise that the interest in research of the ICT industry is great. Data Envelopment Analysis (DEA) is the leading non - parametric mathematical technique for assessing performance and measuring efficiency of complex entities called Decision - Making - Units (DMUs), by conversion of multiple input variables to multiple output variables. DEA has experienced rapid growth in use in many areas ever since its introduction by Charnes, Cooper and Rhodes in 1978. The purpose of this paper is to present and describe Data Envelopment Analysis as the leading mathematical programming technique for data analysis and to provide an extensive literature review, to identify the studies implementing the DEA methodology in Information and Communication Technologies (ICT) and to present its findings. Furthermore, this study's goal is to inspire and encourage researchers to employ this methodology in the fields of ICT and to give guidance for future research in this area.


INTRODUCTION
Information and communication technologies (ICT) are involved in all aspects of life. The emergence and development of the ICT industry in the past two decades has altered the economic landscape and the technological advancement has increased the use of sophisticated hardware and software in all areas of life and business. Namely, ICT have transformed the way in which we communicate and interact with each other, the way we seek and find information, the way we work, do business and operate, and the way we manage our social lives [1][2][3].
As a basic economic need, every business operating in the ICT industry (as well as any business in general) needs to be efficient and to perform. DEA (data envelopment analysis) is a mathematical programming technique that helps evaluate and assess efficiency of peer units concerning multiple performance and efficiency measures. In DEA, the units that are evaluated are known as decision making units (DMUs) and the measures of performance are grouped into input variables and output variables. This methodology is very convenient in cases when the relationships between the input and output variables are unknown [4].
DEA is becoming a globally acknowledged and commonly used technique for assessing efficiencies and performance of decision making units (DMUs). Due to its simplicity and successful application in different research areas, DEA has gained considerable attention and is extensively used by business and academy researchers from all over the world [5].
The remainder of the paper is structured as follows: Section 2 gives an introduction and an overview of the DEA methodology, a short theoretical background and its basic models; Section 3 presents the most important application of the DEA methodology in efficiency measurement in ICT worldwide. Section 4 offers a discussion on the benefits and limitations of this methodology and Section 5 presents concluding remarks and guidance for future research.

DATA ENVELOPMENT ANALYSIS (DEA): THEORETICAL BACKGROUND AND BASIC MODELS
The reason why performance and efficiency are being measured is to compare behaviour (i.e. efficiency) of organizations consistently. Efficiency is one of the key indicators of performance, thus being a vital element for business viability and a precondition for business improvement. These "benchmarking comparisons" (i.e. efficiency comparisons) can be divisional or cross-divisional, national or supranational etc. [6].
When analysing performance and efficiency of entities, there are two existing approaches at hand: the parametric and the non-parametric approach [7]. According to De Borger, B. et al. [6], the main assumption in the parametric approach is that "the boundary of the production possibility set can be represented by a particular functional form with constant parameters", whereas the non-parametric approach "does not impose regularity axioms on the production possibility set and directly constructs a piecewise technology on the sample".
Data Envelopment Analysis (DEA) is the leading nonparametric methodology that has gained a lot of popularity and interest from researchers since its introduction in the literature of the operational research (OR) discipline in 1978 by Charnes, A. et al. [8]. However, Farrell, M. [9] is the one who developed the concept of best-practice frontiers and introduced the first measurement scheme and efficiency concept [6].
The Data Envelopment Analysis (DEA) is a linear mathematical programming methodology whose purpose is to evaluate the performance (i.e. the relative efficiencies) of a group of complex entities referred to as Decision Making Units (DMUs) [10]. The evaluation is based on the implemented inputs and on the produced outputs. One of the main strengths of this approach (unlike the parametric approaches) is the possibility to include more than one output and to not set assumptions for the production function [11].
What is crucial in DEA is the decision-making units (DMUs) to be homogeneous entities, which means they should use the same resources (known as inputs) to produce the same results (known as outputs) [7]. DMUs can be banks, hospitals, supermarkets, shops, police stations, tax offices, defence bases, schools, restaurants, insurance companies, libraries and even university departments. According to Samoilenko, S. & Osei-Bryson, K. M. [12], DMUs can be people, companies, or countries.
In other words, DEA is one of the existing frontier methods, and represents a convex non-parametric frontier. Frontier methods are methods in which an efficiency frontier (which is based on real observations) is used to classify the different DMUs. Thus, Data Envelopment Analysis is a nonparametric approach that implements a linear programming to discover the efficiency frontier for a company or DMU. The relative efficient DMUs form the efficient frontier that presents an envelope for the analysed DMUs [11]. This way, DEA is used to solve an individual linear programming problem for each analysed organization or DMU, considering the organization's "input and output data are assigned a set of weights to maximize the ratio of inputs and outputs" [13]. The DMUs that are on the frontier (i.e. the cases of best practices) are considered relative efficient, whereas DMUs that are not on the frontier are considered relative inefficient [10,6]. The first DEA model, introduced by Charnes, A. et al. [8] was named DEA-CCR in honour of their authors. The DEA CCR model has an input orientation and the main assumption is the existence of constant returns to scale (CRS), i.e. all firms are working at an optimal scale [13,14]. Banker, R. D. et al. [15] have made alterations to the CCR model to incorporate variable returns to scale (VRS) and to estimate the pure technical efficiency of DMUs, thus developing the BCC model (named in honour of its authors as well). The BCC model also recognizes whether a DMU is "operating in increasing, decreasing or constant returns to scale". So it is safe to assume that CCR models are a specific type of the BCC models [5].
Furthermore, DEA models can be performed in the following common orientations: an input-orientation, an output orientation or a non-orientation. In an input orientation, the DEA model should determine the proportion of inputs' reduction that would achieve the same given level of output, with the main assumption that outputs are fixed. In an output orientation, it is not outputs that are fixed, but rather inputs. When using this model orientation, the model determines proportional increase of outputs that would lead to maximization of outputs for a given level of inputs. The non-oriented DEA model is a model in which the input variables are simultaneously reduced and the output variables are increased in order for the DMUs to become efficient [16][17][18][19].
DEA has been gaining a lot of popularity in different areas of research and in different industries. Namely, in the last four decades, there are 10.300 published journal articles since the original DEA paper published in 1978 [20]. However, this was not the case in the beginning. Originally, DEA methodology was used to evaluate and measure the efficiency of non-profit organizations, due to the inability to assess them on the basis of conventional economic and financial indicators, which were used for commercial (profit) organizations [21]. In the past decade, DEA has been employed in different areas. For example, Gardijan, M. & Škrinjarić, T. [22] have used the DEA-based investment strategy to conclude the efficiency of the Croatian stock market; Rabar, D. [23] has measured the socio-economic performance of OECD countries, many scholars have evaluated the efficiency and performance of the banking sectors in different countries (Jemrić, I. & Vujčić, B. [24] for Croatia, Fotova Čiković, K. & Cvetkoska, V. [25] for North Macedonia, Sufian, F. et al. [26] for Malaysian banks etc); it has been used in ecological research of maintenance of forestry mechanisation [27], in the process of selection of Energy-Saving Projects [28], in measuring the productivity of chain restaurants [29], in assessing the efficiency of Brazilian electricity distributors [30], in measuring the technical efficiency of primary public health care centres [31], to examining the efficiency of Formula 1 drivers and teams [32]. But, according to Emrouznejad, A. & Yang, G. [20], there are five areas of research where the DEA methodology is mostly used, and these are the following: agriculture, banking, supply chain, transportation, and public policy. Emrouznejad A. et al. [13] found "banking and finance, education, health care and hospital efficiency, energy and utilities, and transportation" to be the most popular application areas of DEA.
In the next Section, the focus shifts on DEA's application in ICT in ten different studies (with ten different perspectives).

APPLICATION OF DATA ENVELOPMENT ANALYSIS (DEA) IN INFORMATION AND COMMUNICATION TECHNOLOGIES
The ICT industry has not been on the radar for implementation of DEA models very often. Even though the ICT industry has been largely researched and has attracted significant attention in the recent literature, the DEA methodology is not widely accepted by academics and researchers in this area. In this paper the focus is on the application of DEA in information and communication technologies by analyzing 10 studies published in the past two decades, i.e. from 2006 to 2021.
The applications of DEA in the context of information and communication technologies are presented in Tab. 1.
Chen, Y. et al. [4] have presented how the non-linear programming approach DEA can be handled as a parametric approach and thus, they emphasize the need for a more inclusive and thorough approach that takes into consideration wider economic and strategic IT effects on productivity. They have developed and implemented a new two-stage DEA model to evaluate the impact IT has on a company performance when intermediate measures are at hand. Their model uses an outline from a previous study and they have selected fixed assets, employees, IT investment as inputs in the first stage, deposit as an input (intermediate measure) in the second stage, and profit and loan recovered as outputs.  [14] Evaluation of the impact of ICT on productivity in the hotel sector Martinez-Nunez & Perez-Aguiar (2014) [36] Evaluation of the productive efficiency and online-social-networks (OSN) in telecommunications firms from Spain Efendi, Syahputra & Muchtar, (2018) [37] Measuring efficiencies of ICT development in Indonesia region Guccio, Martorana, Mazza & Rizzo (2019) [38] Evaluation of the impact of the use of ICT on the performance of public historical archives Mimbi (2021) [18] Public value framework to evaluate ICT public value creation Anderson, R. T. et al. [33] have developed and presented a framework for characterization, assessment and forecast of the wireless communication technologies. They have employed a DEA-based methodology for forecast of the "state of the art in future wireless communications technologies". Namely, they have implemented the Technology Forecasting using Data Envelopment Analysis (TFDEA), which was created in 2001 as an "alternative quantitative approach for technology forecasting". Their study is very influential from both theoretical and practical perspective. From a theoretical point of view, the study stretches the use of DEA in technology forecasting, whereas from a practical perspective, it offers a specific methodology that has been thus far used to microprocessors, fighter jets, enterprise data systems and USB drives.
Asosheh, A. et al. [5] have implemented the DEA methodology together with the Balanced Scorecard approach to evaluate IT projects. They specifically focus on the critical aspect of IT management of deciding upon the best set of IT project from numerous competing proposals. Their study implements Balanced Scorecard (BSC) as a framework to define the criteria for IT projects' evaluation, whereas DEA helps them rank the IT projects. Their paper offers great scientific and practical contribution, due to the fact it combines two popular managerial methodologies: the DEA and the Balanced Scorecard (BSC), and furthermore give a proposal of how to approach new IT project selection processes.
Emrouznejad, A. et al. [34] have implemented an alternative approach for assessing information and communication technology (ICT), with the application of the Data Envelopment Analysis (DEA), on a sample of 183 economies. They have drawn the data from the International Telecommunications Union and have made a comparison of the existing ICT-Opportunity Index (ICT-OI) with their developed DEA-Opportunity Index (DEA-OI), and found a high correlation between the two indexes. The developed DEA-OI consists of 10 ICT indicators, similar to the ICT-OI. The authors suggest that DEA-OI can be considered as "an alternative measurement approach for benchmarking and measuring progress toward the information society".
Goto, M. [35] explores the financial performance of the global telecommunications industry by implementing the DEA-DA (Data Envelopment Analysis -Discriminant Analysis), whose model has a relation with the Altman's Z score. The sample consists of 44 world telecommunications companies, which were ranked according to their performance, which provides crucial information for investors, corporate management, the public and anyone who is interested in the financial performance of companies in the telecommunications sector. Additionally, this article gives a commentary to the financial performance of AT&T (American Telephone & Telegraph) and NTT (Nippon Telegraph and Telephone) after their divestment and found that AT&T outperformed NTT due to its evolvement in an IT company.
Paço, C. L. & Pérez, J. M. C. [14] demonstrate ways in which applications of information and communication technologies (ICT) can bring competitive advantages in hotel firms, with an application of the DEA (CRS) methodology. Their sample consists of 184 hotel establishments, and the analyzed period is from 2008 to 2011. The inputs used were human, financial and material factors and their relative costs, whereas the outputs were the total number of stays in the establishment from 2007-2010 and the number of stays in this period resulting from Internet bookings. They claim the DEA method allows them to diagnose the limitations and blind spots and possible ways to increase efficiency in hotels.
Martinez-Nunez, M. & Perez-Aguiar, W. S. [36] have implemented the DEA methodology together with various indicators of business "social media" activities and thereafter carried out the super-efficiency analysis together with the bootstrapping techniques to improve the model's robustness and accuracy. In their study, they examine the relationship between the productive efficiency and online-socialnetworks (OSN) on a sample of Spanish telecommunications firms. They chose the DEA model due to its flexibility and non-existent requirements to pre-determine any functional form for the production function. They employed the inputoriented DEA model and developed four models with combinations of the selected input (employees, assets, equity, tweets, following tweets, posts) and output (sales, profits, followers, fans, comment received) variables. This study is valuable to researchers and businesses because it presents divergent factors that present the performance in the use and management of Web 2.0 technologies and OSNs. Their findings imply that the company's capacity to absorb and utilise OSNs are the most significant factor in efficiency improvement.
Efendi, S. et al. [37] implement the DEA BCC method to find the index of efficiencies for all regions in Indonesia and to evaluate the level of readiness of each district and city in each province (DMUs) for the ongoing digital economy era. Their findings show that the readiness of each of the analysed province was not enough, but a level of efficiency is required from each province, in a way that the local government could be able to recognize the weaknesses and inefficiencies of each province.
Guccio, C. et al. [38] explore one rather rarely verified area of research: the impact of ICT (represented by the existence and usage of a website) on the efficiency of public services (in their case the public historical archives -PHAs). They employ a two-stage approach including the DEA and the Window DEA together with the Mann-Whitney and Kolmogorov-Smirnov tests. The selected inputs are the total surface area in square meters; the total shelf dimension in linear meters; the number of personnel; and current PHA expenditure excluding labour costs. Output variables include: the number of visitors; the total number of requests processed by PHA; and the total number of documents inspected. The obtained results imply that the distribution and implementation of ICT does improve efficiency and in the case of their study, Italian PHAs with a website appear to be generally more efficient than others. They focus on the public sector due to the lack of studies in the literature that focus exclusively on efficiency in the public sector.
Mimbi, L. [18] has adopted a "public value framework to interrogate ICT public value creation", which is quite an interesting study. In this study, he implemented the DEA together with a cluster analysis to analyse data for 53 African countries in the period from 2010 to 2019. The clusters were formed using the human development index (HDI). In his study, Main telephone line, Internet users and Mobile cellular subscribers were used as input variables, whereas Accountability, Government effectiveness, Press freedom and Rule of law as output variables. His findings show that ICT has "efficiently transformed public values in Africa" by 14%, 35%, and 63% in countries of clusters 1, 2, and 3, respectively. Moreover, his findings are in line with the public value theory, which forecasts doing more with less. Namely, the clusters that showed highest efficiency of ICT are the ones noting low human development.

DISCUSSION
The presented review of the DEA literature in ICT shows the application of DEA in different aspects of efficiency evaluation, and highlights the broad use of this methodology. In this paper, we have adopted a pragmatic perspective and we seek to objectively point to some of its strengths as well as its limitations.
There has been open criticism and rebuttals on a number of methodological and substantive grounds regarding the DEA methodology [23,39,40]. Nevertheless, Stolp, C. [41] claims DEA should be considered as an informative and useful tool for systematic sensitivity analysis.
In Tab. 2 the most stated strengths, limitations and setbacks of the DEA are presented. The results can be sensitive to the selection of input variables and output variables, so an analysis of their relative importance is essential prior to the calculation The biggest and most important issue in DEA is the validated selection of the RTS (return-to-scale) assumption. Furthermore, the selection of the output or the input orientation has been an important issue of DEA, since Emrouznejad, A. et al. [44] report that "there can be significant differences" between the results of the CCR and BCC model variants [40]. Therefore, it should be advised that DEA results need to be interpreted with much caution so that wrong signals and inappropriate recommendations could not be laid out.

CONCLUSION
The constantly changing environment that we live in requires the ICT industry to be not only innovative, alert and responsive, but also productive and efficient. As Martinez-Nunez, M. et al. [36] state regarding the ICT use in companies, "the key point is not how good a technology is, but how well it is used by members of an organization". This statement highlights the reason why DEA methodology should be more commonly used and implemented for performance and efficiency measurement in the ICT industry by business analysts and academic researchers.
Data Envelopment Analysis is an econometric mathematical programming technique that has gained large interest in different areas, but has not been often used in the ICT industry. In this paper a theoretical background of the data envelopment analysis (DEA) is given and its application in information and communication technologies is presented according to ten published studies in this area in the period from 2006 to 2021. These studies have explored very different perspectives of the ICT industry and have all implemented the DEA methodology, starting with the evaluation of IT investment [4], prediction of the future wireless communications technologies [33], IT project evaluation [5], measuring ICT [34] and ICT development in Indonesia region [37], evaluation of financial performance of US and world telecommunications companies [35], evaluation of the productive efficiency and online-socialnetworks (OSN) in telecommunications firms from Spain [36], evaluation of the impact of the use of ICT on the performance of public historical archives [38] to the use of the public value framework to evaluate ICT public value creation [18]. This paper highlights all the possible ways to implement and benefit from the DEA methodology in ICT.
As any other methodology, DEA has strengths and limitations. Its limitations and pitfalls could present ample opportunities for some alternative approaches in dealing with these issues, but also for improvement of DEA implementation by researchers as well [13]. Other than that, DEA is a significant diagnostic tool that is relatively easy to implement and that gives new insights to stakeholders in different types of companies and different industries.
The main objective of this paper is to present the studies published thus far implementing DEA in information and communication technologies as well as to encourage other scholars and researchers to implement this methodology in ICT, to use the benefits of this approach and to give new insights to academic researchers, business analysts and the public.