The Assessment Model of Robotic Process Automation (RPA) Project Using Benefit Study and Balanced Scorecard (BSC) Approach

246-253


INTRODUCTION
Technological advancements have enabled many companies to automate well-structured business processes with high volumes, error rates, and repetitive routines [1,2].Automation has been applied in various fields, from factory manufacturing processes to office accounting.Extending automation to all departments makes companies more efficient and agile [3].However, automation in the office is not easy to implement.The biggest challenge does not only come from technology but also from decision-makers in the company who are not fully convinced about technology and its benefit to the organization [4][5][6].
One form of office automation technology is Robotic Process Automation.It is a software robot that performs repetitive back-office tasks by mimicking the behavior of people in interaction with computers [2], such as: opening the application, searching data in the database, extracting data from PDF files, sending emails, and so on [2,7].RPA offers a non-invasive and flexible technology that is easy to implement in various business processes, such as the on boarding process in the human resources department, the invoice processing in the purchasing department, the inventory management in the production & logistics department, the order processing in the sales department, and others [7].
RPA has been proven to provide benefits to companies.However, some companies, especially SMEs failed because they considered that the implementation of RPA does not provide benefits that are worth the sacrifices made [8,9].The companies that have implemented RPA sometimes feel stuck, and choosing not to continue the RPA project [10].This is partly because these companies do not know how to assess the performance and success of their RPA Projects.
To avoid mistakes and hasty decision-making, the authors devised a model that can be used to assess RPA Projects with several indicators of benefits.With this model, RPA assessments can be more objective, and decisions to continue or stop RPA are more accountable.This model is inspired by the strategic decision assessment and monitoring model in business management known as the balanced scorecard [11,12].This study will depart with the research questions: • RQ1: What are the benefits of RPA? • RQ2: How to build and use an assessment model of RPA Projects using the benefits study and balanced scorecard (BSC) approach?
The research began by collecting information and data on the benefits of RPA from scientific contributions.The identified benefits were then grouped into four clusters of BSC perspectives, and used as parameters in the assessment model.The assessment model that has been built is then tested directly on one SME (Small&Medium Enterprise).The authors report on the feedback received, regarding the obstacles encountered by SME while using the model.

STATE OF KNOWLEDGE 2.1 An Overview of RPA Benefits
According to Axmann and Harmoko (2020), RPA is one step closer to making a business lean with the benefits acquired in the short-term and long-term, for management, employees, and customers [7].RPA provides the highest efficiency in personnel cost, and it requires the lowest investment cost [9,13,14].As RPA is easy to integrate with processes when compared to other automation technologies [15].And its development is faster, because it uses the existing infrastructures such as networks, desktop, and server [2,15].It can improve data quality and accelerate digitalization in the whole company [7].
RPA brings benefits for customers in the form of excellent service, from ordering items to after-sales service.With RPA, the company can interact more with customers, such as serving their orders, answering their questions, and knowing their preferences for the new product [7].RPA helps employees as assistants (attended mode) to perform mundane tasks [16].The technology assists them in completing tasks faster.While indirectly, RPA can increase creativity, selfconfidence, and satisfaction of the employee, because when employees are free from repetitive tasks, they will have the opportunity to increase skill and knowledge [7], which makes them confident to execute high value-added tasks [13].
Research on the benefits of RPA has been conducted by several researchers before.Koljonen (2023) describes the benefits of RPA and combines them with risks, to illustrate that RPA implementation has two different sides (advantages and risks) that must be considered.Furthermore, Meironke and Kuehnel (2022) designed metrics that can help measure the benefits of RPA with 62 indicator metrics [17,18].In addition, to assess the overall RPA project, RPA providers have equipped their products with cost-benefit analysis features such as UiPath Automation Hub, and Automation Anywhere Business Analyst.
In contrast to all, authors conducted a benefits study to support the development of an RPA project assessment model in SMEs.Based on the experience and interview with RPA consulting company and SMEs [19,20], authors found that not all indicators described in Meironke and Kuehnel's research can be understood by SMEs, besides that the costbenefit analysis method provided by RPA providers is not sufficient to reveal non-monetary benefits of RPA.Starting from this, the authors designed an RPA project assessment model with a balanced scorecard approach as a business management method that is familiar to SMEs, so that it can be used immediately and helps in a fast and accountable decision-making process.

The Balanced Scorecard (BSC) Theory
The Balanced Scorecard is a monitoring tool for strategic decisions taken by a company on the basis of pre-defined indicators that should address four dimensions: financial, customer satisfaction, internal business processes, and learning and growth [12,21].Organizations use BSC (1) to communicate what they are trying to accomplish, (2) to align the day-to-day business to the company's strategy, (3) to optimize projects, products, and services, and (4) to monitor the achievement of strategic targets [11].
The BSC was created as a measurement system and as an answer to various criticisms of unilateral measurement of a company's performance capability [12,22].Nevertheless, the concept of BSC has evolved beyond a simple perspective that focuses on financial, into a holistic system that manages, measures, and tracks the success of organizations in achieving their strategic goals [11].In addition, the BSC has been used to measure the various benefits of information technology projects such as E-business Projects by Grembergen and Amelinckx (2002) [23], Enterprise Identity Management Project EIdM by Royer and Wolfgang (2008) [22], and Information Communication Technology ICT Project in Southern Africa by Ogembo-Kachieng et al (2013) [24].
The balanced scorecard model consists of four tables.Each table represents four perspectives on a company's vision and strategy (see Fig. 1).A table consists of four columns; (1) objectives, (2) goals, (3) indicators, and ( 4) initiatives.The first column, objectives describes what the company wants to achieve (objectives) from the company's strategy or vision.The second column goal is the magnitude or quantification of the objective that has been described in the first column.The third column is indicator or places where goals and objectives can be evaluated and tracked when the company's vision and strategy have been implemented.Meanwhile, the fourth column, or initiative describes what strategic steps are being and will be taken to meet the goals.
Figure 1 The Origin Balanced Scorecard Model [12] To better understand the balanced scorecard model the four section are generally described and an example of a balanced scorecard in a company (Jewellery store) is shown in Fig. 2).
Financial perspective: This perspective is possible if the development, implementation, and execution of new strategies generate profit [12,22].The Profit is a positive amount of total income after deducting costs in a certain period [25].If the company can reduce costs through efficient business processes, profits will automatically increase [25,26].This shows that the financial perspective is closely related to efficiency, therefore the benefits of RPA will classify into efficiency clusters that represent the financial perspective in the BSC.
Customer perspective: The old argument holds that it is essential for companies to develop the best products to achieve the best financial performance, but in reality, the customers and their behaviour bring strong influence to the market [11,12,27].Customers can easily buy products from unknown brands, but on the other hand, they can also refuse products from well-known brands for various justifications.Understanding customer needs and how to satisfy them is not only to keep loyal customers but also to attract new customers [27].In the context of our study, we perceive that the implementation of RPA must satisfy customers, which are not only real customers but the owners and employees of the company, who gain benefit from RPA implementation.Business processes perspective: To satisfy stakeholders, business processes must be value oriented.Processes that have a significant impact on value creation must be more effective.The effectiveness of the process can be seen from the increase in productivity and accuracy [11,12,22].In this study RPA should provide direct benefits to business processes.Therefore, the business process perspective will include in identifying the RPA benefits.
Learning & Development perspective: This perspective complements the other perspectives.However, its role is not less important than the others, because it ensures sustainability [12].From a learning and development perspective, the company knows that a new strategy, such as the RPA implementation will continue to improve the capacity and capability of human resources, infrastructure, and management.Therefore, this perspective is included, to ensure that the benefits of RPA must support sustainability.
The concept of the BSC lies in the fact that strategic objectives are listed by indicators, target values, and action plans.Those make sure the realization of the new strategy must be clear [11].If the success of implementing RPA is indicated by obtaining comprehensive benefits as much as possible, then incorporating the BSC perspective into this study becomes relevant.

RESEARCH METHODOLOGY
Our study employs three research methods: (1) a structured literature review identifying the benefits of RPA, (2) a meta-analysis that presents statistical data on the benefits of RPA in the scientific contributions, and (3) an a first interview for problem identification, model testing, and feedback collection.It is planned to execute more interviews in future research.

Procedure of Structured Literature Review
The structured literature review refers to the guidelines of Webster & Watson (2002) and vom Brocke et al. ( 2015) [29,30], where future academic research is based on the review of previous literature.Effective review creates a solid foundation for advancing knowledge.This facilitates theory development, covers areas with a lot of research, and uncovers areas that need further research [30].[29].Academic contributions related to the topic of RPA using the search query: "RPA OR robotic process automation" have been surveyed.Search queries have been kept deliberately generic to avoid excluding contributions that indirectly address the various benefits of RPA.In this way, 1,522 academic contributions have been found, however only 149 of them addressed RPA.With abstract, title, and keyword analysis, full-text analysis, and forward and backward searches, 119 academic contributions have been found that mention the benefits of RPA (see Fig. 3).

Meta-Analysis
Meta-analysis is a statistical technique that combines two or more similar studies to obtain a quantitative mix of data.It is a retrospective observational study, where the researcher makes recapitulation of the data without conducting experimental procedures.Meta-analysis does not emphasize the findings of various studies, but the data [31] .This method has been used in several scientific papers, including by Axmann et al. (2021) to identify RPA cost drivers [6].Using a similar method information/data on RPA benefits have been collect and classified into four BSC perspectives.Fig. 4 shows its distribution in RPA papers published in the last 5 years (2018-2022).

Interview
The main purpose of the case study in this research is to test and get feedback on the established assessment model.Case study data collection used the one-on-one interview method by Ryan et al. (2009) and Bullock (2016) [32,33], where the experience of the interviewee (company owners) in evaluating RPA pilot projects (1 month) is explored in depth.Furthermore, the interviewer presents an assessment list as in Tab. 2 (see Section. 6) to be filled in by the company owner as an assessment model for his project.During this process, there was an interaction between both (the interviewer and the interviewee) about some terminology on the list which is not understood by the interviewee.Once completed, the interviewer asked for feedback on this kind of assessment model from the interviewee.
In the future the model needs to be tested by more experts.

THE BENEFITS OF RPA
From 149 scientific contributions, at least 27 benefits have been found, which can be classified into four BSC perspectives such as Tab. 1. Efficiency & Financial: RPA implementation provides financial benefits for the company through cost and time efficiency [7,34].The time saving here includes operational time, programming, and testing when the RPA is initiated [35].In the context of investment, RPA is feasible and profitable because the return on investment of RPA is relatively faster than the other automation technology [36].In addition, RPA can support company growth [37] by improving economic performance [38] such as increasing revenue and saving full time employee [39].
Business Process Improvement: To improve business processes, RPA robots offer 24-hour non-stop service without getting tired [40,41].This reliability is followed by the ability to execute fast, effectively, and accurately [13,42], thereby accelerating the decision-making process [13,35].RPA implementation is also easy and not complex [14,15], it can be used in every unit in the company [7], and is easy to integrate with existing systems [2,16].In addition, RPA can also increase employee utilization through refocusing on value-added tasks [41,43].
Table 1 The Benefits of RPA Stakeholder Satisfaction: The implementation of RPA provides satisfaction not only to the customers and owners but also to employees.With the automated process, employees will be liberated from tedious and mundane tasks.They will be stimulated to upgrade their skills and creativity to become more valuable.In terms of customer satisfaction, RPA can help customer service staff in speeding up the order process or complaints made by customers [7,42].
Learning & Development: The implementation of RPA will accelerate the digitization process and other aspects that accompany it, such as transparency, data security, machine learning, and so on [22,37,42].This is because RPA encourages all processes or documents to be automated to be digital, transparent, and secure from cyberattacks [7,13,44].
In Fig. 4, the meta-analysis explains how the benefits of RPA are calculated among scientific contributions over time.The graph shows that scientific contributions in the past 5 years mostly looked at the benefits of RPA from the perspective of business process improvement (50.1%) compared to learning and development (21.6%), efficiency (17.7%), and stakeholder satisfaction (10.6%).The graph also shows from 2018 to 2022, the benefits of RPA have been revealed through scientific contributions, which are different from costs.
Figure 4 The Meta-analysis of RPA Benefits

THE ASSESSMENT MODEL OF RPA PROJECTS
This assessment model is inspired by the balanced scorecard, in which the decision of whether to continue the RPA project is based on an assessment of the expected benefits (target) and actual benefits (actual) of the RPA.When the benefits obtained are by company expectations, the decision maker can continue the project, but if not met expectations, the decision maker can postpone or stop the project.In section 2, an example of a balanced scorecard was explained, along with the accompanying table.
This assessment model adopts part of the concept, especially in the classification of benefits that follow the balanced scorecard perspective.BSC itself is not a rigid method, it is applied flexibly in each company or business unit.Fig. 1, section 2 shows an ideal figure of BSC which includes objective, measure, target, and initiative columns, but in its application at the Jewelry store, the name or definition of the assessment columns differ (see Fig. 2).
The assessment model consists of four lists of benefits, namely: efficiency and finance, stakeholder satisfaction, business process improvement, and learning & growth.Each list consists of four columns; (1) benefit description, (2) target, (3) actual, and (4) measure (see Fig. 5).In the benefit column, the company describes the benefits it wants to get from RPA.To provide insight into the benefits of RPA, and help the company/organization to fill the first column benefits, the authors have conducted a literature study on the benefits of RPA.Identified benefits are classified and entered into four different lists of benefits in the assessment model.The second column is target, where the company can enter how much (magnitude) the obtained benefits from the RPA project.Column three is the actual amount of the respective benefit, regarding the existing project.And the fourth column, measure, contains two things: (1) the differences or gaps between the target and the actual, and (2) comments ok or not ok.

MODEL TESTING
To clarify the assessment model, the authors tested the model on a small and medium-sized company in Germany.The company had implemented a 1-month trial of RPA in its inventory management.Although the RPA project is still a pilot project, the company owner does not mind applying the author assessment model to evaluate the project.Currently, the owner sees the existing assessment tools as too complicated and partial [20].The business analytics features in Automation Anywhere and UiPath can assess the RPA project, but it needs a lot of effort and is time-consuming to apply them.On the other hand, it is not enough for companies to assess the benefits of RPA only from financial benefits, because there are many non-monetary benefits obtained from the implementation of RPA.However, measuring nonmonetary benefits is not easy and requires in-depth observation of each individual in the company.For this reason, the owner of Flamm GmbH thinks that the assessment model offered is simple and logical because it only compares expectations and reality [20].This is also in line with the Expectation-Reality Gap study in SMEs, where the failure of a new method is when it fails to meet predetermined expectations [45].Although the object of our research is not a large company and only SMEs, the need for reliable and real-time inventory management is essential.The limited number of staff makes the company owner interested in using robotic process automation (RPA) solutions to solve problems in inventory management.Automated inventory management allows companies to plan to meet variable demands for components or materials.The Bot of RPA will notify managers when stock is below the level specified in inventory management.For example, when the current stock (CS) touches the reorder point (ROP) or extra safety stock (EES), it creates an alert and automatically places a new order, which is ready to be approved by the manager.With the above use case, a benefits list was created for the RPA project, which was then filled out by the company owner (see Table 2).From the assessment, there is a negative gap between expectations and reality, which is marked in red.This is what the company needs to pay serious attention to.They must find the cause and initiative steps to mitigate this potential failure (see Tab. 2).
The feedback from this test was that the company owners found the model uncomplicated and easy to understand.The model is suitable and sufficient for pilot projects and periodic project assessments.However, for final project assessments or feasibility studies, it should be combined with a detailed cost benefit analysis.From this model testing, the first proof of concept/model was achieved.More test have to be done to further verify the model but the first evaluation shows positive results and the model can be applied at companies to periodically monitor or assess their RPA projects so that the potential for failure (the gap between expectation and reality) can be detected and mitigated immediately.

CONCLUSIONS
The development of the RPA project assessment model is motivated by the need for a decision-making process to start, delay, continue or stop RPA projects quickly and accountable.The parameters in the assessment and model are the benefits identified in the literature study.According to model testing on an SME (section 6), the model is considered suitable for assessing pilot projects of RPA and periodic RPA project assessments because the model considers all important criteria and has the advantage of being quick and easy to use.For the final assessment of RPA projects or RPA feasibility studies, it is better if the model is combined with a cost-benefit analysis.
Future research is directed at testing the model more widely, on many SMEs to better understand its challenges and limitations.This will be done with more interview partners and also in comparing the model with other models.

Figure 2
Figure 2Balanced Scorecard example: Strategic map for a Jewellery store[28]

Figure 3
Figure 3 The Literature Survey Process To guarantee scientific rigor, relevant data have been collect following Brocke et al. (2015) study on Structured Literature Review (SLR)[29].Academic contributions related to the topic of RPA using the search query: "RPA OR robotic process automation" have been surveyed.Search queries have been kept deliberately generic to avoid excluding contributions that indirectly address the various benefits of RPA.In this way, 1,522 academic contributions

Figure 5
Figure 5 The Assessment Model of RPA Project

Table 2
Example of Using the Assessment Model in SMEs