Technology-based Practical Blockchain System Audit Maturity Model

Information system auditing can reveal the quality of such systems, and standard audit items are crucial elements of system and audit quality. Blockchain technology is currently being applied to various areas including the financial, manufacturing, healthcare, distribution, and public sectors, and an increasing number of systems that apply such technologies are also being developed.The current audit model is insufficient for application in the field, and the auditing of systems applying new technologies, such as blockchain, has not been given sufficient attention. Furthermore, it is difficult to evaluate the relative levels of audited systems using audit results. Existing studies have only examined the auditing of systems that apply blockchain. Although the Korea Association of Information Systems Audit has suggested a checklist for systems applying blockchain, it has yet to be adopted. To address this problem, 50 existing audit result reports and technical data were collected, from which sixteen factors of four audit quality properties consisting of blockchain system, technology compliance, software quality, and document were derived. Furthermore, an audit maturity model was presented after evaluating the priorities of the 16 derived factors. The results of the evaluation of the priorities of audit items indicated that auditors give a higher importance to technology-based than document-based audits of information systems. This study contributes to the literature by deriving field-oriented audit items including blockchain technology, thus enabling practical audits to be conducted in a short time. Further, this study enables the maturity of systems to be compared based on audit results by presenting audit maturity.


INTRODUCTION
An information system audit system has been implemented in South Korea, and public organisations are required to undergo an audit when they execute an IS development project that costs 500 million won (KRW) or more [1,2]. The National Information Society Agency (NIA) established the latest Information System Audit and inspection framework in 2013 and proposed standard audit items with which auditors are required to comply when they perform audits. However, the current standard audit items suffer from the following problems [3,25].
First, they do not reflect rapid changes in information technology (IT). For example, system developments that apply technologies such as artificial intelligence (AI), big data, and blockchain, are not considered as the main targets of standard inspection items because they have been actively applied only since 2013. Therefore, new audit items should be included to facilitate proper audits of system establishment projects that apply blockchain technology based on the current characteristics of the technology [3,23,24,25].
Second, the number of standard audit items is excessive. IS audits are generally conducted over a period of 5 to 10 days in proportion to the cost of the development of the system. As shown in Tab. 2, given that 68 audit items are examined in an object-oriented model, it is unlikely that the audit will be completed within the given amount of time.
Third, system development requirements such as conditions on software (SW) security codes and data standardisation, are not reflected in the standard audit items. The system development requirements are as follows:  Guidelines for database standardisation in public institutions (Article 50 of the Electronic Government Act and Article 59 of the Enforcement Decree of the Electronic Government Act) [2,   Compliance guidelinesfor compatibility with electronic government services (Article 50 of the Electronic Government Act) [2];  According to the guidelines for the construction and operation of an IS in administrative agencies and public institutions, to develop safe SW, 20 coding rules should be followed in the design stage and 47 coding rules in the implementation stage (Paragraph 3 of Article 45 of the Electronic Government Act and Articles 51, 52, and 53 of Chapter 6 of the Act) [2];  Compliance with standards on personal information according to encryption, destruction, and access record preservation (Paragraph 2 of Article 23, Paragraph 3 of Article 24, and Article 29 of the Personal Information Protection Act) and standards on technical and administrative personal information protection measures (Act on Promotion of Utilisation of Information and Communications Network) for personal information protection and system development [34].
Fourth, it is difficult to assess the level of IS audits and compare the level between different systems.
The results of a system audit are difficult to assess to make relative comparisons between systems because there is no criterion for determining the extent to which one system is superior to the other. In other words, relative assessment between systems is difficult.
To solve these issues regarding the standard inspection items, this study extracted the minimum unit of problems from audit reports published over the past decade and reclassified the minimum unit of problems into systemspecific characteristics, SW quality properties, documentation, and technical perspectives. The final 16 audit items were prioritised through the analytic hierarchy process (AHP) and grouped based on the proximity of these items to develop a blockchain system audit maturity model (AMM).
The scope of this research was limited to system development projects with permissioned blockchain technology, and the audit area was excluded from the project management area and limited to the application system, database, and system architecture areas. In addition, expert reviews involving techniques, such as secure coding, were included in the scope of the study and the permissionless blockchain system was not considered.
The structure of this paper is as follows. Section 1 describes the necessity of this study. Section 2 discusses previous research related to IS audits and details of blockchain technologies and AHP to provide background information. Section 3 presents the design of the AHP model. Section 4 describes an analysis of the AHP model and its results. Section 5 details the proposed blockchain AMM based on the AHP analysis results. Section 6 discusses the value of this study and future work. Finally, Section 7 summarises the findings of this study and presents concluding remarks.

BACKGROUND KNOWLEDGE AND RELATED WORKS
This section introduces concepts related to IS auditing, blockchain technology, and AHP that may be unfamiliar to help readers better understand this paper and briefly reviews existing studies on IS auditing.

Background
This section covers the IS system audit introduction and procedures, the IS audit framework, and audit inspection items. In addition, the concept of blockchain technologies, DLT-Reference Architecture, and AHP are briefly introduced.

Information System Audits
An IS audit is conducted in three stages-requirements definition, audit design, and termination.
In principle, the scope of IS audits is established according to the details of the project types and audit points stated in the Information System Supervision and Inspection Framework v3.0. As shown in Fig. 1, the Framework consists of three pillars based on the conceptual model: business type/audit point, audit area, and inspection standard.
When auditing the application system area of the analysis/design phase, auditors perform the audit according to the detailed inspection items in Tab. 1 below [1].
The first heading in Tab. 1 is the code, which refers to business types, development models (structural/information engineering model 1, objectoriented/component-based model 2), audit points (requirements 1, design 2, implementation 3), and audit areas (system architecture 1, application 2 and database 3). Therefore, Tab. 1 is a system development project type, SW development methodology is an objectoriented/components-based model and is a detailed inspection item used to check the application system area at the time of completion of the implementation phase.

Blockchain Technologies
A blockchain is a decentralised distribution system that has technical characteristics that differ from existing centralised transaction systems in the following aspects.  Encryption: Transactions recorded in a blockchain are encrypted based on the pairing of public and private keys.  Real-time procedure: Transactions are stored in a blockchain as soon as they occur. Therefore, a reconciliation of accounts can be applied based on realtime transaction records.  Smart contract hosting: A blockchain includes both a programming code and a smart contract. These programs are used to execute a transaction and form a corresponding ledger item when a certain contract condition is satisfied.  Blockchains are classified into permissioned blockchains that can be used by everyone and include bitcoins and permissionless blockchains that can be accessed only by certain participants. In permissionless blockchains, participants manage the same blockchain and continuously synchronise all copies to ensure the transparency, accuracy, and recentness of data and reach an agreement. In permissioned blockchains, only certain participants can obtain partial copies. Moreover, certain participants and information can be restricted according to the access control configuration.
According to the TTA technical report [35], the reference architecture of distributed ledger technologies (DLT) consists of seven layers: infrastructure, DLT platform, interface, DLT application, non-DLT systems, other-DLT system, and the operation management layer. The following figure shows the layer and function elements for DLT-reference architecture.

The Analytical Hierarchy Process
The AHP proposed by Saaty [6] is a structured technique that is used to analyse complex decisions. The AHP compares each factor in pairs and then estimates its importance [7]. As the AHP is based on pairwise comparisons and can identify the most significant factor, it can reflect the intuitive, rational, and irrational decisions of experts. However, such comparisons require an individual's intuitive decisions; therefore, the consistency of the survey must be reviewed. The AHP includes the consistency index and the consistency ratio (CR). In general, a survey is considered to be valid if its CR is less than 0.1 [7]. This study devised a new matric analysis method to utilise in the audit model in order to analyse the AHP factors from multiple angles.

Related Works
Research topics on IS audits can be classified into three main categories: determining the priority of each audit, audit automation, and audit model proposals for technologies not in the standard IS audit framework. However, scarce research has been conducted on audit models for systems using blockchain technologies.

Research on Problems of Existing Audit Systems
The NIA recognised that the existing audit system was inefficiently operated regardless of the size and types of information-oriented projects and that a method for enhancing the system should be developed. Thus, the NIA introduced an audit reference model (plan) in a report published in 2016.
In this report, the following reasons highlighting why existing auditing systems are inefficient were discussed. Audits are difficult to conduct because the inspection items are not specific. The system development project consists of information engineering/structural and objectoriented/component-based models, which do not reflect the three-step audit of the task performance activities and IS audit standards stipulated in the Enforcement Decree of the Electronic Government Act. It is difficult to determine whether the document is identical or similar because the name of the document is given in a random and inconsistent manner. Finally, it is used as an unofficial individual guide other than the six types of businesses included in the current audit framework.
To address these problems, a new Supervision and Inspection Framework was proposed in 2016 as a business type-based inspection system and a business characteristics-based inspection system [8]. However, the audit model is yet to be changed as of 2020. Fig. 3 shows the revised Supervision and Inspection Framework.

Figure 3
Revised audit and inspection framework [8] As indicated above, inspection items and guidelines have failed to keep pace with rapidly changing IS technology. Moreover, problems related to the inspection items in the existing Supervision and Inspection Framework have not been analysed sufficiently. Therefore, this study first examined problems in applying existing inspection items to the latest IS technologies.
Standard audit items according to system development project types are shown in Tab. 2. Two types of inspection items based on development methodologies are objectoriented and component-based models and structural and information engineering models. This study reviewed inspection items during the stages of requirement analysis, analysis and design, and implementation for objectoriented and component-based models. The audit areas were limited to application systems and databases. A total of 26 inspection items were considered for application systems, including five during the requirement analysis stage, 15 during the analysis and design stage, and six during the implementation stage according to the lifecycle stages. Among them, 17 inspection items were considered for databases, including three during the requirement analysis stage, 10 during the analysis and design stage, and four during the implementation stage according to the lifecycle stages. Test activities 9 Operation preparation 4

Studies on Information System Audit Item Extraction
Previous studies of audit item extraction have mainly focused on system characteristics and system development methodologies that have not been considered in the existing audit framework. These studies include the following:  A study [9] on the improvements of an IS-based audit model for Agile testing conducted in 2014 by TaeyongEom;  A study [10] by Dong-ah Park and Mangon Park on the quality improvement of IS auditing for Agile Methodology published in the Journal of Korea Multimedia Society in 2017;  A study [11] on inspection item identification based on Structured Query Language (SQL) for ensuring the timeliness of the data quality and a study [9] on auditing focusing on choice and concentration through an analysis of inspection item prioritisation;  A study [12] proposing enhanced review items for increasing the quality of IS development projects based on the data of IS audit projects and risk occurrence projects conducted for four years;  A study [13] examining the current database security auditing and proposing a new database security audit framework and a study [14] analysing database error types and domain integrity and proposing a database quality certification system for increasing data quality; and  Studies on data modelling suggested standardised audit conditions proposed based on data models.

Studies on Analysing Inspection Item Prioritisation
Studies on analysing the inspection item prioritisation include the following:  A study [15] by Taewon Kyung and Sangkuk Kim proposing system audit evaluation standards considering the difference in perspectives from audit request groups, audit groups, and audit target groups, and numerically prioritizing the developed indices based on the Fuzzy-AHP method;  A study [16] by Boohyung Lee and Heejoon Cho on analysing the priority of basic audit inspection items and their relevant weights based on the hierarchical decision model (HDM) and a constant-sum method to provide a valid foundation for adjusting and changing the audit standards and increasing audit quality;  A 2013 study [17] by Boo-Hyung analysing the priority of inspection items and the weights between them based on the standard audit checklist through the application of AHP and MOGSA decision-making techniques for increasing audit quality according to changes in the standard audit checklist.

Studies on Audit Automation
Audit automation tools were used in a study [4] on SQL tuning [18] and a study [5] on data quality analysis. These studies analysed the database structure, inspected the effects of logical and physical data consistency and the data structure on the performance of SQL tuning and a data quality analysis, and proposed measures to improve performance through the application of audit automation tools.

RESEARCH DESIGN AND METHODOLOGY 3.1 Research Design 3.1.1 Software Quality Properties
ISO 9126 is an international standard that defines the characteristics of SW quality and the metrics for an SW quality evaluation, given that users, evaluators, and developers require guidelines for evaluating the quality of SW products. Fig. 4 below indicates six quality properties according to ISO/IEC 9126.

Determination of Audit Quality Properties
For empirical audit activities, it is important to analyse problems and improvements derived from many existing audit results reports rather than standard audit items to determine audit quality factors. This study on blockchainbased audit methods classified four elements of the upper level of audit quality factors: blockchain characteristics, documents, SW quality, and technology, which are referred to as audit quality properties.
The audit quality factor is determined by grouping the problems of the minimum units extracted from the audit results report into properties, and the audit quality factors are included in the audit quality properties.
Finally, 16 audit factors according to audit quality properties were obtained, as shown in Tab. 4.

Research Methodology
The methodology of this study is presented in this section. Fig. 5 shows the proposed model composed of three stages. In the first stage, 50 IS audit result reports that were published within the last 10 years were collected. The criteria for selecting audit reports were defined as follows: the audit target is the IS development project, and within the audit report, the audit area selected an application, database, and system architecture. The audit point included the requirement phase, the design phase, and the termination phase. The project size is more than 1 billion won. These reports were analysed to extract the terms they employed. Accordingly, an ontology applying a bottom-up approach was formed based on the extracted terms. Moreover, specific improvement suggestions were enumerated and grouped according to topic. Audit quality factors based on the grouping results were then derived and selected. An expert survey applying the Delphi method was conducted, and the survey results were considered for audit quality factor selection. In the second stage, the four quality factors selected were subdivided through a topdown approach. Subsequently, an AHP hierarchical model was established and a survey was conducted. During the third stage, the priority of consistent measurement data was examined, and the prioritisation results were re-classified. During the final stage, 16 factors were classified according to weight, and a five-stage AMM based on the classification results was proposed.
Stage 1: extraction of system quality properties based on blockchain [19]: The following items are extracted from audit result reports.
A1.extraction of an ontology applying a bottom-up approach A2.extraction of audit items according to audit quality factors The following shows pseudo-code for STAGE 1.

Completion of questionnaires and initial verification
The following shows the pseudo-code for the algorithm in stage 2. Final: Five-stageAMM: The five-stage AMM was proposed by grouping 16 factors based on similar prioritisation scores.
The following shows the pseudo-code for the algorithm in the stage 4.

An Ontology Extraction Method Applying a Bottom-Up
Approach [19] An ontology refers to a set of concepts based on words used in a certain field. It has been used in the fields of machine translation for natural language processing and AI. It has recently received significant attention as a method for Internet resource management in a nextgeneration web environment called the semantic web [26][27][28][29]. The stages for extracting an ontology applying a bottom-up approach are as follows.
Stage 1. Determination of the ontology domain and range: This study established the fields of application systems, databases, and system structure as the domains of an IS audit ontology. The range of the ontology was limited to words used in audit result reports except for standard audit items for an ontology formation. Stage 2. Core word extraction: Core words used in the IS audit field were extracted. Examples of words extracted are as follows: secure coding, essential, recommendation, short-term, long-term, importance, status and problems, improvement direction, application system, database, system structure, project management, quality management, general review, improvement recommendation, web compatibility, web standard, personal information protection act, encryption, data structure, domain integrity, and web vulnerability.
Stage 3. Arrangement of repeated or ambiguous words and identification of synonyms or similar words: During this stage, repeated or ambiguous words were identified. Synonyms or similar words were also noted, as shown below.  Fig. 6 presents these classes. Classes derived based on core words were grouped to form a hierarchy of classes. Fig. 7 and Fig. 8 show the hierarchy of the classes formed.
Stage 5. Definition of the hierarchy structure: The ontology was completed by defining the relations and properties of the classes. The relations of synonyms and similar words were also defined. Concepts were connected to each other based on their interrelations in the ontology.

Analytic Hierarchy Process Hierarchy Model
The hierarchy model used for the AHP analysis consists of three stages, as shown in Fig. 9. The ultimate goal of this study, that is, the development of empirical audit measures for blockchain-based systems, was achieved in the first stage. In the second stage, four factors of blockchain characteristics (B), technology (T), SW quality properties (S), and documents (D), which are a hierarchy of classes derived through the process of ontology formation, were included. In the third stage, 16 sub-factors of four properties shown in the second stage were applied. These sub-factors included blockchain characteristic factors (B 1 , B 2 , B 3 , and B 4 ), technology factors (T 1 , T 2 , T 3 , and T 4 .), SW quality factors (S 1 , S 2 , S 3 , and S 4 ), and document factors (D 1 , D 2 , D3, and D 4 ).

Data Collection and Measurement
An expert survey was conducted based on 60 questionnaires distributed to experts working in fields related to an IS. Descriptions of the survey method were provided to enable survey respondents to independently analyse and answer the questionnaires. Survey respondents included industrial and academic experts who represent their respective IS fields, such as professional data processing engineers, IS auditors, and blockchain experts.

Survey and Consistency Verification Results
An expert survey was conducted based on blockchain experts, audit ordering entities, and IS auditors for three weeks from October to November 2020. A total of 60 questionnaires were distributed, 56 of which were collected. After the first survey, 29 questionnaires that maintained logical consistency in all five groups were used for the analysis of the results. With regards to the questionnaires that did not maintain logical consistency, most experts who completed the questionnaires expressed consistent opinions in their professional fields. However, some experts provided inconsistent responses on the audit property group, which was not their professional field. Thus, this study selected only 29 questionnaires that maintained logical consistency after the first survey under the assumption that those who completed the selected questionnaires were professionally informed about IT in general. Tab. 6 indicates the number of questionnaires distributed and collected as well as the status of consistency verification. The consistency ratio (CR) of a survey refers to the reliability of responses during the AHP analysis. Saaty [20], who developed the AHP analysis method, stated that responses can be evaluated as valid only when their CR score is 0.1 or less. Therefore, the CRs of the questionnaires were examined, and only questionnaires with CRs of 0.1 or less were used in the analysis.

Importance and Priority of Analysis Results
A three-stage hierarchical model including four properties (i.e.,permissioned blockchain, technology, SW quality, and document (BTSD)) and 16 factors in its lower class was established, and the development of empirical audit measures for blockchain-based systems was set as the ultimate goal [22]. The hierarchical model consisted of five 4 × 4 matrices and 30 paired comparison questions. Each matrix included four components, and vector values (r) were generated for six comparison pairs. Eq. (1) shows paired comparison matrices M1 and M2 formed during the second and third stages of the proposed model.
The results of the analysis of the importance of the components in the second stage indicated that the expert group evaluated the importance of the BTSD properties as 0.32519 (32%), 0.360074 (36%), 0.214815 (21%), and 0.102296 (10%), respectively. Although this study focused on the audit of blockchain-based systems, the audit quality properties of technology accounted for a higher proportion than that of the audit quality properties of blockchain characteristics. By contrast, the audit quality property of documents accounted for the lowest proportion. The results of the analysis of the importance of the components in the third stage indicate that among the technology factors, T 3 is most closely related to the personal information protection act, showing the greatest importance of 0.1626, followed by B 1 among the factors of blockchain characteristics, showing an importance of 0.1527; S 1 among the factors of SW quality, showing an importance of 0.1123; T 2 among the technology factors, showing an importance of 0.0833; B 2 among the factors of blockchain characteristics with an importance of 0.0783; and data standards (T 1 ) among the technology factors with an importance of 0.072. Thus, two factors (B) of blockchain characteristics, three factors (T) of technology, and a factor (S) of SW quality were included as six upperclass factors with an importance of 0.070 or greater. In particular, technology factors, such as encryption, were found to exhibit significant importance. By contrast, three document factors were included in four lower-class factors and exhibited a significantly low importance.
The results of the analysis were presented based on the importance (G-weight) of the upper-class factors applying the weight and importance (L-weight) of independent factors according to the BTSD properties. Tab. 7 and Fig.  11 indicate the importance of the analysis results.

Figure 11
Relative importance for IS audit factors This figure was rotated and lines were drawn on the weight closest to the corresponding factors, as shown below.

Importance and Priority of Analysis Results
The following table contextualises our work by comparing it to other studies. We commissioned five experts, who evaluated our study as excellent in all four criteria.

BLOCKCHAIN-BASED SYSTEM AUDIT MATURITY MODEL
To develop the AMM for blockchain-based systems, the result of the analysis of the AHP model was classified into five levels based on K-means clustering. Tab. 5 shows the final result.
The priority of factors derived through the AHP method was classified into five groups. A value closest to a certain weight was established as a classification standard.
Consequently, the set of factors close to 0.16 {T 3 , B 1 } was classified as Level 1, the set of factors close to 0.12 {S 1 } was classified as Level 2, the set of factors close to 0.08{T 2 , B 2 , T 1 } was classified as Level 3, the set of factors close to 0.04 {B 3 , D 1 , T 4 , S 2 , B 4 , S 3 } was classified as Level 4, and the set of factors close to 0.02 {S 4 , D2, D 3 , D 4 } was classified as Level 5. The following table summarises the results.

DISCUSSION
In summary, three lower-class factors among the 16 factors of audit quality properties for blockchain-based system audits were found to be document-related factors exhibiting insignificance. Three technology factors, two blockchain characteristic factors, and a SW quality factor were evaluated as upper-class factors in the order of T 3 (technology related to the Personal Information Protection Act), B 1 (node configuration), S 1 (function accuracy), T 2 (secure coding), B 2 (MSP setting), and T 1 (data standards). Technology factors also exhibited significance.
These results indicate that IS audits should focus on technology-based inspection that applies technologies and blockchain characteristics rather than document-based inspection.
However, although IS auditors placed importance on technology-based audits, auditors actually performed document-based inspections in the field. This indicates that document audits are also important; audit items should include all factors.
A maturity model was created using the results of the AHP technique and was reviewed by five experts who evaluated the model as appropriate. If there are numerous problems in the document, the maturity of the system can be judged as low. More research is required regarding whether the system can be evaluated as a well-founded system if fewer problems are found in the technology audit.
The audit level of a blockchain-based system can be determined according to its maturity level. It is necessary to study the causal relationship to determine whether the maturity of the audit can represent the maturity of the system when it is audited.
The numbers in Tab. 2 represent the numbers of codes in Tab. 1. In this study, the contents of Tab. 1 were changed to those used in the field and the numbers were simplified.
In some cases, duplicate inspections were necessary because there are many inspection items in Tab. 2, and the contents of Tab. 1 do not need to be inspected or their boundary with other items is ambiguous. The number of inspection items was reduced by removing unnecessary inspection items and grouping detailed items by inspection unit. The core matters can be audited in a short time by using audit items that have been used in the field. It can thus be considered that audit efficiency has improved.
Future studies must investigate the automation of audit items so that inspection items can be continuously improved and quickly applied to the field based on the audit reports applied to the field. If audit items can be automated, the audit model and audit maturity can be updated quickly even if new technologies emerge.

CONCLUSION
The development of systems applying blockchain technology is increasing. However, as the current IS audit model is inefficient and inappropriate for auditing new technologies, such as blockchain, auditors create and use informal audit items.
There is a need for an audit model that is applicable to new technologies like blockchain and can audit core matters within a short timeframe.
To address this issue, problems were extracted from technical data and 50 audit reports and regrouped to create 16 audit items of four audit quality properties (blockchain, SW quality, technology,and document). Their priorities were then determined using the AHP technique. A fivestage AMM was then proposed by grouping the audit items based on the results.
This study proposed practical audit items that can contribute to the stabilisation of ISs. Furthermore, this study proposed audit maturity that can be used to evaluate systems based on audit results.