Malicious Behavior Detection Method Using API Sequence in Binary Execution Path

: Today, the amount of malware is growing very rapidly, and the types and behaviors of malware are becoming very diverse. Unlike existing malicious codes, new types or variants of malicious codes are being identified, and it takes a lot of time to analyze all malicious codes. To solve these problems malware analysts analyze and research effective ways to reduce analysis time and cost. In this paper, we propose a method to express characteristics and detect malicious codes by using API Sequence for malicious code detection and classification. It compares and analyzes several existing expression methods and verifies the effectiveness through actual malicious code samples. Using the expression method proposed in the paper, we detected six malicious behaviors: DLL Injection, Downloader, IAT Hooking, Key Logger, Screen Capture and Antidebugging. As a result, more detection was detected than by conventional detection methods, and it can be seen that the more complex the malicious behavior, the higher the detection efficiency. In addition, static analysis was adopted as the main method, but because it searches execution compression, the flow of malicious behavior can be analyzed.


INTRODUCTION
In 1949, Von Neumann published a theory that selfreplication and proliferation can be performed in computers as with biological viruses [1]. Thereafter, those computer programs that are equipped with self-replication and proliferation functions started to be called "virus". However, since the term virus is quite limited in expressions, a concept that can comprehensively express the concept became necessary. The term made consequently is malware, which is an abbreviation of malicious software, which means soft-ware that contains malicious codes. Recently, with the rapid development of network and ICT technologies, the amount of malware has been increasing exponentially. To prevent and respond to such security threats, analysts and anti-virus program producers are trying to maximize the efficiency of malware analysis using various analysis methods.
In general, malware analysis is performed in two formats, static analysis and dynamic analysis [2]. First, static analysis is a technique to detect malicious behaviors by analyzing the structure of malware or specific binary patterns at the code level. Although static analysis enables more in-depth and detailed analysis, if technologies that obstruct static analysis such as executable file packing and code obfuscation are applied to malware, much time and effort will be required and the analysis will become considerably more difficult [3]. Second, dynamic analysis is a method of analyzing malicious behaviors by executing actual malware in a virtual machine. This method is advantageous in that malicious behaviors can be clearly observed even when executable file packing or code obfuscation has been applied to malware because malware is actually executed for analysis. However, it is not suitable for analyzing trigger-based malware that runs at a certain time or is executed when the user's specific action is taken.
In this paper, a malicious behavior detection method using static analysis based execution path searches was proposed to maximize the efficiency of malicious behavior detection. It aims to improve detection efficiency by detecting the execution path through static analysis and determining malicious behavior based on the correlation of APIs. Although the main analysis method is based on static analysis because it is conducted at the code level, the effects of dynamic analysis can be expected because it traces the binary execution path to grasp the behavior. Even if the malicious code does not function completely due to problems such as dependencies or has a packing problem, analysis is possible because it targets various branches in the binary code. In addition, the API behavior information in the analysis results will be visualized with graphs for clear under-standing of malicious behaviors and the excellence of the malicious behavior detection method proposed in this paper will be proved through comparison with the existing simple API collection and listing methods.
In Chapter 2, the methods of malicious behavior detection in previous studies are introduced and the limitations of those methods are mentioned. In Chapter 3, the method proposed in this paper is introduced. In Chapter 4, experiments are conducted based on the proposed method, the results are verified, and the efficiency of the method is mentioned and in Chapter 5, this paper is finished with conclusions.

RELATED WORKS
This paper checks the process and limitations of previous malware analysis research. It improves the limitations of the analysis method by referring to existing studies and suggests an efficient analysis method. Previous binary code based static analysis studies have been carried out by extracting the features of attributes in codes. Statistical algorithms were generally grafted on such studies and utilized for analysis. Such statistics are mainly utilized for comparison with normal programs. For instance, the statistics of op codes [4] or strings [5] extracted from the malware code sections are compared with those of general normal programs for utilization in analysis. These methods simply collect signatures and identify malicious behaviors by extracting information on the structures of executable files [6]. The most basic methods among the sequence-based malicious behavior identification methods mentioned above are those that list the sequences of op codes [7] or the sequences of strings [8]. These studies have been developed to carry out studies that identify malicious behaviors by using the n-gram technique [9], which cuts the information in the file according to a certain standard and processes the cut pieces of information. In addition, studies intended to express the byte sequences for binary codes with n-grams with a view to classifying malware were also carried out [10]. Such methods of collecting signatures for the internal structure of a file can be defined with the unique DNA of the file [11] and are used for similarity and classification of malware based on the foregoing. A clear and definite basis for judging malicious behaviors is the discovery of the functions used by malware. Previous studies have attempted to identify or classify malware by processing such APIs within programs. [12][13][14] First, methods that list the sequences of APIs [15], or collect log information on the use of APIs to determine malicious behaviors [16] are representative. Since APIs are functions used when the program is executed, such APIs are either statically collected [17] or dynamically monitored [18]. In addition to the methods that simply list the APIs, there are methods that extract the features of malware according to the frequency of use of the APIs inside the file [19]. The above studies are statistical methods that have advantages such as not so large amounts of data to be stored, small amounts of operation, and high speed. However, they cannot respond to malware in real time and cannot accurately judge diverse malware behaviors because they are based on simple statistics [20]. To compensate for the foregoing, some studies carried out recently grafted various algorithms onto the statistical properties as such to detect malicious behaviors. The eigenvalues of op code based graph images can be calculated by measuring the distances between the nodes based on the K Nearest Neighbor Algorithm (KNN Algorithm), which is one of the machine learning algorithms [21]. In addition, the processed strings can be reprocessed with the Logistic Common Subsequence (LCS) algorithm to measure the eigenvalues of the strings [22]. In the studies introduced above, static analysis-based methods collect signatures or list the signatures in sequence, but cannot identify the accurate features of behaviors because they are based on code-based feature extraction. To compensate for this this problem, dynamic analysis is adopted as the main detection method [23] or a mixture of static and dynamic analyses is adopted [24]. However, since dynamic analysis is a method that directly executes malware for analysis, it has disadvantages of energy efficiency and analysis time [25]. In addition to this, several studies and methods are under way to classify malware [26,27]. In our previous study, we studied how to express the features of malware using APIs [28]. In this paper, a method that is based on static analysis but tracks the execution flow was proposed so that the effect of dynamic analysis can be expected to compensate for studies in which static and dynamic analyses are mixed.

PROPOSED METHOD 3.1 Method Architecture
First, the malware executable file is converted into disassembled binary codes through the binary reverse engineering tool, IDA [29] (IDA PRO 6.6). The IDA extracts the disassembled code for the executable into .asm and uses it as data for static execution path search. Since malicious behaviors are determined by the API call sequences found in the execution path search, an API extraction process is undergone. In this study, graphical imaging is performed based on APIs. However, since not all the extracted APIs can be applied to imaging because they amount tens of thousands in kind, they are made into graph images through classification to represent malicious behaviors, and finally, the mutual similarity relations of pieces of malware are shown through image based similarity determining work.

Static Execution Path Exploration
The core of the static execution path search in this paper is that the instruction set and subroutine are divided into true and false ones according to the branch instruction before they are searched. First, among the assembly instructions, the instructions for branching are jz, jzr, and so on. As for the branch point, comparison instructions such as cmp and test that occur before the branch instructions are issued, are made through logical operation instructions such as xor. We divide true and false marks according to the branch instructions to search all instruction sets and subroutines.
The IDA's disassembly codes can identify the instruction sets and subroutines used in the search. The instruction sets, which are functions that perform some behaviors in the functions, are represented by loc_xxxxxx, and the subroutines or basic blocks are represented by special prefixes such as sub_xxxxxx. The IDA provides IDAPython, which is a python script, to provide powerful processing activities for binary codes. In this study, searches were performed using IDA APIs such as GetFunctionName, CodeRefsFrom, and CodeRefsTo [30]. Tab. 1 is the binary code for displaying static execution path searches. After CMP instruction in the fourth line, the binary code loc_401460 is branched into the instruction sets of loc_40148 due to the JNZ in the eighth line, which is a branch instruction. If the result of the comparison is true, the binary code will be branched into loc_401488, and if false, into loc_401484. To summarize finally, loc_401460 is a binary code, which is branched into loc_401488 when it is true and into loc_401488 when it is false.
Visualizing the binary code in Tab. 1 will look like  This mechanism is applied equally even when there are subroutines in the instruction set. For example, if there is a subroutine in the instruction set as with the binary code shown in Tab. 2, the subroutine is entered and the above execution path search is performed. The relevant binary code is visualized as shown in Fig. 2.

Processing of the RGL Scheme
In this study, malicious behaviors are detected based on Windows APIs. APIs are predetermined functions provided by the operating system or the programming language so that applications can use system resources or libraries. Applications use APIs for the purpose of using system resources or interacting with other applications and APIs are called from the program. Windows applications use Windows APIs in most cases and the APIs are included in dynamic libraries (DLLs). Those pieces of malware that are executed based on Windows also use the Windows APIs, and information on the APIs can be used as good information to determine malware behaviors.
In previous studies, pieces of malware API information were simply collected based on signatures or simply listed as with n-grams to detect the similarity and behaviors of malware. These methods are efficient for simple classification of pieces of malware and the detection of the variants of the relevant pieces of malware because they simply list the APIs but they have a shortcoming that they cannot accurately detect the behaviors of malware.
In this study, malicious behaviors will be detected based on the API information discovered during binary execution path searches according to the proposed method. Tab. 3 is an example of binary codes used to examine the behaviors of APIs in static execution path searches. The corresponding code branches from loc_4017AA4 into loc_41A5DD when it has been found to be true through the comparison instruction on the seventh line and loc_41A5DD branches into loc_428986 when it has been found to be true through the comparison instruction. The 15th through the 21st lines are the same search process as the one shown above and loc_4268DE is defined as a normal mark instead of true or false mark because the subroutine sub_488930 is simply called in line 23 without any comparison instruction. The mutual relationship between the instruction set and the subroutine of the binary code in Tab. 3 can be visualized as shown in Fig. 3 using the method proposed in this paper.
From the binary code in Tab. 3, it can be seen that Windows API functions such as CreateMutexA, GetLastError, and CloseHandle are called in lines 4, 6, 11, 17, and 26 respectively. In this paper, the Normal, True, and False mark application mechanism is equally applied to APIs to analyze the interactions between the APIs and the behaviors of the APIs.   Since there is no instruction for branching by any comparison instruction between lines 4 and 6, the interactions are marked as normal between CreateMuetexA and GetLastError. Since the interaction between the instruction sets loc_417AA4 and loc_41A5DD is marked as true by lines 7 and 8, the interaction between GetLastError, which is called last in loc_417AA4, and GetLastErrorhe, which is called first in loc_41A5DD, is marked as true. One thing noteworthy is the interaction between EnterCriticalSection and LeaveCriticalSection. Even though it is marked as true in loc_428986 by loc_4268DE and loc_4268DE is marked as normal by sub_488930, since there is no instruction to call any API in loc_4268DE and no API is called between EnterCiritcalSection and LeaveCriticalSection, it is marked as true. Finally, only the interactions between APIs are visualized as shown in Fig. 5.

API Classification
In this study, malicious behaviors are visualized in the form of graphs expressed with nodes and intermediate lines as shown in Fig. 5. However, there is a problem that the number of APIs is too large to make the APIs into nodes. In this study, to solve such problems, the APIs will be reclassified into 24 upper categories through the functions of the APIs so that behaviors can be clearly judged and the temporal efficiency can be enhanced [31,32]. For instance, CreateFile and CreateProcess are APIs that perform functions related to "files" or "processes" and APIs such as GetSystemTime and GetLocalTime have the function to collect information on "time" in the system. In addition, APIs such as strcmp and stcat all perform functions related to strings. Such a classification not only has many categories to which APIs commonly belong although they have been already classified in MSDN but also is too abstract to understand behaviors. For instance, all processrelated APIs are included in the category process but whether the relevant APIs created, deleted, or accessed processes cannot be known. Therefore, the functions of APIs were reclassified into three, which are CREATE_OR_OPEN, READ_OR_ACCESS, and CLOSE. Tab. 4 shows the final 24 API categories. However, not all APIs are reclassified into four behaviors. Since APIs such as Strcat and strcmp do not perform the function to create or access strings separately, APIs related strings are determined to be in a single category, and APIs related to "time" such as GetLocalTime and GetSystemTime are included in the category Time despite the fact that they access the system to obtain time information, because the behavior "Time" has the most important value in the identification of malicious behaviors.

EXPERIMENTS AND DISCUSSION 4.1 Malicious Behavior Detection
In this study, malicious behaviors are detected based on the interactions between APIs using binary static execution path searches. Previous API sequence based static analysis studies had a shortcoming of being unable to accurately understand malware behavior because they simply listed or collected APIs. However, the method proposed in this study enables the understanding of the interactions between APIs because it uses execution path searches despite the fact that it is a static analysis so that the effects of dynamic analysis can be expected.
As a representative example, when the malicious behavior of Trojan.Graftor.D4C56B has been analyzed by the method proposed in this paper, the graphic image shown in Fig. 6 appears. Fig. 6 shows that the malware uses APIs such as String, SystemInformation, Module, and Process. In particular, a detailed analysis of the red shaded API behaviors is as follows. In light of the fact that the relevant APIs use processes such as OpenProcess, Process32Next, WriteProcessMemory, and VirutalAllocEx and APIs used for manipulation of dlls, the API behaviors can be confirmed as DLL injection that inserts code into the remote process of calling LoadLibrary to forcibly make the DLL to be loaded into the context of the relevant process.

Comparison with Dynamic Analysis
With regard to the behaviors shown in 4.1, the existing simple API collection and listing method, the API Monitor based [33] dynamic analysis, and the method proposed in this paper are compared as shown in Tab. 5.  The previous sequence listing method can be shown to have API sequences different from those that appear in the dynamic analysis that directly executes APIs to analyze. However, the API sequences in the method proposed in this paper can be identified to be similar to those appearing in dynamic analysis.

Common Graph of Malicious Behaviors
In this study, common graphs of representative malicious behaviors (Dll injection, Downloader, IAT Hooking, Key Logger, Screen Capture, Antidebugging) were identified as shown in Tab. 6 in the same method. Tab. 6 shows the interactions of the behaviors of all APIs. These actions of behaviors are shown after being combined by the categorization of APIs as shown in Tab. 7 for the clarity of analysis methods and the efficiency of analysis time.

Efficiency
DLL injection, downloader, IAT hooking, key logger, screen capture, and anti-debugging. The studies being compared are those that simply collect or list op codes or APIs such as OPCODE, N-gram and API sequence. In this paper, we show the efficiency of the proposed method compared with previous simple signature information collections. Tab. 9 shows seven methods to simply list or collect code-level information such as OPCODE, N-GRAM, BYTE CODE, API, string and etc.
In addition, an example of applying each of the methods to Tab. 8 code is also shown.
The test set is 1236 pieces of randomly generated malware and all of them include an IAT (Import address Table) because the method proposed in this paper analyzes the interactions between APIs. First, through the identified common behavior graphs, each malicious behavior was analyzed based on the data set consisting of 1,236 pieces of malware.
Figs. 7 to 12 are graphs comparing the method proposed in this study and the existing method. It can be seen that the proposed methods show larger numbers of detection of the malicious behaviors, DLL injection, IAT Hooking, Screen Capture, and Anti Debugging when compared to the existing detection methods.   However, it can be seen that the method proposed in this study shows very similar numbers of detection of other malicious behaviors such as Downloader and key logger when compared to the existing detection methods. This is because the relevant behaviors perform API interactions that are too simple to make the relevant behaviors to be judged to be malicious. In other words, the two malicious behaviors, Downloader and Key Logger are composed of two nodes, and the marks of the intermediate lines that show the interactions are True, so that only one sequence of each of the relevant behaviors is identified. This means that the structure of the sequence is too simple to detect malicious behaviors, which is the reason why the accuracy of detection is lowered. However, it can be seen that the more complex malicious behavior, the higher the detection efficiency.  The complexity of malicious behaviors in this study paper may also be regarded as the complexity of sequences. The efficiency of the method proposed in this paper was shown to be 105% compared to existing studies when the complexity of sequences was low because the grounds for detection are reduced when the complexity decreases. On the contrary, the efficiency of the method proposed in this paper was shown to be 158% compared to existing studies when the complexity of sequences was high because the grounds for detection increase when the complexity increases. This means that the more complex the malicious behaviors, the higher the efficiency of detection.

Binary Classification Result
When compared to previous studies, the method proposed in this paper did not show high detection rates for behaviors such as Downloader and KeyLogger. This is because the relevant behaviors conduct simple API interactions so that grounds for judgment as being malicious are insufficient. However, it can be seen that the higher the complexity of malicious behaviors, the higher the detection efficiency of the method proposed in this paper. These results can be proved based on the accuracy and f-measure values based on the wrong detection rates and detection missing rates of existing studies.
In this paper, we compared the proposed method with previous studies and binary classification results. Since this paper is an API sequence based on static analysis, all previous comparative studies are based on static analysis. [19,34] used API frequency, and [35,36] used API sequence as the main method. All the accuracy and fmeasure values of the method proposed in this paper were measured to be higher compared to previous studies. In addition, the pieces of malware detected by the method proposed were identified to show an average Virustotal [37] detection rate of 69%.The summary of the contents can be found in Tab. 10.

CONCLUSIONS AND FUTURE WORK
In this paper, a method to detect execution paths based on static analysis and judge malicious behaviors based on APIs' interrelationships was proposed. Although static analysis is the main analysis, the method proposed in this paper enables analyzing the flow of behaviors because it searches execution paths. This means that although static analysis is adopted as a main method, the advantages of dynamic analysis that directly executes APIs to analyze the APIs are applied to the method proposed in this paper. In this study, execution flows were analyzed according to branch instructions and the interactions of APIs collected during the flows were analyzed. API interactions are marked as normal, true, and false and are reclassified into and listed as 24 upper categories. In this study, the detection method based on the relevant method was compared with the existing simple API collecting method and API listing method. The malicious behaviors used for the comparison are six behaviors, which are dll injection, downloader, IAT hooking, key logger, screen capture, and anti-debugging. The method proposed in this paper showed high efficiencies in the discrimination of four behaviors among the six behaviors except for downloader and the key logger. This is because the API interactions of downloader and key logger are insufficient for judgment of the behaviors as being malicious. This is related to the complexity of malicious behaviors. As malicious behaviors became more complicated, higher efficiencies of detection appeared because the grounds for judgment of malicious behaviors became more sufficient. In future studies, the frequencies of behaviors will be added to prepare grounds for judgment of detailed behaviors. The utilization of such numerical data can be extended to apply machine learning and various statistics based algorithms, and based on such data, malware will be visualized and malware similarity will be calculated.