Minimizing Illumination Effect in License Plate Recognition

The intelligent transportation system is a key technology for efficient traffic control that has been applied in various fields. The existing intelligent transportation system detects the license plates of vehicles mainly through image feature analysis technology by using image-processing techniques. While this method has the advantage of quickly recognizing license plates by simple computing when the environment for recognizing license plate images is favourable, its accuracy is significantly compromised by various environmental changes. This study proposes a method using Faster region-based CNN (R-CNN) and denoising autoencoder technology to improve the recognition performance for tilted and broken plates and false recognition caused by illumination effects in the access control automation system installed at construction sites where these poor conditions frequently occur. This study investigated 3,000 images collected from actual construction sites, comparing the proposed method with the existing Faster R-CNN for license plates affected by various illumination environments, and found an accuracy improvement of more than 30%.


INTRODUCTION
Recent advances in artificial intelligence technology and hardware capabilities have enabled various imageprocessing services to be introduced, and application services using them are diversifying. In particular, services using vehicles, which are part of the telematics field, are also gradually increasing, and research related to automatic license plate recognition has been actively conducted [1][2][3][4][5].
License plate recognition technology is helping to identify license plates efficiently without the need for human resources. As its importance is increasing, many studies have been conducted on vehicle license plate recognition systems applying artificial intelligence technology, and the recognition rate has also increased to a high level.
Despite such advances in technology, the problem of a declining license plate information recognition rate due to license plate detection and recognition under the influence of illumination has been highlighted as a chronic problem.
Dump trucks in construction sites make better examples of such poor conditions than passenger cars. Most plates of dump trucks in construction sites are covered with dirt, and they are often damaged by external impacts from gravel and other objects. Under such circumstances, when effects from various illumination environments are added, their license plates are not likely to be recognized correctly even by the human eye.
The purpose of this study is to improve license plate recognition performance significantly by proposing a license plate recognition technology that minimizes the effect of lighting by utilizing deep learning technology that has been used in object recognition and image quality improvement.
In this paper, we propose a method that greatly improves the recognition of license plates affected by illumination using a deep learning model. Using Faster region-based CNN (R-CNN) [6] and YOLO [7], this method improves current license plate recognition that uses image-processing technology and solves the chronic problem of overfitting afflicting deep learning recognition technology by reproducing and learning the illuminated images for learned images in various environments. In addition, this method represents the first attempt to apply Denoising Autoencoder (DAE) [8] technology, which has been used mainly for the improvement of image quality and conversion of image style, in a structure that can minimize the effects of illumination.

RELATED WORK 2.1 Traditional Approaches
Traditional approaches to existing license plate recognition include finding and recognizing vertical edges in the image, using unique colours or structural features from license plates, and using changes in contrast and geometric structures [9][10][11]. Various algorithms were used to try to detect license plate using edge information of image. Zheng et al [12] proposed a license plate detection method by combining Sobel filter and horizontal edge information for license plate detection. Filter-based approach was able to detect only horizontal and vertical lines, and methods using the Hough transform algorithm were proposed to solve this problem [13]. Edge-based approach has a problem in that the detection accuracy is inferior to the geometric change of the license plate, methods combining edge information and color information have been proposed to solve this problem. In order to use color information, various approaches have been proposed that mainly utilize the meanshift algorithm [14]. The texture-based approach using the pixel density distribution information in the image mainly used the Haar-like feature [15], and this method showed relatively good performance in the change Technical Gazette 28, 2(2021), 363-369 of lighting and brightness, which is a problem in conventional license plate detection. The character-based approach is a method of finding a character corresponding to a license plate in an image, and mainly uses a convolutional neural network to compare the connectivity of characters [16]. Although this method has relatively high accuracy, there is a problem in that the background is incorrectly recognized as a license plate when characters other than the license plate exist in the image.
These conventional methods have the common disadvantage that it is difficult to detect and recognize details due to external factors such as the influence of illumination. Tab. 1 is an analysis of this traditional approach [17].
To solve this problem, object recognition studies using SVM (Support Vector Machine) and HOG (Histograms of Oriented Gradients) have been actively conducted in recent years, but high accuracy, such as human recognition performance, has been difficult to obtain [18,19].

R-CNN Based Approaches
With the advent of R-CNN [20], which combines CNN and region proposal methods, research on object recognition has accelerated. Various improved models based on R-CNN are showing good results in the field of object recognition in natural environments, and various studies using them are being conducted.
However, it is impossible to apply R-CNN based realtime system because it requires a large amount of computation as it searches for information on all pixels in an image at the region proposal stage. To solve this problem, various algorithms have been developed starting with R-CNN [21][22][23].
Dong [21] proposed a CNN-based automatic license plate recognition system and compared its performance with that of Faster R-CNN, showing a performance improvement of over 40%. Silva [22] used YOLO to propose a model that does not require parameter modification or fine-tuning to perform tilt plate detection. Hashmi [23] proposed a CNN-based structure for real-time license plate recognition that showed an accuracy of 85%.
The existing approaches to license plate recognition have mainly focused on developing new architectures based on Faster R-CNN and YOLO. However, if the image itself is severely affected by illumination, it is difficult to discern it even with the human eye, and in this case, the accuracy of license plate recognition is very low. To solve this problem, a preliminary study [24] proposed a DAEbased image reconstruction model to improve license plate recognition accuracy.

PROPOSED METHOD
This paper proposes a license plate area detection and license plate recognition method that improves the RPN of Faster R-CNN, a lighting model to minimize the effect of lighting, and an image improvement method combining this with the DAE algorithm. The proposal algorithm structure is shown in Fig. 1.
The region proposal network (RPN) improves Faster R-CNN and outputs it in the form of bounding boxes. The license plate regions acquired in this way are cut out and used as learning data to recognize the numbers of the license plates. To minimize the effects of illumination and overfitting problems, which are problems of previous studies using deep learning technology, images are reproduced to learn various illumination environments by the illumination model proposed in this study.

Figure 1 Proposal Algorithm Structure
In addition, to minimize the effect of the illumination of the license plate image itself, the illumination correction is conducted by the DAE-based model at the preprocessing stage. In the final stage, the numbers on the license plates in the model environment where the illumination has been improved are recognized, and the results are verified.

Faster R-CNN-Based License Plate Recognition
Faster R-CNN is used to detect the initial area of the license plate and to recognize the information inside the license plate. In the case of detecting the license plate area, a separate pre-processing process is not used because the characteristic is clear. In the case of recognizing the information inside the license plate, it is performed after applying the method proposed in Section 3.2 of this paper.
Here, classification models ZFNet [16] and VGG-16 [26] were applied. Faster R-CNN is an extended structure of R-CNN, SPPNet [27], and Fast R-CNN, and it demonstrates a noticeable performance improvement (in speed and accuracy) by integrating R-CNN's complex training/test pipeline.
The key idea behind Faster R-CNN is to replace the creation of region proposals, which takes up the heaviest computational load for Fast R-CNN, with a new method and integrate it into the internal system of the model. For this purpose, the RPN of Fig. 2 was designed.
The RPN outputs bounding boxes with scores for whether it is an object or a background by taking the feature map extracted through the CNN of the input image as an input. It slides a small network over the feature map to generate region proposals, with the window set to a size of 3 × 3. Multiple region proposals are predicted at the same time at the position of each sliding window. In this case, boxes with predefined sizes and ratios are used, and these are defined as anchors.
In Faster R-CNN, anchors of three sizes (128, 256, 512) and three ratios (2:1, 1:1, 1:2) are used to include the shapes of various objects in natural images. The larger the number of such anchors, the greater the number of computations. In this paper, the number plate region detection and anchors for license plate recognition are classified and defined to reduce unnecessary region proposal calculations and improve the accuracy of bounding box regression. Fig. 3 shows the anchors defined in Faster R-CNN.  Anchor1 for the detection of the license plate area was defined as a 2:1 ratio with sizes of 128, 256, and 512. In the case of dump truck license plates at the construction site to be detected in this paper, all of them correspond to the old license plates, and the aspect ratio is 2:1, so the characteristics are reflected. Anchor2 for detecting the character area in the license plate was defined as a 1:2, 3:1 ratio with sizes of 128, 256, and 512. In the case of the number of license plate and the letter indicating the purpose, the size is 1:2, and the area name has a 3:1 ratio, so an anchor reflecting these characteristics is used.
In the case of the existing Faster R-CNN, as the number of unnecessary region proposals increases, the problem of recognizing the background as an object region occurs. This method can detect the license plate area and recognize the license plate more quickly and accurately than the existing method. In the classification layer of RPN, the classifier is learned by comparing the anchors and ground-truths received for all anchors. The classification of whether the anchor is a background or an object is determined by a numerical value representing the degree of overlap with the ground-truth. To determine this numerical value, Intersection over Union (IoU) is used, and it is defined in Eq. (1) below: Where A denotes the coordinates of the anchor and GT denotes the coordinates of the ground-truth box. The determination of the anchor object/background using IoU is shown in Eq. (2) below: where p * denotes the value of the anchor that distinguishes the object/background. When IoU is over 0.7, it is recognized as an object, and when IoU is less than 0.3, it is regarded as the background. In the paper of the Faster R-CNN [6], the threshold of IoU was set to 0.3 and 0.7. In this paper, the same threshold was used through empirical judgment.
The final output of the RPN becomes the proposed region of different sizes, which means the region in the feature map from CNN. Adjusting the size of the area in the feature map environment can lead to significant information loss. To prevent this loss, feature maps of different sizes are transformed into the same size using ROI pooling. Technical Gazette 28, 2(2021), 363-369 The training of RPN randomly samples as many anchors as used in mini-batch in one image. At this time, the ratio of positive and negative anchors used is 1:1. In general, there are more negative anchors, so if the ratio is not adjusted, learning may not work well. If the number of positive anchors is lower than 128, zero-padding is performed, and if there is no positive anchor, a value with a high IoU value is used.

Minimizing Illumination Effect
To minimize the effect of illumination, which is mostly problem in license plate recognition, in this paper, an illumination model, meaning directional light, and the DAE algorithm, which has been mainly used in the field of image quality improvement, are applied to the field of illumination improvement.
Illumination Model: Learning of DAE requires an image data pair consisting of an original image and a corresponding noise image. In this paper, the original image is defined as a clean license plate image, and the noise image is defined as an image affected by illumination. Since it is difficult to collect such a pair of image sets in an actual environment, this paper proposes an illumination model that can generate an original image as an image affected by illumination. The illumination model, which represents directional light, is shown in Eq.   To create an image that is affected by lighting that is difficult to recognize even with the human eye, an illumination image is generated using a relatively large illumination factor of 30 < M d < 60.
Image Reconstruction Model: We propose a DAEbased image reconstruction model. In the case of the license plate area, since its characteristics are distinct, it is easy to detect even under the influence of illumination, but the information in the license plate is easily deformed, making recognition difficult. To solve this problem, this paper proposes a new approach that minimizes the illumination effect using a DAE-based network structure.
The purpose of the DAE algorithm is to reconstruct noise-free output data from input data. For this purpose, the input data is compressed so that only the main features that can be expressed remain, and the loss function is minimized so that data similar to the original data can be created.
The DAE algorithm generates an image x' that applies random noise such as Gaussian to the input image x. After that, encoder h compresses the image to x' to obtain the latent vector, and decoder g learns to restore the latent vector to be similar to the input image x to generate the output image y. Fig. 5 shows the structure of removing the illumination from the image affected by the illumination by using the virtual illumination image generated by using the illumination model instead of random noise in the existing DAE network structure x denotes a clean license plate image, and x' denotes an illumination image created through the illumination model proposed in this paper. Image y is newly created through the process of compressing and restoring x' through encoder h and decoder g.

EXPERIMENT AND EVALUATION 4.1 Data Set
To experiment and evaluate the method proposed in this paper, 3,000 images of the front of a dump truck collected at an actual construction site were used. This dataset contains images that are difficult even for humans to recognize due to the influence of various illumination effects.

Figure 6 Learning Dataset
The most important requirement of a deep learningbased model is a sufficient amount and includes a variety of environments of learning data. In order to accurately recognize the numbers on the license plate region, we studied the 3,000 collected images of dump trucks and 13,000 images of SVHN (Street View House Numbers), which were reliable and provided sufficiently for learning.

License Plate Region Detection
As a result of detecting the license plate area in the front image of the dump truck, including various illumination environments using the improved Faster R-CNN method proposed in this paper, the recognition rate was over 98%.
To compare the performance, two types of CNNs, ZFNet and VGG-16, were used in the internal structure of Faster R-CNN. Learning was conducted for up to 1,000 epochs. The result of evaluating the accuracy of the model for each step of 100, 200, 500, 800, 1000 epochs, the same detection results were acquired after 500 epochs, and the results are presented in Fig. 8.
Region-based detection models, such as Faster R-CNN, also have a disadvantage in detecting regions that have a shape similar to a license plate in an image. The method proposed in this paper can be used in a vehicle's access control system. It is assumed that only one license plate exists in the image, and only one region with the highest similarity to a license plate is selected among the detected regions, as shown in Fig. 9.

Minimization of Illumination Effects through DAE
Preprocessing technology to minimize the effect of illumination in the research of object detection corrects the brightness of the image mainly by using image-processing techniques such as adaptive threshold and histogram equalization. In a complex environment, however, it is difficult to expect good results by using this method because the resulting values differ depending on the brightness of the input images. Fig. 10 shows the results of denoising various illumination environments. It is clear that input images that are affected by illumination and difficult to discern even with the naked eye are converted into clearer images by using the DAE-based model with light-effect minimization proposed in this study.

License Plate Recognition Result
Finally, in order to recognize the numbers on the license plates, numbers were detected by applying Faster R-CNN to input images with improved illumination. Technical Gazette 28, 2(2021), 363-369 To analyze the accuracy of the proposed method, this study compared the results of Faster R-CNN with those of the proposed method by using the same dataset for both, as in Fig. 11.

Figure 11 Comparison of Faster R-CNN and Proposed Method
Faster R-CNN and 3,000 images and SVHN data collected from actual construction sites were learned and compared with the method proposed in this paper. As a result of comparing the detection accuracy of license plate numbers for 150 license plate images affected by various Illumination, Faster R-CNN showed 52% detection rate, and when the proposed method was used, detection rate of 83% increased by more than 30%.

CONCLUSION
Automatic license plate recognition technology has been increasing in importance in recent years due to the continuous increase of vehicles on the road and the development of digital image-processing technology capable of detecting and identifying details at high speeds. Accordingly, many studies have been conducted on vehicle license plate recognition systems applying artificial intelligence technology, and the recognition rate has also increased to a high level.
However, the performance of license plate recognition can be greatly influenced by changes in illumination, damage to the license plate, and occlusion, and this phenomenon is highlighted as a common problem in existing license plate recognition-related studies. The environment in the dump truck recognition problem at the construction site often suffers from these issues, and the factor that has the greatest influence on the recognition accuracy is the effect of illumination.
In this paper, a deep learning-based license plate recognition method was proposed to minimize the effect of such illumination. By improving the RPN network of Faster R-CNN, the accuracy of detection of the license plate area and objects in the license plate was improved, and a new method applied with the illumination model and DAE algorithm was proposed to restore the license plate affected by illumination to the original image as much as possible.
To evaluate license plate recognition performance using the method proposed in this paper, 3,000 images of the front of the dump truck and 13,000 pieces of SVHN data collected at the construction site were used. As a result, the vehicle license plate recognition rate was 98% and the license plate number recognition rate was 83%. This shows that the recognition rate was improved by more than 30% compared with the case where the influence of illumination was not minimized.
In the future, it is necessary to continue research on the development of algorithms for license plates affected not only by lighting but also damage and occlusion.