An Efficient Feature Extraction Scheme for Mobile Anti-Shake in Augmented Reality

: In recent years, augmented reality on mobile devices has become popular. Mobile shakes are the most typical type of interference in mobile augmented reality. To negate such interference, anti-shake is an urgent requirement. To enhance anti-shake efficiency, we propose an efficient feature extraction scheme for mobile anti-shake in augmented reality. The scheme directly detects corners to avoid the non-extreme constraint such that the efficiency of feature extraction is improved. Meanwhile, the scheme only updates the added corners during mobile shakes, which improves the accuracy of feature extraction. In the experiments, the memory consumption of existing methods is almost double compared to that in our scheme. Further, the runtime of our scheme is only half of the runtime of the existing methods. The experimental results demonstrate that our scheme performs better than the existing classic methods on mobile anti-shake in terms of memory consumption, efficiency, and accuracy.


INTRODUCTION
Augmented reality (AR) [1] is a new research field in virtual reality technology.In recent years, AR technology has been widely used in military, medical, education, culture, entertainment, and other fields [2,3].
With the characteristics of small volume, portability, and popularity, mobiles are the most popular equipment used by people in their daily lives.Mobiles [4,5] have become the ideal platform to achieve an AR system [6], owing to which mobile AR systems can have broad application prospects.
Mobiles are typically limited by their computational power.For a large amount of computation, feature extraction is an important factor affecting mobile systems in real time.In feature extraction, a mobile shake is the most typical event that occurs when a user operates the phone with his/her hand [7].At this time, camera poses change slightly.Tracking the camera pose frame-by-frame decreases the AR performance in real time.Therefore, enhancing anti-shake efficiency has become a key research area for feature extraction in mobile AR [8].
To enhance mobile anti-shake efficiency, we propose an efficient feature extraction scheme for mobile antishake in AR.The scheme is named EAS (Efficient Anti-Shake).First, we propose a new corner judgment model to avoid the non-extreme constraint.Meanwhile, EAS uses the adjacent pixel approximation principle to reduce corner judgment complexity.The principle enhances the efficiency of feature extraction for mobile shakes.Next, we propose a mobile anti-shake algorithm that only updates new corners.On the one hand, the algorithm reduces the computational complexity for mobile shakes.On the other hand, the update is from the corner direction.Thus, the algorithm improves the accuracy of the feature extraction.The experimental results demonstrate that our scheme EAS significantly outperforms other schemes in terms of memory consumption, efficiency, and accuracy.
Our contributions are summarized as follows: (1) We proposed a new corner judgment model to avoid the non-extreme constraint.(2) We used the adjacent pixel approximation principle to reduce corner judgment complexity.(3) We proposed a mobile anti-shake algorithm that only updates new corners.
The remainder of the paper is organized as follows.Section 2 describes the related works.Section 3 presents an efficient feature extraction model.Section 4 describes a mobile anti-shake algorithm.Section 5 provides the algorithm comparison.Section 6 shows the experimental results.Finally, Section 7 concludes the paper.

RELATED WORKS 2.1 SIFT
The scale-invariant feature transform (SIFT) algorithm [9][10][11][12] is a partial matching algorithm.It obtains key points in different scale spaces established by the Gaussian fuzzy function.The algorithm is invariant to image attributes, including rotation, translation, size, and brightness.The operator of the SIFT scheme has great judgment ability.However, the scheme requires a high computing cost, thus leading to poor real-time performance when the mobile is shaking.We propose a mobile anti-shake algorithm that only updates new corners.The algorithm reduces the computational complexity for mobile shakes.Therefore, our scheme is more efficient than SIFT.

FAST and BRIEF
The features of accelerated segment test (FAST) algorithm [13] does not produce multiscale features.In addition, FAST feature points do not contain direction information.Thus, FAST loses rotational invariance.FAST feature points are described using the binary robust independent elementary features algorithm [13].BRIEF establishes the descriptor for these feature points.Its primary task is to describe the detected feature points.As a binary coded descriptor, it is primarily used in visual target tracking, image registration, target positioning, and image fusion.Their updating involves feature points, whereas our updating makes use of corners.Thus, our algorithm improves feature extraction accuracy.Further, our scheme is more efficient than FAST and BRIEF.

ORB
The oriented FAST and rotated BRIEF [14][15][16][17] algorithms deal with the losing rotational invariance issue associated with FAST.Meanwhile, ORB removes the noise sensitivity problem associated with BRIEF.The feature extraction algorithm of the ORB method performs better than SIFT for mobile shakes.However, the ORB feature extraction algorithm detects the pseudo corners that require non-extreme constraints.Thus, ORB increases the computational complexity of feature extraction.We propose a new corner judgment model to avoid the nonextreme constraint.Therefore, our scheme is more efficient than ORB.

SURF
The speeded up robust features (SURF) algorithm [18] is a type of scale and rotation-invariant point feature.It extracts features from coarse granularity to fine granularity through the pyramid structure of multilayer analysis.It simplifies the scale space generation of an integral image.Therefore, it is more efficient than SIFT in feature extraction.However, it cannot fully meet the real-time requirements when it is directly applied to image tracking.We use the adjacent pixel approximation principle to reduce the complexity of corner judgment, thereby enhancing real-time performance.The principle enhances the efficiency of feature extraction for mobile shakes.Thus, our scheme is more efficient than SURF.

Other Schemes
Çalışkan et al. [19] proposed an efficient noisy pixel detection model for CT images using extreme learning machines.Several other schemes exist as well [20,21].However, they did not focus on mobile shake scenarios.

BASE FEATURE EXTRACTION
EAS includes two parts: base feature extraction and feature extraction for mobile anti-shake.
To improve the efficiency of mobile feature extraction, we propose a scale spatial corner detection algorithm that uses a segment detection method to compute the value of the anticipated-testing pixel.
Definition 1 (Corner).Given an anticipated-testing pixel p, if p synchronously satisfies the following three conditions, p is denoted as a pseudo corner, which means that p is not a corner.Otherwise, if the three conditions cannot be synchronously satisfied, p is denoted as a corner.
(1) There is a pair of pixels {i, i'} in the circle with a radius of 3.
To compute the direction of our corners, we first define the field moment shown in Eq. (1).In Eq. ( 1), (x, y) is the coordinate of a random pixel and I(x, y) denotes the pixel value of (x, y).Eq. (1) defines the (q + r)-order moment of the random pixel (x, y) in the image, where q, r ∈{0, 1} and x, y ∈ N+.  1), the corner direction is defined as the angle between the corner and the center of gravity, which is shown in Eq. ( 2).
For simplifying the next illustration, we provide the following definitions.
Definition 2 (Outer ring region).Given a pixel p, the set of pixels in a circular ring with p as the center and with the radius of 3 is defined as p's outer ring region.
Definition 3 (Middle ring region).Given a pixel p, the set of pixels in a circular ring with p as the center and with the radius of 2 is defined as p's middle ring region.
Definition 4 (Inner ring region).Given a pixel p, the set of pixels in a circular ring with p as the center and with the radius of 1 is defined as p's inner ring region.
The core of feature extraction is to obtain corners and corner directions.

Corner
The "Algorithm 1 Corner" regulates the process of obtaining corners.The algorithm takes the anticipatedtesting pixel set P as the input and corner set P as the output.Algorithm 1 removes all pseudo corners from the anticipated-testing pixel set P. The specific steps are illustrated as follows: (1) If a pair of symmetrical pixels {i, i'} exists in p's outer ring region and {i, i'} have similar pixel values with p, {i, i'} are pseudo corners.All pseudo corners should be removed from P. Meanwhile, based on the adjacent pixel approximation principle [22], the other pixels on the line between i and p are also pseudo corners.Thus, we have to remove them from p (Steps 3-7).
(2) If a pair of symmetrical pixels {i, i'} exists in p's middle ring region, and {i, i'} have similar pixel values with p, {i, i'} are pseudo corners.Meanwhile, the other pixels on the line between i and p should be removed as well (Steps 8-11).
(3) If a pair of symmetrical pixels {i, i'} exists in p's inner ring region, and {i, i'} have similar pixel values with p, {i, i'} are pseudo corners (Steps 12-14).  1 shows a test regarding whether an anticipated-testing pixel p is a corner.
Algorithm 1 first tests p's outer ring region.As long as any pair of pixels has similar pixel values with p, p is a pseudo corner.For example, if {1, 1'} have similar pixel values with p, p is a pseudo corner.Meanwhile, as pixels 9 and 15 are on the line between 1 and p, pixels 9 and 15 are also pseudo corners.They do not need to be tested.Therefore, we enhanced the efficiency of feature extraction by directly judging the pixels on the line between i and p without testing them.

Direction
"Algorithm 2 Direction" regulates the process of obtaining corner directions.It takes the corner set P as the input and corner direction set θ as the output.The algorithm computes the direction for each corner based on Eqs. ( 1) and (2).

FEATURE EXTRACTION FOR MOBILE ANTI-SHAKE
Two types of mobile shakes exist in an AR system.One type is where the user consciously shakes the mobile.In this case, there is a significant change in the camera pose.The other type is where the shake happens unconsciously.In this case, the camera poses changes little.We only focus on the second case that is defined as a mobile shake herein.
This case can be attributed to three reasons: (1) Physically unconscious shakes.
(3) Shakes caused by the surrounding environment.
When mobile shakes occur, the camera poses change too little to be noticed by the user.If we use the frame-byframe tracking method [23,24] at this time, the limited mobile computing resources will significantly affect the real-time tracking performance.Therefore, we only update the camera pose rather than recalculate it to enhance the anti-shake performance for dynamic scenarios.
"Algorithm 3 Anti-Shake" regulates the process of camera pose update.The algorithm takes the previous corner set P, previous corner direction set θ, and current anticipated-testing pixel set P' as the input.Meanwhile, the algorithm takes the new corner set P' and the updated corner direction set θ' as the output.When mobile shakes occur, the computing cost of obtaining the corner direction is much higher than that of obtaining a corner.To enhance anti-shake efficiency, the core idea of algorithm 3 is to only compute the new corners' directions.Therefore, we efficiently decreased the computing cost.
First, algorithm 1 is called to obtain the current corner set P' (Step 1).Next, we obtain the new corners by comparing the differences between the current corner set P' and the previous corner set P (Steps 2-4).Subsequently, if no new corner exists, (|Pn| = 0) (i.e., we may only lose some corners), the shake is blurred (Step 5).Thus, we omit the operation.Otherwise, if some new corners exist, the shake is clear.Thus, we need to update the corner directions (Steps 6-14).In the update process, if P and P' have the same corner p (Steps 7 and 8), p's direction θ p does not need to be recalculated (Step 9).That is, we only need to compute a small number of directions by calling algorithm 2 (Step 12).

ALGORITHM COMPARISON
Our algorithm (EAS) reduces the amount of computation and storage compared to the other two algorithms (SIFT and ORB).To more clearly prove our algorithm's advantage, we compare all algorithms in terms of space and time complexity in Tab. 1. Space complexity decides the storage (memory consumption).Time complexity decides the computation (efficiency).

EXPERIMENT EVALUATION
SIFT and ORB are classical methods and are related to our work.Thus, to verify the feasibility and practicality of EAS, we compare the performance of SIFT, ORB, and EAS from the aspects of memory consumption, efficiency, and accuracy.
In our experiments, we used C++ to implement the synthetic datasets.The P/S system was operating on a server with 128 GB memory, 64 KB L1 cache, 512 KB L2 cache, and CentOS 6.6 operating system.Meanwhile, we calculated the average value of 10 similar experiments to eliminate experimental errors.

Memory Consumption
The frame number and corner number primarily affect memory consumption (RAM).The experimental results are shown in Fig. 2, where the y-axis denotes the memory consumption (units: GB).Meanwhile, the x-axis denotes the frame number and corner number separately.
As SIFT has the best judgment ability, SIFT consumes the most memory, as shown in Fig. 2. The ORB feature extraction algorithm detects the pseudo corners that require non-extreme constraints.Thus, ORB consumes more memory than EAS.The memory consumption of SIFT is almost double that of EAS and that of ORB is 1.5-fold that of EAS.Therefore, our scheme consumes less memory than SIFT and ORB during mobile shakes.

Efficiency
Given sufficient computational memory, our efficiency (CPU) experiment tests the runtime for real-time tracking of mobile shakes.The experiment requires the user to hold a mobile with only one hand and attempt to keep his/her hand steady.
Meanwhile, the user should aim at the tag in the scenario wherein the tag area is approximately 1/6 of the entire screen.The window size is 30 × 30 pixels.The given minimum threshold is t = 1.The given positive small integer is n = 20.
We continuously record the runtime of real-time tracking every 20 frames.The experimental results are shown in Fig. 3.The x-axis of Fig. 3 represents the frame number and corner number separately.Meanwhile, the yaxis of Fig. 3 represents runtime (units: ms).
When mobile shakes occur, SIFT does not directly test the corners.Meanwhile, SIFT tracks the camera pose frame-by-frame based on recalculations.Thus, as shown in Fig. 3, the runtime of SIFT is approximately 2.5 times more than EAS.ORB would detect much more pseudo corners than EAS during mobile shakes.Thus, the performance is obviously lower than EAS.When mobile shakes occur, EAS directly detects the corner, only updates the camera pose without recalculation, and avoids detecting pseudo corners.The runtime of EAS is only approximately 40% that of SIFT and 55% that of ORB.Therefore, from Fig. 3, we found that EAS is more efficient than SIFT and ORB for mobile anti-shake.The corner detection of SIFT is not direct, and SIFT incurred much computing cost for mobile shakes.Thus, from Fig. 4, we observed that EAS's image provides more clarity than SIFT's image.As ORB detects a larger number of pseudo corners than EAS, and EAS only updates the added corners, the image accuracy of ORB is lower than that of EAS.Therefore, by Fig. 4, EAS is obviously more suitable for mobile anti-shake than SIFT and ORB.

CONCLUSION
For mobile shake application scenarios, some existing methods exhibit low efficiency and/or accuracy.To solve this issue, we proposed an efficient feature extraction scheme for mobile anti-shake in AR.The scheme directly detected corners avoiding the non-extreme constraint to enhance the efficiency of feature extraction.Meanwhile, the scheme only updated the added corners for mobile shakes to enhance the feature extraction accuracy.The memory consumption of SIFT was almost double that of EAS and of ORB was 1.5-fold that of EAS.The runtime of EAS was only approximately 40% and 55% that of SIFT and ORB, respectively.Therefore, the experimental results demonstrated that our scheme was more suitable for mobile anti-shake than most existing methods.
the total densities of the small area around (x, y).The one-order moments are 10 ,

Figure 1
Figure 1 An example of feature extractionExample 1. Fig.1shows a test regarding whether an anticipated-testing pixel p is a corner.In Fig.1, ∀i ∈[1,18], {i, i'} are symmetrical about p. Thus, 18 pairs of symmetrical pixels exist.p's outer ring region includes 8 pairs of symmetrical pixels-{1, 1'}, {2, 2'}, …, {8, 8'}; p's middle ring region includes 6 pairs of symmetrical pixels-{9, 9'}, …, {14, 14'}; p's inner ring region includes 4 pairs of symmetrical pixels-{15, 15'}, …, {18, 18'}.Algorithm 1 first tests p's outer ring region.As long as any pair of pixels has similar pixel values with p, p is a pseudo corner.For example, if {1, 1'} have similar pixel values with p, p is a pseudo corner.Meanwhile, as pixels 9 and 15 are on the line between 1 and p, pixels 9 and 15 are also pseudo corners.They do not need to be tested.Therefore, we enhanced the efficiency of feature extraction by directly judging the pixels on the line between i and p without testing them.

2 :
for all p ∈ P do 3: let p's coordinate be (x, y)

Figure 2
The comparison of memory consumption

Figure 3
The comparison of efficiency 6.3 Accuracy The experiment extracts features of mobile shake images in real scenarios.A set of experimental results are shown in Fig. 4. Figs.4(a), 3(b), and 3(c) show the accuracy of SIFT, ORB, and EAS, respectively.In Fig. 4, SIFT uses a 3 × 3 image class.The detection radius of ORB and EAS are both 3.All images are detected during mobile shakes.

Figure 4
The comparison of accuracy

Table 1
Algorithm comparison n O n×