Vehicle Detection and Speed Estimation for Automated Traffic Surveillance Systems at Nighttime

: This article proposes a vehicle detection and speed measurement system to estimate a vehicle’s velocity by identifying its headlight properties in a nighttime environment. We present a traffic surveillance system for vehicle detection and tracking in the nighttime along with a background extraction and automatic vanishing point detection process. We show that a single video camera can sufficiently and effectively operate to concurrently calculate and detect a vanishing point during the daytime. We have applied this to preprocessing in an automated traffic surveillance system in the nighttime. The experimental results show that vehicle tracking is possible, even in the nighttime. Preliminary experimental results confirm the possibility and effectiveness of the proposed algorithms for this nighttime vehicle surveillance system. They demonstrate that a single camera with an embedded image processing system can concurrently monitor, detect and track multiple vehicles in multiple lanes during the nighttime as successfully as the daytime.


INTRODUCTION
The objective of a traffic surveillance system is to perceive salient features of the environment and to provide accurate real-time information regarding various vehicular conditions on the road.Data for real-time traffic surveillance systems can be collected from various sources such as loop detectors, radar sensors or video cameras.Recent technological developments in computer vision and image processing have revealed that video cameras are an efficient means of collecting and analysing traffic data [1].Video based surveillance systems are more sophisticated and robust because the information that is associated with image sequences presented in a video allow us to identify and classify vehicles in the most effective manner.Due to rapid advancement in modern computer vision and traffic surveillance techniques, vehicle detection techniques at night based on image processing have gained much attention in recent years [2,3].Detecting and perceiving the presence of vehicles is an important new field for intelligent traffic surveillance systems.Numerous research projects have been carried out in developing traffic management systems.Specifically, many papers give a literature review on various techniques used in monitoring and tracking vehicles during night [4,5].
This paper handles the problem of traffic surveillance in the nighttime, since the typical daytime surveillance system based on moving object detection cannot operate adequately at night.The appearance of a vehicle during the nighttime is different when compared to daylight hours, in terms of colours, edges, shapes, environment lighting, and light reflection against vehicles.In the nighttime, low intensity of illumination creates influential noise that increases the complexity of the sensing task.Furthermore, under adverse illumination conditions, background scenes are substantially affected by the lighting effects of the vehicle, and it is difficult to make realistic assumptions when compared to the daytime.Reflection from beams of headlights can cause false alarms during the detection process.The reflections of headlights resulting from a moving vehicle may obscure the distinction between foreground and background.Additionally, camera images at night have very low contrast and weak light sensitivity, thus making it difficult to use normal daytime detection methods.Therefore, those features that assist in identifying vehicle characteristics in the daytime are not applicable during the nighttime.Thus, a different image processing approach is essential for the nighttime road environment [6,7].
When oncoming vehicles are visible at night, they are mainly identified by their headlights.On the other hand, for the outgoing vehicles, one of the most important features is the illumination of their taillights [8].We can detect headlights using an object pair detection process and remove the unwanted reflections.Specifically, we can use a vanishing point calculation for tracking and pairing a vehicle's headlights.Usually, the detection of a pair of headlights involves three core steps: bright objects segmentation, possible headlights extraction, and candidate headlights pairing.We have applied a multilevel thresholding method to extract candidate headlights by considering multiple characteristics.The extracted bright objects are processed by using the temporal and spatial characteristics of the headlight patterns, and then the moving vehicles are located and analysed from the traffic video sequence.
This study provides an effective system for detecting and tracking traffic conditions for traffic monitoring vehicles in the nighttime.The proposed process uses pattern analysis and image segmentation techniques to detect vehicle headlights and to estimate vehicle speed.This paper is presented in the following order: Section 2 describes the overall traffic surveillance system.Section 3 presents the algorithms for background extraction and vanishing point detection.Section 4 proposes the bright object segmentation method required to extract potential vehicle headlight sources.Section 5 explains the speed estimation algorithm between the input images and the actual roads.Section 6 illustrates the comparative experimental results.Finally, section 7 presents the conclusions of this study.

SURVEILLANCE SYSTEM
Fig. 1 shows typical examples of colour images of an urban traffic road at night.It displays distinct headlights as well as other obstructions that exist in the ordinary nighttime traffic scene, such as road reflection, outdoor billboards, and streetlamps.The purpose of this research is to confirm the feasibility of image processing techniques for the construction of a surveillance system which enables monitoring of several vehicles in multiple lanes during the nighttime.Only one video camera is needed to simultaneously detect, track, and monitor multiple vehicles for both the daytime and the nighttime.The system can simultaneously detect multiple vehicles in transit on up to four parallel lanes even in both directions, and measure all their speeds with good accuracy.Fig. 2 shows the overall flowchart of the proposed traffic surveillance system at nighttime based on headlights tracking techniques.From the input video, we obtain still images the size of 320 pixels by 240 pixels for applying image processing techniques.We extract the background, which will be used as a reference image, from the image sequence during the daytime period.Generally, if a video camera is installed on the top of the road, the foreground image changes rapidly and the background rarely changes.
By utilizing this property, it is possible to extract the foreground, i.e. moving vehicles, from the background.For background extraction, we have utilized the simple moving average method to take advantage of the night time sequence.Then, the lane and the vanishing point detection processes are conducted.From the extracted background image, lane makers and centrelines are detected during the daytime using Canny edge detection and Hough transform.Vanishing points can be calculated using the selected lane markers information, and then we acquire one vanishing point to estimate the speed of each passing vehicle.We also define some regions of interest to save computational time and to detect vehicles efficiently.
At night, we also extract the background from the input image sequence in terms of the moving average process.Note that the average technique and the background extraction procedure in the nighttime are identical processes to those used during the daytime.The only alteration is the input image sequence.Note also that the background extraction process is applied only once.We can extract the information which is acquired during the daytime by additional processes including bright object extraction, spatial classification, and motion tracking.We can extract meaningful objects including vehicle lights by distinguishing bright objects with the multilevel thresholding method.Then we proceed to track and identify actual vehicles based on the temporal and spatial characteristics.Finally, we apply the perspective projection model to estimate the real distance on the road in terms of input image.The overall system shows that a single video camera is sufficient for extracting the background image, and to concurrently calculate and detect a vanishing point during the daytime, which we have applied for preprocessing in traffic surveillance systems in the nighttime.

BACKGROUND AND VANISHING POINT DETECTION
There are three general approaches to detecting moving objects: background extraction, forward or backward time difference, and motion vectors.Among them, background extraction is a very important preprocessing procedure to efficiently develop detection of a real-time moving object.It provides the most complete featured information, but is very sensitive to active scene changes on account of illumination and external events.We therefore suggest a simpler process that analyzes image sequence to extract background.In addition, the vanishing points of an image contain vital information for camera calibration.A vanishing point is the point on the intersection among the projections of the parallel lines.The driving lanes stretched from the vanishing point can be modelled by a set of line equations.Several methods for detecting a vanishing point utilize the line segments detected in images [9].Our objective in this section is to design an automatic background extraction and vanishing point detection system for the preprocessing of tracking and counting vehicle procession.

Background Extraction
The performance of background extraction varies mainly according to the background modelling methods.We have considered approaches varying from simple techniques to more complex stochastic modelling techniques [10].The information about the background can be composed from the image via a Mixture of the Gaussian model [11].The Mixture of Gaussian is one of the most widely-used recursive modelling techniques if there is no prior information about the circumstances.The probability of a certain pixel with a value x K at a particular time K can be verified by the sum of M weighted Gaussian: where w i is the weight of the i th Gaussian and η(•) is the normal distribution.Different Gaussian models can be defined to show different characteristics of each pixel, and each Gaussian sets a different weight depending on how often the same forms appear.One of the key advantages of this method is that it does not destroy the existing model of the background, which can be part of the background.Since Mixture of Gaussian is parametric, the model parameters can be updated adaptively without having to maintain massive video sequences.However, it requires much higher computational complexity compared to other methods.The recently developed ViBe algorithm may be applied to speed up the background subtraction process [12][13][14].
Here we present an algorithm to overcome the complicated computational process of the Mixture of Gaussians method.We have applied a simple moving average method to take advantage of the night time sequence and to effectively process detection of a real-time moving object.If the camera takes images of the same spot on multiple occasions, we will obtain several different images of the same place.At this point, a very simple approach to taking a background image is to simply take the average of all the images, i.e. the mean of images.Suppose we have n copies of the same street image, each with different cars and at different speeds; then the i th captured noisy image will be M + N i , where M is the ideal background image.We can find the mean M ′ of these images by usual averaging methods: where N i can be interpreted as normally distributed with mean 0; it is easily proven that the mean of all N i is close to zero.The greater the number n, the closer it is to zero.We have used the first 200 frames of input images for n.Thus, the average image is very close to the ideal background image and the approximation would be closer for a larger number of input images.

Lanes and Vanishing Point Detection
During the daytime, after we extract the background from the input image sequence, we apply Canny edge detection to detect the edge of the scene where the lane information is later extracted.The algorithm smoothens the image and finds the gradient of a bright area with a large spatial differential value.The non-maxima regions are then suppressed and two threshold values are used.It is considered an edge if the gradient magnitude lies between the two threshold values.The value is set to zero if the magnitude is below the first threshold, it is considered an edge if the magnitude is above the second threshold and a path is obtained if the magnitude lies between the two.We then apply Hough transform to detect straight lines at lane markers.To find the vanishing point of the lane markers and the centreline, we must identify the junction of two lines.From the image with the detected lines, the calculation of the intersection of the two lines, both representing different lane markers in the background image, are required.Suppose the two lines intersect at the point (x, y); that means that this point must lie on those two lines, so the point must satisfy both equations.Solving these two equations simultaneously gives us the vanishing points of the lines.Because of the small deviation of detected lines, the vanishing points do not converge to one point.However, the error is negligible; we can calculate the average of these vanishing points as a reference for future speed estimation in the nighttime.Among the detected straight lines, we consider the angles and choose lines which are placed between −45 degrees and 45 degrees, since the lane markers in the input image are usually located within this range [15].
We apply a region of interest (ROI) for each traffic scene to proactively eliminate things rather than vehicles, such as outdoor billboards, streetlamps, and traffic lights from the top of the image.All straight lines can be localized using the Hough transform and searching for the straight lines after the edge detection process can be limited to inside the suitable regions of interest of the image to increase the algorithm efficiencies and to reduce computational burdens.In this way, we can quickly identify reliable and effective information on moving vehicles, and reduce the cost of computing for the embedded system.This ROI is the area between the extreme left and right lanes, and from the middle of the image to the lower part of the image.This area is initially determined by the lane detection process in terms of straight line detection.The connected component extraction and spatial classification processes are only performed on the bright objects located in the ROI area.Fig. 3 shows a typical original oncoming vehicles image in the daytime, the extracted background image, selected lane edges with vanishing points image, and region of interest image in the daytime.We utilize this information and apply the headlights detection and tracking algorithms during the nighttime, which will be described in the following section.

HEADLIGHTS DETECTION AND TRACKING
In this section, we introduce the headlights detection algorithm which includes three steps: bright objects segmentation, suitable headlights detection, and candidate headlights pairing.The detected headlights are continuously tracked between successive frames of the input video sequences.

Headlights Detection
The background image is subtracted from the input video sequence to discriminate moving vehicles from the background.It is then converted into a binary image for bright objects segmentation.We perform connectedcomponent mining techniques to extract candidate headlights from the acquired bright objects.The extracted objects also contain bright components of other obstructions that exist in the ordinary nighttime traffic scene, such as road reflections, outdoor billboards, streetlamps, and traffic lights.To differentiate the headlights from the obstructions, several methods are applied according to the intensity, location, shape, and features.First, the candidate's bright area needs to satisfy the following conditions: where A(•) is the area of a object, B i is the i th object in one image frame, T min and T max are selected arbitrarily for threshold values for the bright area.The conceivable bright object should be placed somewhere in an area of a certain size.Secondly, the area containing the candidate headlights shall meet the required configuration conditions of the actual headlights.Let H(•) and W(•) indicate the height and width of the bounding boxes which enclose the i th bright object B i ; then the aspect ratio of the area containing the candidate object should meet the following constraints: where the threshold values T 1 and T 2 are properly selected to ensure that the headlights are suitably extracted.All latent headlights are found based on the definitions of bright components which are described earlier, and the unusual components, such as light from the rear cars and extensive reflectors, are removed.First, locate the position of the headlights arranged in pairs to recognize the position of the objective vehicle on the road.While the shape of the headlights may be imbalanced, most headlights are placed in pairs and almost symmetrical.The symmetry of a headlight pair may be explained by the proximity measure and the correlation of the pair.Generally, the aspect ratio of the headlights' bounding box is roughly two.Therefore, the pairing method of the headlights is based on the aforementioned characteristics, and the corresponding criteria are as follows.First, since the candidate headlights are formed in pairs according to the regions and its centers, the following conditions should be met: where C x,y (•) is the center coordinate of the candidate's headlights, and T A , T D are the threshold values for the area and the distance, respectively.The headlights, which belong to the same vehicle, usually have a high correlation with shape, size, and center coordinate.Therefore, the larger the correlation, the greater the likelihood that the headlights belong to the same vehicle.Like other wellknown vehicle detection algorithms, the threshold values for pairing candidate headlights in the above equations are also chosen based on the characteristics of images and the nature of the experiments.Furthermore, if there is a redundant object between the two paired headlights, choosing a single headlight pair with high symmetrical property and small position difference between them should be sufficient.

Headlights Tracking
Once a pair of headlights is confirmed to belong to the same vehicle, time series analysis models are applied to trace the location of the objective vehicle.Most object tracking systems are based on Kalman filters.However, usage based on unimodal distribution of Kalman filters is limited because it cannot support the concurrent unusual motion hypotheses.We expand the basic concept of Kalman filters to maintain a list of different hypotheses.
The procedure for the spatial tracking algorithm is as follows.
First, we predict the positions of known vehicles.It is important to estimate the position of the vehicle in each of the iterative tracking processes in order to ensure computational simplicity and tracking precision.Then, we associate the predicted vehicles with current vehicles.If one object is given in the current frame, we search the most suitable region for the next frame by calculating the correlation between the object's intensity mask over the feasible area.If a vehicle fits exactly in one area, it is the best situation to trace down.The vehicle's trajectory is updated with information from the new movement area and the certainty of it being a vehicle increases.
If tracks split, it creates a new tracking hypothesis.This problem can occur if one object is disjoined into several separate objects or if the detection algorithm fails to accurately isolate the object.In this instance, the best zone is selected as a new vehicle location and the certainty of it being a vehicle increases.Some other moving zones are considered to be the new object hypotheses and updated consequently.Once the traces merge, the follow-up hypotheses are also combined.This situation can occur if two vehicles are blocked out by each other, or if mistakenly split vehicles are grouped back eventually.In this particular case, an object hypothesis needs to be updated for analysing object tracking.Each vehicle that integrates into a single moving zone is individually tracked.Their paths are then investigated in depth.If they share almost the same speed and trajectory for a given period, they are integrated into a single entity.Otherwise, they are tracked separately.This permits the system to trace objects that are occluded with each other or form an object that would be combined together eventually.
The certainty of something being a vehicle is reduced if it does not maintain a persistent position and speed.If any object falls below a certain threshold, it is assumed that it is lost, at which point the object is excluded from the tracking list.If the vehicle is temporarily disturbed and appears again on several frames, the process is reinitiated.We update the vehicle track models, and repeat the process until the vehicle being processed disappears from the scene.

SPEED ESTIMATION
To estimate the real distance on the road in terms of input image, we apply the perspective projection model as introduced in [15,16].At night, when only a vehicle's headlights can be detected, the trajectory of the headlight pairs shall be used to estimate the speed of the vehicle.It is assumed that the origin of the imaging system is located at the upper left corner of the image.Fig. 4 shows an example of vehicle trajectory on the image plane.In the system of image coordinates, the vehicles located relatively far away from the camera will be located at a higher position and will gradually become smaller until they approach the vanishing point.Therefore, the driving lanes departing from the vanishing point may be modeled as a series of straight lines as explained in section 3. The length of a random straight line can be calculated from the measured coordinate values of two vertices of the line on an image plane.Using the vanishing points of lines, the relation of the vehicle trajectory between that on the image plane and that on the ground plane can be calculated.Fig. 4 shows an example of a headlights trajectory on the image plane.In the figure, x and y are coordinate vectors of the image plane according to the origin.Since we only utilize one camera, we may assume that the horizontal vanishing point, O h , is located at infinity and the vertical vanishing point, O v , is the point which we acquired during preprocessing.Suppose that the specific locations A, B, and C on the image plane correspond to A', B', and C' on the ground plane.Then as a vehicle moves from point A' to C' via B' on the ground plane, the vehicle would move from point A to point C via B on the image plane.Point B is the intersection of lines O h B and O v C for calculating the travel distance, the length of line, A'C' is the vector sum of lines A'B' and B'C'.We can calculate the lengths of A'B' and B'C' from the coordinate values of A, B, and C. Therefore, we can get the real distance between two points on the image plane.Once we calibrate the vanishing point and the real distance of the two known points on the road, we are ready to measure the vehicle speed.
The aforementioned process identifies the potential vehicle components by means of a pair of light sources entering the detection area from each image frame.However, since detail information of the potential vehicle cannot be obtained immediately from a single image frame, the movement tracking and grouping procedures are applied to analyse motion information of these potential vehicle components based on successive image frames.The rigid moving vehicle components are classified into a possible target vehicle by evaluating common motion information.Once all the components have been sourced from the scene, it is possible to confirm the speed of the tracked vehicles.We assume that a vehicle travels with a constant velocity along the road.Suppose the vehicle image located at (x, y) is denoted as Bi t (x, y) at time t and B i t+1 (x, y) at time t+1, respectively.Object based tracking is used to locate the motion vector within the search area, which minimizes the following residual function: With minimization of the existing vehicles' residual function, the position of newly matched vehicles is updated.We can utilize the aforementioned relation between the image plane and the ground plane to estimate the velocity of vehicles by recording each vehicle's time and location of entering and exiting the ROI.

EXPERIMENTS
In order to effectively establish background modeling comparisons, we need to capture a stationary background image to be used as a point of reference.Obviously, it is very difficult to capture a stationary background image during the daytime or the nighttime since there are many moving vehicles on the road.For this particular study, we used a simple median method due to its capacity to generate a solid background image.Median filtering has been shown to be a solid application with a level of performance comparable to higher complexity methods.However, a tremendous amount of memory is often required to store and process many frames of video sequences.We acquired the first 300 video frames and calculated their median, which can be used as a source of reference images for the comparison of performances of other methods.A distortion measure is used to measure how much information has been lost compared to the original image.A common distortion measure is the PSNR in dB between the reference image and the input image.
Fig. 5 shows the performance of background extraction in terms of PSNR values between Mixture of Gaussian (MoG) and the proposed method for the daytime and nighttime, respectively.It was found that a PSNR value of 30dB is sufficient for extracting a good quality background image.As additional data sets from calculating the moving average are added, the values from the average method increases exponentially.In the case of MoG, the background model does not adapt quickly enough; therefore, the values gradually increase up to 30dB.The proposed method converges very rapidly to 30dB.The MoG method outperforms the proposed method for the first 190 frames.However, the performance of the proposed method becomes superior to that of MoG after 190 frames.In particular, for the nighttime process, the gaps between these two methods are narrower than that of daytime.Therefore, if the computation time for background extraction is crucial to a system, the proposed method is a good compromise.Since nighttime convergence is faster than the daytime, the proposed method can be applied to a traffic surveillance system at nighttime.An experiment on urban traffic shows that the algorithm we propose is comparable to that of other background subtraction methods, especially in the nighttime, as shown in Fig. 5. Our main goal is to identify vehicles that project a pair of headlights.It is assumed that one can exclude vehicles with damaged headlights or motorbikes, since they do not have a significant impact on a typical traffic surveillance system.Fig. 6 shows a typical example of a gray-level input image, the extracted background image, the region of interest (ROI) overlapped to the original image, and a bright objects detected image.The object detected image shows unwanted reflections from the vehicle body and the road reflection caused by the headlights of the next vehicles.
We are able to detect headlights using multilevel thresholds with some spatial properties and remove the unwanted reflections at the same time.Fig. 7 shows examples of detected vehicles where the bounding boxes are the positions of the individual headlights.We can distinguish headlights using the pair of objects detection process, and then merge them into one bounding box if those individual bounding boxes meet some criteria.Fig. 8 shows some examples of detected vehicles in terms of headlights pairs where the bounding boxes are the positions of the headlights.Within the ROI, the speed of a vehicle is calculated by the centroid of the bounding box of an object from time of entering a lane within the ROI to the time of exiting that same lane, not necessarily to the bottom of the ROI but at any point of that specific lane.From this position, we can calculate the distance which a vehicle travels within a specific ROI.This procedure can be repeated for each frame of the image sequence and traces the vehicles.

CONCLUSIONS
In this paper, we have suggested a simple process that analyzes image sequence from a camera to extract background and calculate vanishing points.In addition, we also propose an efficient way of providing high precision vehicle detection and tracking approach in the nighttime.To overcome the complicated computational process of the Mixture of Gaussians method, we have presented an algorithm based on moving average method to efficiently process and to detect a real-time moving object.Unlike well-known conventional methods, the proposed method is unaffected by dynamic movement in a scene with natural surroundings, especially in the nighttime environment.It is successful in accurately detecting moving objects as well as suppressing most of the false detection caused by dynamic changes.
The experimental results show that vehicle tracking is possible even in the nighttime while calculating the speed of the vehicles.They also validate the proposed method for detecting and identifying vehicles in various circumstances and for monitoring nighttime traffic.We have applied the multilevel thresholding method to extract light objects including vehicle headlights from traffic video sequences.These bright objects are then grouped based on location and size attributes using the linked component clustering process.Therefore, the vehicle lights are identified using their typical characteristics, such as size, pairing and alignment, from the grouped bright objects.A comparison of preliminary tests of various traffic scenes proves that the proposed system can afford fast and efficient vehicle detection and speed monitoring performance in various nighttime circumstances; even considering various lighting conditions, night traffic flow, and road conditions.The results show that the proposed technique is an appropriate nighttime speed monitoring system.The results also show that the proposed method can be applied to real-time surveillance situations, especially for embedded traffic monitoring systems.Such a simplified algorithm appears to exhibit superior performance with low memory requirements when compared to well-established techniques.
In spite of the experimental results which show that the proposed system is a simple but efficient surveillance system for vehicles detection at night, there are a number of concerns that need to be addressed.It must be implemented on embedded hardware platforms to ensure mobility.False perception can be caused by reflection of the road, vehicle bodies, or fluorescent lane markers, and this matter needs to be resolved in future research.Lastly, the video sequence of this experiment is derived through a CCTV camera, which has a limited dynamic range.A high resolution sensor should be utilized to solve this problem.Our current ongoing research focuses heavily on accurate speed estimation by using headlight images as well as developing speed estimation algorithms for outgoing vehicles based on their taillights.

FigureFigure 2
Figure 1Typical examples of colour images from an urban traffic road in the nighttime

Figure 3
Figure 3 An original oncoming vehicles image, the background image, selected edges with vanishing points image, and region of interest image in the daytime (from top left to bottom right)

Figure 4
Figure 4 An example of trajectory on the image plane

Figure 5 Figure 6
Figure 5 PSNR values for Mixture of Gaussians (MoG) and the proposed method daytime (daytime and nighttime)

Figure 7
Figure 7 Examples of detected vehicles in terms of individual headlight

Figure 8
Figure 8 Examples of detected vehicles in terms of headlight pairs