Background

Diverse approaches have been proposed to implement TDMs based on different data acquisition techniques, algorithms, and system architectures. One approach is based on big data (^{Zhu, Yu, Wang, Ning, & Tang, 2019}). The main idea is to capture and exploit data from different sources, such as video cameras, on-board car sensors, on-road sensors, and even social networks (^{D'Andrea, Ducange, Lazzerini, & Marcelloni, 2015}). All acquired data is uploaded to the cloud, where a group of servers and algorithms perform computations to obtain traffic dynamics estimations.

Other approaches simplify the complexity of the TDMs by just using on-road sensors, such as cameras, light and ranging detection (LIDAR), ultrasonic, and acoustic sensors. For instance, TDMs based on computer vision have been reported in the literature about this topic (^{Giridharan, Kadaieaswaran, Arunprasath, & Karthika, 2017}). Taking advantage of the current cameras used for reporting traffic conditions, especially in newscasts, measuring the traffic density at intersections is possible.

For the case of video-based TDMs, four relevant traffic metrics have been modeled and reported in the literature (^{Hall, 1992}): average speed ( ), traffic flow (q), density (𝜆), and occupancy (o).

Traffic density

It is the number of cars per unit of length:

where L is the physical distance measured or estimated from the images.

Occupancy

It is the average time in which a vehicle is analyzed:

where t_1,i and t₂,_i are the times when the i - th vehicle enters and exits from the

analysis zone, respectively.

A Traffic Dynamics Meter

This work proposes a TDM using data from videos, particularly from free-to-access camera feeds from roadsides, to estimate the previously defined traffic metrics. Figure 2 depicts the main stages of this TDM. The Retriever stage takes each video frame, converts it to grayscale, and applies filtering and morphological operators. According to the road and camera feed, this stage uses predefined masks to define regions of interest in the scene, particularly for detection and tracking. The Detector stage identifies vehicles in the detection zone using Haar-like features (^{Viola & Jones, 2001}) and produces an array of detected vehicles. For such an array, the Tracker stage provides a new tracker for each vehicle in the scene. As depicted in Figure 2, the number of trackers changes dynamically as the total number of vehicles being tracked changes. Also, trackers for the previously detected vehicles are refreshed with each new video frame. This tracking is performed using MOSSE filters (^{Bolme, Beveridge, Draper, & Lui, 2010}). Finally, the Meter stage collects new vehicle positions and extracts metrics for each one, and it calculates the global flow, density, occupancy, and average speed.

Baseline Performance

The TDM proposed was implemented using C++ and OpenCV (version 3.3.0); it was deployed on a Zynq XC7Z020, an offthe-shelf SoC available in the ZedBoard

Note: Own source.

Figure 2 Block diagram of the main components in our proposed TDM. 

development platform. We evaluated the performance of this TDM with video samples from three different test scenarios: two camera feeds from roads of San José, Costa Rica, here referred to as General and Multiplaza ^{(Ministerio de Obras Públicas y Transportes, n.d.}); and a third camera feed from Soquel Ave, in Santa Cruz, California, here referred as SoquelAve^{(California Department of Transportation (Caltrans)}). Samples of these camera feeds are presented in Figures 3.a, 3.b, and 3.c, respectively. Each scenario presents different traffic dynamics, which leads to different detection and tracking effort. General represents a low traffic density with high-speed vehicle flow. On the other hand, SoquelAve presents a high density with moderate speed. Multiplaza is an intermediate scenario.

An execution profile of the TDM is presented in Figure 1.d for these three test scenarios. For this baseline performance evaluation, the queue of detected vehicles was processed in serial. Considering a requirement of 30 fps, the current execution time required per frame is above this constraint for the three scenarios. In the experimentation, the time required by the Retriever stage is almost constant for these scenarios. However, the Detector and Tracking stages consume most of the execution time, ranging from 27 % to 79 %, and from 13 % to 225 % of the entire processing time per frame, respectively. As depicted, the required time for actually estimating the traffic dynamic metrics is negligible.

Considering these scenarios, the detection effort in General is caused by false positives during the first stages of the detection. However, its tracking requirement is smaller with respect to the other two scenarios due to a lower traffic density. In SoquelAve, more vehicles are detected and tracked along the region of interest, which

Note: Cámaras Viales CR. Cámaras Viales CR. Retrieved (2018).

Figure 3 Test scenarios are used in the scope of this work and the execution profile of TDM for them. The execution time for each stage of the TDM presented is relative to a frame period (1/fps). A frame rate of 30 fps is defined as a performance constraint. 

is reflected in a high execution time required for each frame. Multiplaza has the least detection, but more tracking effort compared to General. In this case, the detection discards more elements early but tracks more vehicles due to a higher density.

The implementation of the proposed TDM deals with several implementation challenges, for instance, rotation, geometric transformations, scaling of the vehicles, and light variances in the scene. This was considered when selecting object detection and tracking algorithms. Haar-like features were more suitable for object detection than Local Binary Patterns (LBP). Although LBP were faster than Haar-like features (2.3x), its detection accuracy was lower (about 5 %).

Object tracking techniques are employed to complement the detection and make it more robust to catch false negative cases. Thus, once the application detects a vehicle, the Tracker is in charge of analyzing it in the following frames. One requirement for object tracking is transformation invariance. Adaptive-kernel trackers are used for their capability to adapt to the object transformation over time if the object does not change suddenly. The MOSSE filter adapts its kernel according to a valid detection, which helps the Tracker catch most of the transformations of the object during the video sequence. If multiple objects are detected, the same number of trackers are required, leading to linear growth in the computational requirement.

Optimizations

To improve the performance of our TDM, we explored two techniques at the software level: frame skipping and multi-threaded execution for object tracking.

Frame skipping: In dense traffic, vehicles do not move as fast as they do on a free road, affecting the traffic density. Inspired by loop perforation (^{Sidiroglou-Douskos, Misailovic, Hoffmann, & Rinard, 2011}), an approximate computing technique, and considering the traffic density metric, we propose to save execution time by skipping frames to be processed. We explored two approaches for frame skipping. The first considers a fixed number of frames to be skipped in-a-row. The second adjusts this number dynamically, considering the traffic density metric as a reference. The number of frames f to be skipped with this dynamic approach can be defined as:

where 𝜆 is the last known density, and 𝜆i are the minimum and maximum traffic density values reported for a video stream. The Fmax is the maximum number of frames that can be skipped without exceeding a given maximum error with a fixed approach. We use relative error RE as error metric, calculated as follows:

where M corresponds to one of the traffic metrics. For each scenario, the accurate value for each metric was manually determined.

Both frame skipping approaches impact the accuracy of the estimations performed by our TDM, as depicted in Figure 4 for the case of SoquelAve. For the fixed approach, 3 and 2 frames were skipped for the Detector and Tracker stage, respectively. For the dynamic one, Fmax was selected, such as RE < 20 % for all the metrics. As shown, even the baseline implementation is not entirely accurate, and it presents errors with respect to a golden estimation. The fixed approach improves the average speed and occupancy estimations regarding the baseline; however, it significantly increases the error for flow and density. The dynamic approach improves the metrics estimations concerning the baseline, except for average speed.

Note: Own source.

Figure 4 Effect of frame skipping in the accuracy of traffic dynamic metrics. These results correspond to the SoquelAve scenario, which is the worst-case scenario. 

It is interesting to notice that approximate implementations of our TDM can produce better traffic metrics estimations. To understand this, it is required to analyze the effects of a tracking failure within the detection zone. Under certain conditions, such as vehicle rotation or light variance, the trackers cannot find the car and refresh themselves. This results in trackers stuck within the detection zone, affecting the detection of new vehicles. This issue happens less frequently when applying a frame skipping technique since some cars are detected later than the baseline, enhancing the tracking. It is worth mentioning that density and flow metrics depend on the total vehicles tracked and the speed and occupancy of the tracking itself.

Multi-Thread Object Tracking: As previously discussed, a new tracker is required each time the Detector stage finds a vehicle. Each tracker performs several operations: frame preprocessing, correlation using FFT, Peak-to-Sidelobe Ratio (PSR) computation, and filter update. This last step is only executed if PSR > th, where th = 5.7 in the MOSSE filter implementation of OpenCV.

The computing effort spent by the filter update is comparable to the other previous steps performed during the tracking. Hence, the multi-thread optimization consists in using an additional thread, from the standard C++ library to compute the filter updates passed by a queue as soon as the thresholding has been applied and keeping the other steps of the Tracking stage on the main thread. Thus, the correlation and the filter update work in a pipeline manner, enhancing the time-to-solution for each tracker.

Evaluation

Figure 5 highlights the performance improvement obtained by executing our TDM with multi-thread object tracking (MT) and frame skipping (FS). For the case of FS, we have used the dynamic approach as, in general, it provides metric estimations with smaller errors.

Note: Own source.

Figure 5 Runtime optimization for our TDM implementation with multi-thread object tracking (MT) and frame skipping (FS). 

As it can be noticed, for the Multiplaza and General scenarios, using both optimizations (MT + FS), the execution time of the TDM is near the 33.3 ms per frame (noted in the graph) required to achieve a 30 fps throughput. For such scenarios, near 5% and 25% performance improvement was experienced, respectively, compared to the baseline performance of the TDM.

For the SoquelAve scenario, a more significant improvement was achieved. Compared to the baseline implementation a 1.32 x speedup improvement was achieved using the MT optimization. Considering MT + FS, the speedup reached was 1.76 x. Despite this performance improvement, the maximum throughput reached is about 15 fps, far below the desired 30 fps constraint. This is due to the high vehicle density found in this scenario, which, as previously discussed, increases the computational requirements for detection and tracking.

A detailed analysis of the optimized TDM when processing the SoquelAve scenario shows that the individual execution time of the Retriever, Detector, and Tracker stages are 20.4 ms, 17.1 ms, and 26.3 ms, respectively. This means that none of the individual stages of our TDM reaches more than 33.3 ms of the per-frame available execution time if a 30 fps throughput is required. The entire TDM can be further modified to adopt a pipeline implementation in a daisy chain fashion, in which each stage can be allocated to an individual computing engine. This will enable us to get higher throughput by sacrificing a small initial latency of 3 frames.

Conflict of Interest

The authors declare no competing interests.

Author contribution statement

All the authors declare that the final version of this paper was read and approved.

The total contribution percentage for this paper's conceptualization, preparation, and correction was as follows: L.L.V., 55%; and J.C.G., 45%.

Data availability statement

The data supporting the results of this study will be made available by the corresponding author, L.L.V., upon reasonable request.