Deep Understanding of Urban Traffic from Large-Scale City Cameras
Deep understanding of urban mobility is of great significance for traffic management, including for example autonomous driving. This work develops deep learning methodologies to extract vehicle counts from streaming real-time video captured by multiple low resolution web cameras and construct maps of traffic density in a city environment. In particular, we focus on publicly available cameras installed in the Manhattan borough of New York City.
Besides the high compression, the low resolution is challenging: cars may be a few pixels, strong perspective induces high occlusion rate and scale severe scale changes.
Main contributions
-
A multitask CNN based architecture to count cars by mapping images to a density-image. See details in
ShanghangZhang, W. Guanhang , João Paulo Costeira, J. F. Moura , Understanding Traffic Density from Large-Scale Web Camera Data, IEEE Conference on Computer Vision and Pattern Recognition, Proc. of CVPR 2017 Hawaii, USA, 2017 – PDF
-
Explore space-time “correlation” using LSTM’s
ShanghangZhang, G. Wu , João Paulo Costeira, J. M. F. Moura , Fcn-rlstm: Deep spatio-temporal neural networks for vehicle counting in city cameras, – IEEE International Conference on Computer Vision, Proc. of ICCV 2017 ,Venice, Italy, 2017
In the video below we show benchmarks and after sec 35 the video shows counting from several cameras in New York City. In complex places such as FDR drive Mean Absolute Error (MAE) is 1.67 cars. See the video for details.
-
Domain adaptation by adversarial training- Transfer learning from training set from 30 cameras upscaling the method to hundreds (under development)
This work is part of Shanhghang Zhang’s PhD thesis, co-advised by Prof. José Moura (CMU) and João P. Costeira (IST) with support from the Carnegie Mellon|Portugal Program . Carnegie Mellon supported the creation of the training dataset – WebCamT (to become publicly available soon).
There was also a US Patent Application to which this work contributed:
PCT Patent Application No: PCT/US18/26341;
Filed: 4/5/2018
Title: Deep Learning Methods for Estimating Density and/or Flow of Objects, and Related Methods and Software
Applicants: Carnegie Mellon University and Instituto Superior Técnico
Inventors: Jose Moura, Joao Paulo Costeira, Shanghang Zhang and Evgeny Toropov
