The Cloud LSVA consortium will build a software platform to address the needs for annotation, recognition and fusion of video and vehicle data by building a software platform for efficient and collaborative semiautomatic labelling and exploitation of large-scale video data.
By making both the ADAS and the dynamic mapping sectors capable of handling large amount of data we will be able to:
- Create large training datasets of visual samples for training models to be used in vision-based detection.
- Generate ground truth scene descriptions based on objects and events to evaluate the performance of detective algorithms and systems
By automating or semi-automating the video annotation process we can bypass the human factor and open up the possibilities for video analytics triggered innovation in the automotive domain.
Mobile Sensors in Vehicles
These are the vehicles that gather sensor data via recording software and a given specified format. There will be procedures defined for transmitting incremental streams to the Cloud, and ways to update not only raw data from sensors, but also the (pre)processed metadata from ADAS systems. This input corresponds to large volumes of data, in the order of terabytes recorded per day and per car.
Multiple sources/vehicles may differ in type, volume or nature of the sensorial data. Also, 3rd parties may interact with the Cloud-LSVA platform by uploading content from other open datasets. Therefore, a data fusion stage is devised in order to work as a normalization interface layer to the Cloud-LSVA system.
Large-Scale Cloud Infrastructure
Computation and storage will leverage a cloud-based network that provides accurate and secure transfer of large data sets from the sensing location to the cloud. Particular attention will be paid to maximizing data throughput and reliability and the processing capabilities required to launch the video and semantic analytics routines to semi-automate annotation. Cloud-LSVA will progress from TRL 5 to 7.
This is the main orchestration module managing the interoperability and control of the separate modules. It also defines the external interface to 3rd party and public domains.
The main inputs for adaptive automatic analysis are the initial annotations of objects, events and scenes as understood by a human observer. From these, the complex understanding of the entire scene can be transferred to computerised recognition models. A semi-automated annotation task will be considered, supported by automatic large scale video analysis in the Cloud as well as local, task specific, machine learning algorithms. Interaction mechanisms with the human annotators will be specifically designed to minimize the time and cost of the labelling task.
This module will automatically extract the traffic related information, considering ADAS and dynamic cartography scenarios, for automated annotation, traffic security related event recognition and scene classification. It will employ distributed and scalable machine learning approaches in order to account for variable volumes of input data. This is the main process that will automatically generate high value metadata as a concise representation of the inputs.
Human observations in the form of annotations and corrections will be continuously collected for on-line update of the automatic data analytics modules.
Evaluation and Benchmarking
The assessment of the readiness of the generated computer vision models for automatic on-the-cloud analysis as well as for the export of local processing models, a continuous evaluation of the models will be pursued. Task specific ground truth datasets will be synthesised which will be provided for public benchmarking of related automotive (ADAS, maps) systems and novel machine learning algorithms.
The Cloud LSVA approach consists of three steps, which correspond to three stages of the project. The illustration below provides a clear overview of these steps.
The Cloud LSVA project will follow a 3 step approach to provide video annotation standards:
1. Use an open group to develop the video annotation standards.
2. Liaise with a Standard Development Organization to publish the resulting standards.
3. Carry out open TestFest events to validate the standards.