Teaching Cars to Drive and UAVs to Race


Project Description

At the Image and Video Understanding Lab (IVUL) at KAUST, we have developed a photo-realistic simulator (denoted UE4Sim) based on the open-source computer game engine Unreal Engine. The UE4Sim simulator has been designed to facilitate the integration of computer vision and machine learning techniques into a realistic looking 3D environment with the following advantages. (1) By facilitating the generation of 3D worlds, UE4Sim enables the quick and automatic acquisition of large amounts of labelled data to be used for training data-hungry machine learning models (specifically deep neural networks) targeting a multitude of computer vision and machine learning applications ranging from self-navigating cars/drones and aerial tracking to indoor 2D/3D scene understanding and 3D reconstruction. (2) UE4Sim provides simple-to-use connections with third party software to allow for real-time evaluation of AI techniques. The photo-realism of UE4Sim facilitates the transfer of the learned models to the real-world. In this internship project, we plan to develop UE4Sim further, motivated by the goal of teaching a car to drive in previously unseen scenarios and unmanned aerial vehicles (UAVs) to race through obstacle courses. All this functionality will be done within UE4Sim with an ultimate aim at transfer this learned knowledge to real world vehicles. ​​
Program - Electrical Engineering
Division - Computer, Electrical and Mathematical Sciences and Engineering
Center Affiliation - Visual Computing Center
Field of Study - ​Electrical Engineering or Computer Science

About the

Bernard Ghanem

Associate Professor, Electrical and Computer Engineering

Bernard Ghanem
Professor Ghanem's research interests focus on topics in computer vision, machine learning, and image processing. They include:
  • Modeling dynamic objects in video sequences to improve motion segmentation, video compression, video registration, motion estimation, and activity recognition.
  • Developing efficient optimization and randomization techniques for large-scale computer vision and machine learning problems.
  • Exploring novel means of involving human judgment to develop more effective and perceptually-relevant recognition and compression techniques.
  • Developing frameworks for joint representation and classification by exploiting data sparsity and low-rankness.

Desired Project Deliverables

Improvements on the UE4Sim simulator to make it more streamlined, efficient, and developer friendly for future development and integration with various types of learning (e.g. deep learning and reinforcement learning)Development of deep learning methods to estimate future positions of the vehicles (called waypoints) directly from images (perception network) Development of reinforcement learning methods to generate the appropriate vehicle controls, e.g. steering wheel angle or pitch/yaw/roll (control network) System integration of the perception and control network within UE4Sim​