Find a Project

Have you ever imagined having a robot that cooks every meal for you? Have you ever dreamt about experiencing a different world in the Metaverse? Have you ever expected to have your glasses tell you where you left your keys and how to navigate to your favorite restaurant step by step? If yes, egocentric video understanding is what can lead you there. Egocentric videos are videos recorded from the first-person point of view, with the camera mounted on the head (e.g., GoPro) or smart glasses worn alongside the eyes (e.g., Google glasses), and they are what you actually see in your eyes. We need the AI system to automatically analyze and understand this type of videos to achieve the goals mentioned above. There are two key aspects in this problem: 1) large egocentric video data to fuel AI solutions; 2) effective techniques to generate correct predictions. For the first aspect, our IVUL group has devoted two years’ effort, together with 12 other universities as well as Meta (formerly Facebook), to achieve the largest egocentric video dataset called Ego4D. It contains 3000+ hours of egocentric video, spanning hundreds of scenarios captured by nearly 1000 unique camera wearers. Ego4D also defines various research tasks for egocentric video understanding, ranging from querying past memory, interpreting current behaviors, and forecasting future tendencies. For example, given a sentence “where and when did I last see my keys?”, the AI system returns the most recent video clip showing where your keys are. Or, the AI system automatically summarizes the video by telling you who is talking and what is his/her main point. Or the AI system predicts where you are walking to and what you are doing in the following minutes or even hours. For the second aspect, though Ego4D contains baseline solutions to each task, these solutions are far from practical for real-world application. There are two main challenges here. First, current solutions adopt techniques from video understanding tasks for third-person videos (where activities are recorded from a “spectator” view), which are dramatically different from egocentric videos in terms of recording perspective, camera motion, video continuity, etc. As a consequence, representations learned from third-person videos are not optimal to represent egocentric videos. We need to investigate novel feature representations specific to egocentric videos, or explore ways to smartly transfer the knowledge from third-person videos to egocentric videos. Second, egocentric videos pose new challenges for conventional methods due to their characteristics, such as noisy head motion, long videos and fragment actions. We need to address these challenges and improve the performance with novel techniques. In a nutshell, Ego4D is putting an apron on the robot and knocking on the door of the Metaverse, while at the same time, it is unveiling fresh challenges, which AI researchers are the key. It’s time to hop on board and contribute to this grand effort!

BAS/1/1653-01-01

bernard.ghanem@kaust.edu.sa

egocentric video; video understanding; computer vision;

Computer Vision; Machine Learning

(i) Effective feature representations of egocentric videos that benefit downstream tasks of egocentric videos, such as episodic memory, future anticipation; (ii) Novel techniques to transfer/translate between egocentric videos and exocentric videos; (iii) Improvement to retrieve ‘moments’ from past videos using a category, sentence or an object; (iv) Improvement to identify speaking faces in an egocentric video and summarize the speech.

Computer Science

Computer, Electrical and Mathematical Sciences and Engineering

Graduate or Undergraduate

Visual Computing Center

One of the most impressive abilities of human beings is the incremental learning ability. Since birth, we are always learning continuously without forgetting previously acquired knowledge that is useful for us. Could you imagine the amount of real-world applications that could be developed if we extended this human ability to modern-day AI systems (especially deep neural networks)? Modern deep neural networks adapt their parameters based on large-scale datasets to achieve state-of-the-art performance on specific computer vision tasks. However, due to legal and technical constraints and huge label diversity, deep learning models in real-world scenarios would rarely be trained just once time. Instead, they could be trained sequentially in several disjoint computer vision tasks without considering the data from previous tasks (because it may no longer be available for example). Therefore, these networks should learn incrementally without forgetting the previously learned knowledge. This is known as Continual Learning (CL). Currently, there are two families of CL methods. The first is rehearsal or memory-based methods, which select and store the most relevant samples to remember the current task when the following tasks are learned. The second group involves regularized methods that penalize changes to the most relevant parameters for the previous tasks. While the main studied challenge in the literature is learning with the least amount of forgetting on previous tasks, there are several other unexplored factors that affect learning from a stream of data. For instance, How fast is the learner in adapting the parameters of the model when receiving a new batch of data? If the learner is too slow (expensive training routine), then samples from the stream could be missed and not trained on. Thus, how can we benchmark different continual learning methods under budget constraint training? Furthermore, most continual learning benchmarks are focusing solely on the image domain leaving the more challenging video data unstudied. In the video domain, one of the main issues has been the lack of realistic, challenging, and standardized evaluation setups, making direct comparisons hard to establish. Therefore, our group IVUL has developed vCLIMB, a novel video class incremental learning benchmark, to promote and facilitate research on continual learning in the video domain. Video CL comes with unique challenges. (1) Memory-based methods developed in the image domain are not scalable to store full-resolution videos, so novel methods are needed to select representative frames to store in memory. (2) Untrimmed videos have background frames that contain less helpful information, thus making the selection process more challenging. (3) The temporal information is unique to video data, and both memory-based and regularization-based methods need to mitigate forgetting while also integrating key information from this temporal dimension.

BAS/1/1653-01-01

bernard.ghanem@kaust.edu.sa

continual learning; online learning

machine learning; computer vision

(i) A novel memory sampling strategy that learns to select a different number of relevant frames per video to reduce memory consumption while the performance remains almost the same; (ii) Novel training techniques/schemes to reduce forgetting. (iii) Benchmarking different continual learning methods on more practical metrics.

Computer Science

Computer, Electrical and Mathematical Sciences and Engineering

Graduate or Undergraduate

Visual Computing Center

Lots of data in scientific and engineering applications come with natural graph structures such as molecules in Chemistry, proteins in Biology, particles in Physics, planets in Astronomy, Bulk and MOF materials in Material Science, social and citation networks in Data Science, point clouds and meshes in Computer Vision and Graphics and so on. To model the information of such objects with complex structures, Graph Machine Learning, especially Graph Neural Networks (GNNs), has been proven as one of the most promising tools. Graph neural networks are deep learning architectures that can be trained to represent graphs with node features and edge features. For example, a molecule can be represented by a graph, where each atom is a node in the graph, and each bond is an edge. The atom numbers and types of chemical bonds are the associated node and edge features respectively. A GNN model can be trained to predict the quantum properties by learning on density functional theory (DFT) datasets which has huge potential to advance scientific discovery. Our group at IVUL have developed methods for graphs in 3D vision, videos, data mining, and fundamental science. We have developed GNNs with more than 100 layers with DeepGCNs (ICCV’2019, TPAMI’2021), and PU-GCN (CVPR’2021) for 3D point clouds segmentation and generation, G-TAD (CVPR'2020), VLG-Net (ICCVW’2021), MAAS (ICCV’2021) and VSGN (ICCV’2021) for large-scale video understanding, and DeeperGCN (arXiv’2020), 1000-layer GNN (ICML’2021) and FLAG (CVPR'2022) for node, link and graph level property prediction on Open Graph Benchmark (OGB) datasets which have graphs span nature, society and information domains. Are you excited about working on complex graph-structured data to make advances in biology, chemistry, physics, computer science, and so on? Would you like to use artificial intelligence to make fast predictions about the 3D structure of molecules, thereby speeding up the drug discovery process? Are you motivated by applications to precision medicine, and would like to create AI that learns to recommend what specific drug is suitable for a particular patient? Or perhaps you are more interested in higher-level abstractions, and would like to build an AI-based partial differential equation solver. All of these complex problems can be modeled through graph-structured data, and research in Graph Neural Networks can bring us closer to solving them. GNN has untapped potential in tackling graph based problems in Science and Engineering. However, more work is needed to explore the unique challenges to each scientific domain. In this project, you will have the chance to learn how to build large-scale graph neural networks and apply them to scientific and engineering applications.

BAS/1/1653-01-01

bernard.ghanem@kaust.edu.sa

graph neural networks

machine learning, AI for science

(i) Identifying the ground challenges of graph based problems in Science and Engineering; (ii) Collecting or processing the desired data into graph formats; (iii) Proposing novel GNN architectures and training techniques to tackle the challenges of learning on these graph data; (iv) Training and evaluating the proposed methods on specific metrics; (v) Producing well-performing and reproducible results and releasing the modular and reusable codebase to the research community.

Computer Science

Computer, Electrical and Mathematical Sciences and Engineering

Graduate or Undergraduate

Visual Computing Center