Model- vs. Data-Parallelism for Training of Deep Neural Networks
ApplyProject Description
Training of very large Deep Neural Networks is typically performed on large-scale distributed systems using the so-called data-parallelism approach. However, the scalability of this approach is limited by the convergence properties of the training algorithms. In this project, we willstudy a less common approach, called model-parallelism, which has the potential to overcome the convergence limitations. We will deploy and evaluate experimentally the two approaches, in order to understand the trade-offs. We will then design a hybrid method that will attempt to combine the benefits of the existing approaches.



About the
Researcher
Panagiotis Kalnis
Professor, Computer Science

Professor Kalnis's research interests are in Databases and Information management. Specifically, he is interested in: Database outsourcing and cloud computing, mobile computing, Peer-to-Peer, OLAP, data warehouses, spatial-temporal and high-dimensional databases, GIS, Security - Privacy – Anonymity.
Desired Project Deliverables
1. Experimental evaluation of model- versus data-parallelism.
2. Design and implementation of a hybrid approach.