​Training of very large Deep Neural Networks is typically performed on large-scale distributed systems using the so-called data-parallelism approach. However, the scalability of this approach is limited by the convergence properties of the training algorithms. In this project, we willstudy a less common approach, called model-parallelism, which has the potential to overcome the convergence limitations. We will deploy and evaluate experimentally the two approaches, in order to understand the trade-offs. We will then design a hybrid method that will attempt to combine the benefits of the existing approaches.​ ​​​​​