Combining Deep Learning and Model Predictive Control for Safe, Effective Autonomous Driving


At a glance

We are interested in developing autonomous driving algorithms to combine the benefits of recent advances in end-to-end learning for driving with the well-established techniques of optimal control, specifically model predictive control (MPC). By leveraging both approaches in three different architectures, we hope to design more effective estimation and control algorithms that also provide a guarantee on safety.

In academia and industry, there are two different approaches to autonomous driving. The traditional approach in the control community (Fig. 1a) is hierarchical, with separation between scene reconstruction, estimation, motion planning, MPC, and low-level actuation. One advantage of the traditional approach is that the design is modular and the functionality of each module is easy to understand and explain to non- experts in case of failure (such as a lawyer). Another advantage is that constrained MPC is an optimal control policy that provides safety guarantees for collision avoidance and lane keeping [1].

More recently, there has been interest in applying deep learning to autonomous driving. Many studies use end-to-end learning, where input images and sensors are used directly to produce an action (Fig. 1b) [2]. In this method, there is no explicit and man-made hierarchical separation, which may reduce the impact of error propagation through different human-designed modules [3]. Also, it has the potential of enormously speeding up the control tuning and scene reconstruction effort. However, safety validation for deep learning approaches is difficult, and it is hard to collect data to capture all edge cases.

Our objective is to study non-trivial ways to combine benefits of hierarchical control and deep learning. We will investigate architectures that improve performance through learning, while ensuring safety.

Figure 1. Simplified Autonomous Driving Architectures. (1a) Left: classical model-based approach (1b) Right: End-to-End Approach.

Research Challenges
We will study three mixed architectures to combine the hierarchical and deep learning approaches. In all cases, the research challenge to address is formal stability and safety guarantees of the resulting sensing and control algorithms. We build on our recent work on safe data-driven iterative learning [5] and propose to develop novel control design techniques which merge advanced model-based control and deep learning.

Phase 1. The first architecture is a hierarchical approach (i.e. series composition) that uses a deep neural network to combine the sensor fusion and trajectory generation steps, as has been done in [4]. MPC will be solely used to track the reference trajectory generated by the neural network. We do not envision major challenges with this approach and it will be used for basic comparison with the other two approaches next.

Phase 2. We will study a switched control architecture (i.e. parallel composition) that uses the classical optimal control as “training wheels”. There will be a safety verification module to check whether the vehicle is likely to collide or leave the lane in a given time horizon. If so, the MPC control is used to provide optimal inputs to ensure safety. If the probability of collision or lane departure is low, then the end-to-end controller (e.g. as in [3]) is allowed to operate and learn. The expected outcome is that the end- to-end controller can learn and improve while ensuring collision avoidance and lane keeping.

Phase 3. The third architecture will consider interconnections from intermediate layers of the neural network passed to the MPC controller and vice-versa, from intermediate MPC variables (such as Lagrange multipliers) to the deep neural network. The goal is to explore a deep combination of MPC and neural network architectures and evaluate if there are any benefits over serial/parallel compositions.

1. Carvalho, A., Lefévre, S., Schildbach, G., Kong, J., & Borrelli, F. (2015). Automated driving: The role of forecasts and uncertainty—A control perspective. European Journal of Control, 24, 14-32.
2. Xu, H., Gao, Y., Yu, F., & Darrell, T. (2016). End-to-end learning of driving models from large-scale video datasets. arXiv preprint arXiv:1612.01079.
3. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., ... & Zhang, X. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
4. Caltagirone, L., Bellone, M., & Svensson, L. (2017). LIDAR-based Driving Path Generation Using Fully Convolutional Neural Networks.
arXiv preprint arXiv:1703.08987 v2.
5. Rosolia U and Borrelli F (2017), "Learning Model Predictive Control for Iterative Tasks. A Data-Driven Control Framework.", IEEE Transactions on Automatic Control., December, 2017. Vol. PP (99), pp. 1.

Francesco Borrelli
Trevor Darrell
Vijay Govindarajanadvanced machine learning for autonomous driving, computer vision and autonomous perception, model predictive control (MPC), end-to-end control, mixed control architectures