Combining Deep Learning and Model Predictive Control for Safe, Effective Autonomous Driving

ABOUT THE PROJECT

At a glance

We are interested in developing autonomous driving algorithms that combine recent advances in end-to-end learning with the well-established techniques of optimal control, specifically model predictive control (MPC). By leveraging both approaches in mixed architectures, we hope to design more effective estimation and control algorithms with safety guarantees.

Motivation:

In academia and industry, there are two different approaches to autonomous driving. The traditional approach in the control community (Fig. 1a) is hierarchical, with separation between scene reconstruction, estimation, motion planning, MPC, and low-level actuation. One advantage of the traditional approach is that the design is modular and the functionality of each module is easy to understand and explain to non-experts in case of failure (such as a lawyer). Another advantage is that constrained MPC is an optimal control policy that provides safety guarantees for collision avoidance and lane keeping [1]. 

More recently, there has been interest in applying deep learning to autonomous driving. Many studies use end-to-end learning, where input images and sensors are used directly to produce an action (Fig. 1b) [2]. In this method, there is no explicit and man-made hierarchical separation, which may reduce the impact of error propagation through different human-designed modules [3]. Also, it has the potential of enormously speeding up the control tuning and scene reconstruction effort. However, safety validation for deep learning approaches is difficult, and it is hard to collect data to capture all edge cases. 

Our objective is to study non-trivial ways to combine benefits of hierarchical control and deep learning. We will investigate architectures that improve performance through learning, while ensuring safety. 

Figure 1. Simplified Autonomous Driving Architectures. (1a) Left: classical model-based approach (1b) Right: End-to-End Approach.

Research Challenges:

We study mixed architectures, building on our recent work on safe data-driven iterative learning [4]. In all cases, the challenge is formal stability and safety guarantees. 

The first architecture is hierarchical (i.e. series composition), using a deep neural network to combine sensor fusion and trajectory generation, as in [5]. MPC will track the trajectory generated by the neural network. We have completed behavior cloning experiments on the Hyundai Genesis vehicle (Contributions, 2). 

The second architecture (i.e. parallel composition) is a switched control approach that uses the classical optimal control as “training wheels”. A safety verification module checks whether the vehicle is likely to collide or leave the lane in a given time horizon. If so, the MPC control is used to provide optimal inputs to ensure safety. If not, then the end-to-end controller (e.g. [3]) is allowed to operate and learn and improve. 

References:

[1] Carvalho, A., Lefévre, S., Schildbach, G., Kong, J., & Borrelli, F. (2015). Automated driving: The role of forecasts and uncertainty—A control perspective. European Journal of Control, 24, 14-32. 

[2] Xu, H., Gao, Y., Yu, F., & Darrell, T. (2016). End-to-end learning of driving models from large-scale video datasets. arXiv:1612.01079. 

[3] Bojarski, M., ... & Zhang, X. (2016). End to end learning for self-driving cars. arXiv:1604.07316. 

[4] Rosolia U and Borrelli F (2017), "Learning Model Predictive Control for Iterative Tasks. A Data-Driven Control Framework.", IEEE Transactions on Automatic Control., December, 2017. Vol. PP (99), pp. 1. 

[5] Caltagirone, L., Bellone, M., & Svensson, L. (2017). LIDAR-based Driving Path Generation Using Fully Convolutional Neural Networks. arXiv:1703.08987 v2. 

[6] Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V. (2018). Driving Policy Transfer via Modularity and Abstraction. arXiv:1804.09364.