A Test and Training Facility for Autonomous Driving Research with Model Cars


At a glance

In order to realize the full potential of the BDD model car framework for studying autonomous driving, we propose to build a test and training facility for these model cars at the Richmond Field Station (RFS). Previously, aside from outdoor fieldwork, Berkeley model car research was limited to ad-hoc data collection and testing in University hallways, office spaces, as well as in private residences. While this state of affairs did not preclude rigorous performance measurements at each project location, it made it difficult to compare or replicate the capabilities of different neural network models and techniques across locations. This was unfortunate, because comparison and replication of results across projects is perhaps one of the scientifically richest aspects of the model car research. The RFS model car facility will address this need by allowing for the development of standardized environments and localization-system based performance metrics which will allow for quantitative study of model car behavior. This facility will serve as a research hub for several projects within the BDD community and help foster this community with exchange of ideas, code and hardware. It will furthermore serve as an important basis for approaching the problem of transferring model car networks to full-scale BDD cars.

We argue the value of the proposed facility with three examples of the advantage that performance metrics yield when applied in standardized environments.: First, when training networks to exhibit complex behavior as they control the cars, the goal is not only to show that the behaviors approach optimality according to a defined metric: the networks orchestrating these behaviors should also become understandable by external inspection. Scene interpretation and driving decisions by the network must become traceable, predictable and robust in order to ensure safe autonomous driving, and to help to verify their cognitive plausibility. Standardized and controllable conditions of the type we intent to construct at the RFS will support progress through reproducible and fine-tunable experimental setups. Second, it must be shown, through transfer to full sized vehicles, that there is an actual advantage in training model cars to perform safely in dangerous maneuvers and extreme situations. Only a careful characterization of models across generations and across research groups, will allow for the establishment of stable benchmarks which can then be applied to the full-scale car. Third, with multi-car driving scenarios, the possible configurations of cars within the environment is complex in itself. Without standardized environments in which to study multi-car behavioral systems, which involve real-time interaction of multiple neural networks via the environment, the problem of understanding how a given car reacts to the environment becomes much more difficult to define and compare across project iterations or separate research team strategies.

The Model Car Framework

Measuring performance will be a key focus of the test facility. Autonomous driving with model cars in general lacks evaluation metrics which measure and characterize subtle behavioral changes – as opposed to dramatic failures such as counting the number of crashes in a given time. In the past year we investigated different techniques of measuring a given networks driving behavior to quantify the impact of different training procedures and data-sets. Based on our experience we propose to develop evaluation metrics in three different ways: First, by comparing trained trajectories before and driven trajectories after training (Black Box Testing) to  get statistical measures about the progress of training. Second, taking steps towards White Box Testing by characterizing sensory input into the trained system and tracing the impact on navigational performance. Third, by in-depth investigation of individual deep network layers, i.e. through means of visualization, and performance on solving side-tasks as in [1]. Standardized test routes with obstacles and paths which are predetermined to be relevant to validate trained networks will enable us to compare performance differences across network training styles and designs. Effects of different environment conditions, (simulated) times of the day, etc. can be investigated in detail and with fine control. Furthermore for training, objective information about a given car’s own position and all other car positions from a localization system will give richer feedback for machine learning beyond simple least-squared error of steering decisions.

What are the requirements of the training-testing facility? Various research approaches, striving to solve the path following problem for autonomous cars, fell short of producing a robust, generalizable driving behavior when the environment became too challenging and diverse [2]. In our BDD work, a dataset with over 200 hours of video data was created in the past two years; the dataset contains scenes such as collisions and near-collisions, car following, obstacle avoidance, track following, night and day driving as well as driving in snow, in parks and on sidewalks. We showed in the past two years that our model cars can traverse rough terrain when trained  to imitate human behavior under these diverse conditions. Incrementally our network models and training procedures were refined to outperform state of the art results in Behavioral Cloning [3][4][5]. It therefore seems that in addition to the sheer volume of data, the diversity of environments is of key importance.  Building  this possibility of diversity of training data into a fixed-location facility at RFS will entail variable lighting, projection of images and textures, and movable elements, all of which provide ample scope for diversity within a single facility.

Incrementally, obstacles, model-sized scenery and measuring systems can complete the test-facility and pose as an option for research in-between pure driving simulation (e.g., the CARLA simulator) and real-life driving. This will not only facilitate research on im- proving and understanding the networks behavior but also for closing the gap between driving in simulation and transferring the results to real driving. To this end, and to be able to use the same trained network- model on small- and full-scaled cars interchangeably, we intend to use the same computing architecture  as currently in use within the Lincoln MKZ at the RFS. The model cars will have the option of perform- ing their network execution remotely on a multi-GPU System76 workstation by sending their video feeds wirelessly where processing of the input video takes place on the workstation before steering and motor signals are sent back to the car. This means that the more computationaly-expensive models developed for real-car driving will be usable in real time with the model cars – an essential step for research compatibility within the BDD program. Since ROS is already used as Middleware on all systems, changing from small model cars to the Lincoln MKZ will be as easy as routing messages to a different IP address.


  1. Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end Learning of Driving Models from Large- scale Video Datasets. pages 2174–2182, 2016.
  2. Tiffany Hwu, Jacob Isbell, Nicolas Oros, and Jeffrey Krichmar. A Self-Driving Robot Using Deep Convolu- tional Neural Networks on Neuromorphic Hardware. 2016.
  3. Eric Hou, Sascha Hornauer, and Karl Zipser. Fast Recurrent Fully Convolutional Networks for Direct Perception in Autonomous Driving. arXiv, 2017.
  4. Sauhaarda Chowdhuri, Tushar Pankaj, and Karl Zipser. Multi-Modal Multi-Task Deep Learning for Au- tonomous Driving. arXiv preprint arXiv:1709.05581, 2017.
  5. Ye Xia, Danqing Zhang, Alexei Pozdnukhov, Ken Nakayama, Karl Zipser, and David Whitney. Training a network to attend like human drivers saves it from common but misleading loss functions. pages 1–14, 2017.
principal investigatorsresearchersthemes
Karl ZipserStellu Yu
Sascha Hornauer
Bruno  Olshausen