Berkeley DeepDrive | We seek to merge deep learning with automotive perception and bring computer vision technology to the forefront.

Guiding the Next Generation of Research

We are at the forefront of research on deep automotive perception through the integration of two very important technologies: vision and vehicles.

Our Mission

We seek to merge deep learning with automotive perception and bring computer vision technology to the forefront.

ABOUT DEEPDRIVE

We're driving the future of automotive perception.

The Berkeley DeepDrive Industrial Consortium investigates state-of-the-art technologies in computer vision, robotics, and machine learning for automotive applications. Our multi-disciplinary center is housed at the University of California, Berkeley and is directed by Professor Trevor Darrell, Professor Kurt Keutzer, Dr. Ching-Yao Chan and Dr. Wei Zhan. The BDD consortium partners with private industry sponsors and brings faculty and researchers together from multiple departments and centers to develop new and emerging technologies with real-world applications in the automotive industry. Research is proposed by UC Berkeley faculty and approved by a BDD advisory board composed of faculty and sponsor representatives. Although dramatic progress has been made in the fields of computer vision and robotics, many of these technologies and theories have yet to carry over to the automotive field. Thus, the need and driving force behind the Berkeley DeepDrive Center.

Featured projects

We are excited about our work. See why.

Deep Reinforcement Learning

By watching many videos of moving objects, the team’s new tracker learns the relationship between appearance and motion that allows it to track new objects at test time. This tracker thus establishes a new framework for tracking in which the relationship between appearance and motion is learned offline in a generic manner.

Cross-modal Transfer Learning

This project seeks to transfer models for vision tasks like object detection, segmentation, fine-grained categorization and pose-estimation trained using large-scale annotated RGB datasets to new modalities with no or very few such task-specific labels.

Clockwork FCNs for Fast Video Processing

The team will use a novel, frame-asynchronous, “clockwork” convnet, which pipelines processing over time with different network update rates at different levels of the semantic processing hierarchy, significantly reducing frame processing time with little or no degradation in performance.

ALL projects

See all of the ways we’re driving the future.

Integrated Fine-Grained 3D Reconstruction and Narrow-Space Planning for Autonomous Parking

Truckload Optimization via Task Planning based on Deep Reinforcement Learning

Advancing Vehicle Safety Assessment: A Scalable Safety Benchmark Integrating Probabilistic Motion Heatmaps Prediction for Autonomous Vehicles

Searchable Database of Potential Crash Scenario Models for CARLA and the Corresponding Sensor Data Feeds

Model-Based Testing meets Large Language Models: Automating Intelligent Scenario Generation and Safety Assessment with ScenicNL

SE(3)-equivariant vision-based skill acquisition for robotic manipulation in contact-rich environments

Anomaly Detection in Autonomous Driving Systems through Multiple Transformer Interpretations

3D Object Detection Enhanced by Temporal Multi-View Input

Understanding Corner Cases for Vehicle Detection and Tracking

Collision Indeterminacy Prediction via Stochastic Trajectory Generation

Cross-Domain Self-Supervised Learning for Adaptation and Generalization

Analysis and Synthesis of Road Vehicle Conflicts using Vehicle Trajectory Data

Safe and Effective Perception and Control through Formal Simulation

Autonomous Driving in Unstructured Stochastic Intersections

Completed Projects

See all of the ways we’re driving the future.

Efficient and Consistent Joint Multi-Agent Prediction for Interactive Planning

Online Automatic Construction and Update of HD Maps with Rich Semantics

Computationally Efficient Option-based Monte Carlo Planning for POMDP

Theoretical and Empirical Investigations of Active Learning and Multi-agent Learning in Autonomous Vehicles

Systematic Quantization on Vision Models for Real-time and Accurate Inference in ADAS/AV

Improving OOD Generalization Through Metric-informed Weight-space Augmentation and Architecture Search

Factual, Controllable, and Interactive Dialogue Agents

Simultaneous On-Target Domain Adaptation and Performance Prediction

Learning in repeated Stackelberg interactions with partial feedback in autonomous driving

Multi-Modal Self-Supervised Pre-training for Label-Efficient 3D Perception

In-Car AI Assistant: Efficient End-to-End Conversational AI System

Real-time and Accurate Object Detection through Systematic Quantization of Transformer and MLP-based Computer Vision Models

Measuring Prediction Trustworthiness and Safety Through Neural Network Loss Landscape Analysis

Tracking people and objects in 3D

Intention-aware Pedestrian Motion Modeling and Prediction

Safety Evaluation of Automated Driving Systems

Self-supervised Visual Pre-training for Motor Control

Driver Drowsiness Analysis Based on Multiple Facial Features

Sequence Modeling as Reinforcement Learning

Few-Shot Imitation and Embodiment Transfer

Spatio-temporal 3D reconstruction of Pedestrians, Objects, and the Environment from Self-Driving Data

Bridging Simulation and Real Data with Dynamic SCENIC Programs

Automatic Inference of Lane Topology and Connecting Geometry for Scene Understanding, High-Definition Map Construction and Representation

Representation Learning for Person Segmentation and Tracking

Data Efficiency for Point Cloud Perception Model

Efficient Deep Learning for ADAS/AV Through Systematic Pruning and Quantization

Design Automation of Out-of-Distribution Image Data Detectors

Similarity Learning for Multiple Object Tracking

Meta Neural Architecture Search For Computer Vision

Hyperbolic Embedding for Natural Visual Scene Recognition

Learning Video Models Using VQ-VAE and Transformers

Hessian Aware Neural Cleansing: Searching for Optimal Trade-offs between Adversarial Robustness, Accuracy, and Speed

Embedded In-Car AI Assistant: Efficient End-to-End Speech Recognition and Natural Language Understanding for Command Recognition at the Edge

Implicit Communication of Automated Vehicles: Using Vehicle Movement Cues for Coordination with Pedestrians at Intersections

Vehicle Dynamic Estimation based on Image Sensor & Radar Fusion

Pedestrian Trajectory Prediction Combining Posture and Social Features

Offline Reinforcement Learning for Data-Driven Autonomous Driving

Unified Neural Architecture Search Framework for Computer Vision

Infrastructure, Data, and Related Research

Using High-Level Structure and Context for Object Recognition

BDD 100K and Beyond Data Labeling Efforts

Uncertainty-Aware Reinforcement Learning for Interaction-Intensive Driving Tasks

Best of both Worlds: Combining Classical Planning with Learning for Long-horizon Tasks

3D Object Detection with Temporal LiDAR data for Autonomous Driving

How Many Accidents Can AVs Avoid?

Off-Policy Reinforcement Learning for Learning from Logged Data

Test-time Training for Adaptive and Continuous Learning

Predicting Pedestrian Behavior from 3D Cues

Domain Adaptation with Scene Structure

Principled Defenses Against Adversarial Attacks on Deep Learning

Scene Understanding and Forecasting in Contextual Environment – From Pixel to Semantic

Adversarially Robust Perception for Autonomous Driving

Embracing Uncertainties in 3D Object Detection -- from Annotation and Benchmarking to Failure Analysis and Data Augmentation

Bridging Simulation and Real-World Data with SCENIC Queries

Automatic Semantics Extraction and Representation for High-Definition Map Construction and Scene Understanding

Learning Ultra-Lean Classifiers with Structured Functions and Binary Lifting

Visual Landmarks for Long Range Semantic Navigation

Object-centric Spatial-Temporal Interaction Networks for Rare Event Recognition

Agent Behavior Understanding in Crowds – Predicting Future Trajectories and Activities

Mid-Level Visual Representations for Visuomotor Policies

Data-Driven Generation and Analysis of Hazardous Traffic Scenarios

Training Data Augmentation for Autonomous Driving

Efficient Model for Large-Scale Point Cloud Perception

Systematic Study of Neural Network Robustness Against Adversarial Attacks

Real World Traffic Smoothing Demonstration with Deep-RL Trained CAVs

Auto-tuned Sim-to-Real Transfer

Uncertainty-Aware Meta-Learning and Safe Adaptation

Learning to Compile Mobile and Embedded Vision Code with Halide

Common Representations for Perception, Prediction, and Planning

Hybrid Human Driver Predictions

Advisable Deep Driving

Explainable Deep Vehicle Control

Self-driving by multi-objective reinforcement learning with goal-conditioned policies

Robust Perception for Autonomous Driving

Pixel-Level Confidence Prediction for Interpretable Network-Based Driving (ConfPix)

Towards Safe, but not Overly Conservative, Autonomous Cars

Towards safe self-driving by reinforcement learning with maximization of diversity of future options

Human Action and Intention Prediction in Urban Environments

Perception-Driven Safe Autonomy in Uncertain Environments

Investigating driver’s attention while monitoring an autonomous vehicle

Adversarially Robust Visual Understanding

Modeling and Learning Multi-Agent Behaviors for Simulation and Analysis of Autonomous Driving Scenarios

Trust-Region Based Robustness of Neural Networks in the Face of Adversarial Attacks

Model-Based Reinforcement Learning Approach for Trajectory and Intent Prediction of Vulnerable Road Users

Superhuman Vision for Superhuman Driving

An Innovative Approach to the Dual Problems of High-Resolution Input and Video Object Detection

Enabling Valuable Training Data through Domain Adaptation

Combining Deep Learning and Model Predictive Control for Low-Cost ECUs

Data-Driven Synthesis of Hazardous Scenarios at Traffic Intersections

Data Augmentation via Synthetic Point Cloud for 3D Detection Refinement and Domain Adaptation with Different LiDAR Configurations

Localization and 3D Detection via Fusing Sparse Point Cloud and Information from Visual-Inertial Navigation System

Deep Learning-Based Vehicle Control Strategist for Autonomous Vehicles

Life-Long Learning to Drive by Semi-Supervised Reinforcement Learning (SSRL)

Autonomous Aerial Robots in Dense Urban Environments

Learning Dynamic Point Set Neighbourhoods for 3D Object Detection

Safe and Effective Learning and Control through Formal Simulation

Automated Search for Neural Net Architectures

Automated Search for Neural Net Architectures

Self-supervised Representation Learning for Autonomous Driving

Multi-modal Fusion of Deep Convolutional Neural Networks for 3D Object Detection

Embedded natural language processing for in-car speech commands

Learning Human-Like Decision-making Behavior based on Adversarial Inverse Reinforcement Learning

Imitative Models: Learning Flexible Driving Models from Human Data

Efficient Neural Networks through Systematic Quantization

Adaptive Traffic Signal Control based on Deep Reinforcement Learning

CyCADA: Cycle Consistent Adversarial Domain Adaptation

Which Intersections Need I2V and When?

Graph-to-Graph Transfer in Geometric Deep Learning

Connectionist Representations of Compositional Structure for Sensor Integration and Situational Awareness in Autonomous Vehicles

Real-time, Energy Efficient Video Object Detection and Tracking

Domain Adaptation for Realistic LiDAR Data Synthesis

Efficient Neural Nets for Real-Time object-recognition on High-Resolution Images

Time-Series LIDAR and Camera Fusion with Structured Variational Inference

Guaranteed Adaptation and Exploration in Uncertain Environments

Safe and Effective Learning through Formal Simulation

From Optimization to Sampling: Efficient Methods for Inference With High-Dimensional Distributions

Fusion of Deep Convolutional Neural Networks for Semantic Segmentation and Object Detection

Self-supervision, Meta-supervision, Curiosity: The Harder to Learn, the Easier to Generalize

Public-Road Demonstration of the BDD/HCE Autonomous Driving Car

Training and Hardware-Software Co-design for Implementation of Small- Footprint Neural Networks

Uncertainty-Aware End-to-End Prediction for Robust Decision Making

Generation and Mitigation of Hazardous Scenarios at Traffic Intersections

Interaction and Communication Between Pedestrians and Autonomous Vehicles

Implementing and Evaluating Second-Order Optimization for Deep Learning on Diverse Architectures

Interpretable and Data-Efficient Driving Behavior Generation via Deep Generative Probabilistic and Logic Models

Fast Inverse Vehicle Dynamics Adaptation for Driving Policy Transfer via Meta Learning

Impacts on Energy Consumption of Autonomous Vehicles in the Bay Area

Learning Urban Driving Policies from Online Traffic Webcams

AutoPylot: An Open Platform for Autonomous Vehicles

A Programming Language for Differentiable Image Processing

Risk-Averse, Adversarial Reinforcement Learning

Explainable Deep Vehicle Control and User-Defined Constraints

Spatial and Visual Memory for Navigation

Combining Deep Learning and Model Predictive Control for Safe, Effective Autonomous Driving

Model-Based Reinforcement Learning

Robust Visual Understanding in Adversarial Environments

Detecting and Diagnosing Abnormal Behaviors from Autonomous Driving Logs

Bringing in the Human Factor to Resolve Three Fundamental Problems of Deep Learning in Autonomous Driving

Model-based Reinforcement Learning in a Latent Space

Model-Free Reinforcement Learning Through the Optimization Lens

Computing Confidence in Human Driver and Pedestrian Models

Interpreting Deep Policies via Critical States

Maneuver Control based on Reinforcement Learning for Automated Vehicles in an Interactive Environment

Testing and Verification of Safe Network-Based Driving Algorithms

Learning Timely Ego-centric Visual Attention for Smooth Driving

Deep Reinforcement Learning for Robot Navigation

Pedestrian-Vehicle Interaction – Data Sets and Analysis

Exploiting Environment and In-Vehicle Information from Multi-Sensors for Autonomous Driving Policy Adaption

Reinforcement Learning for Automated Driving – A Use Case of Ramp Merge Automation

Infrastructure and Data 2018

Operationalized Active Learning

Fundamental Tradeoffs in Learning and Control

Behavior Forecasting in Structured Environments

Adversarial Examples in Reinforcement Learning and Imitation Learning

The Dynamical View of Machine Learning Systems

Adversarial Deep Learning for Autonomous Driving

Energy-Efficient Deep Neural Nets for Occupancy-Grid Mapping

Optimal Trade-Offs in Distributed Optimization for Large-Scale Learning

Acting and Learning under Uncertainty

Deep RL with Dexterous Hands and Tactile Sensing

Hierarchical Reinforcement Learning & Abstraction Discovery

Training Real-World Robotic Skills in Diverse Simulated Environments

Enabling Passenger Comfort by Increasing Behavior Transparency

Robotic Assembly from CAD

Intelligent Intersection

Generic Motion Generation and Comprehension with Social Interactions

3D Object Detection based on Lidar and Camera Fusion

Offline vs. Online Learning for Deep Driving from Demonstrations

Design Space Exploration for Deep Neural Nets for Advanced Driver Assistance Systems

Unsupervised Representation Learning for Autonomous Driving

Low Latency Deep Inference for Self-Driving Vehicles

Fast Simultaneous Object Detection and Segmentation

Clockwork FCNs for Fast Video Processing

Motion Prediction for Urban Autonomous Driving Based on Stochastic Policy Learned via Deep Neural Network

Secure and Privacy-Preserving Deep Learning

Predictable and Customizable Autonomous Driving

Cross-modal Transfer Learning

Real-Time Perception/ Prediction of Traffic Scene

FPGA PRET Accelerators of Deep Learning Classifiers for Autonomous Vehicles

Outdoor Semantic Scene Segmentation via Multi-modal Sensor

Deep Reinforcement Learning based Optimization of Autonomous Vehicle Traffic

Deep Reinforcement Learning

Improving the Scaling of Deep Learning Networks by Characterizing and Exploiting Soft Convexity

Augmenting Autonomous Vehicle Technology with I2V Information

Understanding Driver Awareness for Smart Vehicles

Learning to Drive Under Unstructured Conditions

Verifiable Control for (Semi)Autonomous Cars that Learns from Human (Re)Actions

Researchers

Our talented team of thought leaders

Alexandre Bayen

Francesco Borrelli

Maria Laura Delle Monache

Amir Gholaminejad

Joseph Gonzalez

Roberto Horowitz

Angjoo Kanazawa

Michael Mahoney

Alberto Sangiovanni-Vincentelli

Masayoshi Tomizuka

Peggy Wang

Fisher Yu

Bin Yu

Wei Zhan

Our sponsors

We are proud to partner with these industry leaders