The autonomous vehicle testing disengagement reports are published annually by the Department of Motor Vehicles, State of California (DMV). The reports published details the figures on total miles travelled in autonomous mode and the number of times driver intervened to take control of the vehicle, also called as disengagement. DMV offers permits to companies willing to test their self-driving systems on roads of California, which in return has to report the figures to DMV. While, there are 64 companies holding permits for autonomous vehicle testing, only 36 companies actually reported autonomous testing on California roads.
Reports shows while testing Waymo and Cruise autonomous vehicles required less human driver assistance
Reports shows that Waymo’s autonomous vehicles performs better with involvement from human driver on an average of 13,219 miles, followed by GM’s Cruise which required disengagement at 12,221 miles. Both these companies also recorded highest on-road testing of total 1.45 million and 831 thousand miles respectively in 2019. Baidu, which is also developing its self-driving system has tested total of 108 thousand miles on California roads and registered disengagement at an average of 18,050 miles. Certainly, Baidu’s autonomous system performed well in terms of disengagements as compared to that of Waymo’s and Cruise’s autonomous systems, however the total on-road tested miles are comparatively less.
Of the top 5 performers, 3 are Chinese companies testing vehicles in US. Apart from Baidu, AutoX and Pony.ai has showed admirable results on disengagement rates. Pony.ai which tested their fleet of 22 autonomous vehicles for 174,845 miles registered disengagement at an average of 6,476 miles. While AutoX with a fleet of 8 autonomous vehicles tested 32,054 miles and recorded disengagement at an average of 10,685 miles.
One of the notable companies listed in the report is Apple. Apple as always is not much open about its developments in the self-driving space although Apple did confirm about their work on autonomous driving system. Even though Apple is a latecomer in the AV development race, they are aggressively working on developing solutions for shared mobility services. Apple tested fleet of 70 autonomous vehicles which drove a total of 7,544 miles and recorded human driver intervention on an average of every 118 miles.
The other prominent competitor in the race is Tesla. Tesla which is aggressively working on its AutoPilot system recently claimed that it will put 1 million vehicles by the end of 2020 for robotaxis services, but clarified that this would be only possible with regulatory approvals. Tesla has made some extraordinary claims about its AutoPilot systems that are currently fitted in all their models at additional cost. The DMV results shows Tesla drove only one autonomous vehicle for approximately 12 miles with no disengagement recorded. The company claims that it is gathering road environment scenario data from its vehicles globally and training its neural networks for better performance.
Below is a comparative graph on the total autonomous miles travelled vs miles per disengagement by 36 permit holders.
Is disengagement a valid metric?
Although autonomous vehicle disengagement rate is the only standardized measure of autonomous testing available to the public, many companies who already hold permits, claims disengagement report is not an accurate metric to offer insights into the capabilities of autonomous vehicle. According to them autonomous vehicles operate under varied circumstances and scenarios in which safety driver takes control from the autonomous system. Most of the time, safety driver takes control of the system just because of the pressure of possible collision. Although DMV regulations does require companies to add details of what triggered the disengagement. Some of the interesting views from the industry are quoted below -
Waymo official twitter handle twitted - “We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights into the capabilities of the Waymo driver or distinguish its performance from others in the self-driving space.”
Founder and CTO, Cruise in official blog posted - “The idea that disengagements give a meaningful signal about whether an AV is ready for commercial deployment is a myth.”
CEO, Aurora Innovation in a recent blog post said - “If the company or the public is measuring by disengagement rate, then you create this kind of odd incentive. The drivers in the vehicles will be like, ‘Oh geez, I don’t know if I want to just engage at that moment, because it’s a pressure.”
Apart from the above mentions there are numerous other views that says DMV results should not be taken seriously for understanding the development of autonomous driving solutions.
Even if the self-driving systems are developing rapidly, the industry still has to work in collaboration to assist regulatory bodies in developing frameworks for future mobility. As regulatory approval is one of the major inhibitors in deploying self-driving systems in near future.
Autonomous Vehicle Development Race
Read in-depth analysis on autonomous vehicle development maturity on the leading OEMs and self-driving solutions providers