Sensors are critical components of the modern vehicle. They are the eyes of a car, enabling everything from existing ADAS (Advanced Driver-Assistance Systems) features such as automated braking and lane keeping to potential removal of the driver altogether. The consequences of these “eyes” not pointing in the right direction or not seeing clearly could be catastrophic; your car could needlessly break in the middle of the highway or suddenly swerve into another lane. Sufficiently high and safe sensor accuracy is essential, and calibration is critical to ensuring that a vehicle’s sensors are operating at the highest fidelity.
Sensors can be miscalibrated due to everything from daily normal use and changes in operating conditions (temperature or vibrations) to something more severe like accidents or part replacements. Unfortunately, very little emphasis has been placed on addressing the issue. This comes as no surprise; the automotive product cycle is incredibly long, and automated vehicles simply haven’t been tested long enough yet to thoroughly expose this issue.
Most standard perception sensors in the market today can perform intrinsic (refers to internal parameters of one sensor) calibration autonomously. However, extrinsic (refers to parameters relating multiple sensors together) calibration poses significant problems to fleets given the ever-increasing reliance on multiple sensors to overcome the shortcomings of individual sensors. Most calibration solutions today rely on picking functionally or economically inferior sensor configurations and/or simply hoping that the sensors never become miscalibrated from initial factory settings in the first place. Yet while this is obviously unsafe, there exist no common metrics to measure what it means for a sensor to be miscalibrated and no common standards that companies can hold their sensor calibrations up against. Every player in this space has their own unique sensor suites and an accompanying set of unique calibration practices, further complicating the matter.
Current aftermarket, maintenance, and return-to-service options are woefully underprepared to address the issue. Consider ADAS calibration at a typical maintenance shop. The procedure takes 15-120 minutes and requires expensive equipment (scanning tools, large and clear paved areas, alignment racks, etc.). The vehicle itself also needs to be prepared to meticulous standards; the fuel tank must be full, the tires must be properly inflated, the vehicle must be perfectly flat on a balanced floor, etc. Most garages and mechanics are underequipped and insufficiently trained to conduct what is an incredibly tedious and technically complex procedure. This ultimately causes improper calibration that endangers the vehicle’s passengers and those around them.
Innovations and opportunities in sensor calibration
Innovations around sensor calibration in academia generally fall into two schools of thought. Labs such as University of Toronto’s STARS Lab and The Stanford Driving Team are developing real-time self-calibration procedures since the task is too sensitive for a human to do reliably and correctly. As per Dr. Jonathan Kelly, these approaches generally rely on ground truth data (typically from IMUs or GPS) augmented by the sensor’s statistical error models and detection of common targets or appearance parameters in the environment to track the drift of sensor calibration parameters over time.
On the other hand, others such as the Takeda Lab at Nagoya University believe that any place, any time self-calibration won’t be feasible in the near term and are thus focused on developing an open-source measuring stick to compare calibrations of vehicles.
The most robust calibration method today relies on targets with unique patterns (e.g., checkerboards). A measuring stick for such procedures would visualize the targets detected, the relevant reprojection errors and statistics on these errors. Given that any target-based calibration can output these results, making them publicly available allows the transportation community to develop standards to hold all sensor calibrations against.
Innovations around sensor calibration in industry are highly variable based on where the relevant industry stakeholders sit in the automotive supply chain:
OEMs
For OEMs, we found a range of sensor calibration readiness. Some have a full lifecycle approach to vehicle development (including calibration), while others did not have much plans for calibration outside of what their after-service organization could handle. If automated vehicles become fleet-only for the near term, it is likely that OEMs become more involved in the calibration function for their customers.
AV fleets
Every fleet nearly reinvents the wheel when it comes to calibration. Calibration is often required almost daily, and current standard procedures are tedious and can take up to 60 minutes for an engineer or field technician to perform. Fleets like May Mobility see the value in a third party calibration-as-a-service provider since calibration is ultimately a maintenance task that takes away valuable time and capital from R & D. However, other fleets are more skeptical given the amount of customization that goes into their fleets’ sensor suites. We at Trucks believe the opportunity for a third party lies somewhere in the middle, and that it will only grow as sensor suites begin to consolidate and standardize to a common set of perception sensors.
Sensor hardware suppliers
Many sensor suppliers have developed their own in-house intrinsic calibration systems. Lidar start-up AEye provides an SDK that can trigger the self-calibration process and display diagnostic results. These results can then be used for maintenance and integration into system-wide extrinsic calibration.
Given the difficulty of implementing accurate self-calibration, it is crucial for sensor suppliers to make intelligent design decisions that mitigate miscalibration. In the same way that current MEMS or IMU sensors don’t need regular calibration or maintenance, OEMs ultimately don’t want sensors that require regular maintenance. In this vein, AEye designs solid-state instead of rotating iDARs. Other such decisions could be making better housings that protect sensors from vibrations or temperature changes.
Software suppliers
Calibration is not limited to just sensors — the AI perception algorithms built upon them need to be calibrated as well. Perception software is often directly trained on sensor data, making them highly dependent on the specific sensors and parameters that they were trained on. When any of these sensors are miscalibrated or replaced, the overlaying AI algorithms need to be recalibrated and retrained as well. While this may be an acceptable task in R & D labs, it is simply not scalable in the real world to expect aftermarket maintenance to recalibrate and retrain deeply technical AI.
LGN’s solution to this problem is to feed AI with a sensor’s latent space instead of its raw data. A sensor’s latent space is the abstraction of its input data derived using its intrinsic parameters. This has two key benefits:
- The latent space captures a level of redundancy within input data. This makes the AI more robust to data perturbations caused by miscalibration, sensor occlusion, etc. and allows one to normalize the AI and data across a more diverse array of sensors and operating conditions.
- In a multisensor system, one can predict what the sensor input data should look like because one can accurately estimate what the latent space should look like.
The ability to predict what sensors should be seeing is extremely powerful and makes it easy to pinpoint data anomalies. Once the system can confirm that an anomaly was caused by miscalibration, the latent space that should have been observed can be decoded to reverse calibrate the extrinsic parameters of the system as well as the intrinsic parameters of the faulty sensors. Ultimately, the latent space can act as the connective tissue that links together intrinsic and extrinsic calibration parameters.
Aftermarket solutions
Companies like Bosch are developing automatic alignment and leveling rigs to help technicians prepare vehicles for target-based calibration more consistently.
However, such equipment is incredibly complex, expensive and requires space and maintenance that most garages cannot afford. This is where Bridgestone and other aftermarket product suppliers/retailers see an opportunity.
Consider first an analogy proposed by Darren Liccardo: maintenance in the aviation space. Aviation maintenance shops are highly regulated and specialized. They also thoroughly record all maintenance done and are equipped with the latest tools and training. As cars become increasingly more automated, we can no longer rely upon today’s underregulated networks of garages and shops.
As mentioned by Reilly Brennan in his post on the future of auto maintenance, the advent of electrification and autonomy makes obsolete most traditional vehicle maintenance tasks. Companies like Bridgestone that have massive available real estate and capital, as well as traditional maintenance centers, should see this as an opportunity to evolve with vehicles and develop specialized shops for AV maintenance that are equipped with the right tools and properly trained technicians.
Ensuring that sensors operate at the highest fidelity
The rate at which automation is hitting the automotive market is much faster than the rate at which sensor calibration is being improved, and this has the potential to cause massive problems in operations, aftermarket and return-to-service scenarios. It is crucial that OEMs, suppliers, fleet operators, insurance providers, academia and creative third parties come together to develop the technology and infrastructure to ensure that our vehicles’ sensors operate at the highest fidelity.
If you have any thoughts on sensor calibration, feel free to contact me at puneeth@trucks.vc.