Sensor Fusion: Enabling Vehicles to Perceive the World Like People Do

Automotive designers today face a range of challenges related to the evolving market in advanced driver assist systems (ADAS), autonomous vehicles and electric vehicles (EVs). ADAS and autonomous vehicle computers are rapidly expanding their capabilities, but they require a suite of sensors to provide the data necessary to function. Designers must match the needs of the vehicle’s autonomous systems to the capabilities of the sensor package, providing enough input for the vehicle to function safely and efficiently. They also need to optimize the space and weight imposed by sensing devices by selecting sensors with multiple applications – a capability known as “sensor fusion.” 

As ADAS technology marches forward and autonomous driving capabilities advance, it is increasingly clear that vehicle sensing cannot stop at visual and radar capabilities. Tomorrow’s vehicles will require automated systems that can monitor the environment, placing more demands on the vehicle’s sensing capabilities. These vehicles need to perceive their surroundings the way a human does: with a mixture of visual, audio and touch cues. The combination of these varied inputs enables the vehicle to build a complete picture of its environment and act appropriately in response.  

The exploding popularity of EVs also places new demands on managing road noise for passenger comfort and safety. In vehicles with internal combustion engines, the engine generates a relatively consistent level of background noise. This helps mask irritating or distracting road noise. Without that engine, occupants of EVs experience higher perceived levels of road noise, causing greater discomfort and distraction. This, in turn, can adversely affect driver concentration and safety. 

The most common methods for driver assist sensing are line-of-sight sensors. These include cameras, radar, and LiDAR, which work together to detect obstructions or hazards that are within the visual range of the vehicle. While these sensing methods are crucial to effective ADAS and autonomous vehicle technology, they don’t deliver the full picture.

Sound and touch are valuable inputs for human drivers, and this is equally true for automated driving systems. External microphones can detect a range of potential hazards that are difficult or impossible for line-of-sight sensors to perceive. Sound allows the detection and triangulation of other vehicles, like emergency vehicles, long before they are visible. Adverse road conditions, such as deteriorated pavement or gravel surfaces, are better assessed by analyzing road noise captured by sound and touch than by visual cues. A camera may be able to detect a rainstorm, but it is easier to judge its severity using sound.

These sensors can also improve safety through ongoing condition-based monitoring. By analyzing the sounds or vibrations made by the vehicle itself, automated systems can detect and address problems or malfunctions early. This is helpful for ADAS and necessary for automated vehicles, which must be capable of taking action to address any issues that arise with their operation.

In addition to sound, drivers rely on the sense of touch to know when the vehicle has struck (or has been struck by) something or when there has been any unusual force placed on the vehicle, such as black ice. Enabling a vehicle to add a sense of touch to its perception using precision accelerometers permits its systems to react to a range of stimuli that visual sensors might not detect. Detecting the direction and force of an impact can tell the vehicle whether to brake, sound an alarm or simply record an occurrence for review or maintenance attention. This capability is especially vital for autonomous vehicles but it also adds value for more traditional vehicles. 

Active Noise Cancellation with internal and external microphones and RNC sensors can be used in very loud vehicles such as construction trucks, agricultural vehicles and aircraft. In these applications, noise cancellation does not simply assist with preventing fatigue, it can help save the operator’s hearing by lowering the ambient noise level. Using sensors to measure sound in real time makes these measures more effective, making in-cabin noise less harmful to the operator.

External audio sensors assist ADAS in building a complete picture of the environment surrounding the vehicle. These microphones enable “seeing with sound” by triangulating the direction and motion of potential hazards, helping fill in the gaps where the vehicle’s visual and radar sensors cannot see. Whether gauging the approach of an emergency vehicle from around the corner or tracking a vehicle hidden in a blind spot, these sensors can provide valuable information. 

Autonomous vehicles depend on sensors to monitor their environment, and audio and touch sensors play a key role. Microphones provide input in a similar fashion to ADAS applications by filling in the gaps between what cameras and radar can perceive and creating a complete picture of the vehicle’s surroundings. Touch sensors make it possible for the vehicle to react to impacts or external forces, a vital capability for driverless vehicles. As vehicles start to achieve advanced ADAS capability and monitor their surroundings automatically, these sensing capabilities will provide a vital margin of safety. 

Improved machine health through condition-based monitoring also helps make vehicles safer. Audio and touch sensors are often capable of detecting mechanical defects earlier than the driver or other sensors by analyzing changes in engine or chassis noise. By detecting defective or worn components earlier, these sensing capabilities lower vehicle maintenance and repair costs while also helping reduce the number of dangerous on-the-road failures.

The concept of sensor fusion means that each sensing device in the vehicle has multiple functions, enabling diverse features and capabilities while optimizing space and weight. This multifunctionality is crucial. The sensing needs of autonomous vehicles and ADAS technologies require more and more information, but it’s impractical to increase the number of individual sensors indefinitely. Designers must use sensors capable of contributing input to multiple systems and features.

Incorporating multifunctional audio and touch sensors into design work is crucial in making autonomous or ADAS-equipped vehicles safer and more capable. The earlier in the design process engineers can integrate these technologies, the better. It is vital to collaborate with a company that brings both a solid foundation in sensor engineering and customer-focused adaptability and flexibility. With over a million sensors in vehicles on the road today, Molex offers specialized engineering expertise, collaborative design flexibility and a global manufacturing footprint to provide customized solutions for vehicle sensing challenges. 

At Sensors Converge 2023, Molex is hosting demonstrations of how RNC sensors in its new Percept product line can help detect and counteract the effects of various road surfaces. Attendees can see a vehicle driving over diverse road surfaces and watch the correlation between what the sensor is perceiving and the noise cancelling measures that are activated. Exhibits include a sample driver’s seat with built-in noise cancelling capability, allowing attendees to experience the road noise moderation for themselves.

Technology that allows vehicles to perceive the world like people do, by incorporating feedback from multiple senses, is a vital next step on the path to autonomous driving. Molex engineers provide consulting to help OEMs understand how Percept audio and touch sensors can elevate the sensing capability of their vehicles. Let’s connect to build a complete sensor package.