Pixel-Perfect Perception: How AI Helps Autonomous Vehicles See Outside the Box

Pixel-Perfect Perception: How AI Helps Autonomous Vehicles See Outside the Box


In autonomous vehicle software, deep neural
networks are often trained to detect an object using a bounding box around the object. Today in DRIVE Labs, we’re going to talk about
using a DNN to provide pixel-level accurate object shape information. This is known as panoptic segmentation. In the top video, we see pixel-level shape
segmentation results for different object classes, with vehicles in blue, pedestrians
in red, and drivable space in green. In the bottom video, we see segmentation of
these object classes into individual instances, as shown by their unique object IDs. So, each of the pedestrians at this intersection
is detected and tracked as a different instance. We detect object shape details, including
the rearview mirror on the bus coming towards us, unusually shaped trucks, and the precise
shape of a trailer. Despite occlusion by the tree, this car is
still detected as a single instance. Here we see accurate segmentation of drivable
space around traffic cones and on-road traffic signs. In this clip, the panoptic segmentation DNN
is segmenting both cars in drivable space accurately in an unstructured environment. Panoptic segmentation of objects and free
space provides a deeper and richer understanding of complex scenes, such as dense traffic with
vehicles occluding each other, or construction zones with unusual object shapes, or corner
cases, such as pedestrians carrying large objects.