Details about preprocessing for evaluation

Youngseok_Kim · December 18, 2019, 2:49am

Hi, I am trying to detect objects using camera and radar data, without LiDAR

It says, All boxes (GT) without LiDAR or radar points in them are removed.

Should I consider a bounding box without any LiDAR points as a valid object even if I do not use lidar point cloud?

holger-motional · December 18, 2019, 3:04am

I think it won’t make a significant difference as the vast majority of the objects is in close range and has lidar points. It only matters for a few objects that are beyond the lidar range.

Youngseok_Kim · December 18, 2019, 3:44am

Thank you for the quick answer.

You said it won’t make a significant difference, but there are many objects with a lot of LiDAR points but without any radar point if you plot radar and LiDAR points.

I attach some examples, red point: radar (5 sweeps), black point: lidar (1 sweep), red box: ground truth

I guess one of the major reason can be the poor azimuth resolution and occlusion.

Also, radar is mounted on the bumper, not the top as LiDAR, so the FoV can be easily blocked by obstacles as guardrails.

And another reason can be the nature of the wavelength of the radar, because it cannot detect through the wired fence.

I didn’t count the number of vehicle with LiDAR and without radar point yet, but I guess there will be the number of vehicles do not contain radar point.
Moreover, this is even worse with side cameras.
So I wonder if it will be fair to use the same preprocessing method as LiDAR.

holger-motional · December 18, 2019, 5:01am

You said it won’t make a significant difference, but there are many objects with a lot of LiDAR points but without any radar point if you plot radar and LiDAR points.

In your original question you were asking the other way around (radar, but no lidar). I do agree that the majority of boxes has only lidar. I also agree with all your observations.

So I wonder if it will be fair to use the same preprocessing method as LiDAR.

What does “fair” mean? Shouldn’t the goal be to use whatever you can to build the best system? Another big problem is that in many cases cars that only have radar returns are not annotated. That is because it is extremely difficult to draw an accurate 3d bounding box around a car that is 100m away with 1 radar return. So even if you develop a perfect radar detector, using 3d evaluation metrics may not lead to perfect results, as the ground-truth is incomplete/noisy.