Fusing LIDAR sweeps with keyframe

#1

Hi, I’m trying to apply sensor fusion to the front camera and lidar. I’m having problems as single keyframe sweep of the lidar data is too sparse. Is it possible to combine adjacent sweeps of the lidar to get a more detailed view?
Thanks!

#2

Just to further elaborate. I’m converting the dataset to kitti-format using the provided script:


I want to increase the lidar details in the data.

#3

Hi. That should be possible, but we don’t support it for KITTI. I think you just need to plug in this method, which loads multiple sweeps: https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/utils/data_classes.py#L56

#4

When you say you don’t support it for KITTI, do you mean I won’t be able to extract the data to KITTI-format, or just that it’s a feature you’ve not yet implemented?

#5

I think you will be able to do it with both of the code snippets from above. I just meant that we did not implement it and likely won’t do so in the future.
KITTI simply wasn’t made with multiple sweeps in mind. In fact, a lot of the improvement that we see from multiple sweeps may be due to the added time dimension in the pointcloud, rather than just a denser pointcloud, which may confuse the network. And since KITTI format does not support that time dimension, having multiple sweeps may not give you that much of an improvement.

1 Like
#7

Hi Holger!
I’m having some issues and hopefully you can provide some insight. When i display my multi-pointcloud in nuscenes everything looks good. But when I try to implement it in “export_kitti.py” and try to visualize the lidar file it does not look good. I changed line 174 in “export_kitti.py” from:

pcl = LidarPointCloud.from_file(src_lid_path)

to:

pcl, _ = LidarPointCloud.from_file_multisweep(self.nusc, sample, ‘LIDAR_TOP’, ‘LIDAR_TOP’, nsweeps=10)

Is this not a valid way to import the pointcloud? Can’t i use the return of “from_file_multisweep” in the place of “from_file”.

Thanks!

#8

I think it should work. What is the error you are seeing?

#9

I had problems visualizing the points after saving them to a file. I change:

with(open(dst_lid_path, “w”) as lid_file:
pcl.points.T.tofile(lid_file)

to:

pcl.points.T.astype(‘float32’).tofile(open(dst_lid_path, “wb”))

#10

Thanks for reporting this. I was able to reproduce the problem with:

%matplotlib inline
from nuscenes.nuscenes import NuScenes
from nuscenes.utils.data_classes import LidarPointCloud
import os.path as osp

nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=True)

# Single sweep
sample_data = nusc.get('sample_data', sample['data']['LIDAR_TOP'])
pcl1 = LidarPointCloud.from_file(osp.join(nusc.dataroot, sample_data['filename']))
print(pcl1.points.T.dtype)

# Multisweep
sample = nusc.sample[0]
pcl, _ = LidarPointCloud.from_file_multisweep(nusc, sample, 'LIDAR_TOP', 'LIDAR_TOP', nsweeps=10)
print(pcl.points.T.dtype)

The former prints float32, the latter float64. We will fix this issue in https://github.com/nutonomy/nuscenes-devkit/pull/581.