Relation between camera frame and lidar data

#1

Hello,
I am trying to run 2d and 3d detections using nuScenes data, I have downloaded v1.0-mini data and after extraction in the samples folder I see CAMERA_FRONT, LIDAR_TOP and other files.
How to match which lidar bin to the jpg file, I would like to run 2d detections on camera frame, 3d detection on the point cloud and combine both to track the object. Not able to understand how to match point clouds with the images.
Can you please help me understand the structure.

Thanks,
Gayathri.

#2

Hi @Gayathri81,
You can retrieve the lidar and camera data by doing something like this:

import os.path as osp

from nuscenes.nuscenes import NuScenes


# Load the nuScenes dataset (mini-split, in this case).
nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=False)

sample_record = nusc.sample[100]

lidar_sensor_token = sample_record['data']['LIDAR_TOP']
lidar_sensor = nusc.get('sample_data', lidar_sensor_token)
print(lidar_sensor)

cam_sensor_token = sample_record['data']['CAM_FRONT']
cam_sensor = nusc.get('sample_data', cam_sensor_token)
print(cam_sensor)

lidar_sensor will look something like:

{
'token': '693114499b864bb39e32676e0fb5ea61',
'sample_token': 'ac452a60e8b34a7080c938c904b23057', 
..., 
'filename': 'samples/LIDAR_TOP/n008-2018-08-28-16-43-51-0400__LIDAR_TOP__1535489306446341.pcd.bin',
...,
'channel': 'LIDAR_TOP'
}

While cam_sensor will look something like:

{
'token': 'defde01fa5594f18b24be8fe68e6f2c4', 
'sample_token': 'ac452a60e8b34a7080c938c904b23057', 
..., 
'filename': 'samples/CAM_FRONT/n008-2018-08-28-16-43-51-0400__CAM_FRONT__1535489306412404.jpg', 
..., 
'channel': 'CAM_FRONT'
}

With these pieces of info, you should be able to easily match the camera and lidar data files to do what you want

#3

Thanks for the clarification.