Project LiDAR to image plane only for annotated items

#1

I am trying to get LiDAR pointcloud project to image only for the annotated (box) items, with the below code. But an error of getting all zeros after masking. It would be helpful if you can give some hints.

(lidar_xyz_points, lidar_z_points , image)=NuScenesExplorer.map_pointcloud_to_image(NuScenesExplorer(nusc), pointsensor_token=lidar_pointsensor_token, camera_token=camera_token, min_dist=1.0)
    #pointcloud <np.float: 2, n)>, coloring <np.float: n>, image=NuScenesExplorer.map_pointcloud_to_image()

    lidar_xyz_points[2]=lidar_z_points # add z points (depth) to the main array.

       boxes = nusc.get_boxes(lidar_nusc_sample_data['token']) 
       lidar_gt_mask = np.zeros(lidar_xyz_points.shape[-1])
       for box in boxes:
               ##### Check if points are inside box #####
               cur_mask = nuscenes.utils.geometry_utils.points_in_box(box, lidar_xyz_points)
               lidar_gt_mask = np.logical_or(lidar_gt_mask, cur_mask)
       lidar_gt_mask = np.clip(lidar_gt_mask, a_min=0, a_max=1)
       lidar_xyz_points = np.compress(lidar_gt_mask, lidar_xyz_points, axis=-1)

I also tried to have the box using "boxes = nusc.get_boxes(camera_nusc_sample_data[‘token’]) ", but same issue of all zeros.

An Example data of lidar_xyz_points (with z points) is shown below.
image

An example of one box is shown below

image
image.png749×347 94.9 KB

#2

Hi. My guess is that since the points are being masked inside map_pointcloud_to_image, they cannot be easily combined with lidar_gt_mask afterwards. I suggest you simply create a copy of map_pointcloud_to_image and modify it with 1-2 lines to take only the points inside the boxes.