Projection from Camera to Lidar

How can I project from the camera plane to the lidar coordinates system? I have created a depth image and I need to transform that image to the lidar coordinates system.

Unfortunately we don’t have a convenient routine for that. I guess the steps are:

  1. Take a look at the transformations we apply in https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/nuscenes.py to go from camera to ego vehicle to lidar.
  2. Figure out which line of points in the 3d world is projected to a particular pixel. This will require the transformation matrix and the calibration parameters.
  3. Plug in the depth to figure out where along that line your 3d point is.
1 Like