How can I project from the camera plane to the lidar coordinates system? I have created a depth image and I need to transform that image to the lidar coordinates system.
Unfortunately we don’t have a convenient routine for that. I guess the steps are:
- Take a look at the transformations we apply in https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/nuscenes.py to go from camera to ego vehicle to lidar.
- Figure out which line of points in the 3d world is projected to a particular pixel. This will require the transformation matrix and the calibration parameters.
- Plug in the depth to figure out where along that line your 3d point is.