MapExpansion - semantic Segmentation Labels

CarlSchwedes · October 31, 2019, 2:17pm

Is there a quick way to get semantic Segmentation labels for Velodyne Point Clouds from the MapExpansion with the nuScenes DevKit?

Basically would need classes for: drivable_area, ego_lane, opposite_lane and parallel_to_ego_lane with same direction as ego.

holger-motional · November 1, 2019, 1:18am

This function renders the map annotations in the camera view:

github.com

nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/map_expansion/map_api.py#L759


        ax.text(x_min + local_width / 100, y_min + local_height / 2, "%g m" % local_height,
                fontsize=14, weight='bold')
        ax.text(x_min + local_width / 2, y_min + local_height / 100, "%g m" % local_width,
                fontsize=14, weight='bold')


    if render_legend:
        ax.legend(frameon=True, loc='upper right')


    return fig, ax


def render_map_in_image(self,
                        nusc: NuScenes,
                        sample_token: str,
                        camera_channel: str = 'CAM_FRONT',
                        alpha: float = 0.3,
                        patch_radius: float = 10000,
                        min_polygon_area: float = 1000,
                        render_behind_cam: bool = True,
                        render_outside_im: bool = True,
                        layer_names: List[str] = None,
                        verbose: bool = True,

I suggest you create a modified copy and return the results, rather than printing them.
Note however that this method is not optimized for speed (you should probably render these to disk).
Furthermore the results can be inaccurate on sloped roads as we do not model z/pitch/roll in the localization.
Here is an example: https://www.youtube.com/watch?v=wUBdMrnL6BU

CarlSchwedes · November 1, 2019, 12:27pm

Thank you so much holger,

that should make some things much easier.
Is there also a quick way to project labels from images into point clouds?

The result I try to get would be a per point label of the velodyne point clouds.

holger-motional · November 2, 2019, 4:21am

Yes, you can use render_pointcloud_in_image(). Again the same caveat that this function is designed for displaying it on screen. If you just want the data you need to change it a bit.

CarlSchwedes · December 4, 2019, 3:06pm

I could manage to render the hd-map data into the velodyne point cloud.

I used use_flat_vehicle_coordinates=True due to render underlay_map mask. There is always a slight offset from the underlay_map to the actual polygon patches from map layers.

Could somebody give me a hint please, if that is caused by the viewpoint transformation matrix used in use_flat_vehicle_coordinates section and how to correctly apply affine_transformations to the 2D polygon coordinates to rectify map layers?

holger-motional · December 9, 2019, 3:35am

Yes, there is currently an issue with a few scenes where we used a different version of the map, which is not compatible with the ego poses in that scene. Apologies for that. We are working on fixing it.

CarlSchwedes · December 9, 2019, 12:13pm

I’m primarilly working with the nuscenes mini dataset, are there specific scenes where this problem is not occurring?

Will this be fixed in the next major release?

holger-motional · December 18, 2019, 9:18am

The scenes were this is particularly apparent are listed in https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/map_expansion/map_api.py#L944 . Yes we hope to fix this in the next major version.

CarlSchwedes · January 16, 2020, 12:29pm

Related to the segmentation labels of lidar point clouds, is there a fast way available from the nuscenes-devKit to identify which points of a point cloud are within some specific polygon, e.g. of drivable_area?

holger-motional · January 17, 2020, 1:29am

We work with shapely which should be sufficiently fast for us: https://streamhacker.com/2010/03/23/python-point-in-polygon-shapely/ . However, we haven’t done that for an entire point cloud, which may need custom speedups.