Detect reappearing objects

#1

Hello,

I have two questions concerning the detection of newly appearing objects in each scene.
1 - At the moment, I gather a list of unique instances for each camera frame per scene like so:

unique_instances = []
while has_more_frames:
     img_path, boxes, camera_intrinsics = 
     nusc.get_sample_data(current_cam_data['token'], box_vis_level=BoxVisibility.ANY)
     for box in boxes:
         sample_annotation = nusc.get('sample_annotation', box.token)
         instance = nusc.get('instance', sample_annotation['instance_token'])
         if instance not in unique_instances:
             # new appearing object

This approach seems to work fine. Can you suggest another way to detect newly appearing objects to build redundancy?

2 - In certain cases objects that already appeared can be completely occludded and reappear. I noticed using the approach above these reappearing instances are not detected. My question is can you point me into a direction with which I can detect reappearing objects. I thought about using the points_in_box() function as a criterion to detect when an object disappears completely (https://github.com/nutonomy/nuscenes-devkit/blob/28765b8477dbd3331bacd922fada867c2c4db1d7/python-sdk/nuscenes/utils/geometry_utils.py#L111). But I am not sure whether the LiDAR points are a determining factor for an object disappearing completely.

#2

  1. An alternative approach could be (in pseudo-code):

    for camera in list_of_cameras:
  unique_instances = []
  for sample annotation in instance:
    if sample_annotation_in_cam(camera) and instance not in unique_instances:
      # new appearing object

  2. I don’t quite follow what you mean by “In certain cases objects that already appeared can be completely occluded and reappear” - if you could provide an example code snippet to observe this behavior, it would help

#3

Thanks for the reply.

  1. To clarify I uploaded three pictures showing the specific case of reappearing objects.
    a) In the first picture, you can see three persons walking on the board walk.
    b) In the second picture, the persons “disappear” behing the small building.
    c) In the third picture, these persons “reappear” again.

s002_082
s002_082.jpg1600×900 303 KB
s002_112
s002_112.jpg1600×900 299 KB
s002_150
s002_150.jpg1600×900 285 KB

#4

@aserbremen in the case you showed above, I believe the three persons should still have the same instance ID before and after disappearing behind the small building, and so your code snippet should still work

#5

I realized that they still have the same instance ID, but I would like to find a way to robustly detect the persons appearing from occlusion as a newly appearing object. As they have been visible before, they are now part of the unique_instances list and would not be detected as a new instance ID.

#6

Ah ok, I now understand your use case better - if you sent me the scene token for the above example, I can take a look and let you know if I come up with anything

#7

I misread your answer. Here are the details for the specific scene:

name: scene-0003
token: e7ef871f77f44331aefdebc24ec034b7
description: Parking lot, barrier, exit parking lot
first_sample_token: fd8420396768425eabec9bdddf7e64b6
log_token: 92af2609d31445e5a71b2d895376fed6

I hope that info is enough, camera channel is ‘CAM_FRONT’. Thanks!

#8

@aserbremen one idea could be to make use of the visibility attribute, and count a box as newly appearing if its visibility changes from a higher visibility to a lower one and to a higher one again

You could get the visibility of a box via:

visibility_token = nusc.get('sample_annotation', anntoken)['visibility_token']