nuPlan paper comparison table

Hello ! I was reading the nuPlan paper ( where the Table 1 (shown below):

Is showing that the Lyft dataset doesn’t have Closed-loop evaluation and that it has a “prediction” in the Type column.

As a suggestion, I would review the table because in L5kit there is a closed-loop evaluation with many closed-loop metrics available since 2021 (before the nuPlan paper publication). Not only there is a simulator but also a RL environment where rollouts can be done and use rewards from closed-loop metrics as well.
There is also an entire section about Planning where there was even a paper released using it for training a planner in closed-loop. Anyway, the table doesn’t seem to reflect the current state of the framework.

cc @holger-motional

@christian.perone thanks for your interest in nuPlan! Do post your suggestion under nuPlan here:

1 Like