Hello ! I was reading the nuPlan paper (https://arxiv.org/abs/2106.11810) where the Table 1 (shown below):
Is showing that the Lyft dataset doesn’t have Closed-loop evaluation and that it has a “prediction” in the Type column.
As a suggestion, I would review the table because in L5kit there is a closed-loop evaluation with many closed-loop metrics available since 2021 (before the nuPlan paper publication). Not only there is a simulator but also a RL environment where rollouts can be done and use rewards from closed-loop metrics as well.
There is also an entire section about Planning where there was even a paper released using it for training a planner in closed-loop. Anyway, the table doesn’t seem to reflect the current state of the framework.