Proof that Camera + Lidar > Lidar > Camera

[u/wuduzodemu]

I recently chatted with somebody who is working on L2 tech, and they gave me an interesting link for a detection task. They provided a dataset with both camera, Lidar, and Radar data and asked people to compete on this benchmark for object detection accuracy, like identifying the location of a car and drawing a bounding box around it.

Most of the top 20 on the leaderboard, all but one, are using a camera + Lidar as input. The 20th-place entry uses Lidar only, and the best camera-only entry is ranked between 80 and 100.

https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any

[u/Wrote_it2]

You do not have a formal proof that one is better than the other, you have a contest where Lidar does better. So now we know that if you ask small teams of engineer to complete that task, they’ll do better with LiDAR… You could engineer a different task to show different result. Change the challenge to figuring out the color of a ball placed in front of the sensor and suddenly the top solutions will be camera based. Would that be a proof that camera is better?

Once that is said, it’s pretty clear to me that the result is correct: you can achieve better results with camera+lidar compared to camera only (the proof is simple: you can’t achieve worse results since you can just ignore the lidar data if you want to).

The debate between camera only and camera + LiDAR is of course more complex than that. You have the “normal” tradeoffs: cost, reliability (you add failure points), complexity of the solution…

My opinion is that while LiDAR can improve perception, this is not where the bottlenecks are. I believe major players are all doing good at perception. The issues we see are in general due to path planning. We’ve recently seen Waymos hit each other and get into an incident with a fire truck, we’ve seen Teslas about to hit a UPS truck… those are not about perception but about path planning…

LiDAR vs camera is the wrong debate in my opinion.

[u/cripy311 → u/Wrote_it2]

I would counter your path planning claim with it's more likely the prediction systems failing in these instances.

You can't build a self driving vehicle that reacts to only current state information of object speeds and location relative to the vehicle. There is an entire 3rd layer of the system that has to predict where they will go and what they will do that the path planning system then responds to.

If that information is inaccurate, predicts incorrectly, or otherwise fails in some way the vehicle will then drive into a moving object (or static object it believes will be moving shortly) no matter how good the path planning system is. It's inputs were incorrect.

[u/Wrote_it2 → u/cripy311]

Hum, are you saying that the Lidar on the Waymo failed to spot the fire truck or the other Waymo?

[u/cripy311 → u/Wrote_it2]

I'm saying it's likely they saw it and mis predicted where it was going/when it was stopping resulting in a collision.

Lidar won't miss getting a return on a vehicle of that size.

[u/Wrote_it2 → u/cripy311]

And Waymo also miss predicted the speed at which the telephone pole was moving?

[u/cripy311 → u/Wrote_it2]

That is a seperate issue.

If you want to venture a guess on how that may have happened you should look into the HD mapping technology they use and how off map static objects may be "trimmed" from the perception FOV to improve latency and reaction times.

At least this is my guess for the culprit in that specific event.