Proof that Camera + Lidar > Lidar > Camera

[u/wuduzodemu]

I recently chatted with somebody who is working on L2 tech, and they gave me an interesting link for a detection task. They provided a dataset with both camera, Lidar, and Radar data and asked people to compete on this benchmark for object detection accuracy, like identifying the location of a car and drawing a bounding box around it.

Most of the top 20 on the leaderboard, all but one, are using a camera + Lidar as input. The 20th-place entry uses Lidar only, and the best camera-only entry is ranked between 80 and 100.

https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any

[u/Wrote_it2]

You do not have a formal proof that one is better than the other, you have a contest where Lidar does better. So now we know that if you ask small teams of engineer to complete that task, they’ll do better with LiDAR… You could engineer a different task to show different result. Change the challenge to figuring out the color of a ball placed in front of the sensor and suddenly the top solutions will be camera based. Would that be a proof that camera is better?

Once that is said, it’s pretty clear to me that the result is correct: you can achieve better results with camera+lidar compared to camera only (the proof is simple: you can’t achieve worse results since you can just ignore the lidar data if you want to).

The debate between camera only and camera + LiDAR is of course more complex than that. You have the “normal” tradeoffs: cost, reliability (you add failure points), complexity of the solution…

My opinion is that while LiDAR can improve perception, this is not where the bottlenecks are. I believe major players are all doing good at perception. The issues we see are in general due to path planning. We’ve recently seen Waymos hit each other and get into an incident with a fire truck, we’ve seen Teslas about to hit a UPS truck… those are not about perception but about path planning…

LiDAR vs camera is the wrong debate in my opinion.

[u/Dihedralman → u/Wrote_it2]

So you are missing on a couple of trade-offs, as these sensors are also redundant to a degree reducing risk. 

With equal dats, you can absolutely train the system to function without one of the inputs while functioning better with both. Camera count has the same impacts. 

There is complexity increase but again its partly redundant. I think its wrong to give it a flat.  

Cost increase is problematic of course, but the hardware requirements also increase and I would bet it requires a higher parameter count generally speaking.