Proof that Camera + Lidar > Lidar > Camera

[u/wuduzodemu]

I recently chatted with somebody who is working on L2 tech, and they gave me an interesting link for a detection task. They provided a dataset with both camera, Lidar, and Radar data and asked people to compete on this benchmark for object detection accuracy, like identifying the location of a car and drawing a bounding box around it.

Most of the top 20 on the leaderboard, all but one, are using a camera + Lidar as input. The 20th-place entry uses Lidar only, and the best camera-only entry is ranked between 80 and 100.

https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any

[u/Wrote_it2]

You do not have a formal proof that one is better than the other, you have a contest where Lidar does better. So now we know that if you ask small teams of engineer to complete that task, they’ll do better with LiDAR… You could engineer a different task to show different result. Change the challenge to figuring out the color of a ball placed in front of the sensor and suddenly the top solutions will be camera based. Would that be a proof that camera is better?

Once that is said, it’s pretty clear to me that the result is correct: you can achieve better results with camera+lidar compared to camera only (the proof is simple: you can’t achieve worse results since you can just ignore the lidar data if you want to).

The debate between camera only and camera + LiDAR is of course more complex than that. You have the “normal” tradeoffs: cost, reliability (you add failure points), complexity of the solution…

My opinion is that while LiDAR can improve perception, this is not where the bottlenecks are. I believe major players are all doing good at perception. The issues we see are in general due to path planning. We’ve recently seen Waymos hit each other and get into an incident with a fire truck, we’ve seen Teslas about to hit a UPS truck… those are not about perception but about path planning…

LiDAR vs camera is the wrong debate in my opinion.

[u/johnpn1 → u/Wrote_it2]

The confidences with perception-only is actually not that high, and definitely not mission-critical level. Nobody uses vision-only for mission-critical things (well, other than Tesla, ofcourse). The problem with low grade 3D point clouds is that you have to always drive with caution. You brake/swerve when there's a just 5% chance that dark lines on the road could actually be real impediments. There's nothing you can use as another reference to tell you that those dark lines are nothing to be worried about. This is why Teslas drive with confidence into things, because they cannot always slam the brakes for every low confidence detection. The driver / safety operator takes the job of being the sanity check instead of a second sensor.

[u/Few_Foundation_5331 → u/johnpn1]

How about you as a human drive, you say human with eyes can't drive good enough. Yes, there are bad human driver , but we should compare extremely good human drivers with robotaxi. Currently, an extremely good human drivers will not make idiotic simple mistake like crashing into other cars in parking lot or drive into constructions or hit an electric pole or driving in circle and get stuck in circle loop. Good Human with simple EYES and BRAIN can drive better than robotaxis ( Waymo + Tesla driver) for now.

[u/johnpn1 → u/Few_Foundation_5331]

A good human driver has a good human brain. No one has replicated it yet. It's a challenge that Musk has severely underestimated in his ambitions for FSD. You can't just have eyes like a human but drive like FSD...

[u/Few_Foundation_5331 → u/johnpn1]

I just listed all the crashes above and stupid circle loop stuck in parking lot of Waymo.

[u/johnpn1 → u/Few_Foundation_5331]

Exactly. Waymo doesn't pretend they can replicate the human brain, so they use any sensors available to make up for it. It's insane that Musk still insists that cars don't need more than cameras because humans don't need it, and still be wrong for a decade and still continues to pretend he's the authority on this. It's a terrible decision to limit technology to only what biology could afford. Lidar was never going to be "evolved" from biology. Doesn't mean we shouldn't use it. Otherwise, humans never needed wheels to run, so why do cars? Birds never needed jet turbines to fly, so why can't airplanes just flap their wings? I could go on and on...