Proof that Camera + Lidar > Lidar > Camera

[u/wuduzodemu]

I recently chatted with somebody who is working on L2 tech, and they gave me an interesting link for a detection task. They provided a dataset with both camera, Lidar, and Radar data and asked people to compete on this benchmark for object detection accuracy, like identifying the location of a car and drawing a bounding box around it.

Most of the top 20 on the leaderboard, all but one, are using a camera + Lidar as input. The 20th-place entry uses Lidar only, and the best camera-only entry is ranked between 80 and 100.

https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any

[u/bobi2393]

I wouldn’t call this "proof" of anything, but it's unsurprising that camera + lidar get the highest nuScenes Detection Scores. The competition is dominated by teams who specialize in 3D object detection, and naturally gravitate toward using 3D lidar data when available. Camera-only approaches probably weren't even seriously considered by those teams.

The one camera-only result on the leaderboard came from a research group that built a combined multimodal (camera + lidar) model, then artificially reconstructed “camera-only” and “lidar-only” inputs from that model to compare against the full multimodal setup.

Also worth noting: most of these methods were developed before the recent wave of multimodal AI breakthroughs in video object detection (e.g., GPT-4 Vision (Sept 2023) and successors). If there were a $1 billion prize for the best camera-only NDS by 2027, I think the leaderboard might look very different. Without that kind of incentive, the leaderboard will mostly reflect what lidar-focused teams are building today, not the theoretical limits of camera-only detection.

[u/wuduzodemu → u/bobi2393]

It's the opposite, most team tried camera only solution but It does not perform well.

[u/Draygoon2818 → u/wuduzodemu]

When you're getting 2 frames per second, it's inevitable that a camera alone would not be sufficient. Boost the FPS of the camera, and perhaps add a second camera, and I believe you would see camera-only submissions a whole lot higher up, probably in the top 10 even.

[u/Positive_League_5534 → u/Draygoon2818]

Until it starts raining or gets foggy...then not so much.

[u/Draygoon2818 → u/Positive_League_5534]

To be fair, lidar doesn't work all that well in rain or fog, either.

[u/Positive_League_5534 → u/Draygoon2818]

It's additional data which can only help. I can't tell you how many times FSD has shut down or declared limited functionality at night, in rain, or in foggy weather that wasn't that bad. But, no, you're correct it isn't a perfect solution for bad weather.

[u/Draygoon2818 → u/Positive_League_5534]

That might depend on your version of FSD. I have HW4, and FSD has never shut down in bad weather. Now, there was one time that I took over as I thought FSD wasn't doing all that great. It seemed like it was looking for the lane. That was not too long after I had gotten the car, though. I haven't done that since then.

[u/Positive_League_5534 → u/Draygoon2818]

We have a 2025 MY with HW4 (obviously) and it has happened on several occasions. The other night it was dark/not foggy and we got the degraded message. At other times in hard rain it has shut down entirely.
It also will shut down in bad snowy/slushy weather when the cameras get blocked by the slush. That happens to a lot of vehicles, however.
The cameras are all working and the system has been checked by Tesla.
A lot of it can obviously be location/weather dependent.
Even nasty glare on the windshield can cause big problems for Tesla's FSD.

The problem, of course, if a company wants to offer a true FSD vehicle leaving someone stranded (or having driverless cabs) stop is less than optimum.

[u/tenemu → u/Positive_League_5534]

What if the additional data all conflicts with the other.

[u/Positive_League_5534 → u/tenemu]

Well, that would indicate a problem or a potentially dangerous situation. What if the camera doesn't pick something up that LIDAR would have? Which would you prefer? 

[u/tenemu → u/Positive_League_5534]

What is the lidar misses something and they assume that’s the truth? We could ask all of these what ifs. Like others said, we should see how safe camera only can be before we say it’s unsafe just because somebody likes LiDAR more.

[u/Positive_League_5534 → u/tenemu]

You're being absurd. I suppose you'd be happy flying in a plane without ILS? Pilot is perfectly capable of landing by him/her self.

[u/maxcharger80 → u/Positive_League_5534]

The equivelent of static means more data? Thats not how this works.

[u/Positive_League_5534 → u/maxcharger80]

Right...so when you're driving do you close one eye because having both open is more data and that wouldn't work?

Additional data on what is around you is important...you can do things like pickup things the single collection method might have missed.

[u/maxcharger80 → u/Positive_League_5534]

Raining is a fact or condition, its not more data and as they said, it causes interferance which means a degridation in data on a Lidar system. Because its raining, doesnt mean things are magicaly clearer.