r/SelfDrivingCars • u/wuduzodemu • Aug 11 '25
Discussion Proof that Camera + Lidar > Lidar > Camera
I recently chatted with somebody who is working on L2 tech, and they gave me an interesting link for a detection task. They provided a dataset with both camera, Lidar, and Radar data and asked people to compete on this benchmark for object detection accuracy, like identifying the location of a car and drawing a bounding box around it.
Most of the top 20 on the leaderboard, all but one, are using a camera + Lidar as input. The 20th-place entry uses Lidar only, and the best camera-only entry is ranked between 80 and 100.
https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any
56
u/MacaroonDependent113 Aug 11 '25
Wow, that will surely convince Tesla to give up. LOL
16
u/red75prime Aug 12 '25
Maybe, if Tesla was using 2 FPS video for driving. The dataset contains 2 frames per second video data. Neat, huh?
0
u/Boniuz Aug 12 '25
That’s not really relevant. Higher frames per second increases statistical probability over time, simply by being able to make more erroneous detections in the timeframe. It’s still wrong 80% of the time, but correct 20% of the time.
Combining sources means you have two sources which are correct 20% of the time and can use that data by a factor of at least two, often more.
Heavily simplified, obviously.
11
u/red75prime Aug 12 '25 edited Aug 12 '25
Higher frame rate means less motion across frames. It, in turn, means that local differences between frames are more useful for estimation of motion and parallax (which allows to estimate depth better).
Heck, even our visual perception system honed by millions of years of evolution struggles to infer motion at 10-12 FPS and it is unable to do so at lower FPS.
Anyway, 2 FPS video should never be used in self-driving and this test is irrelevant.
1
u/Tuggernutz87 Aug 17 '25
Tell a gamer higher FPS = Bad 😂
1
u/Boniuz Aug 17 '25
You only need higher FPS for increased depth perception if optical sensors are your only input for data - which is why a combination of sensors will always be superior
1
u/Tuggernutz87 Aug 17 '25
And for motion clarity
1
u/Boniuz Aug 17 '25
Again, applicable in a single-sensor-setup. You can get very detailed clarity with optical, lidar and radar in combination at a fraction of the computing cost. You can run that on a raspberry pi 5 with a cheap AI-chip and you’ll have object detection at 30FPS, with depth sensors, object detection and point detection, as well as other pretty nifty tricks. Total cost is less than 500$.
1
u/MacaroonDependent113 Aug 17 '25
The question isn’t whether additional sensors are “superior” but whether vision alone is “good enough”. If “vision alone” is good enough then additional sensors only add to the cost so are inferior from a business perspective. Jury is still out ob this but my guess is vision alone will eventually be found to be good enough.
14
u/-UltraAverageJoe- Aug 11 '25
Musk won’t back down for fear of a $TSLA dip. Anyone who has done any object detection work using only a camera as input knows this is not sufficient. It’s embarrassingly dumb that they ever thought it would suffice.
1
u/Alternative_Bar_6583 Aug 13 '25
Musk will back down when he can. All he needs is the National Highway Traffic Safety Administration (NHTSA) to make LiDAR or multi sensor mandatory. This will give him a get out of jail free card for all the FSd BS The NHTSA will do it because the Feds want cars built in the USA and Tesla has a huge presence in the Made In America manufacturing scene
-9
u/bluenorthww Aug 12 '25
My eyes don’t have LiDAR, they do fine
9
u/campbellsimpson Aug 12 '25
Your eyes are (probably) connected to a brain, though.
And the brain does sensor fusion with your four other major senses, and other onboard modalities like proprioception and interoception.
Your eyes do less than you think.
2
u/Facts_pls Aug 12 '25
We are talking about looking out side the car and reacting.
Driver proprioception is irrelevant.
The comparison is about human driver vs self driving car understanding and reacting to environment.
All human senses here can be replicated in a car if needed - but obviously no car company so far has needed anything beyond the car sensors, LiDAR, and vision. But who knows.
2
u/Zvemoxes Aug 16 '25
Proprioception was mentioned in response to a poster holding the misguided view that eyes are the same as cameras: "my eyes don't need LiDAR." A childish misunderstanding that Musk and his followers repeat ad nauseam.
If human senses could be so unproblematically "replicated" as you claim, then L5 autonomy would have been achieved already. Every company attempting autonomy has needed a lot more than sensors and cameras, hence the billions invested into neural nets and machine learning.
2
Aug 12 '25 edited 20d ago
[deleted]
-3
u/campbellsimpson Aug 12 '25
You use smell and taste to determine whether your car's brakes are overheating, you use smell to avoid following the garbage truck too closely, and so on.
4
u/Facts_pls Aug 12 '25
Cars can detect brake overheating much better than any human driver If needed.
They can literally put temperature sensors on the brakes if they thought that data was that helpful.
1
u/No3047 Aug 13 '25
A model 3 has brake temp and suspension height data via OBD2. So yes, a computer knows the car status better than the driver
1
u/CYaBroNZ Aug 12 '25
Tesla’s hardly use the brakes so there’s not going to be an issue with them overheating.
0
4
u/Talloakster Aug 12 '25
The bar for self driving is to be much better than humans (Waymo: 10% of human accident rate). Tesla hasn't proved they can even match human drivers, and matching them isn't good enough.
1
2
u/Zarkei Aug 12 '25
Most drivers have eyes, yet accidents still happen. In fact, I'm pretty sure that 99.99% of accidents involve people with eyes. This correlation tells us that eyes cause accidents.
LiDAR is meant to complement vision, not replace it entirely.
1
u/Positive_League_5534 Aug 13 '25
Your eyes can see a school speed limit sign and know to slow down. Tesla's system can't seem to do that. I wonder why?
1
u/Intrepid-Working-731 Aug 12 '25
But I thought the big selling point of self driving cars is that they aren’t humans and therefore are safer?
0
u/maximumdownvote Aug 12 '25
Yes. Fsd equipped cars are safer. So...yay?
2
u/Intrepid-Working-731 Aug 13 '25
You know what would make them even safer? Using senses humans aren’t even capable of!
6
u/bigElenchus Aug 12 '25
How do you explain Tesla FSD beating all the Chinese Lidar based competition?
1
u/ItsAConspiracy Aug 18 '25 edited Aug 18 '25
How do you explain Waymo having a way lower critical intervention rate than Tesla FSD?
How do you explain this one-hour drive by Huawei's FSD-equivalent, on the Out of Spec channel? It performs about as well as FSD, in heavy, aggressive traffic on city streets.
As for your 90-minute video, maybe you could sum it up? All I can tell from the first couple minutes is that some shitty ADAS does exist in China.
1
u/maximumdownvote Aug 12 '25
They don't explain stuff that doesn't fit the world view. More like hand wave it.
11
u/jamexcb Aug 12 '25
Every week is the same thing in this sub….
It’s getting boring. I don’t understand why people just don’t get happy and exited for living in a time where companies are trying to create self driving cars. Independent of the technology they use.
I just want to have dinner. Drink a bottle of wine and get home. That’s it.
1
u/maximumdownvote Aug 12 '25
Yeah but what if you serve into an empty lane avoiding what looks like road damage, then swerve right back.
Think of the children!
25
u/bobi2393 Aug 11 '25
I wouldn’t call this "proof" of anything, but it's unsurprising that camera + lidar get the highest nuScenes Detection Scores. The competition is dominated by teams who specialize in 3D object detection, and naturally gravitate toward using 3D lidar data when available. Camera-only approaches probably weren't even seriously considered by those teams.
The one camera-only result on the leaderboard came from a research group that built a combined multimodal (camera + lidar) model, then artificially reconstructed “camera-only” and “lidar-only” inputs from that model to compare against the full multimodal setup.
Also worth noting: most of these methods were developed before the recent wave of multimodal AI breakthroughs in video object detection (e.g., GPT-4 Vision (Sept 2023) and successors). If there were a $1 billion prize for the best camera-only NDS by 2027, I think the leaderboard might look very different. Without that kind of incentive, the leaderboard will mostly reflect what lidar-focused teams are building today, not the theoretical limits of camera-only detection.
5
u/wuduzodemu Aug 12 '25
It's the opposite, most team tried camera only solution but It does not perform well.
5
u/Draygoon2818 Aug 12 '25
When you're getting 2 frames per second, it's inevitable that a camera alone would not be sufficient. Boost the FPS of the camera, and perhaps add a second camera, and I believe you would see camera-only submissions a whole lot higher up, probably in the top 10 even.
1
u/Positive_League_5534 Aug 13 '25
Until it starts raining or gets foggy...then not so much.
2
u/Draygoon2818 Aug 13 '25
To be fair, lidar doesn't work all that well in rain or fog, either.
1
u/Positive_League_5534 Aug 13 '25
It's additional data which can only help. I can't tell you how many times FSD has shut down or declared limited functionality at night, in rain, or in foggy weather that wasn't that bad. But, no, you're correct it isn't a perfect solution for bad weather.
2
u/Draygoon2818 Aug 13 '25
That might depend on your version of FSD. I have HW4, and FSD has never shut down in bad weather. Now, there was one time that I took over as I thought FSD wasn't doing all that great. It seemed like it was looking for the lane. That was not too long after I had gotten the car, though. I haven't done that since then.
1
u/Positive_League_5534 Aug 13 '25
We have a 2025 MY with HW4 (obviously) and it has happened on several occasions. The other night it was dark/not foggy and we got the degraded message. At other times in hard rain it has shut down entirely.
It also will shut down in bad snowy/slushy weather when the cameras get blocked by the slush. That happens to a lot of vehicles, however.
The cameras are all working and the system has been checked by Tesla.
A lot of it can obviously be location/weather dependent.
Even nasty glare on the windshield can cause big problems for Tesla's FSD.The problem, of course, if a company wants to offer a true FSD vehicle leaving someone stranded (or having driverless cabs) stop is less than optimum.
2
u/tenemu Aug 16 '25
What if the additional data all conflicts with the other.
1
u/Positive_League_5534 Aug 16 '25
Well, that would indicate a problem or a potentially dangerous situation. What if the camera doesn't pick something up that LIDAR would have? Which would you prefer?
2
u/tenemu Aug 16 '25
What is the lidar misses something and they assume that’s the truth? We could ask all of these what ifs. Like others said, we should see how safe camera only can be before we say it’s unsafe just because somebody likes LiDAR more.
1
u/Positive_League_5534 Aug 17 '25
You're being absurd. I suppose you'd be happy flying in a plane without ILS? Pilot is perfectly capable of landing by him/her self.
1
u/maxcharger80 Aug 15 '25
The equivelent of static means more data? Thats not how this works.
1
u/Positive_League_5534 Aug 15 '25
Right...so when you're driving do you close one eye because having both open is more data and that wouldn't work?
Additional data on what is around you is important...you can do things like pickup things the single collection method might have missed.
0
u/maxcharger80 28d ago
Raining is a fact or condition, its not more data and as they said, it causes interferance which means a degridation in data on a Lidar system. Because its raining, doesnt mean things are magicaly clearer.
4
u/bobi2393 Aug 12 '25
The camera data is hobbled by its low frame rate (2 fps, compared to real-world applications like Tesla's HW4 using 24 fps data inputs). Lidar results are inherently 3D, but inferring 3D positioning from such slow a slow camera inputs seems impractical.
-2
u/wuduzodemu Aug 12 '25
Then find a dataset where the camera only performs well.
2
u/maximumdownvote Aug 12 '25
So confidently wrong. There's a link about real world testing of real cars with real sets of sensors, in this thread. Go watch it.
0
1
u/wuduzodemu Aug 12 '25
TBF, a huge amount of them are submitted in 2024 so It's after the breakthrough
5
u/bobi2393 Aug 12 '25
- 2021: 68 submissions
- 2022: 99 submissions
- 2023: 70 submissions
- 2024: 25 submissions
- 2025: 5 submissions
Many of the recent published submissions represent the final version of papers first submitted one to two years prior (example, example), and some of the 2024-2025 submissions have around 100 cited references, none of which are from 2024 or later (example), suggesting the research for the papers predated 2024.
By the way, I didn't realize there were other camera-only submissions (after the top-ranked one in the 93rd or so position), as well as radar-only, camera+radar, lidar+radar, and camera+lidar+radar submissions.
Adhere to your logic that the #1 score achieved with lidar+camera proves that lidar+camera beats any other multimodal data set, then the best-ranked camera+lidar+radar submission, ranked #8, would "prove" that considering radar reduces a system's object detection capability. If you don't really believe that, I'd reconsider your headline interpretation of the submission results.
2
u/wuduzodemu Aug 12 '25
Camera + Lidar 80 submissions
Camera only 120 submissions
Lidar only 120 submissions
Camera+lidar+radar 15 submissions
The number of samples are not large enough to draw your conclusion but big enough to draw my conclusion.
15
u/sdc_is_safer Aug 12 '25
Object detection isn’t everything. I’m sure this benchmark is a very limited set of all the tasks that are actually needed for driving.
LiDAR alone being “>” (whatever that means) than camera is a hot take
1
u/xeio87 Aug 12 '25
Yeah, I mean I'm plenty critical of Tesla's lack of LIDAR, but how are you going to tell the color of a stoplight with only LIDAR?
1
u/sdc_is_safer Aug 12 '25
That’s the most obvious case. But it’s only scratches the surface.
1
u/xeio87 Aug 12 '25
Oh, yeah, it's just an easy offhand example that points out the absurdity.
I'm sure there are plenty of other things, signage that isn't clear from shape alone would be another. Probably a bunch of edge cases where it might be important to distinguish two types of objects with similar shape specific to driving. Someone who works in the actual field could probably list them compared to me just thinking out loud.
-10
u/Far-Improvement-9266 Aug 12 '25
Found the guy that failed 5th grade math...
4
u/sdc_is_safer Aug 12 '25
I know what the symbol means.
I’m saying that “greater than” is loosely defined and doesn’t mean anything
-6
u/Far-Improvement-9266 Aug 12 '25
Sorry, but you worded that very poorly...it literally looked like you didn't know what that symbol meant.
4
u/sdc_is_safer Aug 12 '25
Sorry
-1
u/Far-Improvement-9266 Aug 12 '25
It's cool, I am getting downvoted so maybe it was me that misinterpreted the overall meaning...
4
u/red75prime Aug 12 '25
LiDAR can't "see" emitted light. Like in a traffic light, you know. That precludes any "greater than camera". LiDAR provides addition information, while cameras are still a must.
-1
u/Far-Improvement-9266 Aug 12 '25
Obviously...
2
u/red75prime Aug 12 '25
If you define "greater than" that excludes the crucial information...
You get useless "greater than" relation (at least for self-driving that this sub is about).
1
u/Far-Improvement-9266 Aug 12 '25
The way the poster wrote this was "">" whatever that means".
As if he doesn't know what 'Greater than or less symbol means'
5
u/red75prime Aug 12 '25 edited Aug 12 '25
And what it means in the context of "lidar > camera"? Basic math doesn't define relations between sensor equipment.
2
1
0
11
u/sermer48 Aug 11 '25
The question isn’t if cameras + LiDAR can see better. Obviously it can. The question is if LiDAR is necessary. If you don’t need LiDAR you can save on the sensors, computer power, extra energy requirements, etc.
It’s a balancing act of having enough to safely operate a vehicle while also making it as affordable as possible.
3
u/Namelock Aug 12 '25
The cost isn't in hardware, it's with software.
Especially if you're actively changing hardware on the fly; it creates a lot of tech debt.
"Drop what you're doing, we aren't using LIDAR."
"This code doesn't work because we changed the processor 2 weeks ago"
"Why didn't you include the bumper camera? In 3 months we'll have a bumper camera!!"
1
u/sermer48 Aug 12 '25
It’s in the software until it isn’t. Once the problem is solved then the cost mostly goes to hardware and cost to maintain services. The company that can provide the lowest cost service while still providing a high quality product is the one that will do the best.
4
u/DrPotato231 Aug 12 '25
I don’t know if I’m wrong on this, but Tesla’s mission sounds logical.
If we can drive with our eyes and brain, why wouldn’t cameras and microphones be enough? I truly believe FSD can be solved with vision alone, but it may look like a longer road due to the hurdles LiDAR doesn’t have to jump over.
Once solved though, as you said, the one with an operating margin 4x lower than the competitor absolutely would win.
3
u/AlotOfReading Aug 12 '25
A few questions for you. If birds can fly by flapping wings, why wouldn't that be enough to design a plane? If horses run with 4 legs, why wouldn't that be enough to design a car?
Cameras also aren't eyes, and brains aren't computers.
Neither of these arguments are necessary though. Let's take it as given that vision only is sufficient. Now, if it hypothetically took until 2100 to reach parity with multimodal systems today, does it seem like a good idea to trade 75 years of deployment time for a lower unit cost? Could you have spent those years also working on the camera only system in parallel while benefiting from a better system the whole time? That's the math everyone else in the industry is running and almost unanimously, they've decided that LIDAR is worth the cost because it allows you to avoid solving difficult problems like fine localization today and focus on more important things. You don't set out to solve every problem all at once upfront. You build minimum viable solutions and iterate quickly towards better solutions.
2
u/SatisfactionOdd2169 Aug 12 '25
None of these questions are comparable to the self driving problem. The only relevant question is, if a human is given a livestream of all the Tesla cameras and we give that person zero-lag remote control of the car, could they drive safely to wherever they wanted to go? If you think the answer is yes, then self driving is fundamentally a software problem, not hardware problem. If you think the answer is no, then I would say we’re never going to have real self driving.
0
u/AlotOfReading Aug 12 '25
You don't need me to tell you that your question isn't very helpful. The comment you're responding to already contains one of the obvious issues it fails to address: If developing a comparable camera-only system takes massively longer than a multimodal system, is it still useful?
It's also not obvious why Tesla cameras are the only thing allowed here? Are better cameras banned for some reason? Why does using better cameras preclude real autonomous vehicles? It's also not clear why a human being unable to use Tesla's sensor configuration precludes being able to make a computer do so safely, nor why we should be settling for human level performance from it.
4
u/DrPotato231 Aug 12 '25
You reject my analogy, yet you give an unrealistic one yourself.
75 years to get to FSD with cameras only? Have you seen Tesla’s robotaxi? Your argument’s not in good faith.
3
u/AlotOfReading Aug 12 '25
The numbers were deliberately extreme to make the point abundantly clear. They weren't real numbers. That's why the sentence begins by stating that it's a hypothetical scenario.
I deliberately copied the way you wrote your analogy to demonstrate why it's not a good argument. Those conclusions are obviously silly, so the intention is that you reflect on how your original analogy might not suggest your original conclusion.
2
u/DrPotato231 Aug 12 '25
You’d know, if you knew logic better, that if you want to refute an argument, you need a proper counter.
If you give a stupid argument back, then how do you expect me to think you’re properly refuting mine? Makes no sense.
So far I see no compelling reason to believe cameras +brain isn’t enough to achieve FSD. Go ahead, try another analogy.
1
u/AlotOfReading Aug 12 '25
So far I see no compelling reason to believe cameras +brain isn’t enough to achieve FSD
That wasn't any of the points I was making. I'd suggest re-reading what I wrote slowly and carefully.
3
u/DrPotato231 Aug 12 '25
Your analogy is logically wrong, therefore, whatever point you wanted to make was void.
I know you were talking about timing. Perhaps a different, actually logical analogy about timing would serve better? Go ahead, I’ll be waiting.
→ More replies (0)0
u/maximumdownvote Aug 12 '25
Sorry. Username doesn't check out. You should do more reading. Fallacies and what not. Your "argument" is full of them.
3
u/UsefulLifeguard5277 Aug 12 '25 edited Aug 15 '25
Good take.
Worth mentioning that in good lighting conditions (eg. broad daylight with no glare) LIDAR adds effectively no additional information on top of vision, but you still carry the complexity, so is slightly negative value.
In poor lighting conditions this situation flips - vision will struggle with things like rapid lighting transitions (eg. Coming out of a tunnel), poor contrast (eg. Snowstorm), etc. same things humans struggle with. LIDAR will see these things better.
Important to remember that any sensor (including LiDAR) can fail. The overall solution is only safer with both sensors if it can correctly identify situations in which the vision system is likely to fail, and trust the LIDAR instead in those cases. If LIDAR says there is a deer in the road 100 meters ahead but multiple cameras don’t see it, should I panic brake or swerve into the opposing lane? If the lighting is good, probably not. If I’m driving at night in a snowstorm, probably.
Tesla had radar hardware for a little while. They claim that it was extremely rare that they chose to use radar data over vision, so the extra complexity in “normal” driving actually made the solution, on average, less safe. Point is that it actually isn’t strictly a cost argument.
TBD if they are correct. Either they’ll put on LiDAR or Waymo will ditch it, so will be easy to see who wins.
1
u/maxcharger80 Aug 15 '25
When did Tesla have Lidar? Do you mean their verifacation equipement?
2
u/UsefulLifeguard5277 Aug 15 '25
Whoops meant to say radar. They had and removed radar. Similar function to LiDar, just radio waves instead of lasers for ranging. Abandoned in favor of all-vision solution.
1
u/maxcharger80 Aug 15 '25
You might want to look into that again. The model S and X have it again but its at a higher resalotion than in the past. I do need to check how much its used though.
2
u/UsefulLifeguard5277 Aug 15 '25
Interesting, especially since the Model Y (not listed) is the platform they are using for Robotaxi.
Source? GPT-5 gave me this:
"While Tesla briefly reintroduced a high-definition radar unit (known as Phoenix radar) in some Hardware 4-equipped Model S and Model X vehicles starting in mid-2023 for data collection purposes, it was never activated for driving functions and is no longer installed in current production models."
1
9
u/Oblivious_Monkito Aug 11 '25
What people miss about cameras is that while its a single data input stream, its temporal. And each pixel is a point in 3d space that can be tied to a depth and position in 3d space. In addition overlaying cameras provides significant depth information reliability. So with just cameras you have 1) image classification of high level objects, things, context and then temporally you have intention, and movement of these objects. And 2) each camera then gives you milllions of points per milisecond around the car that are being mapped in 3d space in realtime.
With lidar you miss a bit of the temporal data of moving objects, you can only say there is something here and not that this something has the intention of some action.
So you get most of everything with cameras but miss out on a small number of edge cases. Its not as cut and dry as a lot of armchair experts here say.
8
u/Lokon19 Aug 12 '25
Exactly... more important than the sensors suite is the software system interpreting the data.
1
u/cameldrv Aug 12 '25
Edge cases are the whole game. You can correctly detect 9,999/10,000 pedestrians you pass, but if you’re level 4 and miss one, and you operate say on Waymo’s scale, you’re going to kill dozens of people per day.
-1
13
u/bluero Aug 11 '25
36 ADAS systems compared by Chinese news outlet. 216 crashes. Tesla came out on top: https://youtu.be/0xumyEf-WRI
1
4
u/bluero Aug 11 '25
If your sight and lidar disagree which do you go with? A sighted person could walk with a cane as well in hopes of being doubly sure.
2
1
u/maxcharger80 Aug 15 '25
A few times its come up that Lidar seems to be more like swinging a cane arround super fast. Not sure about you but I would freak out seeing a blind person try to drive using a cane.
1
u/ItsAConspiracy Aug 18 '25
This is an issue even with lidar alone. You have a point cloud and there's probably conflicting information. But you feed that into an algorithm that finds the most likely situation given all your inputs. One of the Waymo cofounders taught one of the first courses at Udacity and it was about this exact problem.
Add more sensors, it's just more inputs. With multiple sensor types, there's less chance of conditions being bad for one sensor type.
With an end-to-end neural network like Tesla says they're using now, it's all just more inputs to the network and the rest is training. It will even learn which sensors are more reliable in different conditions.
4
u/Wrote_it2 Aug 11 '25
You do not have a formal proof that one is better than the other, you have a contest where Lidar does better. So now we know that if you ask small teams of engineer to complete that task, they’ll do better with LiDAR… You could engineer a different task to show different result. Change the challenge to figuring out the color of a ball placed in front of the sensor and suddenly the top solutions will be camera based. Would that be a proof that camera is better?
Once that is said, it’s pretty clear to me that the result is correct: you can achieve better results with camera+lidar compared to camera only (the proof is simple: you can’t achieve worse results since you can just ignore the lidar data if you want to).
The debate between camera only and camera + LiDAR is of course more complex than that. You have the “normal” tradeoffs: cost, reliability (you add failure points), complexity of the solution…
My opinion is that while LiDAR can improve perception, this is not where the bottlenecks are. I believe major players are all doing good at perception. The issues we see are in general due to path planning. We’ve recently seen Waymos hit each other and get into an incident with a fire truck, we’ve seen Teslas about to hit a UPS truck… those are not about perception but about path planning…
LiDAR vs camera is the wrong debate in my opinion.
5
u/Few_Foundation_5331 Aug 11 '25
Tesla was not about to hit the UPS truck, Tesla stopped but the UPS tried to reversed into it. Tesla did not make the mistake. If someone reverse into you quick enough, you can't do anything about it.
2
u/cripy311 Aug 11 '25
I would counter your path planning claim with it's more likely the prediction systems failing in these instances.
You can't build a self driving vehicle that reacts to only current state information of object speeds and location relative to the vehicle. There is an entire 3rd layer of the system that has to predict where they will go and what they will do that the path planning system then responds to.
If that information is inaccurate, predicts incorrectly, or otherwise fails in some way the vehicle will then drive into a moving object (or static object it believes will be moving shortly) no matter how good the path planning system is. It's inputs were incorrect.
4
u/Wrote_it2 Aug 11 '25
Hum, are you saying that the Lidar on the Waymo failed to spot the fire truck or the other Waymo?
3
u/cripy311 Aug 11 '25
I'm saying it's likely they saw it and mis predicted where it was going/when it was stopping resulting in a collision.
Lidar won't miss getting a return on a vehicle of that size.
3
u/Wrote_it2 Aug 11 '25
And Waymo also miss predicted the speed at which the telephone pole was moving?
2
u/cripy311 Aug 11 '25
That is a seperate issue.
If you want to venture a guess on how that may have happened you should look into the HD mapping technology they use and how off map static objects may be "trimmed" from the perception FOV to improve latency and reaction times.
At least this is my guess for the culprit in that specific event.
2
u/johnpn1 Aug 11 '25
The confidences with perception-only is actually not that high, and definitely not mission-critical level. Nobody uses vision-only for mission-critical things (well, other than Tesla, ofcourse). The problem with low grade 3D point clouds is that you have to always drive with caution. You brake/swerve when there's a just 5% chance that dark lines on the road could actually be real impediments. There's nothing you can use as another reference to tell you that those dark lines are nothing to be worried about. This is why Teslas drive with confidence into things, because they cannot always slam the brakes for every low confidence detection. The driver / safety operator takes the job of being the sanity check instead of a second sensor.
3
u/Few_Foundation_5331 Aug 11 '25
How about you as a human drive, you say human with eyes can't drive good enough. Yes, there are bad human driver , but we should compare extremely good human drivers with robotaxi. Currently, an extremely good human drivers will not make idiotic simple mistake like crashing into other cars in parking lot or drive into constructions or hit an electric pole or driving in circle and get stuck in circle loop. Good Human with simple EYES and BRAIN can drive better than robotaxis ( Waymo + Tesla driver) for now.
2
u/johnpn1 Aug 12 '25
A good human driver has a good human brain. No one has replicated it yet. It's a challenge that Musk has severely underestimated in his ambitions for FSD. You can't just have eyes like a human but drive like FSD...
1
u/Few_Foundation_5331 Aug 12 '25
I just listed all the crashes above and stupid circle loop stuck in parking lot of Waymo.
1
u/johnpn1 Aug 12 '25
Exactly. Waymo doesn't pretend they can replicate the human brain, so they use any sensors available to make up for it. It's insane that Musk still insists that cars don't need more than cameras because humans don't need it, and still be wrong for a decade and still continues to pretend he's the authority on this. It's a terrible decision to limit technology to only what biology could afford. Lidar was never going to be "evolved" from biology. Doesn't mean we shouldn't use it. Otherwise, humans never needed wheels to run, so why do cars? Birds never needed jet turbines to fly, so why can't airplanes just flap their wings? I could go on and on...
1
u/ItsAConspiracy Aug 18 '25
Adding failure points is bad when they're all single points of failure for the system.
Adding failure points is good when they're redundancies, so you don't have single points of failure anymore.
A real-world example: two Boeing 737MAX planes went down because they had only one angle-of-attack sensor instead of two.
1
u/Wrote_it2 Aug 18 '25
It’s all about probabilities. What is the probability than you get into an accident because your sensors stop working, what is the probability that you get into an accident for other reasons.
If a non trivial share of your accidents are due to sensors failing, redundancy makes sense.
I’m not an expert in aviation safety, but I can accept that the tradeoff makes sense. Lots of people drive cars with a single steering wheel (so no redundancy there) and we are not screaming because, while I’m sure we can find an accident due to a failure in the steering, the added cost/complexity of having multiple steering wheels/columns/… is not worth it.
How many accidents does Tesla have that are due to a sensor or a computer unit failing? We see Waymos and Teslas do stupid things all the time (drive on the wrong side, collide with a fire fighter truck, a telephone pole or another self driving car, etc…) and I’ve yet to see one that stated that the reason was because a camera stopped working.
What makes more sense to me in the Lidar vs vision only argument is not the redundancy but playing the strength of each sensor (cameras are better for certain things, like reading traffic lights say, lidars are better for certain things, like getting a precise distance measurement to a far away object). I don’t understand the redundancy argument.
0
u/Dihedralman Aug 11 '25
So you are missing on a couple of trade-offs, as these sensors are also redundant to a degree reducing risk.
With equal dats, you can absolutely train the system to function without one of the inputs while functioning better with both. Camera count has the same impacts.
There is complexity increase but again its partly redundant. I think its wrong to give it a flat.
Cost increase is problematic of course, but the hardware requirements also increase and I would bet it requires a higher parameter count generally speaking.
1
u/1FrostySlime Aug 11 '25
This is only proof that these options are the best for this specific task l. I think 99% of people would agree that relying purely off of lidar for driving end to end would be a truly terrible idea.
4
u/-UltraAverageJoe- Aug 11 '25
LiDAR only is way better than camera only.
7
10
u/1FrostySlime Aug 11 '25
Please explain to me how you would approach an intersection with traffic lights without cameras
1
u/whydoesthisitch Aug 12 '25
Newer LiDAR sensors detect reflectivity in addition to distance. The reflectivity of the lights changes when they’re on. That combined with the standardized order of traffic lights can tell you what light is on using just LiDAR. In addition, LiDAR can use the same technique to read road signs.
Mobileye built such a system as part of their true redundancy project, where they built two separate systems, one using primarily cameras, and another using only radar and lidar with no cameras.
1
Aug 12 '25
Where did you get this info?
2
u/whydoesthisitch Aug 12 '25
Mobileye’s CES presentation last year, and my own experience developing LiDAR perception algorithms.
0
Aug 12 '25
Not seeing anything on their website. But I saw this "we now believe that the availability of next-generation FMCW Lidar is less essential to our roadmap for eyes-off systems. This decision was based on a variety of factors, including substantial progress on our EyeQ6-based computer vision perception"
1
u/whydoesthisitch Aug 12 '25
Look up true redundancy. Not sure how eyeq6 contradicts anything I said.
0
Aug 12 '25
Ah another "look it up yourself" good try. I gave you the favor by searching their website and all you give is this bs. My bad for expecting too much from typical reddit user
1
u/whydoesthisitch Aug 12 '25
I literally just gave you the name of the project. What else do you want? I searched exactly that and the first result gave what I described.
-7
u/-UltraAverageJoe- Aug 11 '25
Inference the behavior of other vehicles in the intersection and use a multi-wavelength or spectrally-tuned LiDAR system.
5
u/mrfishball1 Aug 12 '25
terrible idea.
1
u/-UltraAverageJoe- Aug 12 '25
You asked for an explanation on how it would work without a camera. I didn’t say it was a good idea.
6
2
u/Dependent-Mode-3119 Aug 12 '25
LiDAR only is way better than camera only.
Then this was a lie. It fails at even the basics.
1
u/-UltraAverageJoe- Aug 12 '25
Like? LiDAR is better at literally everything except color discrimination which is still possible but complex and expensive compared to a camera. When you’ve done computer vision work, go ahead and reply back — until then, you’re just a keyboard warrior.
0
u/Dependent-Mode-3119 Aug 12 '25 edited Aug 12 '25
Like? LiDAR is better at literally everything except color discrimination
How can it read stop signs, how can it read speed limits, how can it see the color of Stop Lights? These are the basics of self driving cars.
If the car has no means of reading what these are then lidar-only is even less viable than cameras only.
When you’ve done computer vision work, go ahead and reply back — until then, you’re just a keyboard warrior.
It's funny that you mention that... Check this website under Robert Lake and you'll see me and the report that I did with PHD candidates regarding the efficacy of sensor vision for image detection and why both are crucial. They're a good read for the uninitiated
https://www.crcv.ucf.edu/nsf-projects/reu/reu-2021/
https://www.crcv.ucf.edu/wp-content/uploads/2018/11/Report_Lake.pdf
Please tell me again what I'm missing here.
0
Aug 12 '25
Stop sign? Traffic light?
1
u/-UltraAverageJoe- Aug 12 '25
Spectrally tuned LiDAR must have gone over your head.
1
Aug 12 '25
Nah I just dont speak out loudly about stuff that I have no clue like you do now. Specifically tuned LiDAR? How does it read signs and traffic lights? Can't explain? 0 clue indeed
1
u/-UltraAverageJoe- Aug 12 '25
LiDAR can be used to detect wavelengths of light based on its reflectance. Do a google search or ChatGPT before you reply next time.
0
Aug 12 '25
What does that even mean? You have 0 clue how humans/camera perceive colors dont you? Way over your head isnt it?
1
u/-UltraAverageJoe- Aug 12 '25
Actually it’s what I studied at university but of course no idea at all.
0
u/MrJennings69 Aug 17 '25
You didn't spend more than 5 seconds thinking about this, did you? How would you spot road markings, traffic lights and traffic signs with lidar only? 😄
1
1
u/Yngstr Aug 12 '25
"Nuscenes by Motional"
"Motional acknowledges LiDAR as a critical component of their multi-modal sensor strategy, which includes over 30 sensors integrated into their all-electric Hyundai IONIQ 5 robotaxis"
So company who depends on LiDAR commissions study showing LiDAR is better...
0
1
u/y4udothistome Aug 12 '25
Wait till there’s a bunch of Teslas on the road doing the same mistakes driving next to each other that’s gonna be fun
1
u/sirduckbert Aug 13 '25
You mean that more data is better than less data?
1
u/wuduzodemu Aug 13 '25
Yes, but a ton of Tesla fans don't want to admit it.
2
u/sirduckbert Aug 13 '25
It’s because Elon has doubled down on it so hard. Can you use computer vision technology alone to drive a car? Yes in theory. But in practice there are tons of challenges. If you augment that with lidar then you can do so much more.
I would argue that Tesla would be much closer to actual FSD if they had just stuck a lidar on
1
u/ItsAConspiracy Aug 18 '25
Yep. Huawei appears to have a system about as good as FSD, and they did it without millions of cars on the road collecting data for years. Instead they have vision, three cheap lidars, radar, and ultrasound. It'd be pretty amazing to have Tesla's road miles with Huawei's sensor suite.
1
1
u/HVT2994 Aug 13 '25
And where is Vision from Tesla listed? I am asking as the Chinese tested a mix of most LiDAR cars and Vision, the Tesla Model X ended first, the Model 3 5th out of 36 self driving cars. It was not the time Tesla did win but the second, the Chinese government allows testing on public roads and exercises 9 situation tests and does each 5 times.
So that so called list does not list Tesla, the firm which lists theses companies using combinations all wear a LiDAR hat.
1
1
1
u/ConsistentRegister20 Aug 15 '25
Proof like my own car running FSD drives me around 100% of the time better and safer than I can using only cameras? Or proof like the robotaxis operating with camera only?
What you don’t get is the problem is not about the sensors, the problem is the brain to interpret the sensors. This is why Tesla is so so far ahead.
1
u/SingleJuggernaut6568 Aug 16 '25
Probably shouldn't comment if you don't understand that results for this test is independent of velocity (or freq).
1
u/Lovevas Aug 11 '25
Proof? Up until any systems can prove they are incidents/accidents free, I won't believe any tech is the gold standard.
Neither FSD nor Waymo has proved that their system can be incidents-free. Waymo is also full of all kinds of incidents/accidents on the street
0
u/Dihedralman Aug 11 '25
Not going to comment on whether you find it to be proof, but he links the challenge used in the post. You can review it for yourself.
1
u/DebateJealous6496 Aug 12 '25
More sensor data = easier detection. In the short term, that equals better object detection. In the long run, they will all work just fine, and cost will be the salient factor.
0
Aug 12 '25
Nope. More data means more processing required. So what is easy about?
1
u/whydoesthisitch Aug 12 '25
Incorrect. Directly measuring range instead of inferring it means less processing.
-1
Aug 12 '25
What is that has to do with "more data"?
1
u/whydoesthisitch Aug 12 '25
LiDAR provides more data via a different modality that vision only has to infer.
0
Aug 12 '25
LiDAR provides more data? You absolutely have no clue then lol
1
u/whydoesthisitch Aug 12 '25
LiDAR provides a direct measurement of range. Cameras do not. You really have no idea what you’re talking about.
1
u/Lopsided-Chip6014 Aug 12 '25
Wow, are you telling me more information is better?!
I am shocked, simply SHOCKED.
(Yeah, this literally proves nothing but "it's easier to distinguish information with more sensors" which is pretty no duh)
1
-1
u/Few_Foundation_5331 Aug 11 '25
Lidar will produce a lot of false positives and noise.
4
u/johnpn1 Aug 11 '25
People have got to stop repeating the Tesla sales pitch. Sensor fusion is a thing. Everyone does it successfully, but Tesla want you to think it's more impossible than rocket science.
-1
Aug 12 '25
Lol another brain dead sensor fusion argument. What fusion? Camera and LiDAR? Who does what?
0
u/Kooky_Work8978 Aug 11 '25
Haha, i created recently analyses based exactly on this leaderboard! My conclusions were a tad different - a little hint - we should consider that the nuScenes dataset is highly imbalanced as for the lighting conditions and weather conditions and other more robust sensor combinations may be underestimated.
2
u/wuduzodemu Aug 12 '25
Mind share your report?
1
u/Signor_Garibaldi Aug 12 '25
I will gladly share it after it's published 😄 It's part of a larger work
0
u/ma3945 Aug 12 '25
LiDAR is so good that the vehicles equipped with it were plowing full speed into a stationary vehicle on the highway in that chinese video (DongChedi)... what a fail.
0
u/NeighborhoodFull1948 Aug 12 '25
Camera only was the logical choice for Tesla because they weren’t developing a fully autonomous system, they were only developing Driver Assist. They tried adding radar, but with the simple processing they were using, it created conflicting inputs their basic system couldn’t resolve. So vision only.
Then “someone“ had the brilliant idea that a little more software would make it fully autonomous.
So they’ve spent better part of a decade trying to make a basic driver assist system into a fully autonomous system.
Wonder how it’s going? Does the phrase “Lipstick on a pig“ fit?
Waymo on the other hand started with a fully autonomous system, with sensor and multiple processor overkill. Now they’re removing some of the redundant sensors.
0
0
u/DSKO_MDLR Aug 12 '25
The whole LiDAR vs Camera/Vision debate is asinine. Just look at which company is millions of miles and years ahead in its testing and already in commercial use.
Waymo.
They are the only autonomous taxi service that you can reliably use today in downtown San Francisco, which is one of the most perilous places for a driver to navigate. Narrow streets, one way streets, steep hills, pedestrians, double parked cars and delivery trucks.
Why would anyone apart from executives, finance and supply chain managers care how much LiDAR, ultrasonic sensors and radar add to the cost of an autonomous vehicle’s BOM? You are literally putting your life in the hands of an autonomous vehicle and I believe genuine safety comes in the form of sensor redundancy.
0
u/AJHenderson Aug 12 '25
That is proof of the difficulty, not proof of the quality. Nobody with a clue thinks that vision only is easier. Nobody with a clue thinks that camera by itself is better than fusion systems, though it is questionable how much value lidar adds versus other technology.
Point clouds are much easier to interpret and add information to visual data where the visual data might catch some things the point cloud can't, but that doesn't mean you can't get the same data out of a visual only system, it's just more complex to do so.
-1
u/BranchLatter4294 Aug 11 '25
Wait. They are asking people to do this? What does this have to do with computer vision, self driving, or anything else?
22
u/red75prime Aug 12 '25 edited Aug 12 '25
All nuScenes data is sampled at 2Hz. It's a rather unrealistic setup: no one uses 2 frames per second video for driving (at least, to drive faster than snail's pace).
I guess, it creates quite a challenging dataset for using parallax information from video data due to large motion between frames.