US probes Tesla's Full Self-Driving software after fatal crash

- It failed with a cryptic system error while driving

- It started making a left turn far too early that would have scraped the left side of the car on a sign. I had to manually intervene.

- In my opinion, the default setting accelerates way too aggressively. I'd call myself a fairly aggressive driver and it is too aggressive for my taste.

- It tried to make way too many right turns on red when it wasn't safe to. It would creep into the road, almost into the path of oncoming vehicles.

- It didn't merge left to make room for vehicles merging onto the highway. The vehicles then tried to cut in. The system should have avoided an unsafe situation like this in the first place.

- It would switch lanes to go faster on the highway, but then missed an exit on at least one occasion because it couldn't make it back into the right lane in time. Stupid.

After the system error, I lost all trust in FSD from Tesla. Until I ride in one and feel safe, I can't have any faith that this is a reasonable system. Hell, even autopilot does dumb shit on a regular basis. I'm grateful to be getting a car from another manufacturer this year.

When I did the trial on my Tesla, I also noted these kinds of things and felt like I had to take control.

But at the end of the day, only the numbers matter.

Maybe that could be a problem with future versions, but I don't see it happening with 12.3.x. I've also heard that driver attention monitoring is pretty good in the later versions, but I have no first hand experience yet.

Even if it is true that the data show that with FSD (not Autopilot) enabled, drivers are in fewer crashes, I would be worried about other confounding factors.

For instance, I would assume that drivers are more likely to engage FSD in situations of lower complexity (less traffic, little construction or other impediments, overall lesser traffic flow control complexity, etc.) I also believe that at least initially, Tesla only released FSD to drivers with high safety scores relative to their total driver base, another obvious confounding factor.

Happy to be proven wrong though if you have a link to a recent study that goes through all of this.

We all know that some humans are sometimes terrible drivers!

We also know what that looks like: Driving too fast or slow relative to surroundings. Quickly turning every once in a while to stay in their lane. Aggressively weaving through traffic. Going through an intersection without spending the time to actually look for pedestrians. The list goes on..

Bad human driving can be seen. Bad automated driving is invisible. Do you think the people who were about to be hit by a Tesla even realized that was the case? I sincerely doubt it.

You have a Mercedes, Mercedes takes responsibility.

You have a Tesla, you take the responsibility.

Says a lot.

That said, there is a night and day difference between FSD 12.3 that you experienced earlier this year and the latest version 12.6. It will still make mistakes from time to time but the improvement is massive and obvious. More importantly, the rate of improvement in the past two months has been much faster than before.

Yesterday I spent an hour in the car over three drives and did not have to turn the steering wheel at all except for parking. That never happened on 12.3. And I don't even have 12.6 yet, this is still 12.5; others report that 12.6 is a noticeable improvement over 12.5. And version 13 is scheduled for release in the next two weeks, and the FSD team has actually hit their last few release milestones.

People are right that it is still not ready yet, but if they think it will stay that way forever they are about to be very surprised. At the current rate of improvement it will be quite good within a year and in two or three I could see it actually reaching the point where it could operate unsupervised.

It got stuck in a side street trying to get to a target parking lot, shaking the wheel back and forth.

It's no better so far and this is the first day.

As I said, it still makes mistakes and it is not ready yet. But 12.3 was much worse. It's the rate of improvement I am impressed with.

I will also note that the predicted epidemic of crashes from people abusing FSD never happened. It's been on the road for a long time now. The idea that it is "irresponsible" to deploy it in its current state seems conclusively disproven. You can argue about exactly what the rate of crashes is but it seems clear that it has been at the very least no worse than normal driving.

Don't get me wrong without a concerted data team building maps a priori, this is pretty incredible. But from a pure performance standpoint it's a shaky product.

Oh dear.

Glad you're okay!

I wonder how things are inside your head. Are you ignorant or affected by some strong bias?

I'll believe it when I see it. I'm not sure "quite good" is the next step after "feels dangerous".

Oh? Who are presenting the numbers?

Is a crash that fails to trigger the airbags still not counted as a crash?

What about the car turning off FSD right before a crash?

How about adjusting for factors such as age of driver and the type of miles driven?

The numbers don't make sense because they're not good comparisons and are made to make Tesla look good.

I'm literally saying that it is visible, to me, the passenger. And for reasons that aren't just bad vibes. If I'm in an Uber and I feel unsafe, I'll report the driver. Why would I pay for my car to do that to me?

This is what bugs me about ordinary autopilot. Autopilot doesn't switch lanes, but I like to slow down or speed up as needed to allow merging cars to enter my lane. Autopilot never does that, and I've had some close calls with irate mergers who expected me to work with them. And I don't think they're wrong.

Just means that when I'm cruising in the right lane with autopilot I have to take over if a car tries to merge.

This is probably the worst way to evaluate self-driving for society though, right?

Confounding factors are what allows one to tell appart "the system cause less loss of life" from "the system causes more loss of life yet it is only enabled in situations were fewer lives are lost".

There’s a simple answer to this. As soon as it’s good enough for Tesla to accept liability for accidents. Until then if Tesla doesn’t trust it, why should I?

Also, Tesla is known to disable self-driving features right before collisions to give the appearance of driver fault.

And the coup de grace: if Tesla's own data showed that FSD was actually safer, they'd be shouting it from the moon, using that data to get self-driving permits in CA, and offering to assume liability if FSD actually caused an accident (like Mercedes does with its self driving system).

>And I don't even have 12.6 yet, this is still 12.5;

How am i supposed to take anything you say seriously when your only claim is a personal anecdote that doesn't even apply to your own argument. Please, think about what you're writing, and please stop repeating information you heard on youtube as if it's fact.

The is one of the reasons (among many) that I can't take Tesla booster seriously. I have absolutely zero faith in your anecdote that you didn't touch the steering wheel. I bet it's a lie.

When I feel as safe as I do sitting in the back of a Waymo.

12.5.6 Still fails to read very obvious signs for 30 Km/h playgrounds zones.

The current vehicles lack sufficient sensors, and likely do not have enough compute power and memory to cover all edge cases.

I think it's a matter of time before Tesla faces a lawsuit over continual FSD claims.

My hope is that the board will grow a spine and bring in a more focused CEO.

Hats off to Elon for getting Tesla to this point, but right now they need a mature (and boring) CEO.

Technically, that is the easiest way to get a perfect safety record and journalists will seemingly just go along with the charade.

I didn't say I didn't touch the steering wheel. I had my hands lightly touching it most of the time, as one should for safety. I occasionally used the controls on the wheel as well as the accelerator pedal to adjust the set speed, and I used the turn signal to suggest lane changes from time to time, though most lane choices were made automatically. But I did not turn the wheel. All turning was performed by the system. (If you turn the wheel manually the system disengages). Other than parking, as I mentioned, though FSD did handle some navigation into and inside parking lots.

Here's an unintentionally hilarious meta-post on the subject https://news.ycombinator.com/item?id=29531915

I've ridden in an X, S and Y with it on. Talk about vomit inducing when letting it drive during "city" driving. I don't doubt it's OK on highway driving, but Ford Blue Cruise and GM's Super Cruise are better there.

But the rate of progress I see right now has me thinking that it may not be more than two or three years before that threshold is finally reached.

(you probably misinterpreted what I said. I do sometimes change lanes, even well in advance of a merge I know is prone to problems, if that's the safest and most convenient. What I am saying is the guidance I have read indicates that staying in the same lane is generally safer than changing lanes, and speeding up into an empty space is better for everybody than slowing down, especially because many people who are merging will keep slowing down more and more when the highway driver slows for them)

for whatever does exist, it is also easy to imagine how they could be misleading. for instance i've disengaged FSD when i noticed i was about to be in an accident. if i couldn't recover in time, the accident would not be when FSD is on and depending on the metric, would not be reported as a FSD induced accident.

The average for all USA cars seems to be around $2000/year, so even if FSD was half as dangerous Tesla would still be paying $1000/year equivalent (not sure how big insurance margins are, assuming nominal) per car.

Now, if legally the driver could avoid paying insurance for the few times they want/need to drive themselves (e.g. snow? Dunno what FSD supports atm) then it might make sense economically, but otherwise I don't think it would work out.

I'm curious, what is the alternative that you are considering? I've been delaying an upgrade to electric for some time. And now, a car manufacturer that is contributing to the making of another Jan 6th, 2021 is not an option, in my opinion.

Car insurance payments include people stealing your car, uninsured motorists, rental cars, and other issues not the drivers fault. Further insurance payments also include profits for the insurance company, advertising, billing, and other overhead from running a business.

Also, if Tesla was taking on these risks you’d expect your insurance costs to drop.

I have no illusions about Tesla's ability to deliver an unsupervised self-driving car any time soon. However, as far as I understand, their autosteer system, in spite of all its flaws, is still the best out there.

Do you have any reason to believe that there actually is something better?

"To ensure our statistics are conservative, we count any crash in which Autopilot was deactivated within 5 seconds before impact"

https://www.tesla.com/en_gb/VehicleSafetyReport

Situations which inevitably cause a crash more than 5 seconds later seem like they would be extremely rare.

How far are we now, 6 years later than I expected?

Dunno.

I suspect it's gonna need an invention on the same level as Diffusion or Transformer models to be able to get all the edge cases we can get, and that might mean we only get it with human level AGI.

But I don't know that, it might be we've already got all we need architecture-wise and it's just a matter of scale.

Only thing I can be really sure of is we're making progress "quite fast" in a non-objective use of the words — it's not going to need a re-run of 6 million years of mammilian evolution or anything like that, but even 20 years wall clock time would be a disappointment.

Bring on the videos of you in the passenger seat on FSD for any amount of time.

Perhaps FSD isn't ready for city streets yet, but it's great on the highways and I'd 1000x prefer we make progress rather than settle for the status quo garbage that the legacy makers put out. Also, human drivers are the most dangerous, by far, we need to make progress to eventual phase them out.

I've been on the receiving end of this with the offender being a Tesla so many times that I figured it must be FSD.

Germans are not known for poor driving.

Note that I personally do change lanes at times when it's safe, convenient, I am experienced with the intersection, and the merging driver is being especially unaware.

They may well best Tesla to this, though.

But people get upset about this. We need corporations to take responsibility.

Maybe you have a deathwish, but I definitely don't. Your choices affect other humans in traffic.

> If you write software that people rely on with their lives, and it fails, you should be held liable for that criminally.

Easy to type those words on the internet than make it a policy IRL. That sort of policy IRL would likely result in a) killing off all commercial efforts to solve traffic deaths via technology and vast amounts of other semi-autonomous technology like farm equipment or b) government/car companies mandating filming the driver every time they turn it on, because it's technically supposed to be human assisted autopilot in these testing stages (outside restricted pilot programs like Waymo taxis). Those distinctions would matter in a criminal court room, even if humans can't always be relied upon to always follow the instructions on the bottle's label.

I’m not GP, but I can share video showing it driving across residential, city, highway, and even gravel roads all in a single trip without touching the steering wheel a single time over a 90min trip (using 12.5.4.1).

Texas highways allow for much higher merging speeds at the cost of far large (land area), 5-level interchanges rather than 35 mph offramps and onramps common in California.

Any defensive driving course (which fall under instruction IMO) states that you don't always have to exercise your right of way, and indeed it may be unsafe to do so in some circumstances. Anticipating the actions of other drivers around you and avoiding potentially dangerous are the other aspects of being a defensive driver, and those concepts are consistent with freeing up the lane slower-moving vehicles are merging onto when it's safe to do so.

I do wonder if people who have wildly different experiences than I have are living in a part of the country that, for one reason or another, Tesla FSD does not yet do as well in.

Upside: you get paid a 200k salary, if all your code works perfectly. Downside: if it doesn't, you go to prison.

The users aren't compelled to use it. They can choose not to. They get to choose their own risks.

The internet is a gold mine of creatively moronic opinions.

I would also say that if something gets signed off on by management, that carries an implicit transfer of accountability up the chain from the individual contributor to whoever signed off.

Holding software developers (or their companies) liable for defects would definitely kill off a part of the industry: the very large part that YOLOs code into production and races to get features released without rigorous and exhaustive testing. And why don’t they spend 90% of their time testing and verifying and proving their software has no defects? Because defects are inevitable and they’re not held accountable for them!

And also, of course some people would take that deal, and of course some others wouldn't. Your argument is moot.

- Certain exits are not detected as such and the car violently veers right before returning to the lane. I simply can't believe they don't have telemetry to remedy this.

- Sometimes the GPS becomes miscalibrated. This makes the car think I'm taking an exit when I'm not, causing the car to abruptly reduce its speed to the speed of the ramp. It does not readjust.

- It frequently slows for "emergency lights" that don't exist.

- If traffic comes to a complete stop, the car accelerates way too hard and brakes hard when the car in front moves any substantial amount.

At this point, I'd rather have something less good than something which is an active danger. For all intents and purposes, my Tesla doesn't have reliable cruise control, period.

Beyond that, though, I simply don't have trust in Tesla software. I've encountered so many problems at this point that I can't possibly expect them to deliver a product that works reliably at any point in the future. What reason do I have to believe things will magically improve?

It doesn't matter, Tesla does neither. It always does the worst possible non-malicious behavior.

I'll say the autopark was kind of neat, but parking has never been something I have struggled with.

I'm not quite sure what you meant by your second paragraph, but I'm sure I have my blind spots and biases. I do have direct experience with various versions of 12.x though (12.3 and now 12.5).

Scroll down to Methodology at https://www.tesla.com/VehicleSafetyReport

I’ve been pretty skeptical of FSD and didn’t use the last version much. Today I used the latest test version, enabled yesterday, and rode around SF, to and from GGP, and it did really well.

Waymo well? Almost. But whereas I haven’t ridden Waymo on the highway yet, FSD got me from Hunters Point to the east bay with no disruptions.

The biggest improvement I noticed was its optimizations on highway progress.. it’ll change lanes, nicely, when the lane you’re in is slower than the surrounding lanes. And when you’re in the fast/passing lane it’ll return to the next closest lane.

Definitely better than the last release.

The line should be where a person tells others that they can rely on the software with their lives - as in the integrator for the end product. Even if I was working on the software for self driving, the same thing would apply - if I wrote some alpha level stuff for the internal demonstration and some manager decided "good enough, ship it", they should be liable for that decision. (Because I wouldn't be able to stop them / may have already left by then)

I don't think people believe this as such. It may be the short way to write it, but actually what devs mean is "bugs are inevitable at the funding/time available". I often say "bugs are inevitable" when it practice it means "you're not going to pay a team for formal specification, validated implementation and enough reliable hardware".

Which business will agree to making the process 5x longer and require extra people? Especially if they're not forced there by regulation or potential liability?

Literally everywhere else, we accept that the costs of 100% safety are just unreasonably high. People would rather have a mostly safe device for $1 than a definitely safe one for $5. No one wants to pay to have every head of lettuce tested for E Coli, or truckers to drive at 10mph so they can’t kill anyone.

Software isn’t different. For the vast majority of applications where the costs of failure are low to none, people want it to be free and rapidly iterated on even if it fails. No one wants to pay for a formally verified Facebook or DoorDash.

Most code has bugs from time to time even when highly skilled developers are being careful. None of them would drive if the fault rate was similar and the outcome was death.

Being in the passenger seat is still off limits for obvious reasons.

I suspect the free trials let tesla collect orders of magnitude more data on events requiring human intervention. If each one is a learning event, it could exponentially improve things.

I tried it on a loaner car and thought it was pretty good.

One bit of feedback I would give tesla - when you get some sort of FSD message on the center screen, make the text BIG and either make it linger more, or let you recall it.

For example, it took me a couple tries to read the message that gave instructions on how to give tesla feedback on why you intervened.

EDIT: look at this graph

https://electrek.co/wp-content/uploads/sites/3/2024/10/Scree...

The fact that people generally consume drugs voluntarily and make that decision after being informed about most of the known risks probably mitigates that to some extent. Being killed by someone else’s FSD car seems to be very different

If the software is not reliable enough, then don’t use it in a context where it could kill people.

Yes, but also in none of these situations would the consumer/customer/patient be held responsible. I don’t expect a system to be perfect, but I won’t accept any liability if it malfunctions as I use it the way it is intended. And even worse, I would not accept that the designers evade their responsibilities if it kills someone I know.

As the other poster said, I am happy to consider it safe enough the day the company accepts to own its issues and the associated responsibility.

> No one wants to pay for a formally verified Facebook or DoorDash.

This is untenable. Does nobody want a formally verified avionics system in their airliner, either?

Look, there are well established procedures in a lot of industries where products are relied on to keep people safe. They all require quite rigorous development and certification processes and sneaking untested alpha quality software through such a process would be actively malicious and quite possibly criminal in and of itself, at least in some industries.

For a huge part of the industry, the reason is entirely different. It is because software that mostly works today but has defects is much more valuable than software that always works and has no defects 10 years from now. Extremely well informed business customers will pay for delivering a buggy feature today rather than wait two more months for a comprehensively tested feature. This is the reality of the majority of the industry: consumers care little about bugs (below some defect rate) and care far more about timeliness.

This of course doesn't apply to critical systems like automatic drivers or medical devices. But the vast majority of the industry is not building these types of systems.

You're in violent agreement with me ;)

It's completely normal in other fields where engineers build systems that can kill.

Please stop presenting arguments of the form "I haven't seen problems so people who have problems must be extreme outliers". At best it's ignorant, at worst it's actively in bad faith.

Would you want FSD software to be developed or not? If you do, do you think holding devs or companies criminally liable for half of all crashes is the best way to ensure that progress happens?

Sounds like Tesla drivers have been at the Kool-Aid then.

But to be a bit more serious, the problem isn't necessarily that people don't think it's improving (I do believe it is) or that they will likely succeed (I'm not sure where I stand on this). The problem is that every year Musk says the next year will be the Year of FSD. And every next year, it doesn't materialize. This is like the Boy Who Cried Wolf; Musk has zero credibility with me when it comes to predictions. And that loss of credibility affects my feeling as to whether he'll be successful at all.

On top of that, I'm not convinced that autonomous driving that only makes use of cameras will ever be reliably safer than human drivers.

I would love for self-driving to succeed. I do long-ish car trips several times a year, and it would be wonderful if instead of driving, I could be watching a movie or working on something on my laptop.

I've tried Waymo a few times, and it feels like magic, and feels safe. Their record backs up that feeling. After everything I've seen and read and heard about Tesla, if I got into a Tesla with someone who uses FSD, I'd ask them to drive manually, and probably decline the ride entirely if they wouldn't honor my request.

> fortunately, it's clear as day to some of us that tesla approach will work

And based on my experience with Tesla FSD boosters, I expect you're basing that on feelings, not on any empirical evidence or actual understanding of the hardware or software.

That’s not a reasonable assumption. You can’t just extrapolate “software rate of improvement”, that’s not how it works.

(That's for merging onto a highway from an entrance ramp, at least. If you're talking about a zipper merge due to a lane ending or a lane closure, sure, cooperation with other drivers is always the right thing to do.)

Realistically speaking, nuclear power is pretty safe. In the history of nuclear power, there were two major incidents. Considering the number of nuclear power plants around the planet, that is pretty good. However, as those two accidents demonstrated, the potential fallout of those incidents is pretty severe and widespread. I think this massively contributes to the perceived risks. The warnings towards the public were pretty clear. I remember my mom telling stories from the time the Chernobyl incident became known to the public and people became worried about the produce they usually had from their gardens. Meanwhile, everything that has been done to address the hazards of fossil based power generation is pretty much happening behind the scenes.

With coal and natural gas, it seems like people perceive the risks as more abstract. The radioactive emissions of coal power plants have been known for a while and the (potential) dangers of fine particulate matters resulting from combustion are somewhat well known nowadays as well. However, the effects of those danger seem much more abstract and delayed, leading people to not be as worried about it. It also shows on a smaller, more individual scale: people still buy ICE cars at large and install gas stoves into their houses despite induction being readily available and at times even cheaper.

How often does an autonomous driving system get the driver into a dicey situation, but the driver notices the bad behavior, takes control, and avoids a crash? I don't think we have publicly-available data on that at all.

You admit that you ran into some of these sorts of situations during your trial. Those situations are unacceptable. An autonomous driving system should be safer than a human driver, and should not make mistakes that a human driver would not make.

Despite all the YouTube videos out there of people doing unsafe things with Tesla FSD, I expect that most people that use it are pretty responsible, are paying attention, and are ready to take over if they notice FSD doing something wrong. But if people need to do that, it's not a safe, successful autonomous driving system. Safety means everyone can watch TV, mess around on their phone, or even take a nap, and we still end up with a lower crash rate than with human drivers.

The numbers that are available can't tell us if that would be the case. My belief is that we're absolutely not there.

Consider: what I expect is actually true based on the data is that Tesla FSD is as safe or safer than the average human driver, but only if the driver is paying attention and is ready to take over in case FSD does something unsafe, even if FSD doesn't warn the driver it needs to disengage.

That's not an autonomous driving system. Which is potentially fine, but the value prop of that system is low to me: I have to pay just as much attention as if I were driving manually, with the added problem that my attention is going to start to wander because the car is doing most of the work, and the longer the car successfully does most of the work, the more I'm going to unconsciously believe I can allow my attention to slip.

I do like current common ADAS features because they hit a good sweet spot: I still need to actively hold onto the wheel and handle initiating lane changes, turns, stopping and starting at traffic lights and stop signs, etc. I look at the ADAS as a sort of "backup" to my own driving, and not as what's primarily in control of the car. In contrast, Tesla FSD wants to be primarily in control of the car, but it's not trustworthy enough to do that without constant supervision.

That's not a crash (fortunately!), but it is a failure of the autonomous system.

This is hard to track, though, of course: people might take over control for reasons unrelated to safety, or people may misinterpret something that's safe as unsafe. So you can't just track this from a simple "human driver took control".

Also how about: it turns into oncoming traffic, but there isn't much oncoming traffic, and that traffic swerves to get out of the way, before FSD realizes what it's done and pulls back into the correct lane. We certainly don't have data on that.

So we accept that there are some bad drivers on the road because the alternative would be cruel.

But we don't have to accept bad software drivers.

The only practical way around this which permits autonomous vehicles (which are apparently dependent on much more complex and intractable codebases than, say, avionics) is a much higher threshold of criminal responsibility than the "the serious consequences resulted from the one-off execution of an dangerous manoeuvre which couldn't be justified in context" which sends human drivers to jail. And of course that double standard will be problematic if "willingness to accept liability" is the only safety threshold.

I will refrain from commenting on whether or not that's a fair assumption to make, but I think that's where the frustration comes from.

I think when people make "WFM" posts, it would go a long way to acknowledge that the person who had a problem really did have a problem, even if implicitly.

"That's a bummer; I've driven thousands of miles using FSD, and I've felt safe and have never had to intervene. I wonder what's different about our travel that's given us such different experiences."

That kind of thing would be a lot more palatable, I think, even if you might think it's silly/tiring/whatever to have to do that every time.

But ok, great, so what? If that wasn't the case, FSD would be an unmitigated disaster with a body count in the tens of thousands. So in a comment thread about someone talking about the problems and unsafe behavior they've seen, a "well it works for me" reply is just annoying noise, and doesn't really add anything to the discussion.

That makes a lot of sense and not just from a selfish point of view. When a person drives a vehicle, then the person is held responsible for how the vehicle behaves on the roads, so it's logical that when a machine drives a vehicle that the machine's manufacturer/designer is held responsible.

It's a complete con that Tesla is promoting their autonomous driving, but also having their vehicles suddenly switch to non-autonomous driving which they claim moves the responsibility to the human in the driver seat. Presumably, the idea is that the human should have been watching and approving everything that the vehicle has done up to that point.

If a construction firm takes an arbitrary design and then tries to build it in a totally different environment and for a different purpose, then the construction firm is liable, not the original designer. It'd be like Boeing taking a child's paper aeroplane design and making a passenger jet out of it and then blaming the child when it inevitably fails.

By that token, no action could ever apply to a company—including approving, producing, or releasing—since it is a legal entity, a concept, not a physical thing. For all those actions there was a person actually doing it in the name of the company.

It’s perfectly normal to say, for example, “GenericCorp wrote a press-release about their new product”.

The regressions are getting worse. For the first release anouncement it was only hitting regulatory hurdles and now the entire software stack is broken? They should fire whoever is in charge and restore the state Elon tried to release a decade ago.

Climate change is very visible in the present day to me. People are protesting about it frequently enough that it's hard to claim they are not worried.

The FSD numbers are much more hidden.

The general accident rate is 1 per 400’000 miles driven.

FSD has one “critical disengagement” (aka before accident if human or safety braking doesn’t intervene) every 33 miles driven.

It means to reach unsupervised with human quality they would need to improve it 10’000 times in few months. Not saying it is impossible, just highly optimistic. In 10 years we will be there, but in 2 months, sounds a bit overpromising.

Which brings it right back to the original criticism of Tesla's "self driving" program. What you're describing is assisted driving, not anything close to "full self driving".

So nobody will accept it.

Of course that's wildly oversimplifying and multiple entities can be at fault at once. My point is that these are normal things considered in regular engineering and manufacturing

Until the car is ready to take over as legal driver, it's foolish to set the human driver up for failure in the way that Tesla (and the humans driving Tesla cars) do.

Has it really? How much data is each car sending to Tesla HQ? Anybody actually know? That's a lot of cell phone bandwidth to pay for, and a lot of data to digest.

Vast amounts of data about routine driving is not all that useful, anyway. A "highlights reel" of interesting situations is probably more valuable for training. Waymo has shown some highlights reels like that, such as the one were someone in a powered wheelchair is chasing a duck in the middle of a residential street.

May I ask how this initial trust was established?

This is a idiosyncrasy of the US (maybe other places too?) and I wonder if it's easier to do self driving at junctions, in countries without this rule.

Basically (_very_ basically, there's more to it) the insurance company insures itself against large claims.

Never really understood the supposed dilemma. What happens when the brakes fail because of bad quality?

Like for the COVID-19 vaccines? Experimental yet given to billions without ever showing them a consent form.

Correct, including when the other driver may not have the strictly interpreted legal right of way. You don't know if their vehicle is malfunctioning, or if the driver is malfunctioning, or if they are being overly aggressive or distracted on their phone.

But most of the time, on an onramp to a highway, people on the highway in the lane that is being merged into need to be taking into account the potential conflicts due to people merging in from the acceleration lane. Acceleration lanes can be too short, other cars may not have the capability to accelerate quickly, other drivers may not be as confident, etc.

So while technically, the onus is on people merging in, a more realistic rule is to take turns whenever congestion appears, even if you have right of way.

Take the Boeing 777 Primary Flight Computer for example. This is a fully digital fly-by-wire aircraft. There are 3 separate racks of equipment housing identical flight computers; 2 in the avionics bay underneath the flight deck, 1 in the aft cargo section. Each flight computer has 3 separate processors, supporting 2 dissimilar instruction set architectures, running the same software built by 3 separate compilers. Each flight computer captures instances of the software not agreeing about an action to be undertaken and wins by majority vote. The processor that makes these decisions is different in each flight computer.

The power systems that provide each flight computer are also fully redundant; each computer gets power from a power supply assembly, which receives 2 power feeds from 3 separate power supplies; no 2 power supply assemblies share the same 2 sources of power. 2 of the 3 power systems (L engine generator, R engine generator, and the hot battery bus) would have to fail and the APU would have to be unavailable in order to knock out 1 of the 3 computers.

This system has never failed in 30 years of service. There's still a primary flight computer disconnect switch on the overhead panel in the cockpit, taking the software out of the loop, to logically connect all of your control inputs to the flight surface actuators. I'm not aware of it ever being used (edit: in a commercial flight).

I don’t see how this is in any way objective besides the fact that you want it to be objective.

You can support things for society that scare you and feel unsafe because you can admit your feelings are subjective and the thing is actually safer than it feels to you personally.

As they say, they take "full responsibility"

Civil engineer kills someone with a bad building, jail. Surgeon removes the wrong lung, jail. Computer programmer kills someone, “oh well it’s your own fault”.

A big example is why do the other carmakers not yet offer camera recording on their cars? They are all using cameras all around, but only Tesla makes it available to you in case you want the footage? Bizarre. And then they want to charge you an extra $500+ for one dash cam on the windshield.

I even had Carplay/Android Auto as a basic requirement, but I was willing to forgo that after trying out the other brands. And not having to spend hours at a dealership doing paperwork was amazing. Literally bought the car on my phone and was out the door within 15 minutes on the day of my appointment.

You can still cook. You can still get hot water. If you have gas logs you still have a heat source in the winter too.

These trade offs are far more important to a lot of people.

The Koch brothers, famous "anti-regulatory state" warriors, have fought oversight so hard that their gas pipelines were allowed to be barely intact.

Two teens get into a truck, turn the ignition key - and the air explodes:

https://www.southcoasttoday.com/story/news/nation-world/1996...

Does anyone go to jail? F*K NO.

If you're the one merging? It's on you. If you're the one being merged into? Also you.

If you assume that every other driver has a malfunctioning vehicle or is driving irresponsibly then your odds of a crash go way down because you assume that they're going to try to merge incorrectly.

Not making an argument. Asking a clarifying question about someone else’s.

> It'd be like Boeing taking a child's paper aeroplane design and making a passenger jet out of it and then blaming the child when it inevitably fails.

Yes exactly. You are using the same example I used to say the same thing. So which part of my message was ridiculous?

It’s like a children’s book explanation of the nice things you can have (no speed limits) if everyone could just stop being such obscenely selfish people (like sitting in the left lane or preventing merges because of some weird “I need my car to be in front of their car” fixation).

For example:

I use this onramp fairly frequently. It’s rural and rarely has much traffic, but when there is you can get stuck for a while trying to get on because it’s hard to see the coming cars, and there’s not much room to accelerate (unless people move over, which they often do). https://maps.app.goo.gl/ALt8UmJDzvn89uvM7?g_st=ic

Preemptively getting in the left lane before going under this bridge is a defensive safety maneuver I always make—being in the right lane nearly guarantees some amount of conflict with merging traffic.

https://maps.app.goo.gl/PumaSM9Bx8iyaH9n6?g_st=ic

Are these the numbers reported by tesla, or by some third party?

Take a deep breath. Read my message one more time carefully. Notice the question mark at the end of the last sentence. Think about it. If after that you still think I’m trolling you or anyone else I will be here and happy to respond to your further questions.

Even a system that is 99% reliable will honestly feel very, very good to an individual operator, but would result in huge loss of life when scaled up.

Tesla can earn more trust be releasing the data necessary to evaluate the system’s performance. The fact that they do not is far more informative than a bunch of commentators saying “hey it’s better than it was last month!” for the last several years — even if it is true that it’s getting better and even if it’s true it’s hypothetically possible to get to the finish line.

The first question that comes to their minds is not "how can we prevent this accident?" but it's "how can we further inflate this bubble?"

Less capable driver assistance type systems might help the driver out (e.g. adaptive cruise control), but leave no doubt that the human is still driving. Tesla though goes far enough that it takes over driving from the human but it isn't reliable enough that the human can stop paying attention and be ready to take over at a moment's notice. This seems like the worst of all possible worlds since you are both disengaged by having to maintain alertness.

Autopilots in airplanes are much the same way, pilots can't just turn it on and take a nap. But the difference is that nothing an autopilot is going to do will instantly crash the plane, while Tesla screwing up will require split second reactions from the driver to correct for.

I feel like the real answer to your question is that having reasonable confidence in self-driving cars beyond "driver assistance" type features will ultimately require a car that will literally get from A to B reliably even if you're taking a nap. Anything close to that but not quite there is in my mind almost worse than something more basic.

Otherwise every car brand will have to kill a whole heap of people too until they manage to make a FSD system.

I agree, but my observation has been that the majority of drivers are absolutely trash at doing that and I'd rather they not crash into me, even if would be their fault.

Honestly I think Tesla's self-driving technology is long on marketing and short on performance, but it really helps their case that a lot of the competition is human drivers who are completely terrible at the job.

I don't think anyone in this thread has talked about an exemption from civil liability (sue for money), just criminal liability (go to jail).

Civil liability is the far less controversial issue because it's transferred all the time: governments even mandate that drivers carry insurance for this purpose.

With civil liability transfer, imperfect FSD can still make economic sense. Just as an insurance company needs to collect enough premium to pay claims, the FSD manufacturer would need to reserve enough revenue to pay its expected claims. In this case, FSD doesn't even need to be better than humans to make economic sense, in the same way that bad drivers can still buy (expensive) insurance.

This is important because I'd bet similar data on the use of standard, non-adaptive cruise control would similarly show it's much safer than human drivers. But of course that would be because people use cruise control most in long-distance highway driving outside of congested areas, where you're least likely to have an accident.

Socially, that's not quite the standard. As a society, we're at ease with auto fatalities because there's often Someone To Blame. "Alcohol was involved in the incident," a report might say, and we're more comfortable even though nobody's been brought back to life. Alternatively, "he was asking for it, walking at night in dark clothing, nobody could have seen him."

This is an emotional standard that speaks to us as human, story-telling creatures that look for order in the universe, but this is not a proper actuarial standard. We might need FSD to be manifestly safer than even the best human drivers before we're comfortable with its universal use.

If you didn't intend your comment as a criticism, then you phrased it poorly. Do you actually believe that your scenario happens in reality?

You would likely also be liable if you overloaded an electrical cable, causing a fire that killed someone.

“Using it in the way it was intended” is largely circular reasoning; of course it wasn’t intended to hurt anyone, so any usage that does hurt someone was clearly unintended. People frequently harm each other by misusing items in ways they didn’t realize were misuses.

> This is untenable. Does nobody want a formally verified avionics system in their airliner, either?

Not for the price it would cost. Airbus is the pioneer here, and even they apply formal verification sparingly. Here’s a paper from a few years ago about it, and how it’s untenable to formally verify the whole thing: https://www.di.ens.fr/~delmas/papers/fm09.pdf

Software development effort generally tends to scale superlinearly with complexity. I am not an expert, but the impression I get is that formal verification grows exponentially with complexity to the point that it is untenable for most things beyond research and fairly simple problems. It is a huge pain in the ass to do something like putting time bounds around reading a config file.

IO also sucks in formal verification from what I hear, and that’s like 80% of what a plane does. Read these 300 signals, do some standard math, output new signals to controls.

These things are much easier to do with tests, but tests only check for scenarios you’ve thought of already

That just seems like a theoretical possibility (even if that). I don’t see how any engineer or even someone in management could go to jail unless intent or gross negligence can be proven.

> drivers carry insurance for this purpose.

The mandatory limit is extremely low in many US states.

> expected claims

That seems like the problem. It might take a while until we reach an equilibrium of some sort.

> that bad drivers can still buy

That’s still capped by the amount of coverage + total assets held by that bad driver. In Tesl’s case there is no real limit (without legislation/established precedent). Juries/courts would likely be influenced by that fact as well.

The whole point is that is somewhat of an unreasonable expectation but it’s what Tesla expects you to do today

The liability lies with the driver, and all Tesla needs to prove is that input from the driver will override any decision made by the ADAS.

So the blame won't be on a guy who got an if statement backwards, but signing off on stopping training, failing to have certain kinds of pictures in the set, or other similar, higher order problem. Blame will be incredibly nebulous.

Depends on the root cause of the failure. Manufacturing faults would put the liability on the manufacturer; installation mistakes would put the liability on the mechanic; using them past their useful life would put the liability on the owner for not maintaining them in working order.

And the only reason the commenter I linked to had that response is because its parent comment was slightly careless in its phrasing. Probably just change “write” to “deploy” to capture the intended meaning.

Then don't write software to be used in things that are literally always critical situations, like cars.

It relies on inferred depth from a single point of view. This means that the depth/positioning info for the entire world is noisy.

From a safety critical point of view its also bollocks, because a single birdshit/smear/raindrop/oil can render the entire system inoperable. Does it degrade safely? does it fuck.

> recognizes that training exceptional deep neural networks requires vast amounts of data,

You missed good data. Recording generic driver's journeys isn't going to yield good data, especially if the people who are driving aren't very good. You need to have a bunch of decent drivers doing specific scenarios.

Moreover that data isn't easily generalisable to other sensor suites. Add another camera? yeahna, new model.

> Tesla recently held a robotaxi event, explicitly informing investors of their plans

When has Musk ever delivered on time?

> his ability to achieve results

most of those results aren't that great. Tesla isn't growing anymore, its reliant on state subsidies to be profitable. They still only ship 400k units a quarter, which is tiny compared to VW's 2.2million.

> attract top engineering and management talent is undeniable

Most of the decent computer vision people are not in tesla. Hardware wise, their factories aren't fun places to be. He's a dick to work for, capricious and vindictive.

Besides, this is what flight testing is for. Aviation certification authorities don't let an aircraft serve passengers unless you can demonstrate that all of its safety-critical systems work properly and that it performs as described.

I find it hard to believe that automotive works much differently in this regard, which is what things like crumple zone crash tests are for.

Tesla collects their own data, builds their own training clusters with both Nvidia hardware and their own custom hardware, and deploys their own custom inference hardware in the cars. There is no obstacle to them scaling up massively in all dimensions, which basically guarantees significant progress. Obviously you can disagree about whether that progress will be enough, but based on the evidence I see from using it, I think it will be.

Also, "years of death and injury" is a bald-faced lie. NHTSA would have shut down FSD a long time ago if it were happening. The statistics Tesla has released to the public are lacking, it's true, but they cannot hide things from the NHTSA. FSD has been on the road for years and a billion miles and if it was overall significantly worse than normal driving (when supervised, of course) the NHTSA would know by now.

The current investigation is about performance under specific conditions, and it's possible that improvement is possible and necessary. But overall crash rates have not reflected any significant extra danger by public use of FSD even in its primitive and flawed form of earlier this year and before.

One of these years, he is going to be right. And at that point, the fact that he was wrong for a long time won't diminish their achievement. As he likes to say, he specializes in transforming technology from "impossible" to "late".

> I'm not convinced that autonomous driving that only makes use of cameras will ever be reliably safer than human drivers.

Believing this means that you believe AIs will never match or surpass the human brain. Which I think is a much less common view today than it was a few years ago. Personally I think it is obviously wrong. And also I don't believe surpassing the human brain in every respect will be necessary to beat humans in driving safety. Unsupervised FSD will come before AGI.

And you believe that, despite experiencing what happens if you speak up?

We shouldn’t simultaneously require people to take heroic responsibility, while also leaving them high and dry if they do.

While this has a dramatic rhetorical flourish, I don’t think it’s a good proxy. Even if it was safer, it would be an unnecessarily high burden to clear. You’d be effectively writing a free insurance policy which is obviously not free.

Just look at total accidents / deaths per mile driven, it’s the obvious and standard metric for measuring car safety. (You need to be careful to not stop the clock as soon as the system disengages of course. )

You've been lied to.

As for whether it's worthwhile to solve that problem when having more sensors will always be safer, that's another issue...

Didn't Tesla have an issue a couple of years ago where pressing the brake did not disengage any throttle? i.e. if the car has a bug and puts throttle to 100% and you stand on the brake, the car should say "cut throttle to 0", but instead, you just had 100% throttle, 100% brake?

Indeed, and that goes for the person or persons who say that the products they sell are safe when used in a certain way.

I’ll give you this one.

> In my opinion, the default setting accelerates way too aggressively. I'd call myself a fairly aggressive driver and it is too aggressive for my taste

Subjective.

> It started making a left turn far too early that would have scraped the left side of the car on a sign. I had to manually intervene.

Since you intervened and don’t know what would’ve happened, subjective.

> It tried to make way too many right turns on red when it wasn't safe to. It would creep into the road, almost into the path of oncoming vehicles

Subjective.

> It would switch lanes to go faster on the highway, but then missed an exit on at least one occasion because it couldn't make it back into the right lane in time. Stupid.

Objective.

You’ve got some fair complaints but the idea that feeling safe is what’s needed remains subjective.

On the other hand, I fear/suspect it is asymptotically, rather than linearly, approaching good enough to be unsupervised. It might get halfway there, each year, forever.

While it ought to be possible to solve for just RGB… making it needlessly hard for yourself is a fun hack-day side project, not a valuable business solution.

Intention matters when passing crime judgement. If a mother causes the death of her baby due to some poor decision (say feed her something contaminated), no one proposes or tries to jail the mother, because they know the intention was the opposite.

The report measures accidents in FSD mode. Qualifiers to FSD mode: the conditions, weather, road, location, traffic all have to meet a certain quality threshold before the system will be enabled (or not disable itself). Compare Sunnyvale on a clear spring day to Pittsburgh December nights.

There's no qualifier to the "comparison": all drivers, all conditions, all weather, all roads, all location, all traffic.

It's not remotely comparable, and Tesla's data people are not that stupid, so it's willfully misleading.

This report does not include fatalities. It also doesn't consider any incident where there was not airbag deployment to be an accident. Sounds potentially reasonable until you consider:

- first gen airbag systems were primitive: collision exceeds threshold, deploy. Currently, vehicle safety systems consider duration of impact, speeds, G-forces, amount of intrusion, angle of collision, and a multitude of other factors before deciding what, if any, systems to fire (seatbelt tensioners, airbags, etc.) So hit something at 30mph with the right variables? Tesla: "this is not an accident".

- Tesla also does not consider "incident was so catastrophic that airbags COULD NOT deploy*" to be an accident, because "airbags didn't deploy". This umbrella could also include egregious, "systems failed to deploy for any reason up to and including poor assembly line quality control", as also not an accident and also "not counted".

You can't imagine that the stopping distances will be the same.

Or, they could buy sufficient buffer land around the pipeline such that the gas leak will be found and stopped before it could explode down the road

Musk in 2021: "Right now our highest priority is working on solving the problem."

But no. I want self-driving to succeed, and when it does (which I don't think is that soon, because the last 10% takes 90% of the time), I don't think Tesla or their approach will be the "winner".

I did not represent or misrepresent anything. I have asked a question to better understand their thinking.

> If you didn't intend your comment as a criticism, then you phrased it poorly.

Quite probably. I will have to meditate on it.

> Do you actually believe that your scenario happens in reality?

With railway bridges? Never. It would ring alarm bells for everyone from the fabricators to the locomotive engineer.

With software? All the time. Someone publishes some open source code, someone else at a corporation bolts the open source code into some application and now the former “toy train bridge” is a loadbearing key-component of something the original developer could never imagine nor plan for.

This is not theoretical. Very often I’m the one doing the bolting.

And to be clear: my opinion is that the liability should fall with whoever integrated the code and certified it to be fit for some safety critical purpose. As an example if you publish leftpad and i put it into a train brake controller it is my job to make sure it is doing the right thing. If the train crashes you as the author of leftpad bear no responsibility but me as the manufacturer of discount train brakes do.

You have two imperfect systems for highway driving: Autopilot with human oversight, and humans. The first has far far less death. Yet you are choosing the second.

I am not an engineer, but I have watched clearly bad decisions take place from technical perspective so that a person with title that went to their head and a bonus that is not aligned with right incentives mess things up for us. Maybe some proffesionalization of software engineering is in order.

It heated my whole house and we used a pan to cook over it. When we moved the first thing I did was install gas logs, gas stove and a gas water heater.

It’s nice to have options and backup plans. That’s one of the reasons I was a huge fan of the Chevy Volt when it first came out. I could easily take it on a long trip but still averaged 130mpg over 3 years (twice). Now I’ve got a Tesla and when there are fuel shortages it’s also really nice.

A friend of ours owns a cybertruck and was without power for 9 days, but just powered the whole house with the cybertruck. Every couple of days he’d drive to a supercharger station to recharge.

But we should look down on them and speak poorly of them same as we look down on and speak poorly of everyone else who's discourteous in public spaces.

Right. But neither of these examples are following guidelines or proper use. If I turn the car into people on the pavement, I am responsible. If the steering wheel breaks and the car does it, then the manufacturer is responsible (or the mechanic, if the steering wheel was changed). The question at hand is whose responsibility it is if the car’s software does it.

> “Using it in the way it was intended” is largely circular reasoning; of course it wasn’t intended to hurt anyone, so any usage that does hurt someone was clearly unintended.

This is puzzling. You seem to be conflating use and consequences and I am not quite sure how you read that in what I wrote. Using a device normally should not make it kill people, I guess at least we can agree on that. Therefore, if a device kills people, then it is either improper use (and the fault of the user), or a defective device, at which point it is the fault of the designer or manufacturer (or whoever did the maintenance, as the case might be, but that’s irrelevant in this case).

Each device has a manual and a bunch of regulations about its expected behaviour and standard operating procedures. There is nothing circular about it.

> Not for the price it would cost.

Ok, if you want to go full pedantic, note that I wrote “want”, not “expect”.

If you treat the car like it's a student driver or someone else driving, disengagements will go do. If you treat it like you're driving there's also something to complain about.

https://www.notebookcheck.net/Tesla-vs-Mercedes-self-driving...

I want some of these tesla bulls to PROVE that they are actually "not intervening". I think the one's who claim they aren't doing things for hours are liars.

Sensor platforms deployed at scale, that you have the right to take data from, are difficult to replicate.