Rivian says “fat finger” caused software update to brick infotainment systems, physical servicing may be required::Today’s cars are more like computers on wheels, and even a seemingly routine software update can lead to unexpected consequences. Rivian unfortunately experienced a “fat finger” mishap with their latest software update, bricking infotainment systems […]
I don’t have to update my car in order to drive it. Sucks to be them
I thought the Rivian vehicles were still drivable? It was just the infotainment that’s impacted, no?
Imagine paying $90k for a vehicle and you can’t even turn on the radio.
That’s always going to be a problem for early adopters of new tech. Plus, since it’s automotive you’d run that risk with the first model year or two after a major redesign too.
If what Rivian has reported is true it’s not really a failing of the equipment but human error. That’s always going to be tough to account for, though there’s likely improvements to be made to their testing methodology before pushing to prod.
Congrats on having a nicer experience 2 out of the 365 days in a year. Bravo.
Ok man what if it’s an emergency. Here’s an example: My wife is in labour, but my car has to update so I can’t drive her to the hospital. Now I’ll have to pay a more than 2 grand ambulance ride since it was out of network.
Fortunately, you can still do that with a Rivian. From the article that no one read:
Speedometer, charging, backup cameras, locks, lights, wipers, and turn signals are all still functional with the 2023.42 error.
At least there’s a speedometer on a separate display.
On the Tesla Model Y there’s only one centre display for everything. I was on a road trip with someone in one and the display froze then automatically rebooted itself. Took a good 5 minutes.
Nothing happened thankfully and the car kept running, but it’s a scary experience when you’re driving 65-70 MPH on a highway. I was worried the car would refuse to continue driving and lock itself or something while it rebooted.
That’s not really a problem outside of USA.
hahah nothing says america like absolutely obscene healthcare costs… cept maybe all the human killing devices they love so very very much
Considering I have a decent chance of getting a “Please wait – updating” message when trying to play a damn videogame nowadays, that’s not out of the realm of possibility.
You are inconsistent. If you are going to adopt a luddite way of life, have her deliver at home not at the fancy high tech hospital where the computerized anesthesia machine might need to update before the c section incision is made.
At least with a tesla you have to manually confirm the update, and can cancel out of it for 2 minutes after doing so. If something happens between then and 15 minutes later then i guess you’re fucked, but with the amount of control you have this really should never be a problem.
If you were going to drive you’d get a taxi instead. I agree but just saying.
most taxis won’t cover medical emergencies
That’s fucked up.
I would think they would keep at least 1 of each model/trim of vehicle for testing these things. This leads me to believe one of the following:
- Too tight of deadlines
- Cheap management won’t pay for testing time or units
- Culture of pencil whipping
- A bad apple didn’t do their job, which should be caught by procedures
I got an offer for software engineering role at Rivian a few years ago. The pay was low.
I think it’s more likely just typical dysfunctional software industry workflows. Companies that actually test their software adequately before deployment are the exception, not the norm. That’s different from what you said in your second bullet point because it’s not even an issue of cheapness, it’s an issue of not actually understanding what the best practices need to be.
I mean… those “typical dysfunctions” are what OP described. You’re just describing the general state of the industry, not providing further examples of disfunction.
No, the parent commenter attributed it to management not wanting to spend money. I’m attributing it to management being incompetent instead, which isn’t the same thing. Spending even unlimited amounts of money is not sufficient to fix what’s wrong.
¿Porque no los dos?
All of the above.
OTA updates are very much a double-edged sword.
Usually it’s good, but sometimes you get crap like this or when Tesla downgraded a bunch of Model S batteries when they saw they weren’t holding up as well as expected and wouldn’t line up with their battery warranty.
I’m a tech enthusiast, but I increasingly am slowing my roll on all updates but security updates. The era of heavily QAing feature updates is ending, which is leading to some frankly annoying issues across the board, from car software, to phone and computer software. Even GPUs don’t seem to be immune from occasional bugs in the latest updates.
Companies and orgs really need to dedicate more time to making sure updates are vetted and clean before releasing them. It might cost you some money up front, but it saves you PR, security issues, and hotfixes in the longer run.
in the longer run.
I have yet to work for a company where management cares about this. It’s always about what can be sold next week if we rush it enough. Or, more commonly, what was sold weeks ago without any consultation with the technical team.
It’s a problem everywhere. Rush to build something, then don’t actually finish it, or iron out the bugs before moving on to the next sexy thing. Management wonders why everyone is on the super ancient, rocksolid platform and hasn’t really taken the bait on the series of products that have an “agile” development cycle and were dropped after about 5 years.
Guys…my entire team knows the issue. Support your damn products for years. Commit to improvements for a decade, and help people transition to a new product when the old one has to be sunsetted for technological or knowledge worker retirement reasons.
I have a Sync 3 system that filled its self up with files and no longer has free memory. Its a software bug that was fixed, but I’d have to pay to replace the entire unit rather than snag a utility to clear space and upgrade. That is soooo dumb.
But an unaffiliated community created a fix before Ford. Thanks hackers!
Weird. My primitive old Samurai started right up this morning, and it’s “infotainment” system (bluetooth boombox bungee-corded to the floor behind the seats) still works.
Maybe if I’d paid $90,000 for it I could be privileged enough to have a broken functionality.
How are you replying from an i386 based pc running windows 3.11?
Slowly
I think while the topic is up it’ll be fun mentioning that the Colorado/Canyon does not have a physical headlight control anymore - in favor of defaulting to Auto and touchscreen controls and the project lead(?) claimed that the system was 100% bug-free.
Also later there was a bug with some OTA update for that model that’ll kill the battery.
Anyways, I bought a 4Runner and immediately called to disable its cellular radio. (dubbed DCM in Toyota-land)
How do you disable the DCM?
There have been some experimentation but the best (IMO) option is hitting the SOS button to call and navigating the phone menu to get to a representative, not emergency services and having them disconnect your car. You may need your VIN, you’ll need to confirm that you do not want connected services and it may take a day or so to take effect. Now, my SOS button doesn’t have the green light and while the radio - according to the infotainment - is still powered, it is no longer connected to the network.
Another way if you don’t plan on using the microphone (like for calls) you can pull the DCM fuse but I prefer the above option.
How though
“According to Rivian the glitch occurred when the wrong build with incorrect security certificates was inadvertently sent out. This error resulted in bricking the infotainment systems of an unknown number of vehicles.”
I always thought that ‘fat finger’ revered to having fingers that are too big for buttons, but I guess it means sending out wrong software versions.
Fat finger in this context typically means that the person knew what they wanted to do, but typed the wrong thing. Their “fat finger” hit the wrong button. Possible that they wanted certificate 123 in the build, but they accidentally typed 133, or something like that.
This. And the guy who did it was fat.
I think it’s also worth noting that “security certificates” in this context are almost certainly there for the purpose of locking the system away from modification by the vehicle owner, which means they weren’t necessary (and indeed, ought to be prohibited by law) to begin with.
Rivian was hoist on their own consumer-hostile petard.
I don’t think so in this context. This is probably more like SSL cert trusting or some private/public key pair.
Based on the fact that the screen is blank, I think @grue was correct. The certificate that tells the system that the infotainment was valid was incorrect, so the infotainment couldn’t load. They used the dev-infotainment cert instead of the prod cert.
They did
Rm -rf /
To force remove everything recursively instead of
Rm -fr /
To remove the French language pack. Rookie mistake really.
Dumb-ass managers who thought they could run their embedded software group all Agile and shit, and push stuff out to Production without full testing to meet deadlines, not realizing that they can’t just “roll back” this one.
They had 2 files in one directory. One said “good update” the other said “bad update-don’t use”. When they wanted to push out the update they used a tablet and tapped on “good update” but they accidentally touched the wrong one a bit more with the thumb.
Something like this. But maybe they don’t want to say wat really happened.
Between the massive privacy issues and cars getting bricked by automatic OTA updates, I currently see zero reason whatsoever to buy a car made after ~2015.
I just recently upgraded to a 2013 Rav4 and it has all the tech I could ever want. I’m gonna keep it as long as I can.
