How a Tesla is like a PlayStation

AI and the return of "next-gen software"

Earlier this year my car learned how to drive itself.

Dropped off at our home in May 2022, it was equipped with a “Full Self-Driving”-ready computer and suite of cameras, as has come standard on Tesla’s cars for a number of years now.

But even as an ardent early adopter of new tech, we hadn’t opted to pay thousands of dollars to get access to the beta version of the actual “Full Self-Driving” capability. Keeping an eye on clips on Reddit from those who would post how the system handled driving in their neighborhood made it clear that the value was not there vs the fairly bulletproof “Autopilot” that would keep the car centered in its lane at a set distance from other vehicles.

Even as a Tesla enthusiast, the path forward for FSD seemed fraught. Yes, they could collect video clips from their fleet of drivers to iron out odd behavior or poorly handled situations, but they needed engineers in the loop defining how the vehicle should behave in those situations. It meant playing whack-a-mole with every edge case, with an increasingly complex set of rules for the system to try to follow.

Then, with version 12 of the FSD software, Tesla changed their approach. They pivoted to a true end-to-end machine learning approach, going from “pixels to motor controls.” Wiping away hundreds of thousands of lines of code defining how a car should act, they trained a model on footage from its customers’ journeys, and voila, proper driving behavior in a wide range of situations emerged from the data.

By version 12.3, Tesla was clearly feeling very confident about this new approach. In March 2024, they offered a fleet-wide free trial of this new software, rebranded from Full Self-Driving (Beta) to Full Self-Driving (Supervised).

It was very impressive. While definitely deserving of the Supervised note, the system was surprisingly smooth in a wide range of circumstances. While occasionally jerky or indecisive when a lane split, it genuinely felt like a step up from Autopilot as a convenience feature that let you experience driving from a slightly more removed perspective.

At this point, Tesla began offering a subscription to FSD at a reasonable price vs the thousands of dollars it had previously cost, so we kept it on. In the months since, it has had some progress and regressions in overall driving capability. It’s better at picking a lane; some updates have worse “phantom braking” problems than others. It’s also become more convenient: rather than periodically nudging the steering wheel to communicate to the vehicle that you’re watching the road, it uses the car’s interior camera to watch the driver’s gaze.

Compute Rules Everything Around Me

Alas, my 2022 Tesla, with its third-generation Full Self-Driving computer from 2019, will likely not get another leap forward in capability.

After unveiling its CyberCab at an event in Los Angeles, Tesla shared their plans for upcoming software releases:

For context, “AI3” and “AI4” are the newer names for the third- and fourth-generation Full Self-Driving computers, previously known as “HW3” and “HW4.”

That in mind, the new v13 of Full Self-Driving seems like a fantastic update. While AI4 has been shipping in new Teslas since 2023, it’s been working with a hand-tied behind its back, processing fewer pixels at lower resolutions than the system ought to be able to handle so that the software running on AI3 and AI4 could be as similar as possible.

The above post indicates we’re reaching the end of that approach. Sure, AI3 vehicles will get some capability gains and better interfaces over time for using FSD, but the bitter lesson from AI scaling generally is that way more data and compute will lead to more emergent capability. If it’s going to get good enough to drive people without the option to Supervise and even take over with a steering wheel, they have to move on from the limits of AI3’s compute and work on bigger models.

“Start Thinking PlayStation, Blow S### Up!”

This whole experience oddly reminds me of being a buyer of game consoles across their generations of hardware.

In the 80s and 90s, chips were progressing so fast that game consoles released half-a-decade apart would be 3-10x as powerful across key performance metrics, leading to obvious leaps forward in graphical capabilities, like the jump from 2D to 3D:

Game developers needed to learn to skills and build new technology to keep up with this progression in underlying capabilities. The effort needed to actually take advantage of the extra horsepower, which often came from a chip with a completely different architecture than the previous generation console, can be seen in the difference between games released early in the PlayStation 2’s lifespan vs later:

This dynamic meant there was a golden period to each console’s lifespan.

After a few years on the market, games were built to take advantage of that generation’s unique hardware and looked fantastic. But if you waited too long, you’d only have that great accumulated back catalogue, as developers would have moved on to figuring out how to take advantage of powerful new next-gen hardware and the marketing allure of even more impressive graphics.

For the most part, we no longer have to think this way about games or most other kinds of software. Instead, things are written to run on a range of systems, and users can expect varying levels of fidelity and performance based on how fast of a computer they get. This cycle of game consoles, the “last-gen” PS4 and Xbox One haven’t quite faded like their predecessors: instead, they’re the boxes that play modern games on Low settings with some occasional visible drops in performance, like a budget laptop struggling with too many Chrome tabs. A new app might load slowly on an old iPhone, but it will still be the same fundamental experience.

But this v12 to v13 Tesla FSD generational hand-off feels like “the old days.”

Nearly 5 years into the AI3 computer’s lifespan, Tesla got really good at making the software for it. Capable and optimized, it was the very best time to have the underlying hardware. But now its developers have to focus on what they can make the bigger, badder AI4 computer do. And while maybe some of the work can trickle down, you can’t bring 3D Mario to the Super Nintendo.

And we’re seeing this dynamic anywhere companies are trying to sell AI that runs locally on a user’s device. Apple spent years acting as if 8 GB of RAM was sufficient as their entry-level offering on its laptops, finally relenting this fall with its Apple Intelligence rollout and marketing campaign. Similarly constrained on phones, Apple Intelligence is only available on iPhones from the last two years, and only the Pro models from last year at that. In the Windows ecosystem, new laptops need dedicated AI compute or a decent GPU to get the new “Copilot PC” moniker.

So far, local AI doesn’t feel like that strong of a driver of device upgrades vs more straightforward improvements to cameras on phones and battery life (via the introduction of decent ARM chips) on laptops. Can the work being done to improve the smaller models that run on a consumer’s device keep pace with the giant models running in data centers such that software running on a newer phone feels “next-gen” compared to last year’s? ¯\_(ツ)_/¯

I feel much more confident anticipating my next cycle of car purchases. We’ll hold onto the 2022 Model Y as long as I can muster despite my early-adopter inclination, and then get a late-model used Tesla with AI4 once AI5 is rolling out to the new fleet more broadly, hopefully getting the most out of that generation’s “golden period” before Tesla again has to shift its focus to what can be done with much more compute and bigger models.