Change `.to("cuda")` to `.to("tpu")` and It Runs: Google Open-Sources TorchTPU as the Anti-Nvidia Alliance Takes Shape

At PyTorch Conference, Google Dropped an "Anti-Nvidia" Bomb

On October 20, 2026, on the stage of PyTorch Conference 2026, Mark Lohmeyer (VP & GM of AI Infrastructure at Google Cloud) and Bill Jia (VP of Engineering) jointly announced that TorchTPU is now open source under the Apache 2.0 license. It is a project Google has been co-developing with Meta's PyTorch team since mid-2025 — a backend that runs PyTorch natively on Google TPUs.

The official language was restrained: "TPUs join GPUs and CPUs as a native option for PyTorch." But anyone paying attention could smell the gunpowder: roughly 85% of AI apps in the PyTorch ecosystem are written, by default, to run on Nvidia GPUs. What Google is going after is the deepest part of Nvidia's moat — not the silicon, but the software layer.

The timing is even more telling: just one day earlier (October 19), AWS released TorchNeuron — letting PyTorch apps move to its homegrown Trainium chips with no rewrite — and submitted code to the main PyTorch project decoupling features from Nvidia's chips. Google and Amazon fired at the same target in the same week, with nearly identical tactics. That is not a coincidence. That is a coordinated hunt.

TorchTPU in One Sentence: Change .to("cuda") to .to("tpu") and You're Done

TorchTPU's core promise is absurdly simple: your existing PyTorch code runs on TPUs with barely any changes. The official migration example is literally swapping torch.device("cuda") for torch.device("tpu"), and backend="nccl" for backend="tpu_dist" in distributed training. Google even recommends going further — just use torch.accelerator.current_accelerator() and let the same code run on GPUs and TPUs alike.

Technically, TorchTPU goes through PyTorch's official PrivateUse1 backend mechanism — the "front door" PyTorch reserves for third-party hardware vendors. Tensors are ordinary PyTorch tensors: no subclasses, no wrappers. This is a completely different route from Google's earlier PyTorch/XLA.

And the single most important design decision of this route is four words: eager-first.

Why Eager-First Is the Soul of the Whole Thing

To understand TorchTPU's value, you first need to understand what it is here to bury: PyTorch/XLA. In 2020, Google shipped the first generation of "PyTorch on TPU," built on the Lazy Tensor mechanism. The way it worked: record your ops into one big graph first, then compile and execute it all at once. It sounded elegant. In practice, it was a three-part disaster:

First, it required intrusive code changes. You had to manually mark which code regions were "compilable" — framework compiler internals, smeared across every developer's face. Second, compile latency was unpredictable — "mystery recompiles" were a daily complaint in the TPU developer community. Third, there was no real eager mode — and eager execution (write it, run it, debug line by line) was exactly the killer feature PyTorch used to steal developers from TensorFlow.

More fatal was the pace problem: the PyTorch community evolves extremely fast, and PyTorch/XLA was perpetually half a step behind, with emerging distributed primitives (like DTensor) missing for long stretches. A backend that can't keep up with the community gets abandoned by the community.

TorchTPU's design philosophy is a total rebellion against that dark history. Its three iron rules, published on the PyTorch blog: zero JAX dependencies in the runtime, solid eager mode, and compilation via TorchDynamo (not Lazy Tensor) with performance comparable to JAX compile. Eager mode even comes in three flavors — Strict, Debug, and DeferAndFuse: debug line-by-line in eager, then hand the same model to torch.compile(backend="tpu") for production, lowering through TorchDynamo → StableHLO → XLA.

On the numbers: roughly 450 Aten ops are supported (forward and backward for all core ops needed by major architectures), and out-of-box execution has been validated on the 5,000 most-downloaded Hugging Face models — the full zoo of Qwen, DeepSeek, Gemma, Llama, Mistral, Whisper, DinoV2, BERT, T5, and more. Custom kernels can be authored via JAX/Pallas or the Helion DSL.

Google's Decade-Long Detour: TPUs Finally Bow to PyTorch

There is a historical angle worth savoring here. TPUs are nearly a decade old, and for most of that time they lived in the TensorFlow/JAX world. It is not that Google didn't want PyTorch developers — PyTorch/XLA in 2020 was exactly such an attempt. But the posture was wrong: it asked PyTorch developers to learn to think like JAX (record graphs first, compile later, mark regions by hand) instead of teaching TPUs to think like PyTorch (eager-first, write-and-run).

The outcome is well known: PyTorch won the developer community (today ~85% of AI apps are built with it), while TPUs remained second-class citizens in the PyTorch world. Google trains Gemini on TPUs internally, of course — but external developers who wanted their own PyTorch models on TPUs had to survive PyTorch/XLA first. Many tried once and went back to GPUs.

TorchTPU is a 180-degree turn. The first design principle listed on the PyTorch blog is "PyTorch Citizenship": no JAX dependencies in the runtime. That sentence is addressed to every developer ever burned by PyTorch/XLA: this time, the TPU is here to be a good PyTorch citizen, not to proselytize.

Several "people-pleasing" technical details show real craft. Take DeferAndFuse, the new eager mode: ops don't have to execute immediately — they can be deferred to an automatically determined "materialization point" (e.g., a .to("cpu") statement) and then aggregated and compiled together. It finds a middle ground between eager's debugging experience and compiled execution efficiency. Or bounded dynamism: the XLA compiler wants shapes fixed at compile time, which fights PyTorch's dynamic shapes, so TorchTPU allows "bounds fixed at compile time, actual values at runtime," cutting down recompilation churn. Then there is precision mapping: TPU MXUs natively speak bfloat16, so TorchTPU defaults to medium precision (BF16 multiply + FP32 accumulate) and makes you opt in explicitly for higher precision — telling developers the hardware's temperament honestly instead of hiding it.

These details say one thing: Google actually "read" the PyTorch community this time, instead of just translating the TPU manual.

Three Shots in One Week: AWS, Google, and DeepSeek

Zoom out on the timeline and October 19–20, 2026 looks like an "anti-CUDA week":

Shot one came from AWS (Oct 19): TorchNeuron, moving PyTorch apps to homegrown Trainium chips with no rewrite. More ruthlessly, AWS simultaneously submitted code to the main PyTorch project to decouple a batch of Nvidia-entangled features — cutting at the root. In the future, PyTorch would treat all hardware equally by default, with Nvidia as just one option.

Shot two came from Google (Oct 20): TorchTPU plus the full vLLM/SGLang/llm-d/TorchTitan/VeRL/Ray/HF family. Google's play differs slightly from AWS's: AWS is betting on its own cloud (Trainium lives only on AWS), while Google brought in Meta and Hugging Face — one holding framework mindshare, the other model distribution. Bigger coalition, louder statement.

Shot three came from DeepSeek (Oct 20): the same day, DeepSeek open-sourced its serving stack. Note the signal: even the lab most famous for squeezing GPUs to the extreme is now open-sourcing its inference know-how. When the players who understand inference optimization best all go open, the returns on closed optimization keep shrinking.

All three shots land in the same place: turning "switching chips" from an engineering epic into a config change. Nvidia hasn't responded yet, but expect "CUDA ecosystem stickiness" to morph from a moat into a risk item questioned on earnings calls over the next year.

Not Just a Backend: Google Brought the Whole Family

If this were only an open-sourced backend, it would be "yet another hardware adapter." Google's real bet is the ecosystem slot — it plugged TPUs into the entire toolchain PyTorch developers use daily, in one shot:

Inference: with Inferact (the company behind vLLM), a native PyTorch vLLM-TorchTPU backend (continuous batching, quantization, multi-chip parallelism included); with RadixArk (behind SGLang), SGLang-TorchTPU (RadixAttention KV-cache reuse, prefill/decode disaggregation included). These two are the de facto standards for LLM serving today — winning them means winning production.

Fleet orchestration: llm-d 1.0 released with Red Hat, IBM, and 500+ contributors (a CNCF Sandbox project), making prefix-cache-aware request routing a Kubernetes-native solution. On GKE, the same endpoint can pick GPUs or TPUs per workload — the subtext being: ops teams can switch hardware without replacing their whole scheduling stack.

Training: TorchTitan (the PyTorch community's reference implementation for distributed training) got a production-grade integration — official numbers claim 52% MFU on Qwen 3 8B, scaling near-linearly to 8,192 Ironwood chips (TPU v7x). The popular RL framework VeRL upstreamed a TPU backend, and Ray made TPUs a first-class accelerator — CPUs, GPUs, and TPUs orchestrated in a single workflow.

Out-of-the-box: in Hugging Face Transformers v5, deploying to TPUs is now as simple as passing device_map="tpu"; Diffusers runs natively with eager mode, torch.compile, and tensor parallelism; Google also contributed LiteRT (formerly TensorFlow Lite) to the PyTorch ecosystem — one litert-torch.convert() call sends your model to phones and embedded devices.

And one more intriguing item: the TPU agent suite (in preview on GitHub) — packaged as plugins for coding agents, automating TPU onboarding, error diagnostics, and inference tuning. Even "learning TPUs" is something Google wants AI agents to do for you now.

Why Meta Is the Loudest Cheerleader

The most forceful testimony of the whole launch came from Jana van Greunen, who runs PyTorch engineering at Meta: "Advancing the frontier of AI requires immense compute scale and deep hardware diversity. We cannot be constrained by a single hardware architecture. TorchTPU is a massive step forward."

That carries a lot of information. Meta invented PyTorch (2016, donated to the Linux Foundation in 2022) and is one of the world's largest GPU buyers. Publicly backing Google's TPUs is, at its core, a negotiating move against Nvidia: when your training framework natively supports a second chip vendor, your posture at the procurement table changes completely. Meta doesn't want to "replace Nvidia" — it wants to "always have a choice."

Jeff Boudier (VP of Product at Hugging Face) spoke from a different angle: "Hardware diversity is essential to making open-source AI accessible to everyone." Translation: when only one vendor's chips can smoothly run open models, the open-source ecosystem's throat is in one company's hands.

So TorchTPU's real narrative isn't "how great Google's TPUs are" — it is "the united front against lock-in": Google brings chips and engineering, Meta brings framework mindshare, Hugging Face brings model distribution, vLLM/SGLang bring the inference stack — each adding a brick to the "no more Nvidia hostages" wall.

The Truth About the CUDA Moat: It Was Never Just Silicon

There is a layer here many people haven't thought through. Nvidia's moat has been discussed for a decade, and most assume it is "stronger GPU compute." But TorchTPU proves the opposite: CUDA's real barrier was that the software layer made leaving expensive.

Your training code says .to("cuda") by default. Your serving engines' (vLLM/SGLang) optimized paths default to CUDA kernels. Your tutorials, your CI, your performance-tuning instincts are all CUDA-shaped. Switching chips meant rewriting — not the model, but the entire engineering apparatus around it. That migration cost is the moat's true body.

And what Google and AWS did this week is systematically dismantling that wall: making frameworks (PyTorch) and serving engines (vLLM/SGLang) treat non-Nvidia accelerators as native options. When the cost of leaving drops from "rewrite" to "change one line of config," Nvidia's pricing power starts to loosen. Note this doesn't mean developers will abandon Nvidia overnight — Google itself admits that after migration, model shapes and sharding strategies still need retuning for TPUs to hit peak performance. But between "it runs" and "it runs fast" lies bargaining power.

Why the Inference Side Is the Real Battlefield

A close read of Google's launch list reveals a tilt: inference-side investments (vLLM, SGLang, llm-d) outweigh training-side ones. That's no accident — as reasoning and agentic models go mainstream, production's primary bottleneck has shifted from "can we train it" to "can we afford to serve it." A single agent task can invoke the model dozens of times; inference cost decides whether such products live or die.

Google cited a real case in its blog: Zenteiq used the GKE Inference Gateway (llm-d powered) on TPU v6e for synthetic data generation, and with load-aware, prefix-cache-aware routing pushed daily token generation on 1,000 TPU v6e chips from 1.47 billion to 5.97 billion — over 4× throughput, achieved not with stronger chips but with smarter scheduling. This is the most grounded part of the TPU narrative: not peak FLOPS, but throughput per dollar.

For vibe-coding and indie developers, the trend means this: over the next two to three years, the inference API price war will intensify under the cover of "hardware diversity." If your app's architecture isn't tied to a specific backend, you'll catch every price-drop wave first. Write "cuda" into every line of code, and you've voluntarily surrendered your right to comparison-shop.

Cooling Down: Four Honest Caveats

Having written all that, four buckets of cold water are in order — all facts findable in the official materials, none emphasized in the launch:

First, "runs out of the box" ≠ "runs fast out of the box." Google admits on the PyTorch blog that just changing .to("tpu") gets models running, but not peak performance, because model shapes, sharding, and op choices may be tuned for GPU architectures. A TPU TensorCore is a single-threaded giant matrix-multiply unit (MXU) that needs sufficiently wide shapes to be "fed" — the opposite of the GPU intuition of launching many small kernels in parallel. Migration is the start; tuning is the main course.

Second, the 52% MFU is self-reported. 52% Model FLOPs Utilization on Qwen 3 8B across 8,192 Ironwood chips sounds beautiful, but it's Google's own model, own hardware, own test conditions. Until independently reproduced, it's a reference point, not a promise.

Third, the version is very early. PyPI hosts torch_tpu==0.2.0, and the README says "active development." A 0.2.0 claiming production readiness takes courage — but production potholes (e.g., each TPU device accessible by only one user-space process, so RL colocation isn't supported out of the box) will need the community to step through one by one.

Fourth, TPUs can still only be bought from Google. This is the most fundamental difference from CUDA: Nvidia GPUs can be bought from any cloud or reseller, while TPUs live only inside Google Cloud. Today's "hardware diversity" still means "getting quotes from two cloud vendors instead of one chip monopoly." True openness arrives the day TPUs can be bought and sold as freely as GPUs — and that day is one Google probably doesn't want to come.

A Note for Developers in China

One honest word: TPUs live inside Google Cloud, so developers in China are unlikely to touch TorchTPU directly anytime soon. But it still matters, for three reasons:

First, a direct win for going-global businesses. If your inference services run on Google Cloud, mature vLLM-TorchTPU will likely make TPU instances cheaper than equivalent GPU instances for the same throughput — inference cost is among the largest line items for AI apps going global, and 20–30% off is real profit.

Second, the methodology is replicable. TorchTPU proves a path: for a third-party chip to tap the PyTorch ecosystem, you don't need to rebuild the framework — you need three things done right: a PrivateUse1 backend, an eager-grade experience, and the mainstream serving engines. Chinese AI chipmakers face exactly the same exam — whoever ships the "one config line to run" experience first eats the PyTorch ecosystem spillover.

Third, a mindset point: stop treating "CUDA-only" as a given. Spend 30 extra seconds on new code using torch.accelerator.current_accelerator() instead of a hardcoded "cuda" — one day, that line may save you a hefty migration bill.

What Developers Should Do Now

If you're an AI app developer, the practical implications split into three horizons:

Short term (now): if you already run inference on Google Cloud, vLLM-TorchTPU and SGLang-TorchTPU deserve a POC. The TPU value story has always been "cheaper for the same throughput" — now that serving-stack friction is gone, the savings could be real money. Transformers v5's device_map="tpu" brings experiment cost down to near zero.

Medium term (6–12 months): in new projects, replace hardcoded "cuda" with torch.accelerator.current_accelerator(). It's one line of insurance — for your future freedom to comparison-shop across three chip vendors. Google and AWS already wrote the standard answer this week: stop hardcoding hardware names into code.

Long term: watch the "de-Nvidia-ization" of PyTorch's main branch. AWS has already submitted decoupling code upstream. When the framework treats all hardware equally by default, "Nvidia-first" downgrades from an architectural assumption to an option. That day, the AI infrastructure cost curve may genuinely bend.

One last honest word: TorchTPU's success won't depend on how strong Google's TPUs are, but on how many developers are willing to spend time on a "second option." Open-sourcing is the starting gun; the ecosystem is the marathon. The good news: when the gun fired this time, Meta, Hugging Face, vLLM, and SGLang were already on the track.

And more worth remembering than TorchTPU itself is the new common sense established this week: AI infrastructure competition has moved from "whose chip is faster" to "whose software layer is more open." Nvidia won the last decade on the CUDA ecosystem; the next decade's winner may be whoever makes developers feel equally at home wherever they run. Google bet on it with TorchTPU, AWS bet the same way with TorchNeuron — two cloud giants rarely agree, and that consensus speaks volumes.

For ordinary developers, the best strategy was never picking sides — it is staying portable. Chips get refreshed, clouds brawl, prices fluctuate; the only constant is that assumptions hardcoded today become tomorrow's tech debt. Starting today, every hardcoded "cuda" you don't write is a lifeline left for your future self.