What is a GPU used for in AI? Short answer: GPUs do the maths. Artificial intelligence is, at its core, an enormous number of simple calculations happening in parallel, and graphics processing units (GPUs) are built to do exactly that. They are why your phone can answer questions, why ChatGPT responds in seconds, and why data centres full of servers exist at all. This guide explains the chips behind AI in plain English — GPUs, TPUs, NPUs, VRAM and the supply chain that makes them — with real numbers from the research.
What Is a GPU Used for in AI?#
A GPU is a chip designed for parallel processing: it breaks a big problem into thousands of smaller pieces and solves them at the same time. In AI, both training (building a model) and inference (running it) are giant parallel-maths jobs, so GPUs do both. Data-centre GPUs like NVIDIA's H100 sit in racks of thousands; consumer GPUs like the RTX 4090 do the same work on your desk. The machines that house them — the buildings we covered in what is an AI data center — exist mainly to keep these chips powered and cool.
Why AI Needs Parallel Maths (and Why CPUs Are Bad at It)#
Your computer's central processing unit (CPU) is brilliant at doing one task quickly, in order. AI models do the opposite: every answer requires millions of nearly identical calculations — multiply, add, compare — repeated across billions of parameters. A CPU does these one at a time and stalls; a GPU does thousands at once. That is why AI needs GPUs. The trade-off is real: GPUs consume roughly 10–15 times more power per processing cycle than CPUs, which is why AI data centres need gigawatts and why chip efficiency is such a big deal.
GPU vs TPU vs NPU vs AI Accelerator#
The four names describe where a chip lives and what it was narrowed for, not four unrelated technologies. A GPU is the general-purpose parallel processor that runs most AI, a TPU is Google's in-house variant built for tensor maths, and an NPU is the low-power version sitting inside laptops and phones for on-device work.
| Chip type | What it is | What it's for | Examples |
|---|---|---|---|
| GPU (graphics processing unit) | General-purpose parallel processor | Training and running most AI models | NVIDIA H100, RTX 4090, AMD cards |
| TPU (tensor processing unit) | Custom-built to speed up tensor maths | AI in Google's data centres | Google TPUs |
| NPU (neural processing unit) | Low-power AI accelerator inside devices | On-device AI on laptops and phones | Copilot+ PC NPUs, Snapdragon AI Hub |
| AI accelerator / ASIC | Any chip specialised for AI workloads | Custom clouds and specific tasks | AWS Trainium |
Table: GPU vs TPU vs NPU vs AI accelerator, from the IBM, Cisco, Fortinet, Lenovo and Spiceworks materials.
The umbrella term is "AI chip" — anything specialised for AI workloads. GPUs are the workhorses because they are general enough to train and run almost any model; TPUs and NPUs trade that generality for speed or efficiency in specific places.
What Is VRAM and Why It's the Number That Matters#
VRAM is the memory on the graphics card, and for AI it decides which models will run at all, because a model's weights have to fit in it while it works. When they do not fit, the software spills over into slower system memory and speed drops sharply. The hardware guide sets out how much VRAM each model size needs, and the VRAM calculator works it out for a specific card and model.
Why the Same Chip Trains and Serves Models#
Training and inference are different jobs, but they run on the same kind of hardware. Training is the one-time "mountain climb": thousands of GPUs crunch data for weeks to build the model. Inference is the "daily commute": every prompt you send runs the finished model on GPUs again, over and over. AI models train and run on data-centre GPUs (IBM's explainer says it directly), and the accelerator lifecycle is the same in cloud and local setups. The workloads are so similar that the same H100s that trained a frontier model go on to serve millions of queries — and because inference never stops, it now uses an estimated 80–90% of AI's computing power.
AI Chip Performance: A Few Real Numbers#
The spread between the top and the bottom of this list is more than two orders of magnitude. An RTX 4090 is rated at 1,321 TOPS, a Jetson Orin Nano manages 67 TOPS on about 15 watts, and a Raspberry Pi 5 does roughly 2, so the right chip depends far more on the job than on the headline number.
| Chip / device | What it is | Key number | Source |
|---|---|---|---|
| NVIDIA H100 | Data-centre GPU | 700 W rated; up to 989 trillion FLOP/s | Epoch AI |
| NVIDIA A100 | Previous-generation GPU | 800 W max | Epoch AI |
| RTX 4090 | Consumer desktop GPU | 1,321 TOPS | Top AI Hardware 2026 |
| Mac Mini M4 Pro | Apple unified-memory machine | ~67 TOPS, 24 GB | Top AI Hardware 2026 |
| Jetson Orin Nano | Edge AI board | 67 TOPS at ~15 W | Top AI Hardware 2026 |
| Raspberry Pi 5 | Budget board | ~2 TOPS | Top AI Hardware 2026 |
| DGX Spark (GB10) | NVIDIA desktop appliance | 128 GB unified; "1 PFLOP FP4" is positioning, not measured LLM speed | kingy.ai guide |
Table: real chip numbers from the supplied materials. TOPS and FLOPs measure peak capability, not everyday AI speed — see the hardware guide for real-world tokens per second.
The Chip Supply Chain in One Section#
A finished AI chip carries costs that never appear on its spec sheet. Design happens largely in the US, fabrication is over 90% concentrated in Taiwan, and a single fab uses around 10 million gallons of ultrapure water a day, which is why one chip has already consumed thousands of gallons before it reaches a data centre.
| Stage | Fact | Source |
|---|---|---|
| Design | NVIDIA, AMD, Google and Apple design the chips | IBM |
| Fabrication | Taiwan produces over 90% of the world's advanced semiconductors; TSMC used 24 billion kWh in 2023 (~2.7 GW average) | Chatham House; Epoch AI |
| Water | A chip fab uses ~10 million gallons of ultrapure water a day; ~1,500 gallons of piped water per 1,000 gallons of UPW | EESI |
| Water (per chip) | A single chip has consumed thousands of gallons of water by the time it reaches a data centre | EESI |
| Recycling | Wafer plants in Singapore recycle ~45% of water; semiconductor plants ~23% | "Making AI Less Thirsty" |
| Supply-chain share | Apple reports its supply chain accounts for 99% of its total water footprint | "Making AI Less Thirsty" |
Table: the chip supply chain's hidden costs. This is why the environmental footprint of AI chips extends far beyond electricity — our [AI water use explainer](https://learnaitodayonline.com/how-much-water-does-ai-use/) covers the full picture.
What's in Your Computer Right Now#
Most computers already contain some AI hardware. Discrete GPUs (like the RTX series) have VRAM you can check in the system settings; integrated graphics handle small models on CPU; newer Copilot+ PCs include NPUs designed for on-device AI, and Microsoft's Phi-class models run offline on them. Apple Silicon uses unified memory, so the GPU can access all system RAM — the reason a 64 GB Mac can run models that would not fit on a 24 GB graphics card.
If you have a laptop with 16 GB of VRAM (like an RTX 5080-class machine), you already have a genuinely capable local-AI machine; if you are buying used, the community consensus is that a second-hand RTX 3090 with 24 GB is the best value VRAM per dollar.
What You Need for Local Models#
Running models at home needs a graphics card with enough VRAM for the models you want, enough system RAM and fast storage. The guide to the hardware for running AI models locally explains how to size each one, and the VRAM calculator checks a card against a model before you buy anything.
Sources#
- What is an AI data center? (IBM, 2025)
- What is an AI data center? (Cisco)
- What is an AI data center? Key capabilities (Fortinet)
- How much energy does ChatGPT use? (Epoch AI, 2025)
- We did the math on AI's energy footprint (MIT Technology Review, 2025)
- Data Centers and Water Consumption (EESI, 2025)
- Making AI Less "Thirsty" (arXiv, 2025)
Last reviewed: 22 August 2026. Chip generations move quickly; this article will be re-checked quarterly.
