---
title: "What Is a GPU Used for in AI? 5 Surprising Facts"
url: "https://learnaitodayonline.com/what-is-a-gpu-used-for-in-ai/"
description: "What is a GPU used for in AI? GPUs, TPUs and NPUs explained for beginners — VRAM, why chips matter, and what's inside your computer right now."
author: "Robert Waithaka"
published: "2026-08-25"
last_reviewed: "2026-08-22"
categories: ["Foundations"]
tags: ["level-beginner"]
site: "Learn Artificial Intelligence"
approx_tokens: 2615
---

# What Is a GPU Used for in AI? 5 Surprising Facts

What is a GPU used for in AI? Short answer: GPUs do the maths. Artificial intelligence is, at its core, an enormous number of simple calculations happening in parallel, and graphics processing units (GPUs) are built to do exactly that. They are why your phone can answer questions, why ChatGPT responds in seconds, and why data centres full of servers exist at all. This guide explains the chips behind AI in plain English — GPUs, TPUs, NPUs, VRAM and the supply chain that makes them — with real numbers from the research.

## What Is a GPU Used for in AI?

A GPU is a chip designed for parallel processing: it breaks a big problem into thousands of smaller pieces and solves them at the same time. In AI, both training (building a model) and inference (running it) are giant parallel-maths jobs, so GPUs do both. Data-centre GPUs like NVIDIA's H100 sit in racks of thousands; consumer GPUs like the RTX 4090 do the same work on your desk. The machines that house them — the buildings we covered in [what is an AI data center](https://learnaitodayonline.com/what-is-an-ai-data-center/) — exist mainly to keep these chips powered and cool.

## Why AI Needs Parallel Maths (and Why CPUs Are Bad at It)

![What is a GPU used for in AI — parallel maths compared with a CPU](https://cdn.sanity.io/images/gfihpee1/production/d82e058928f71a708670b4d39e0991bf61e0f8ac-1614x696.png)

_Figure: a CPU has a few very fast cores that work sequentially; a GPU has thousands of simpler cores that work in parallel. Based on the CPU-vs-GPU explanations in IBM, Cisco and Fortinet materials._

Your computer's central processing unit (CPU) is brilliant at doing one task quickly, in order. AI models do the opposite: every answer requires millions of nearly identical calculations — multiply, add, compare — repeated across billions of parameters. A CPU does these one at a time and stalls; a GPU does thousands at once. That is why AI needs GPUs. The trade-off is real: GPUs consume roughly 10–15 times more power per processing cycle than CPUs, which is why AI data centres need gigawatts and why chip efficiency is such a big deal.

## GPU vs TPU vs NPU vs AI Accelerator

| Chip type | What it is | What it's for | Examples |
| --- | --- | --- | --- |
| GPU (graphics processing unit) | General-purpose parallel processor | Training and running most AI models | NVIDIA H100, RTX 4090, AMD cards |
| TPU (tensor processing unit) | Custom-built to speed up tensor maths | AI in Google's data centres | Google TPUs |
| NPU (neural processing unit) | Low-power AI accelerator inside devices | On-device AI on laptops and phones | Copilot+ PC NPUs, Snapdragon AI Hub |
| AI accelerator / ASIC | Any chip specialised for AI workloads | Custom clouds and specific tasks | AWS Trainium |

_Table: GPU vs TPU vs NPU vs AI accelerator, from the IBM, Cisco, Fortinet, Lenovo and Spiceworks materials._

The umbrella term is "AI chip" — anything specialised for AI workloads. GPUs are the workhorses because they are general enough to train and run almost any model; TPUs and NPUs trade that generality for speed or efficiency in specific places.

## What Is VRAM and Why It's the Number That Matters

VRAM is the memory on the graphics card, and for AI it is the number that matters most. A model's weights — the billions of numbers that define its behaviour — must fit in memory while it runs. If they don't fit, the software "offloads" to slower system RAM and speed collapses.

The rule of thumb from the 2026 hardware guides: a 7B model needs roughly 4–5 GB at Q4 quantization, a 70B model about 40 GB, and a 120B model 65–70 GB. Memory bandwidth matters just as much as capacity: an RTX 3090 moves data at 936 GB/s, while a Strix Halo mini PC runs at 256 GB/s — which is why the same model can feel dramatically faster on one machine than another. We go deep on sizing in our [hardware guide](https://learnaitodayonline.com/what-hardware-do-you-need-to-run-ai-models-locally/).

## Why the Same Chip Trains and Serves Models

Training and inference are different jobs, but they run on the same kind of hardware. Training is the one-time "mountain climb": thousands of GPUs crunch data for weeks to build the model. Inference is the "daily commute": every prompt you send runs the finished model on GPUs again, over and over. AI models train and run on data-centre GPUs (IBM's explainer says it directly), and the accelerator lifecycle is the same in cloud and local setups. The workloads are so similar that the same H100s that trained a frontier model go on to serve millions of queries — and because inference never stops, it now uses an estimated 80–90% of AI's computing power.

## AI Chip Performance: A Few Real Numbers

![AI performance across devices: from $80 boards to desktop GPUs](https://cdn.sanity.io/images/gfihpee1/production/37d35793cb8c3488b531fe86f9870f072948fa96-1179x762.png)

_Figure: AI performance measured in TOPS (trillions of operations per second) across devices. Source: Top AI Hardware 2026 ranking (supplied material)._

| Chip / device | What it is | Key number | Source |
| --- | --- | --- | --- |
| NVIDIA H100 | Data-centre GPU | 700 W rated; up to 989 trillion FLOP/s | Epoch AI |
| NVIDIA A100 | Previous-generation GPU | 800 W max | Epoch AI |
| RTX 4090 | Consumer desktop GPU | 1,321 TOPS | Top AI Hardware 2026 |
| Mac Mini M4 Pro | Apple unified-memory machine | ~67 TOPS, 24 GB | Top AI Hardware 2026 |
| Jetson Orin Nano | Edge AI board | 67 TOPS at ~15 W | Top AI Hardware 2026 |
| Raspberry Pi 5 | Budget board | ~2 TOPS | Top AI Hardware 2026 |
| DGX Spark (GB10) | NVIDIA desktop appliance | 128 GB unified; "1 PFLOP FP4" is positioning, not measured LLM speed | kingy.ai guide |

_Table: real chip numbers from the supplied materials. TOPS and FLOPs measure peak capability, not everyday AI speed — see the hardware guide for real-world tokens per second._

![AI chips are power-hungry at every scale](https://cdn.sanity.io/images/gfihpee1/production/4023b5ec8b7962d4a37526b002fbcb9c6ef2bdc7-1291x762.png)

_Figure: an H100 chip is rated at 700 W, draws up to ~1,500 W installed, and sits in racks that need 50–150 kW each. Sources: Epoch AI; Hanwha Data Centers._

## The Chip Supply Chain in One Section

![The AI chip supply chain in one picture](https://cdn.sanity.io/images/gfihpee1/production/de790149df9e2557278b61a7e915a0314574c552-1325x645.png)

_Figure: from design to your data centre — with the hidden inputs. Sources: Chatham House; EESI; Epoch AI; Hanwha._

| Stage | Fact | Source |
| --- | --- | --- |
| Design | NVIDIA, AMD, Google and Apple design the chips | IBM |
| Fabrication | Taiwan produces over 90% of the world's advanced semiconductors; TSMC used 24 billion kWh in 2023 (~2.7 GW average) | Chatham House; Epoch AI |
| Water | A chip fab uses ~10 million gallons of ultrapure water a day; ~1,500 gallons of piped water per 1,000 gallons of UPW | EESI |
| Water (per chip) | A single chip has consumed thousands of gallons of water by the time it reaches a data centre | EESI |
| Recycling | Wafer plants in Singapore recycle ~45% of water; semiconductor plants ~23% | "Making AI Less Thirsty" |
| Supply-chain share | Apple reports its supply chain accounts for 99% of its total water footprint | "Making AI Less Thirsty" |

_Table: the chip supply chain's hidden costs. This is why the environmental footprint of AI chips extends far beyond electricity — our [AI water use explainer](https://learnaitodayonline.com/how-much-water-does-ai-use/) covers the full picture._

## What's in Your Computer Right Now

Most computers already contain some AI hardware. Discrete GPUs (like the RTX series) have VRAM you can check in the system settings; integrated graphics handle small models on CPU; newer Copilot+ PCs include NPUs designed for on-device AI, and Microsoft's Phi-class models run offline on them. Apple Silicon uses unified memory, so the GPU can access all system RAM — the reason a 64 GB Mac can run models that would not fit on a 24 GB graphics card.

If you have a laptop with 16 GB of VRAM (like an RTX 5080-class machine), you already have a genuinely capable local-AI machine; if you are buying used, the community consensus is that a second-hand RTX 3090 with 24 GB is the best value VRAM per dollar.

## What You Need for Local Models

For local AI, the shopping list is short: a GPU with enough VRAM for the models you want, enough system RAM, fast NVMe storage, and one of the tools from our [Ollama guide](https://learnaitodayonline.com/ollama-download-install-run-local-ai-models/). Start small, measure real tokens per second, and size up only when a task genuinely needs it — our [hardware requirements guide](https://learnaitodayonline.com/what-hardware-do-you-need-to-run-ai-models-locally/) has the full sizing tables and tier-by-tier expectations. And when the terminology piles up, the [AI glossary](https://learnaitodayonline.com/ai-glossary-essential-terms/) is the place to look things up.

---

## Sources

- [What is an AI data center? (IBM, 2025)](https://www.ibm.com/think/topics/ai-data-center)
- [What is an AI data center? (Cisco)](https://www.cisco.com/site/us/en/learn/topics/computing/what-is-an-ai-data-center.html)
- [What is an AI data center? Key capabilities (Fortinet)](https://www.fortinet.com/resources/cyberglossary/ai-data-centers)
- [How much energy does ChatGPT use? (Epoch AI, 2025)](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use)
- [We did the math on AI's energy footprint (MIT Technology Review, 2025)](https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/)
- [Data Centers and Water Consumption (EESI, 2025)](https://www.eesi.org/articles/view/data-centers-and-water-consumption)
- [Making AI Less "Thirsty" (arXiv, 2025)](https://arxiv.org/abs/2304.03271)

_Last reviewed: 22 August 2026. Chip generations move quickly; this article will be re-checked quarterly._