Can I run a local model without a dedicated graphics card?

You can, on the processor and system memory, but generation slows to a few tokens per second on anything beyond a very small model. That is workable for batch jobs left running unattended and frustrating for anything interactive.

Does quantisation hurt the model's answers?

Somewhat, and less than most people expect. Four-bit formats in the Q4_K_M family retain most of the quality of the full-precision original while cutting memory to roughly a quarter, which is why they have become the common default. Dropping to three-bit or below is where degradation starts to show in reasoning and code.

How much system RAM do I need alongside the graphics card?

Enough to load the model file before it moves onto the card, so match or exceed its size on disk. Thirty-two gigabytes is a comfortable starting point and sixty-four helps if you switch between large models often. System memory also absorbs any layers you deliberately offload, at the speed penalty that implies.

What else should the machine have for this workload?

Fast NVMe storage, since model files run to tens of gigabytes and are read from disk each time you switch between them. A power supply rated for the card's peak draw with real headroom, because inference holds a card near full load for minutes at a stretch rather than in the bursts a game produces. Cooling able to sustain that without the card throttling back.

Does the choice of card affect which models I can use at all?

Only through capacity, and to a lesser degree format support. Any modern card runs the same open-weight models; what changes is which sizes and quantisations will fit. A card that cannot hold your preferred model at a usable quantisation effectively removes it from your options, which is why the model you want to run should decide the card rather than the other way round.

Best GPU for Local LLM South Africa

►Home

►Specials

►

Graphics cards selected for running large language models locally

Start With VRAM, Then Look at Bandwidth

Running a language model on your own machine is first a capacity question. The weights have to fit in graphics memory, and if they do not, layers spill across the PCIe bus into system RAM and generation speed collapses from tens of tokens per second to low single digits. Everything else about a card matters only once that threshold is met.

After a model fits, memory bandwidth decides how fast it runs, because producing each token reads the entire set of weights once. That is why two cards holding identical capacity can differ by roughly a factor of two in tokens per second, and why the quickest gaming card is not automatically the right one for this job.

Start With VRAM, Then Look at Bandwidth

Start With VRAM, Then Look at Bandwidth

Top 5 Reasons to Buy Gaming Laptops

How to Size a GPU for Local Language Models

Top 5 Reasons to Buy Gaming Laptops

How to Size a GPU for Local Language Models