I consider the RTX 4060 Ti as the best entry level GPU for running small models.... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		elorant on Dec 29, 2024 \| parent \| context \| favorite \| on: How I run LLMs locally I consider the RTX 4060 Ti as the best entry level GPU for running small models. It has 16GBs of RAM which gives you plenty of space for running large context windows and Tensor Cores which are crucial for inference. For larger models probably multiple RTX 3090s since you can buy them on the cheap on the second hand market. I don’t have experience with AMD cards so I can’t vouch for them.

fnqi8ckfek on Dec 29, 2024 [–]

I know nothing about gpus. Should I be assuming that when people say "ram" in the context of gpus they always mean vram?

elorant on Dec 29, 2024 | | [–]

Not always, because system RAM also has to be equally adequate, but mostly yes, it's about the total VRAM of the GPU(s).

layer8 on Dec 29, 2024 | | [–]

“GPU with xx RAM” means VRAM, yes.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact