I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		onion2k on May 22, 2024 \| parent \| context \| favorite \| on: Ask HN: Which LLMs can run locally on most consume... I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable if you're patient. I'm glad I also have a Hugging Face account though.

Liquix on May 22, 2024 [–]

seconded - IMHO Jan has the cleanest UI and most straightforward setup out of all LLM frontends available now.

https://jan.ai/

https://github.com/janhq/jan

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact