Run Deepseek Coder LLM locally

rightbyte · on Dec 3, 2023

Note, I haven't vetted the code. Use sandbox. Something like this should get you started:

    wget https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf

    git clone https://github.com/ggerganov/llama.cpp

    cd llama.cpp
    make

    ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins

rightbyte · on Dec 3, 2023

I think this LLM is quite fascinating, since it is self-hosted. It seems about as good as ChatGPT free for coding tasks, but write better descriptions.

Somehow these models does not seem to get that much hype here, that I would anticipate.