Ollama using Deephat

This is the second of a three-part series about how to host your own LLM. Today we are discussing how to locally host your own model on Ollama. Previously we discussed hosting on LM Studio and our next article will be about setting up your own rig using an NVIDIA Orin.

Most people interact with AI models through official interfaces, but did you know you can also host your own model locally? Hosting your own model not only allows you to get hands-on experience with AI infrastructure and learn the intricacies of model deployment and management, but it can also allow more flexibility than using the official interfaces. For example, you do not have the same usage limits that you may face on official interfaces. Most importantly, for people and organizations with unreliable internet access or wishing to use the model with sensitive information, locally hosting your model allows you to operate the model without an internet connection.

Let’s walk through how to locally host WhiteRabbitNeo using the Ollama application.

Why WhiteRabbitNeo?

WhiteRabbitNeo is the #1 uncensored, open-source AI model for red and blue team cybersecurity. It has a software engineering base and has been trained on a wide range of red team cybersecurity data, making it the perfect pair for any cybersecurity enthusiast, from the n00b to the seasoned hacker, and like I said in the first post, I use it to teach myself how to hack for CTF challenges.

Why choose Ollama?

Ollama allows you to chat with models directly in the command line, which you may prefer over switching back and forth between other programs. Ollama also allows you to locally run any AI model available on HuggingFace, as long as your computer can support it.

If you haven’t already, view the supported GPU requirements for Ollama on their GitHub repository https://github.com/ollama/ollama/blob/main/docs/gpu.md. Answers to issues encountered running Ollama may also be found in the repository.

Once you’ve verified that your computer can support Ollama, open your terminal and go to the directory where you want to download Ollama (~/Downloads is a good place) and install Ollama.

cd ~/Downloads (or other directory)
curl -fsSL https://ollama.com/install.sh | sh

For more information on installing Ollama, visit Ollama’s website.

Steps to locally hosting WhiteRabbitNeo in Ollama:

Open a terminal and go to the directory where you want the model to be downloaded (~/Downloads is a good place).

cd ~/Downloads (or other directory)

Install GitHub Large File Storage, which helps handle larger files.

sudo apt-get install git-lfs
#If this command doesn't work, try 'git lfs install'

Go to https://huggingface.co/WhiteRabbitNeo and choose the model you wish to download. From the model page, click on the three dots on the right side of the screen and then click Clone Repository.

You can also search “WhiteRabbitNeo” and click on WhiteRabbitNeo under Organizations.

Click on the 3 dots on the model card then clone repository.

The pop-up screen will remind you to install GitHub Large File Storage and give you the command to download the repository. Copy the command using the copy icon and paste it into the same command line you’ve been using. It may take a few minutes for the repository to clone.

If you encounter an error message after running the command, try the following steps:

#Identify the name of the model directory
ls

#Remove any model files that were downloaded.
rm -r [name of model directory]  #Approve any override requests using Y

#Retry cloning the directory
git-lfs clone [model repository URL link from Hugging Face]

You now have the option to run the model using .safetensors or to convert the model files to .gguf and then run the model using the .gguf file. I will first go through the steps to convert the files to .gguf, if you wish to run using .safetensors (a much shorter process), jump here.

.gguf files are model files that are optimized for faster loading. You can learn more about .gguf files on Hugging Face: https://huggingface.co/docs/hub/en/gguf
.safetensors are secure model files that are much faster than PyTorch. You can learn more about .safetensors files on Hugging Face: https://huggingface.co/docs/safetensors/index

Converting to a .gguf file

Clone the llama.cpp GitHub repository.

git clone https://github.com/ggerganov/llama.cpp

Change to the llama.cpp directory.

cd llama.cpp

Set up a virtual environment.

'PyTorch does not currently support beyond Python 3.12, meaning newer versions 
of Python will not support some of the commands below.

To circumvent this, I downloaded Python version 3.12.9 from 
https://www.python.org/downloads/. Once it is installed, run the 
following commands:'

python3 --version #If it says Python 3.13.0, run the command
alias python3='python3.12', 
#then run 
python3 - version again. 
#It should now read Python 3.12.9.

Once you have the correct Python version installed, set up the virtual environment.

python3 -m venv path/to/venv
source path/to/venv/bin/activate

Next, install the required Python dependencies.

pip install -r requirements.txt

Convert the model to .gguf

python3 convert_hf_to_gguf.py [full path to WhiteRabbitNeo model
directory]

#To get the full path to the WhiteRabbitNeo model directory:
cd ~/Downloads #or whichever directory you started in
ls #to see the name of the WhiteRabbitNeo directory
cd [name of WhiteRabbitNeo directory]
pwd #this result is the full pathname

#To return back to llama.cpp now that you have the pathname:
cd ..
cd llama.cpp

A new .gguf file should now be in the WhiteRabbitNeo model directory. To proceed, you’ll need the full pathname to that file.

The full name of the .gguf file should be in the last line of the conversion output following “Model successfully exported to,” but if not you can find it with the following steps:

cd [full pathname of WhiteRabbitNeo directory]
ls #copy the .gguf filename

Deactivate the virtual environment.

deactivate

Change into the directory you cloned the WhiteRabbitNeo repository into.

cd ~/Downloads #or whichever directory you started in

Create a model file.

nano modelfile

#Inside the file write:

FROM ./[name of WhiteRabbitNeo directory]
#do not include the brackets

#To exit the file:
press Control + X
press "Y" to save
press "Enter" to accept the filename

Import the model (this may take a few minutes).

ollama create [name you want for the model] -f modelfile --quantize q4_0

Chat with the model.

ollama run [model name you chose]

Running using .safetensors

These steps pick up after cloning the repository.

Create a model file inside of the current directory. Ensure that the model directory is nested inside of the current directory.

#To ensure the model directory is nested:
ls #you should see the model directory listed

nano modelfile

#Inside the file write:

FROM ./[name of the .gguf file created by the conversion]
#do not include the brackets

#To exit the file:
press Control + X
press "Y" to save
press "Enter" to accept the filename

Create the model (this may take a few minutes).

ollama create [name you want for the model]