Chapter 15: Talk to Your Pi with Ollama
Large Language Models, or LLMs, are very trendy right now. ChatGPT and its peers are starting to completely replace traditional web searches for information. When using a chat bot like ChatGPT, your queries get passed to their server, as it would with any web request. But what if we could run our own chat bot?
One of the most compelling reasons to run an LLM on a Raspberry Pi is the privacy benefits it offers. Unlike cloud-based models, running an LLM locally on a Raspberry Pi ensures that all data processing happens on your own device, without the need to send information to external servers. This setup is particularly advantageous for sensitive applications where data privacy is paramount, such as personal assistants, home automation, and private data analysis. By keeping the data local, users have full control over their information, mitigating risks associated with data breaches or unauthorized access. Moreover, this approach aligns with growing concerns about digital privacy, providing a practical solution for those wary of extensive data collection practices prevalent in many modern AI applications.
There are a lot of options available for running LLMs. All require quite a bit of RAM, so a 2GB Raspberry Pi 3 might not be the best candidate. Even the mighty 8GB Pi 5 will struggle with a more robust model. For that reason, we're going to use tinyllama, a LLM built for lightweight devices. On a Pi 5, it will provide quick answers. Don't expect the depth and accuracy of ChatGPT. If you want a more high performing LLM, phi3 runs well on a Pi. It's several times bigger than tinyllama and provides much better answers. Answering each prompt will take up to several minutes, however.
For the truly masochistic, you can run llama3, a very capable LLM from Meta. Bear in mind that Meta recommends 64GB of RAM for llama3. You'll get answers on a Pi 5 but they will take a very long time.
Step 1: Install Ollama and Download a Model
- Run the following command to get the installation script:
curl -fsSl https://ollama.com/install.sh | sh - Download tinyllama.
ollama run tinyllama
Step 2: Talk to Your Pi!
Input a question or prompt. After a couple of seconds, you should get a response. You can try other models like phi3 or even llama3, but be warned that they are big downloads and will take forever to run.