Running Multiple Open Source LLMs Locally with Ollama

Introduction

Large language models are used everywhere, from chatbots to content generation. Cloud-based LLMs are popular, but running them locally has real advantages: better privacy, lower latency, and more room to customize. Ollama is a tool built for this — it lets you run open-source LLMs like Mistral, Llama2, and Llama3 on your own PC.

This post shows how to use Ollama to run multiple open-source LLMs, covers its basic and advanced features, and walks through building a local LLM setup.

Why run LLMs locally?

• Privacy: your data stays on your machine, cutting the risk of unauthorized access.
• Reduced latency: processing happens locally, so responses come back faster than from cloud LLMs.
• Customization: you have full control over the models, which allows deeper tuning and optimization.

Getting started with Ollama

Ollama runs on macOS, Windows, and Linux. You want a modern multi-core processor and at least 8 GB of RAM for large models. To install, download the latest version from the Ollama website or GitHub repository.

Create a virtual environment to manage dependencies with python -m venv ollama_env, then activate it (source ollama_env/bin/activate on macOS and Linux, or ollama_env\Scripts\activate on Windows). Install the supporting libraries with pip install numpy torch torchvision.

Ollama supports many open-source models, including Mistral, Llama2, Llama3, Vicuna, GPT-J, and GPT-NeoX. Load one from the command line with ollama load llama2, or from Python: from ollama import Ollama, then ollama = Ollama(model_name='llama2').

Building a simple chatbot with Ollama

To build a chatbot, load a model and define a function that takes user input and returns a response. A basic bot built on Llama2 answers questions through a chatbot.query(question) call. You can expand this into more complex bots or wire it into other applications.

Creating a FastAPI server with Ollama

Ollama can also power a FastAPI server so users can choose which model — or models — answer their questions. Install the web framework with pip install fastapi uvicorn, then create an /ask endpoint that accepts JSON with a question and a model choice, validates the choice against the supported models, and returns the model name with its response.

A user sends a POST request to the local server, for example curl -X POST "http://127.0.0.1:8000/ask", and gets back JSON such as a llama2 response reading "The capital of France is Paris." You can also send the same question to two models — say mistral and llama2 — and compare their answers side by side.

Exploring the Ollama API for advanced features

The Ollama API exposes a rich set of endpoints for interacting with and managing local models — generating completions, listing local models, creating models from Modelfiles, and more. It runs on localhost at port 11434; start it with ollama serve.

To generate a completion, use the POST /api/generate endpoint. This is a streaming endpoint, so by default the response is a series of JSON objects. For example: curl http://localhost:11434/api/generate -d '{"model": "llama2", "prompt": "Why is the sky blue?"}'. Set stream to false to get the full response in one JSON object, or set format to json to get a well-formed JSON response — just be sure to instruct the model to answer in JSON within the prompt.

Beyond completions, the API offers several management endpoints:

• Create a model from a Modelfile: ollama create mymodel -f ./Modelfile
• List local models: ollama list
• Pull a model from the library: ollama pull llama3
• Delete a model: ollama rm llama3
• Copy a model to a new version: ollama cp llama3 my-model

A few conventions matter: model names follow the model:tag format, and durations are returned in nanoseconds. For streaming endpoints like POST /api/generate, the final JSON object carries statistics such as total_duration (time generating the response), load_duration (time loading the model), prompt_eval_count (tokens in the prompt), and eval_count (tokens in the response).

Best practices and tips

• Resource management: make sure your system has enough memory and CPU to run the models.
• Data security: since models run locally, take steps to protect your data and privacy.
• Regular updates: keep Ollama and your models current for the latest features and fixes.

If you hit problems, restarting Ollama often clears minor issues, reinstalling key dependencies can fix broken setups, and checking system logs helps you find errors or bottlenecks.

Conclusion

This guide covered Ollama from setup through advanced API usage for running LLMs locally. The examples let you build chatbots and a FastAPI server for question-answering, and the API gives you full control over managing models. Experiment with configurations and parameters to fit your projects, and follow the best practices to keep performance reliable while keeping your data private on your own machine.

Originally published on the FutureSmart AI blog.