Most LLM tutorials for Go start with an OpenAI API key and end with a monthly bill. You don’t always need either. Ollama lets you run models like DeepSeek-R1, Gemma 3, Llama 3 and Mistral on your own machine, and because Ollama itself is written in Go, the Go client library is a first-class citizen rather than an afterthought.

At over 158,000 GitHub stars it’s comfortably the most popular way to run LLMs locally. Here’s how to use it from Go.

What is Ollama?

Ollama handles the tedious parts of running a model locally: downloading weights, managing versions, and exposing a simple API for inference. The model library covers most bases. DeepSeek-R1 for reasoning-heavy work, Google’s Gemma 3 and Gemma3n, Meta’s Llama 3, Mistral when you want something fast, Microsoft’s compact Phi-4, Alibaba’s multilingual Qwen, and LLaVa for vision tasks.

The part that matters for us: the official Go client library lives in the same repository as Ollama itself, so it tracks the server closely.

Setting up Ollama

Install Ollama from ollama.com, then pull a model:

ollama pull llama3.2

That’s the whole setup. Everything else happens from Go.

Using the Go client library

Install the official client:

go get github.com/ollama/ollama/api

Here’s a basic example that generates text:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/ollama/ollama/api"
)

func main() {
	client, err := api.ClientFromEnvironment()
	if err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()
	
	req := &api.GenerateRequest{
		Model:  "llama3.2",
		Prompt: "Explain goroutines in one paragraph.",
	}

	var response string
	err = client.Generate(ctx, req, func(resp api.GenerateResponse) error {
		response += resp.Response
		return nil
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(response)
}

The callback receives the response in chunks as they arrive, which is what makes streaming possible. Every call also takes a context, and that matters more here than in most APIs: generation can be slow, and cancellation is how you bail out. If you’re hazy on context in Go, sort that first.

Streaming responses in real time

Nobody wants to stare at a blank terminal while a full response is generated. For chat, print tokens as they arrive:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/ollama/ollama/api"
)

func main() {
	client, err := api.ClientFromEnvironment()
	if err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()
	
	messages := []api.Message{
		{
			Role:    "system",
			Content: "You are a helpful Go programming assistant.",
		},
		{
			Role:    "user",
			Content: "What's the difference between a slice and an array?",
		},
	}

	req := &api.ChatRequest{
		Model:    "gemma3",
		Messages: messages,
	}

	err = client.Chat(ctx, req, func(resp api.ChatResponse) error {
		fmt.Print(resp.Message.Content)
		return nil
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println()
}

The callback fires for each token, which gives you the familiar typing effect from ChatGPT-style interfaces.

Building a simple CLI chat

Enough snippets. Here’s a minimal interactive chat you can actually run:

package main

import (
	"bufio"
	"context"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/ollama/ollama/api"
)

func main() {
	client, err := api.ClientFromEnvironment()
	if err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()
	scanner := bufio.NewScanner(os.Stdin)
	
	var history []api.Message
	history = append(history, api.Message{
		Role:    "system",
		Content: "You are a helpful assistant. Keep responses concise.",
	})

	fmt.Println("Chat with Ollama (type 'quit' to exit)")
	
	for {
		fmt.Print("\nYou: ")
		if !scanner.Scan() {
			break
		}
		
		input := strings.TrimSpace(scanner.Text())
		if input == "quit" {
			break
		}
		if input == "" {
			continue
		}

		history = append(history, api.Message{
			Role:    "user",
			Content: input,
		})

		req := &api.ChatRequest{
			Model:    "llama3.2",
			Messages: history,
		}

		fmt.Print("Assistant: ")
		var response strings.Builder
		
		err = client.Chat(ctx, req, func(resp api.ChatResponse) error {
			fmt.Print(resp.Message.Content)
			response.WriteString(resp.Message.Content)
			return nil
		})
		if err != nil {
			log.Printf("Error: %v\n", err)
			continue
		}
		fmt.Println()

		history = append(history, api.Message{
			Role:    "assistant",
			Content: response.String(),
		})
	}
}

The important detail is the history slice. The model has no memory between calls; every request sends the whole conversation again, and that’s how it “remembers” what you said. It also means long chats get slower over time, because each turn ships more tokens.

Handling errors gracefully

Plenty can go wrong here. The model might not be pulled yet. The server might not be running. A query might take longer than you’re prepared to wait:

func generateWithRetry(ctx context.Context, client *api.Client, model, prompt string) (string, error) {
	var result strings.Builder
	
	req := &api.GenerateRequest{
		Model:  model,
		Prompt: prompt,
	}

	err := client.Generate(ctx, req, func(resp api.GenerateResponse) error {
		result.WriteString(resp.Response)
		return nil
	})
	
	if err != nil {
		// Check if it's a context cancellation
		if ctx.Err() != nil {
			return "", fmt.Errorf("request cancelled: %w", ctx.Err())
		}
		return "", fmt.Errorf("generation failed: %w", err)
	}

	return result.String(), nil
}

Wrapping errors with what you were actually trying to do pays off quickly in this kind of code. There’s more on this in error handling patterns in Go.

Switching between models

One of my favourite things about Ollama is how cheap it is to swap models. DeepSeek for complex reasoning, Gemma for general tasks, LLaVa for image understanding:

type ModelConfig struct {
	Name        string
	Description string
}

var models = map[string]ModelConfig{
	"reasoning": {Name: "deepseek-r1", Description: "Best for complex reasoning"},
	"general":   {Name: "llama3.2", Description: "Good all-around model"},
	"fast":      {Name: "gemma3", Description: "Quick responses"},
	"code":      {Name: "qwen2.5-coder", Description: "Code generation"},
}

func getModel(task string) string {
	if cfg, ok := models[task]; ok {
		return cfg.Name
	}
	return models["general"].Name
}

Naming models by the job rather than the model means upgrading to a better one later is a one-line change.

Performance tips

Local inference isn’t free; you’re paying in RAM and compute instead of API bills. Reach for the smallest model that does the job, because Gemma 2B will feel snappy where Llama 70B grinds. Set sensible timeouts, since some queries genuinely take minutes. Keep prompts short where you can, as shorter context means faster responses. And look at quantised models if memory is tight.

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()

// Use context for all Ollama calls
err := client.Chat(ctx, req, callback)

Wrapping up

The Go client is one of the nicer LLM libraries I’ve come across: small API surface, streaming by default, no surprises. Start with Llama 3.2 or Gemma 3, wire it into something you already run, and see how far it gets you. The official Ollama GitHub repository has solid documentation and the model library keeps growing.

The trade is straightforward. No API costs, no rate limits and complete privacy, in exchange for your own hardware doing the work. For internal tools and side projects, I find that an easy yes.