Build an AI Wiki from Your Notes with Go and sage-wiki
You have a pile of research papers, markdown notes, and bookmarked articles scattered across folders. You’ve told yourself you’ll organize them “later.” Later never comes. I know because I’ve been doing this for years.
sage-wiki takes a different approach. You drop your documents into a directory, and it uses an LLM to extract concepts, discover cross-references, and produce a structured, interlinked wiki. The whole thing is written in Go, and honestly, the way it’s built is more interesting to me than the wiki output itself. There are some really clean patterns in here for building LLM-powered CLI tools.
What sage-wiki actually does
sage-wiki reads your input documents (markdown, PDFs, plain text) and sends them through an LLM pipeline. The LLM pulls out concepts from each document, finds relationships between concepts across documents, and generates wiki-style pages with cross-links.
The output is a set of interlinked markdown files you can browse locally or host as a static site. Each concept gets its own page. References between concepts are automatic.
But we’re here to talk about Go, so let’s look at how it’s built.
CLI structure with Cobra
sage-wiki uses cobra for its CLI, which is the standard choice for Go command-line tools. The project follows a clean command structure that separates concerns:
package cmd
import (
"fmt"
"os"
"github.com/spf13/cobra"
)
var rootCmd = &cobra.Command{
Use: "sage-wiki",
Short: "Compile documents into a structured wiki",
RunE: func(cmd *cobra.Command, args []string) error {
inputDir, _ := cmd.Flags().GetString("input")
outputDir, _ := cmd.Flags().GetString("output")
model, _ := cmd.Flags().GetString("model")
compiler, err := wiki.NewCompiler(wiki.Config{
InputDir: inputDir,
OutputDir: outputDir,
Model: model,
})
if err != nil {
return fmt.Errorf("creating compiler: %w", err)
}
return compiler.Run(cmd.Context())
},
}
func Execute() {
if err := rootCmd.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Notice the use of cmd.Context(). This passes the cobra command’s context through the entire pipeline, so if you hit Ctrl+C, the context gets cancelled and the LLM calls stop. This is how well-behaved Go CLI tools should work. If you want a refresher on how context works, check out this post on context in Go.
The document processing pipeline
The concurrency design here is where things get interesting. sage-wiki needs to:
- Read all input files
- Extract concepts from each file (LLM call)
- Cross-reference concepts across all files (LLM call)
- Generate output wiki pages
Steps 1 and 2 can be parallelized per-document. Step 3 needs all results from step 2. Classic fan-out/fan-in.
Here’s a simplified version of how this pipeline looks:
package wiki
import (
"context"
"fmt"
"sync"
)
type Document struct {
Path string
Content string
}
type ConceptResult struct {
DocPath string
Concepts []Concept
Err error
}
type Concept struct {
Name string
Description string
References []string
}
func (c *Compiler) extractAllConcepts(ctx context.Context, docs []Document) ([]ConceptResult, error) {
results := make([]ConceptResult, len(docs))
var wg sync.WaitGroup
// Semaphore to limit concurrent LLM calls
sem := make(chan struct{}, c.config.Concurrency)
for i, doc := range docs {
wg.Add(1)
go func(idx int, d Document) {
defer wg.Done()
select {
case sem <- struct{}{}:
defer func() { <-sem }()
case <-ctx.Done():
results[idx] = ConceptResult{
DocPath: d.Path,
Err: ctx.Err(),
}
return
}
concepts, err := c.llm.ExtractConcepts(ctx, d.Content)
results[idx] = ConceptResult{
DocPath: d.Path,
Concepts: concepts,
Err: err,
}
}(i, doc)
}
wg.Wait()
// Check for errors
for _, r := range results {
if r.Err != nil {
return nil, fmt.Errorf("extracting concepts from %s: %w", r.DocPath, r.Err)
}
}
return results, nil
}
A few things worth pulling apart here.
The sem channel limits how many goroutines call the LLM at once. LLM APIs have rate limits, and firing 50 concurrent requests just gets you throttled. The buffered channel acts as a counting semaphore. Simple, effective, and I reach for this pattern constantly.
The select statement checks if the context has been cancelled before waiting for a semaphore slot. Without this, a cancelled operation would still sit in the queue behind other goroutines, burning time doing nothing.
And the pre-allocated results slice is a nice touch. Instead of collecting results through a channel, each goroutine writes directly to its own index. No race condition, no channel coordination overhead.
LLM client integration
sage-wiki supports multiple LLM backends: OpenAI, local models via Ollama, and others. The Go interface pattern makes swapping between them straightforward:
package llm
import (
"context"
)
// Client defines what any LLM backend must support
type Client interface {
Complete(ctx context.Context, prompt string) (string, error)
CompleteWithSchema(ctx context.Context, prompt string, schema any) error
}
// OpenAIClient implements Client for the OpenAI API
type OpenAIClient struct {
apiKey string
model string
baseURL string
client *http.Client
}
// OllamaClient implements Client for local Ollama models
type OllamaClient struct {
model string
baseURL string
client *http.Client
}
CompleteWithSchema is the interesting method. When extracting concepts, you don’t want free-form text back from the LLM. You want structured data. This method sends a JSON schema along with the prompt, telling the LLM to respond in a specific format. The Go code then unmarshals the response directly into a struct:
package llm
import (
"context"
"encoding/json"
"fmt"
"net/http"
"bytes"
)
type ConceptExtractionResponse struct {
Concepts []struct {
Name string `json:"name"`
Description string `json:"description"`
RelatedTo []string `json:"related_to"`
} `json:"concepts"`
}
func (c *OpenAIClient) ExtractConcepts(ctx context.Context, content string) ([]Concept, error) {
var response ConceptExtractionResponse
prompt := fmt.Sprintf(`Extract key concepts from this document.
For each concept, provide a name, description, and related concepts.
Document:
%s`, content)
err := c.CompleteWithSchema(ctx, prompt, &response)
if err != nil {
return nil, fmt.Errorf("LLM completion: %w", err)
}
concepts := make([]Concept, len(response.Concepts))
for i, c := range response.Concepts {
concepts[i] = Concept{
Name: c.Name,
Description: c.Description,
References: c.RelatedTo,
}
}
return concepts, nil
}
This pattern (define a response struct with JSON tags, pass a pointer to the LLM client, let the client handle unmarshaling) keeps extraction logic separate from the HTTP/API plumbing. It’s the same approach you’d use when wrapping any REST API in Go. Nothing fancy, but it stays clean as you add more extraction types.
Cross-reference discovery with maps
Once concepts are extracted from all documents, sage-wiki needs to find connections. If Document A mentions “gradient descent” and Document B has a full explanation of it, they should link to each other.
The Go code builds a concept index:
package wiki
// ConceptIndex maps concept names to the documents that mention them
type ConceptIndex map[string][]DocumentRef
type DocumentRef struct {
Path string
Section string
}
func buildIndex(results []ConceptResult) ConceptIndex {
index := make(ConceptIndex)
for _, result := range results {
for _, concept := range result.Concepts {
normalized := normalize(concept.Name)
index[normalized] = append(index[normalized], DocumentRef{
Path: result.DocPath,
})
}
}
return index
}
func normalize(name string) string {
// Lowercase, trim whitespace, collapse spaces
return strings.ToLower(strings.TrimSpace(name))
}
func (idx ConceptIndex) FindCrossRefs(concept string) []DocumentRef {
return idx[normalize(concept)]
}
The normalization step matters more than it looks. “Gradient Descent”, “gradient descent”, and “Gradient descent” should all resolve to the same concept page. Basic, easy to forget, annoying to debug when you don’t do it.
File walking with io/fs
sage-wiki reads input documents by walking a directory tree. Go 1.16 introduced io/fs and fs.WalkDir, which is cleaner than the older filepath.Walk:
package wiki
import (
"io/fs"
"os"
"path/filepath"
"strings"
)
var supportedExts = map[string]bool{
".md": true,
".txt": true,
".pdf": true,
}
func loadDocuments(inputDir string) ([]Document, error) {
var docs []Document
err := filepath.WalkDir(inputDir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
ext := strings.ToLower(filepath.Ext(path))
if !supportedExts[ext] {
return nil
}
content, err := os.ReadFile(path)
if err != nil {
return fmt.Errorf("reading %s: %w", path, err)
}
docs = append(docs, Document{
Path: path,
Content: string(content),
})
return nil
})
return docs, err
}
Straightforward Go. The map for extension lookup instead of a chain of if statements is a small thing, but it’s cleaner and easier to extend when you inevitably want to support .rst or .org files.
Structured output generation
The final step is generating wiki pages. sage-wiki uses Go’s text/template package to produce markdown:
package wiki
import (
"os"
"path/filepath"
"text/template"
)
const wikiPageTemplate = `# {{.Name}}
{{.Description}}
## References
{{range .CrossRefs -}}
- [{{.ConceptName}}]({{.Link}})
{{end}}
## Source Documents
{{range .Sources -}}
- {{.Path}}
{{end}}
`
func generateWikiPage(outputDir string, concept Concept, crossRefs []CrossRef, sources []DocumentRef) error {
tmpl, err := template.New("wiki").Parse(wikiPageTemplate)
if err != nil {
return fmt.Errorf("parsing template: %w", err)
}
filename := filepath.Join(outputDir, slugify(concept.Name)+".md")
f, err := os.Create(filename)
if err != nil {
return fmt.Errorf("creating %s: %w", filename, err)
}
defer f.Close()
data := struct {
Name string
Description string
CrossRefs []CrossRef
Sources []DocumentRef
}{
Name: concept.Name,
Description: concept.Description,
CrossRefs: crossRefs,
Sources: sources,
}
return tmpl.Execute(f, data)
}
Using text/template instead of html/template is the right call since the output is markdown, not HTML. The anonymous struct for template data is a pattern you see everywhere in Go. It avoids creating a named type for something used exactly once, which I appreciate. No reason to pollute your package namespace for a one-off data shape.
Error handling patterns
One thing sage-wiki does well is error wrapping. Throughout the codebase, errors get wrapped with fmt.Errorf("context: %w", err). When something fails, you get a full chain:
creating compiler: loading documents: reading papers/attention.pdf: open papers/attention.pdf: no such file or directory
This is idiomatic Go error handling. Every function adds context about what it was trying to do when the error occurred. If you’re building CLI tools in Go, this pattern will save you hours of debugging. You can actually trace what went wrong without scattering print statements everywhere. For more on Go patterns that hold up in production, the functional options post covers another one I use all the time.
Running it yourself
Getting started is simple:
go install github.com/xoai/sage-wiki@latest
sage-wiki --input ./my-notes --output ./wiki --model gpt-4o
For local LLM usage with Ollama:
sage-wiki --input ./my-notes --output ./wiki --model llama3 --backend ollama
The output directory will contain interlinked markdown files. Serve them with any static site generator, or just browse them in your editor.
Why this codebase is worth reading
What I like about sage-wiki is that it’s not a toy example. It’s a real tool that happens to use several Go patterns well: interfaces for swappable LLM backends, fan-out/fan-in concurrency with semaphore-based rate limiting, context propagation from CLI through to HTTP calls, error wrapping for debuggable error chains, and text/template for output generation.
If you’re building any kind of document processing pipeline or LLM-powered tool in Go, read this codebase. The patterns transfer directly to other projects, whether you’re building API clients, data pipelines, or other CLI tools.
For a completely different domain that uses similar caching and optimization thinking, check out the Advent of Code memoisation post. Different problem, overlapping instincts.