Every Kubernetes cluster stands on etcd. All cluster state lives there: pods, services, secrets, everything. If etcd goes down, your cluster stops behaving like a cluster. Yet plenty of us run Kubernetes daily without ever having looked at the thing underneath it, which is a shame, because it’s a Go project and a genuinely instructive one.

Etcd is a distributed key-value store written in Go, designed for the most critical data in a distributed system. The CNCF graduated it as a project, the same tier as Kubernetes itself.

What makes etcd special?

Etcd uses the Raft consensus algorithm to keep data consistent across nodes, so some nodes can fail and your data stays both consistent and available. On top of that you get watch functionality for real-time updates, automatic leader election, and transactional compare-and-swap operations.

If you build distributed systems in Go, these patterns are worth learning properly. Consensus, leader election, and distributed locking turn up everywhere in modern infrastructure, and etcd is a good place to see them done well.

Connecting to etcd with Go

Let’s start with a basic connection. You’ll need the official etcd client library:

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
)

func main() {
	// Create a client connection
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"localhost:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	// Put a key-value pair
	_, err = cli.Put(ctx, "service/config", "production")
	if err != nil {
		log.Fatal(err)
	}

	// Get the value back
	resp, err := cli.Get(ctx, "service/config")
	if err != nil {
		log.Fatal(err)
	}

	for _, kv := range resp.Kvs {
		fmt.Printf("%s : %s\n", kv.Key, kv.Value)
	}
}

Notice how context is used throughout. Etcd operations can hang when the cluster is unavailable, and “can hang” in a distributed system means “will hang, eventually, in production”. Set timeouts from day one.

Watching for changes

My favourite etcd feature is watching keys for changes. This is the mechanism behind Kubernetes controllers reacting to state changes:

func watchConfig(cli *clientv3.Client) {
	// Watch for changes to any key with the "service/" prefix
	watchChan := cli.Watch(context.Background(), "service/", clientv3.WithPrefix())

	for watchResp := range watchChan {
		for _, event := range watchResp.Events {
			switch event.Type {
			case clientv3.EventTypePut:
				fmt.Printf("Key updated: %s = %s\n", event.Kv.Key, event.Kv.Value)
			case clientv3.EventTypeDelete:
				fmt.Printf("Key deleted: %s\n", event.Kv.Key)
			}
		}
	}
}

The watch channel stays open and streams events as they happen, which is far more efficient than polling. Once you’ve seen this pattern, the whole “reconciliation loop” model that Kubernetes is built on starts to make sense.

Distributed locking

Etcd also provides distributed locking through its concurrency package, useful when you need to be sure only one process runs a task:

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"localhost:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// Create a session with a 10 second TTL
	session, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	// Create a mutex on the /my-lock/ prefix
	mutex := concurrency.NewMutex(session, "/my-lock/")

	ctx := context.Background()

	// Acquire the lock
	if err := mutex.Lock(ctx); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Lock acquired, doing critical work...")

	// Simulate work
	time.Sleep(2 * time.Second)

	// Release the lock
	if err := mutex.Unlock(ctx); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Lock released")
}

The important detail is the session TTL. If your process crashes while holding the lock, the lock releases itself once the TTL expires, so a dead process can’t deadlock the whole system. Getting this right yourself is harder than it looks, which is exactly why you should let etcd do it.

Best practices

A few things I’d want you to know before using etcd in anger. Structure your keys like a filesystem (/services/api/config, /services/worker/status); prefixes make watching and listing far easier later. Always use contexts with timeouts, because network partitions happen and a client hanging forever is worse than one that fails fast. Expect the cluster to be temporarily unavailable now and then, and retry with backoff rather than falling over. And keep your values small. Etcd is tuned for values under 1MB, so for anything bigger, store a reference in etcd and put the actual data somewhere built for it.

When to use etcd

Etcd shines for configuration management, service discovery, leader election, distributed locking, and coordination between services. In other words, the small, critical data that everything else depends on.

It is not a general-purpose database, and it will punish you if you treat it like one. High-throughput data storage belongs in a traditional database or a purpose-built system.

If you’re building infrastructure tools in Go, etcd repays deep understanding. The Raft consensus algorithm it implements is the same one used by many other distributed systems, so time spent here transfers. My suggestion for getting started: run a single-node etcd locally, point the watch example above at it in one terminal, and write keys from another. Watching the events stream in makes the Kubernetes controller model click in a way no architecture diagram ever has for me.