Syncthing syncs files between devices without a cloud. It's open source, peer-to-peer, and built entirely in Go.

Syncthing: The P2P file sync tool written in Go

Syncthing is an open source continuous file synchronization program written in Go. It implements peer-to-peer file synchronization between devices using the Block Exchange Protocol (BEP) over TLS 1.3. Unlike cloud-based sync solutions, devices communicate directly with each other.

The project consists of approximately 200,000 lines of Go code and demonstrates several advanced Go patterns: structured concurrency with goroutines and channels, custom protocol implementations, and cross-platform system integration.

Go’s Role in Syncthing’s Architecture

Syncthing compiles to native binaries for Windows, macOS, Linux, FreeBSD, and Android from a single Go codebase. The cross-compilation is straightforward using GOOS and GOARCH:

GOOS=linux GOARCH=amd64 go build
GOOS=windows GOARCH=amd64 go build
GOOS=darwin GOARCH=arm64 go build

The binary size is approximately 12-15 MB with no external dependencies—Go’s static linking produces fully self-contained executables.

Syncthing’s architecture relies heavily on Go’s concurrency primitives. The program maintains persistent TCP connections to multiple devices, monitors filesystem changes, manages block transfers, and handles conflict resolution—all concurrently. Each device connection runs in its own goroutine, with channels coordinating state between components.

Filesystem Monitoring with Go

Syncthing uses filesystem watchers to detect changes without constant polling. On Linux, this means inotify; on macOS, FSEvents; on Windows, ReadDirectoryChangesW. Go’s fsnotify package abstracts these platform-specific APIs:

package main

import (
	"context"
	"log"

	"github.com/fsnotify/fsnotify"
)

type FileWatcher struct {
	watcher *fsnotify.Watcher
	changes chan string
}

func NewFileWatcher() (*FileWatcher, error) {
	watcher, err := fsnotify.NewWatcher()
	if err != nil {
		return nil, err
	}
	return &FileWatcher{
		watcher: watcher,
		changes: make(chan string, 100),
	}, nil
}

func (fw *FileWatcher) Watch(ctx context.Context, path string) error {
	if err := fw.watcher.Add(path); err != nil {
		return err
	}

	go func() {
		defer fw.watcher.Close()
		for {
			select {
			case <-ctx.Done():
				return
			case event, ok := <-fw.watcher.Events:
				if !ok {
					return
				}
				// Syncthing is interested in writes, creates, removes, and renames
				if event.Op&(fsnotify.Write|fsnotify.Create|fsnotify.Remove|fsnotify.Rename) != 0 {
					select {
					case fw.changes <- event.Name:
					case <-ctx.Done():
						return
					}
				}
			case err, ok := <-fw.watcher.Errors:
				if !ok {
					return
				}
				log.Printf("watcher error: %v", err)
			}
		}
	}()

	return nil
}

func (fw *FileWatcher) Changes() <-chan string {
	return fw.changes
}

Key Go patterns here:

  • Context for cancellation: The watcher stops cleanly when the context is cancelled
  • Deferred cleanup: defer fw.watcher.Close() ensures the file descriptor is released
  • Buffered channels: The changes channel has a buffer to prevent blocking when multiple files change rapidly
  • Select with default: Prevents blocking when sending to a full channel during shutdown

The Block Exchange Protocol

Syncthing’s Block Exchange Protocol (BEP) divides files into blocks and transfers only blocks that differ between devices. Syncthing uses variable block sizes based on file size—smaller files use smaller blocks (typically 128 KiB for files under 250 MB), while larger files use larger blocks (up to 16 MiB) to reduce metadata overhead.

Each block is identified by a SHA-256 hash. When synchronizing, devices exchange block lists (file metadata containing offset, size, and hash for each block). Only blocks with mismatched hashes are transferred.

Here’s a simplified implementation of block hashing in Go:

package main

import (
	"crypto/sha256"
	"fmt"
	"io"
	"os"
)

type Block struct {
	Offset int64
	Size   int64
	Hash   [32]byte // SHA-256 produces exactly 32 bytes
}

// calculateBlockSize returns the block size for a given file size.
// This mimics Syncthing's variable block sizing strategy.
func calculateBlockSize(fileSize int64) int64 {
	const (
		minBlockSize = 128 << 10  // 128 KiB
		maxBlockSize = 16 << 20   // 16 MiB
	)

	// Use 128 KiB blocks for files up to 250 MB
	if fileSize < 250<<20 {
		return minBlockSize
	}

	// Scale block size with file size, capped at 16 MiB
	blockSize := fileSize / 2000
	if blockSize > maxBlockSize {
		return maxBlockSize
	}
	if blockSize < minBlockSize {
		return minBlockSize
	}
	return blockSize
}

func hashFile(path string) ([]Block, error) {
	f, err := os.Open(path)
	if err != nil {
		return nil, fmt.Errorf("open file: %w", err)
	}
	defer f.Close()

	stat, err := f.Stat()
	if err != nil {
		return nil, fmt.Errorf("stat file: %w", err)
	}

	blockSize := calculateBlockSize(stat.Size())
	numBlocks := (stat.Size() + blockSize - 1) / blockSize
	blocks := make([]Block, 0, numBlocks)

	buf := make([]byte, blockSize)
	offset := int64(0)

	for {
		n, err := io.ReadFull(f, buf)
		if n > 0 {
			hash := sha256.Sum256(buf[:n])
			blocks = append(blocks, Block{
				Offset: offset,
				Size:   int64(n),
				Hash:   hash,
			})
			offset += int64(n)
		}
		if err == io.EOF || err == io.ErrUnexpectedEOF {
			break
		}
		if err != nil {
			return nil, fmt.Errorf("read file: %w", err)
		}
	}

	return blocks, nil
}

Key Go implementation details:

  • Fixed-size arrays vs slices: Using [32]byte for hashes instead of []byte eliminates heap allocations since the size is known at compile time
  • Pre-allocated slices: make([]Block, 0, numBlocks) pre-allocates capacity to avoid repeated allocations during append
  • io.ReadFull: Unlike Read, ReadFull attempts to fill the entire buffer, which is important for consistent block sizes
  • Error wrapping: Go 1.13+ error wrapping with %w preserves error chains for better debugging

When devices sync, they exchange these block lists. A simple comparison identifies which blocks need transfer:

Device Discovery and Connection Establishment

Syncthing implements a multi-layered discovery system to establish device connections:

Local Discovery: UDP broadcasts on port 21027 to the IPv4 broadcast address and IPv6 multicast group ff12::8384. Devices on the same LAN respond with their listening address. This discovery runs every 30 seconds.

Global Discovery: Devices announce their presence to discovery servers via HTTPS. The announcement includes the device ID (derived from its TLS certificate) and current external IP addresses. Other devices query these servers to find connection endpoints. The discovery server protocol is documented in the BEP specification.

Relay Servers: When direct connections fail (due to NATs or firewalls), Syncthing falls back to relay servers using the Relay Protocol. Relays forward encrypted data between devices—they never see the plaintext since the BEP connection itself is TLS-encrypted.

Here’s how device IDs work in Go:

package main

import (
	"crypto/sha256"
	"encoding/base32"
	"strings"
)

// DeviceID generates a Syncthing device ID from a TLS certificate's
// SHA-256 fingerprint. Syncthing uses base32 encoding (Luhn mod N check digit algorithm).
func DeviceID(certSHA256 []byte) string {
	// Syncthing uses a base32 alphabet without padding
	encoded := base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(certSHA256)

	// Split into chunks of 13 characters separated by hyphens
	// (Syncthing uses 7-character chunks with Luhn check digits, simplified here)
	var chunks []string
	for i := 0; i < len(encoded); i += 13 {
		end := i + 13
		if end > len(encoded) {
			end = len(encoded)
		}
		chunks = append(chunks, encoded[i:end])
	}

	return strings.Join(chunks, "-")
}

Every Syncthing device generates a TLS certificate on first run. The device ID is derived from this certificate’s SHA-256 fingerprint, ensuring each device has a globally unique, cryptographically verifiable identity.

The REST API and Embedded Web Server

Syncthing embeds an HTTP server (Go’s net/http) that serves both a web UI and REST API on port 8384. The web UI is compiled into the binary using Go’s embed package (introduced in Go 1.16):

package main

import (
	"embed"
	"io/fs"
	"net/http"
)

//go:embed gui/*
var webUI embed.FS

func setupHTTPServer() *http.Server {
	// Extract the gui subdirectory
	guiFS, _ := fs.Sub(webUI, "gui")

	mux := http.NewServeMux()

	// Serve static files from embedded filesystem
	mux.Handle("/", http.FileServer(http.FS(guiFS)))

	// REST API endpoints
	mux.HandleFunc("/rest/system/status", handleSystemStatus)
	mux.HandleFunc("/rest/db/completion", handleCompletion)

	return &http.Server{
		Addr:    "127.0.0.1:8384",
		Handler: mux,
	}
}

func handleSystemStatus(w http.ResponseWriter, r *http.Request) {
	// Return JSON system status
	w.Header().Set("Content-Type", "application/json")
	// ... implementation
}

The //go:embed directive embeds the entire web UI at compile time, eliminating external dependencies. The resulting binary contains HTML, JavaScript, CSS, and images—no separate installation step required.

Protocol Implementation in Go

Syncthing implements BEP using Protocol Buffers for message serialization. The .proto files define message structures, and protoc-gen-go generates Go code:

// Generated from BEP protocol buffer definition
type Index struct {
	Folder string
	Files  []FileInfo
}

type FileInfo struct {
	Name         string
	Size         int64
	ModifiedS    int64
	Blocks       []BlockInfo
	Version      Vector
	Permissions  uint32
}

The protocol runs over TLS 1.3. Go’s crypto/tls package handles the cryptography. Syncthing uses mutual TLS authentication—both sides present certificates, and the device ID is verified against the certificate.

For concurrent connections, Syncthing spawns a goroutine per device connection. Each goroutine handles message encoding/decoding and block transfer. A connection manager coordinates state:

type ConnectionManager struct {
	connections map[DeviceID]*Connection
	mu          sync.RWMutex
}

func (cm *ConnectionManager) Add(deviceID DeviceID, conn net.Conn) {
	cm.mu.Lock()
	defer cm.mu.Unlock()

	c := &Connection{
		deviceID: deviceID,
		conn:     conn,
		sender:   make(chan Message, 100),
	}

	cm.connections[deviceID] = c

	// Start goroutines for send/receive
	go c.sendLoop()
	go c.receiveLoop()
}

The separation of send and receive into different goroutines is a common Go pattern for full-duplex protocols. Each direction operates independently, with channels coordinating message flow.

Go Implementation Insights

Syncthing demonstrates several production Go patterns:

  1. Structured concurrency: Each component (file watcher, connection manager, block exchanger) runs in its own goroutine with clear communication via channels
  2. Context propagation: Shutdown signals propagate through context.Context, allowing clean termination
  3. Interface-based design: Platform-specific code (filesystem operations, network discovery) hides behind interfaces, with implementations selected at compile time
  4. Careful memory management: Block buffers are pooled using sync.Pool to reduce GC pressure during transfers
  5. Observability: Extensive use of Go’s expvar package for runtime metrics and pprof for profiling

The codebase is worth studying for anyone building networked systems in Go. It handles real-world challenges: NAT traversal, conflict resolution, incremental syncing, and cross-platform compatibility.