Rclone is written in Go — here is what you can learn from it
Rclone calls itself “rsync for cloud storage”, which rather undersells it. It syncs files to and from Google Drive, S3, Dropbox, Backblaze B2, OneDrive, Azure Blob Storage, SFTP, and about 70 other backends. It also handles encryption, chunked uploads, and bandwidth throttling, and it will mount remote storage as a FUSE filesystem if you ask.
I often tell people that once you’re comfortable with Go, the fastest way to improve is to read other people’s code, and rclone is one of my favourite codebases to send them to. The problem it solves is genuinely hard: one sync engine, roughly 70 storage APIs that all behave slightly differently, and no drowning in special cases. The answer is interfaces, used about as well as I’ve seen anywhere. Let’s dig in.
The fs.Fs interface: one interface to rule them all
How do you write sync logic that works identically against S3, Google Cloud Storage, Azure Files, Dropbox, WebDAV, FTP, and OpenStack Swift? You don’t write it seventy times. You define one interface and make every backend meet it.
At the heart of rclone is the fs.Fs interface:
// Fs is the interface a cloud storage system must provide
type Fs interface {
Info
// List the objects and directories in dir into entries
List(ctx context.Context, dir string) (DirEntries, error)
// NewObject finds the Object at remote
NewObject(ctx context.Context, remote string) (Object, error)
// Put uploads to the remote path with the modTime given of the given size
Put(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
// Mkdir creates the directory if it doesn't exist
Mkdir(ctx context.Context, dir string) error
// Rmdir removes the directory
Rmdir(ctx context.Context, dir string) error
}
Every backend implements this interface, whether it’s talking to Google Drive’s REST API or an SFTP server. The sync engine never knows or cares which provider it’s working with. It calls List, Put, NewObject, and gets on with its day.
This is implicit interface satisfaction doing exactly the job it was designed for. The backend/s3 package never declares “I implement fs.Fs”. It simply has the right methods, and the compiler checks the rest. If you’ve read about Go interfaces and how they work, you’ll recognise the shape.
Optional capabilities with interface upgrades
Not every backend can do the same things. S3 supports server-side copy. Google Drive supports moving files. FTP supports neither. The lazy fix would be to bolt Copy and Move onto the base interface and have half the backends return “not supported” errors. Rclone doesn’t do that.
It uses optional interfaces instead, a pattern sometimes called “interface upgrades”:
// Mover is an optional interface for Fs
type Mover interface {
// Move src to this remote using server-side move operations
Move(ctx context.Context, src Object, remote string) (Object, error)
}
// Copier is an optional interface for Fs
type Copier interface {
// Copy src to this remote using server-side copy operations
Copy(ctx context.Context, src Object, remote string) (Object, error)
}
Then, at runtime, rclone checks whether a backend supports a given capability using type assertions:
func serverSideCopy(ctx context.Context, fdst fs.Fs, src fs.Object, remote string) (fs.Object, error) {
do, ok := fdst.(fs.Copier)
if !ok {
return nil, errors.New("server-side copy not supported")
}
return do.Copy(ctx, src, remote)
}
I really like this pattern. The base fs.Fs stays small and clean. Backends opt in to extra capabilities by implementing additional interfaces, and the caller checks at runtime with a type assertion. No feature flags, no config booleans, no inheritance hierarchy to untangle.
Rclone defines about 20 of these optional interfaces: Purger, Mover, DirMover, PublicLinker, Abouter, UserInfoer, and more. Each one is a capability a backend may or may not have.
It also lets you extend the system without modifying existing code, which is the Open/Closed Principle without any of the ceremony. If you’ve worked with functional options in Go, this kind of extensible design will feel familiar.
Backend registration with init()
Rclone has 70+ backends, so how does it wire them all up without a giant switch statement somewhere? With Go’s init() function and a global registry.
Each backend package registers itself on import:
// In backend/s3/s3.go
func init() {
fs.Register(&fs.RegInfo{
Name: "s3",
Description: "Amazon S3 Compliant Storage Providers",
NewFs: NewFs,
Options: []fs.Option{
// ... provider-specific config options
},
})
}
The fs.Register function adds the backend to a global map:
var registry []*RegInfo
func Register(info *RegInfo) {
registry = append(registry, info)
}
Then in the main package, blank imports pull in every backend:
import (
_ "github.com/rclone/rclone/backend/s3"
_ "github.com/rclone/rclone/backend/drive"
_ "github.com/rclone/rclone/backend/dropbox"
_ "github.com/rclone/rclone/backend/azureblob"
_ "github.com/rclone/rclone/backend/b2"
// ... many more
)
Each blank import triggers the init() function, which registers the backend. By the time main() runs, all backends are available.
This is the same plugin pattern Go’s own database/sql package and image decoders use. It works, but go in with your eyes open: init() functions run before main(), so registration errors can be miserable to debug. The Go team has documented this pattern and its caveats.
Concurrency: parallel transfers with goroutines and semaphores
Syncing thousands of files one at a time would be painfully slow, so rclone runs transfers in parallel with goroutines and keeps them in check with a bounded semaphore.
The core idea fits in a page: a buffered channel used as a pool of tokens.
package main
import (
"context"
"fmt"
"sync"
)
type Semaphore struct {
tokens chan struct{}
}
func NewSemaphore(n int) *Semaphore {
return &Semaphore{
tokens: make(chan struct{}, n),
}
}
func (s *Semaphore) Acquire(ctx context.Context) error {
select {
case s.tokens <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func (s *Semaphore) Release() {
<-s.tokens
}
func main() {
sem := NewSemaphore(4) // max 4 concurrent transfers
var wg sync.WaitGroup
files := []string{"file1.txt", "file2.txt", "file3.txt", "file4.txt", "file5.txt", "file6.txt"}
for _, f := range files {
wg.Add(1)
go func(name string) {
defer wg.Done()
ctx := context.Background()
if err := sem.Acquire(ctx); err != nil {
fmt.Printf("cancelled: %s\n", name)
return
}
defer sem.Release()
fmt.Printf("transferring: %s\n", name)
// simulate transfer work here
}(f)
}
wg.Wait()
}
Rclone’s actual implementation is more sophisticated. There’s a transfer manager that tracks in-progress operations, retries with exponential backoff, and respects context cancellation throughout. But the bones are the same: buffered channels as semaphores, goroutines for parallelism, context.Context for cancellation.
The --transfers flag controls how many parallel transfers rclone runs (default is 4). The --checkers flag controls parallel hash-checking goroutines. Both use this same bounded concurrency approach.
If you want to understand how context cancellation works in practice, check out what is context in Go.
The io.Reader pipeline for encryption and streaming
Rclone supports client-side encryption via its crypt backend, so files are encrypted before they leave your machine. The implementation leans on io.Reader composition, and it’s my favourite bit of the codebase.
Rather than reading a file into memory, encrypting it, and uploading the result, rclone wraps readers:
// Simplified version of how rclone chains readers
func encryptedUpload(ctx context.Context, dst fs.Fs, plaintext io.Reader, key []byte) error {
// Wrap the plaintext reader with an encrypting reader
encrypted, err := newEncrypter(plaintext, key)
if err != nil {
return err
}
// The dst.Put call streams from the encrypted reader
// No full file buffered in memory
_, err = dst.Put(ctx, encrypted, objectInfo)
return err
}
The newEncrypter returns an io.Reader that encrypts bytes on the fly as they’re read. The upload function just sees an io.Reader. It doesn’t know or care that encryption is happening.
This is the io.Reader composability pattern that makes Go’s I/O model so effective. You can stack readers for encryption, compression, progress tracking, and bandwidth limiting without ever holding the whole file in memory.
Rclone’s crypt backend uses NaCl secretbox for file content encryption and scrypt for key derivation. File names get encrypted too, using EME (ECB-Mix-ECB) wide-block encryption.
FUSE filesystem mount
The feature that impressed me most is rclone mount, which presents remote cloud storage as a local filesystem. It uses bazil.org/fuse, a Go FUSE library, to pull this off.
The mount code implements FUSE callbacks that translate filesystem operations into fs.Fs method calls:
// Simplified FUSE read handler
func (f *File) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error {
// Open the remote object
reader, err := f.obj.Open(ctx, &fs.SeekOption{Offset: req.Offset})
if err != nil {
return err
}
defer reader.Close()
buf := make([]byte, req.Size)
n, err := io.ReadFull(reader, buf)
resp.Data = buf[:n]
return err
}
When you cat /mnt/remote/file.txt, the kernel sends a FUSE read request. Rclone translates that into an HTTP range request to your cloud provider, and the file streams back through the FUSE layer. Your application thinks it’s reading a local file. That the plumbing for this is the same fs.Fs interface as everything else tells you the abstraction was right.
This works on Linux and macOS. On Windows, rclone uses WinFsp instead.
What you can take away
Rclone is a big project, but nothing it does is exotic. Keep your core interfaces small and let implementations opt in to extra capabilities through type assertions. Use init() and blank imports when you want a plugin registry that grows without touching the core. Reach for buffered channels when you need bounded concurrency you can actually reason about. And chain io.Readers instead of buffering whole files; your memory graphs will thank you.
None of this requires cleverness, just consistency, and that’s why the source rewards reading. If you have 30 minutes this week, start in the fs package, find the interface definitions, then pick a backend you actually use and trace how it satisfies them.
You can browse the full source at github.com/rclone/rclone and the official docs at rclone.org.