Why MinIO chose Go for S3-compatible storage at scale
If you need S3-compatible object storage but don’t want to depend on AWS, MinIO is probably on your radar. It’s written entirely in Go and designed for high-performance workloads. What makes it fast? Let’s find out.
What is MinIO?
MinIO is an object storage server. It speaks the S3 protocol, so any application that works with Amazon S3 works with MinIO. You can run it on your laptop, in Kubernetes, or across multiple data centers.
The project is open-source under AGPLv3. It’s become the go-to choice for cloud-native storage, especially in multi-cloud setups where you want consistent APIs across providers.
Why Go for High-Performance Storage?
Go might seem like an odd choice for storage software. Isn’t C or Rust faster? In theory, yes. In practice, Go’s advantages outweigh the raw speed difference.
MinIO benefits from Go’s:
- Excellent concurrency model with goroutines
- Simple deployment (single binary, no dependencies)
- Strong standard library for networking
- Fast compilation for rapid iteration
The goroutine model is particularly important. Storage systems handle thousands of concurrent connections. Each client request can be a lightweight goroutine instead of a heavy thread.
Connecting to MinIO with Go
Here’s how to interact with MinIO from a Go application:
package main
import (
"context"
"fmt"
"log"
"strings"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
ctx := context.Background()
// Connect to MinIO
client, err := minio.New("localhost:9000", &minio.Options{
Creds: credentials.NewStaticV4("minioadmin", "minioadmin", ""),
Secure: false,
})
if err != nil {
log.Fatal(err)
}
// Create a bucket
bucketName := "my-bucket"
err = client.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{})
if err != nil {
// Check if bucket already exists
exists, errExists := client.BucketExists(ctx, bucketName)
if errExists != nil || !exists {
log.Fatal(err)
}
}
// Upload an object
content := "Hello, MinIO!"
_, err = client.PutObject(ctx, bucketName, "greeting.txt",
strings.NewReader(content), int64(len(content)),
minio.PutObjectOptions{ContentType: "text/plain"})
if err != nil {
log.Fatal(err)
}
fmt.Println("Object uploaded successfully")
}
Notice how we pass context as the first parameter. This is standard practice in Go and enables proper cancellation and timeout handling.
Performance Tuning with Concurrent Uploads
For large files, MinIO supports multipart uploads. You can tune the concurrency:
package main
import (
"context"
"log"
"os"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func uploadLargeFile(filePath, bucketName, objectName string) error {
ctx := context.Background()
client, err := minio.New("localhost:9000", &minio.Options{
Creds: credentials.NewStaticV4("minioadmin", "minioadmin", ""),
Secure: false,
})
if err != nil {
return err
}
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
stat, err := file.Stat()
if err != nil {
return err
}
// Upload with concurrent parts
_, err = client.PutObject(ctx, bucketName, objectName, file, stat.Size(),
minio.PutObjectOptions{
ContentType: "application/octet-stream",
NumThreads: 4, // Concurrent upload threads
PartSize: 64 * 1024 * 1024, // 64MB parts
})
return err
}
Running MinIO in Kubernetes
MinIO shines in Kubernetes environments. Here’s a minimal deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: minio
spec:
replicas: 1
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
spec:
containers:
- name: minio
image: minio/minio:latest
args:
- server
- /data
- --console-address
- ":9001"
env:
- name: MINIO_ROOT_USER
value: "minioadmin"
- name: MINIO_ROOT_PASSWORD
value: "minioadmin"
ports:
- containerPort: 9000
- containerPort: 9001
For production, use the MinIO Operator which handles distributed deployments across multiple nodes.
Performance Tips
MinIO can saturate a 100Gbps network link. To get there:
- Use SSDs - MinIO’s erasure coding is CPU-bound, not I/O-bound on spinning disks
- Enable direct I/O - Bypasses the kernel page cache for large objects
- Tune your Go client - Increase
MaxIdleConnsPerHostin your HTTP transport - Use distributed mode - Spread load across multiple servers
// Custom HTTP transport for better performance
transport := &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 100,
IdleConnTimeout: 90 * time.Second,
}
client, err := minio.New("localhost:9000", &minio.Options{
Creds: credentials.NewStaticV4("minioadmin", "minioadmin", ""),
Secure: false,
Transport: transport,
})
When to Use MinIO
MinIO fits well when you need:
- S3 compatibility without vendor lock-in
- On-premises object storage
- Multi-cloud storage with consistent APIs
- High-throughput data pipelines
It’s not the right choice if you need a general-purpose filesystem. MinIO is optimized for object storage patterns: write once, read many.
Wrapping Up
MinIO proves that Go can power high-performance infrastructure. The combination of goroutines, simple deployment, and a solid standard library makes Go an excellent choice for storage systems.
If you’re building applications that need object storage, give MinIO a try. The official documentation covers distributed deployments and advanced configuration.