A worker stalls. The system writes it off. Thirty seconds later it wakes up, finishes its job, and tries to publish results into a run that was supposed to be dead. That failure mode is the reason fencing exists, and it is central to Skein’s design.

Skein is a swarm orchestrator in Go that runs a DAG of tasks and stops admitting results once someone pulls the emergency brake. The interesting part is the small set of primitives behind that behaviour: a monotonic epoch in a file, a LeaseValid predicate, stop checks around task publication, and a CLI that turns engine outcomes into distinct exit codes.

Here’s how those pieces look in the source, and what’s worth stealing.

The epoch file and the temp-file-and-rename pattern

The fencing state lives in one file named EPOCH inside the state directory, .skein by default. internal/engine/epoch.go reads it in two steps.

CurrentEpoch(stateDir) first calls readStop(stateDir). If it reads a valid stop record with a non-zero epoch, that value wins. Otherwise it falls through to storedEpoch, which reads the EPOCH file: a missing file returns 1, and a file that fails strconv.ParseUint or parses to 0 returns errors.New("invalid epoch file"). engine.Run checks for the existence of STOP before this, so even a malformed stop file prevents a normal run from starting.

The ordering is the point. Once a stop is latched, the epoch everything else sees is the one pinned in the stop record, not whatever a later run would prefer to write.

Writing it uses the temp-file-plus-rename dance, which is worth burning into muscle memory for any Go program that persists small state:

// Illustrative version of the pattern used in internal/engine/epoch.go.
func writeAtomic(dir, name, contents string) error {
	temp, err := os.CreateTemp(dir, ".tmp-*")
	if err != nil {
		return err
	}
	path := temp.Name()
	defer os.Remove(path) // no-op once the rename succeeds

	if _, err := io.WriteString(temp, contents); err != nil {
		temp.Close()
		return err
	}
	if err := temp.Sync(); err != nil { // commit file contents before publishing
		temp.Close()
		return err
	}
	if err := temp.Close(); err != nil {
		return err
	}
	return os.Rename(path, filepath.Join(dir, name))
}

Three details carry the weight. CreateTemp puts the scratch file in the same directory, avoiding a cross-filesystem rename. Sync commits the file contents before the rename publishes the new name. The deferred Remove cleans up on error and becomes harmless after a successful rename.

There are limits to the pattern. Go documents Rename as non-atomic on non-Unix platforms, and the code does not sync the containing directory after the rename, so this is not a universal guarantee that the directory entry survives sudden power loss. It does prevent readers from observing an in-place partial write on platforms with atomic replacement. The real version uses fmt.Fprintf(temp, "%d", epoch) and a .epoch-* prefix, and storedEpoch fails rather than silently turning corrupt state back into epoch 1.

LeaseValid: a local fencing check

Fencing tokens are the standard answer to the zombie-worker problem. Skein’s implementation is one function in epoch.go:

func LeaseValid(lease Lease, currentEpoch uint64) bool {
	return lease.Epoch == currentEpoch && lease.RunID != "" && lease.Sequence != 0 && lease.TaskID != "" && lease.Pass > 0
}

Five conditions, one boolean, no locks or consensus protocol. A lease carries the epoch it was issued under, so it fails when checked against a newer epoch. The other four conditions are zero-value guards: a bare Lease{} can never pass. Notice that the caller supplies currentEpoch; the predicate does not read shared state itself.

Real runs also persist leases as audit artifacts. Run passes its per-run directory to WriteLease, which creates <runDir>/leases/ with mode 0o700, marshals to JSON, and writes <dataName>.<pass>.lease with mode 0o600. The pass number keeps a task’s multi-pass records separate. These writes use os.WriteFile; unlike the epoch path, they are neither temp-file replacements nor explicitly synced.

The engine makes the same commitment in its Options doc comment in internal/engine/engine.go:

// InMemory is reserved for scheduler benchmarks. It skips process artifacts,
// lease files, locks and stop-file polling while retaining fencing checks.
InMemory bool

The in-memory benchmark path still creates and validates leases in memory, but it deliberately skips process artifacts, lease-file I/O, locks, and stop-file polling. Its numbers therefore cover scheduler bookkeeping, not the complete cost of a real run.

Fail-closed startup and the CLI exit-code contract

engine.Run won’t start on a stopped swarm. After creating the state directory:

if !options.InMemory && StopRequested(options.StateDir) {
	return Report{}, ErrStopped
}

ErrStopped is a package-level sentinel, errors.New("swarm stopped"), and cmd/skein/main.go checks it with errors.Is before it checks anything else:

report, err := engine.Run(ctx, manifest, engine.Options{...})
if errors.Is(err, engine.ErrStopped) {
	fmt.Fprintln(os.Stderr, "swarm is stopped; inspect it, then run skein reset before a new run")
	return 4
}

For why errors.Is beats == here, see Go Error Chains.

Every outcome gets its own exit code: 2 for bad usage, 4 for a stopped swarm, 3 for engine.Failed, 1 for anything else, 0 on success. A report.Status of Stopped also exits 4, so the latch surfaces whether it fires at startup or mid-run.

main is one line, os.Exit(run(os.Args[1:])), which keeps every defer inside run alive until the work is done. Each subcommand builds its own flag.NewFlagSet(name, flag.ContinueOnError) and returns 2 on a parse error rather than letting the flag package call os.Exit out from under it. Standard library subcommands, no framework. More on that in Go flag Package.

Cancellation is equally plain:

ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer cancel()

Strict manifest decoding

Manifests are JSON, and internal/engine/manifest.go is deliberately unforgiving:

decoder := json.NewDecoder(file)
decoder.DisallowUnknownFields()
var manifest Manifest
if err := decoder.Decode(&manifest); err != nil {
	return Manifest{}, fmt.Errorf("decode manifest: %w", err)
}
if decoder.More() {
	return Manifest{}, errors.New("decode manifest: multiple JSON values")
}

DisallowUnknownFields turns a typo like "needz" into a loud error instead of a dependency that silently disappears. In this implementation, the decoder.More() check also rejects another top-level JSON value. The more conventional stream check is a second Decode that must return io.EOF, because More is documented for elements inside an array or object. json.Unmarshal, meanwhile, already rejects trailing non-whitespace after one JSON value.

Validate handles the graph before a task runs. Version must be 1. The task list cannot be empty. Workers has to land in [0, 65536], limits cannot be negative, every block needs a non-blank Argv[0], and environment entries must contain = without starting with one. It also rejects unknown blocks and dependencies, duplicate task IDs and dependencies, cycles, conflicting stdin sources, and invalid pass or timeout values. Failure returns before the scheduler starts.

Nil-receiver methods on the event logger

internal/engine/logger.go has a trick that’s easy to skim past. The eventLogger methods work on a nil pointer:

func (logger *eventLogger) emit(kind, taskID string, pass int, epoch uint64, status Status, message string) {
	if logger == nil {
		return
	}
	logger.mu.Lock()
	defer logger.mu.Unlock()
	...
}

close() does the same. That’s what lets engine.Run call scheduler.logger.close() on the prepareInputs error path even when --no-events left the logger nil. Methods on pointer receivers dispatch statically, so a nil *eventLogger is a legal receiver as long as the method checks. The payoff is a caller with no if logger != nil scattered through it.

The logger also carries a sticky err. Once a write fails, every later emit bails immediately and close reports the first error it saw, so one bad write doesn’t turn into a thousand duplicate failures. Writes go through a 64KB bufio.Writer. With SyncEvents on, each event triggers buffer.Flush() followed by file.Sync(). That’s the durability knob behind skein run --sync, and also why it defaults to off.

Where the concurrency happens

The scheduler in engine.go is a channel-based worker pool with carefully sized buffers:

ready: make(chan string, len(manifest.Tasks)),
done:  make(chan taskExecution, WorkerCount(manifest, options.Workers)),

Sizing ready to the full task count means seeding the roots can never block. Run walks manifest.Tasks, builds a dependents map from each task’s Needs, sets remaining to len(task.Needs), and pushes every zero-dependency task onto ready before a worker exists. done is sized to the worker count, so a worker that finishes always has somewhere to drop its result.

Shared counters are sync/atomic types declared directly on the struct, sequence atomic.Uint64 and stopped atomic.Bool, rather than atomic.AddUint64 against a plain field. And if WorkerCount comes back below one, the engine falls back to runtime.GOMAXPROCS(0).

The executor seam

Tasks run through an Executor interface, with ProcessExecutor{} filled in when Options.Executor is nil. There’s a function adapter in the http.HandlerFunc mold, and cmd/skein-bench/main.go uses it to time the scheduler with no processes involved:

executor := engine.ExecutorFunc(func(ctx context.Context, lease engine.Lease, task engine.Task, block engine.Block, stdin, stdout string) error {
	return nil
})

The lease travels to the executor alongside the task and block. ProcessExecutor also exposes its fields as SKEIN_* environment variables, so a cooperating workload can pass the token to a storage system that enforces fencing. The benchmark builds manifests in three shapes, independent, chain, and fanout, runs Skein with InMemory: true, and compares against an explicitly unstructured goroutine baseline. Each sample is bracketed by runtime.GC() and runtime.ReadMemStats, and the Go baseline runs ten inner repetitions per sample.

For a single-host runner, the useful combination is an epoch, a lease that carries it, explicit stop checks, and a start path that returns a sentinel while the latch exists. Skein validates a lease before execution, checks stop state before publishing the final output, and rejects completion after its scheduler has latched a stop. That protects Skein’s own final output and report state; it cannot undo arbitrary side effects performed inside a command. If a task writes to an external database or service, that destination must compare and enforce the fencing token too. For more on fencing agents specifically, see CrabTrap.