Retry-Safe Cloud Operations in go-bot
Creating a manual Alibaba ECI image cache incurs temporary-resource and snapshot charges, and the operation can take minutes. If your process dies halfway through, the next run has to answer an awkward question: did I already start one? Guess wrong in one direction and you pay for a duplicate. Guess wrong in the other and you wait for a cache that was never created.
xyc999-ai/go-bot at the revision reviewed for this post answers that question in cmd/buildeci/workflow.go. The pattern is useful for any Go program that makes a side-effecting remote call it cannot safely repeat. The repository is a self-hosted agent with three binaries (host, controller, and buildeci) plus a memory package for the agent’s long-term notes. This post focuses on that supporting code rather than the agent loop.
Three binaries, three different CLI shapes
cmd/host/main.go is the minimal case. It calls flag.Parse(), loads config with config.LoadHost(), builds a context with signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM), and hands both to host.Run. The exit codes are distinct: os.Exit(2) for a config failure and os.Exit(1) for a runtime failure. A supervisor can therefore distinguish invalid configuration from a failure after startup.
cmd/controller/main.go and cmd/buildeci/main.go both implement subcommands without a CLI framework. The controller selects a command with a small piece of manual parsing:
command := "serve"
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
command, args = args[0], args[1:]
}
That gives you controller serve, check, doctor, host, and web-available, while a bare controller falls back to serve. The remaining arguments go to one flag.FlagSet. buildeci instead requires an explicit prepare, cache, or check command, then builds a flag set with flag.NewFlagSet("buildeci "+command, flag.ContinueOnError), so usage output includes the selected command. For the standard library version of this, we covered the Go flag package; if you’d rather not hand-roll it, there’s the Cobra approach.
Strict YAML: reject unknown fields and extra documents
cmd/buildeci/config.go reads an optional tuning file with three fields. The decoder deliberately rejects schema mistakes:
d := yaml.NewDecoder(bytes.NewReader(data))
d.KnownFields(true)
if d.Decode(&t) != nil || d.Decode(new(any)) != io.EOF {
return options{}, errors.New("invalid buildeci tuning YAML")
}
Two checks are working together. KnownFields(true) makes gopkg.in/yaml.v3 fail when a mapping key does not correspond to a field in the target struct, so writing size_gb instead of size_gib is an error rather than a setting that silently does nothing. The second Decode into a throwaway any must then return io.EOF. YAML can contain multiple documents separated by ---; demanding EOF after the first decode rejects a second document.
Here’s the same idea as a reusable helper (illustrative, not copied from the repo):
package strictyaml
import (
"bytes"
"errors"
"fmt"
"io"
"gopkg.in/yaml.v3"
)
// DecodeOne decodes exactly one YAML document into out and rejects
// unknown fields.
func DecodeOne(data []byte, out any) error {
d := yaml.NewDecoder(bytes.NewReader(data))
d.KnownFields(true)
if err := d.Decode(out); err != nil {
return fmt.Errorf("decode yaml: %w", err)
}
if err := d.Decode(new(any)); !errors.Is(err, io.EOF) {
return errors.New("expected exactly one YAML document")
}
return nil
}
go-bot collapses both failure modes into one message, discarding the underlying parse error. That keeps this three-field path simple, but a larger configuration surface should preserve the decoder error and its line information.
The same file validates the push image without trying to reproduce URL parsing in a hostname regex. It parses the registry portion as a URL and checks the result:
u, urlErr := url.Parse("https://" + server)
The check requires urlErr == nil, u.User == nil (no embedded credentials), u.Host == server, a non-empty u.Hostname(), and a . or : in the server so it cannot be a bare path segment. Comparing u.Host with the original string confirms that net/url treated the entire value as the authority rather than reinterpreting part of it as user information or a path.
Making a cloud create call safe to retry
execute in cmd/buildeci/workflow.go has the most interesting Go in the repo. It has to create an image cache that costs money and takes minutes, and survive being killed partway.
It starts by deriving a deterministic directory name from the operation’s identity:
identity, _ := json.Marshal([]any{o.Region, o.Cloud.AccessKeyID, req.Image, req.VSwitchID,
req.SecurityGroupID, req.RegistryServer, req.SizeGiB, req.RetentionDays, name})
key := sha256.Sum256(identity)
store, err := localstate.Open(filepath.Join(root, hex.EncodeToString(key[:16])))
Marshalling a []any to JSON before hashing avoids the ambiguous encodings that delimiter-based string concatenation can create. Change the region or image and the encoded identity changes, producing a different directory and therefore a different in-flight operation.
That directory holds operation.json: a clientToken (a uuid.NewString() from google/uuid), the image, an optional digest, an optional cache ID, and a publishing bool. The token goes to the cloud API as req.ClientToken. Alibaba’s CreateImageCache documentation defines it as the request’s idempotency token, so retrying the same operation does not create a duplicate cache.
The publishing flag handles the part the client token can’t. Pushing an image to a registry has no dedup token, so the code records its intent on disk before doing the thing:
if op.Publishing {
return result{}, errors.New("previous image publication is uncertain; ...")
}
op.Publishing = true
if err := store.Write(filename, op); err != nil {
return result{}, err
}
digest, err := publish(ctx, req.Image)
If the process dies during the push, the next run sees publishing: true, refuses to guess, and tells the operator to inspect the registry. Retrying blindly could move a tag that another system depends on; skipping the push could leave the cache pointed at an image that was never published. The flag is cleared only after the returned digest passes imagebuild.ValidDigest and the updated operation is written to disk.
Polling runs under context.WithTimeout(ctx, o.Wait) with a time.Ticker whose interval main.go supplies as three seconds. Passing the interval into execute lets tests use a much shorter delay. The same idea appears in cacheAPI, a two-method interface declared at the point of use:
type cacheAPI interface {
CreateImageCache(context.Context, aliyun.CreateImageCache) (string, error)
DescribeImageCache(context.Context, string) (aliyun.ImageCache, error)
}
main.go supplies the concrete Aliyun client; tests can pass a small fake that implements those two methods.
The controller CLI talks to the controller over HTTP
controller host status|start|stop|delete --user name doesn’t reach into the manager. It builds an HTTP request against the controller’s own listen address:
method, route := http.MethodPost, "/v1/host/"+action
switch action {
case "status":
method, route = http.MethodGet, "/v1/host"
case "delete":
method, route = http.MethodDelete, "/v1/host/instance"
case "start", "stop":
default:
return errors.New("unknown host action")
}
The account name rides in controller.UserHeader, and the body comes back through io.ReadAll(io.LimitReader(res.Body, 1<<20)), a 1 MiB cap so a misbehaving server can’t balloon the CLI’s memory. Anything at or above status 300 becomes a Go error carrying the trimmed body.
One limitation is that the request is built with http.NewRequest, not http.NewRequestWithContext. The client timeout still bounds the call, but Ctrl-C does not cancel an in-flight host command. A context-aware request would make signal cancellation immediate. We walked through context-driven CLI cancellation in the Agent Tincan post.
The doctor path does propagate cancellation: it wraps the signal context with context.WithTimeout(ctx, 30*time.Second) before running preflight checks.
Memory limits counted in runes
harness/memory/store.go keeps two files per account: USER.md capped at 1375 characters and MEMORY.md capped at 2200, each described by a memoryFile value holding a filename, a display label, and a limit. Entries inside a file are joined with the separator "\n§\n".
The cap is enforced with utf8.RuneCountInString, not len:
body := strings.Join(entries, separator)
chars := utf8.RuneCountInString(body)
if chars > file.limit {
// reject the operation
}
Counting runes makes this application-defined character cap consistent across UTF-8 text: a Chinese character is three bytes but one rune. Prompt() reports the same count back to the model as "%s (%d/%d characters)", so the agent sees the same budget that Apply() enforces.
Concurrency is a plain sync.Mutex on the Store. The Host shares one store across its session runtimes, and both Prompt() and Apply() take the lock, serializing reads and writes in that process. Prompt() also degrades instead of failing outright: it accumulates per-file errors and returns errors.Join(problems...), so an invalid USER.md does not suppress valid notes from MEMORY.md.
The most reusable idea here is the publishing flag. Instead of assuming an operation either happened or did not, go-bot records intent first and represents an uncertain result explicitly. That is a useful trade whenever retrying a side effect can move a tag or incur another charge.
If you want to see how another self-hosted Go agent arranges its CLI and runtime layers, the Agentbox writeup covers overlapping ground from a different direction.