base64/base64.go in MoeForge/moego implements RFC 4648 behind an API that closely follows encoding/base64. Standard and URL alphabets use assembly kernels where the build and CPU allow it; custom alphabets, short inputs, unsupported amd64 CPUs, other architectures and purego builds fall back to the standard library.

Three parts of the design are useful beyond base64: build-tagged assembly with a portable implementation, runtime AVX2 detection on amd64, and an optional allocator that can skip clearing pointer-free result buffers before they are filled.

Where the SIMD path actually starts

The package doesn’t vectorize everything. base64/codec_amd64.go sets two floors:

// Inputs from these lengths use the AVX2 kernels: one 24-byte encode block or
// one 32-byte decode block.
const encodeMin, decodeMin = 24, 32

Encode checks the floor and the alphabet mode before anything else:

if len(src) < encodeMin || e.mode == 0 {
	e.stdlib.Encode(dst, src)
	return
}

mode is a uint8 on the Encoding struct: 0 for a custom alphabet, 1 for standard, 2 for URL. NewEncoding sets it by comparing the alphabet string against the two package constants. Custom alphabets never reach assembly, because the kernels are written against fixed character tables, and the struct carries a one-byte flag that says whether the fast path is legal for this encoding.

After the floor check, the package asks the kernel how much input it consumed:

n := encodeBlocks(dst, src, &e.alphabet)
if len(src)-n >= 3 {
	e.stdlib.Encode(dst[n/3*4:], src[n:])
	return
}
e.encodeTail(dst[n/3*4:], src[n:])

encodeBlocks returns how many source bytes the kernel consumed. The n/3*4 arithmetic converts that into a destination offset, since three input bytes become four output bytes. Three or more source bytes left over, and the stdlib finishes the whole groups. Fewer, and encodeTail handles one or two bytes by hand.

If encodeBlocks returns 0, len(src)-n >= 3 is true for every input that reached this point, so e.stdlib.Encode handles the entire input. On amd64 this is how a machine without usable AVX2 reaches the portable implementation without executing an unsupported instruction.

Feature detection as a package-level variable

codec_amd64.go is gated with //go:build amd64 && !purego. Detection looks like this:

var haveAVX2 = supportsAVX2()

// supportsAVX2 checks both CPU features and operating-system YMM state support.
//
//go:noescape
func supportsAVX2() bool

The detection runs during package initialization and lands in a package-level bool, so each operation checks the stored result instead of executing CPUID again.

And the comment names the mistake people make: the AVX2 CPUID bit alone isn’t enough. The OS has to have enabled YMM register state in XCR0, or your 256-bit instructions fault on a CPU that supposedly supports them. Both checks, or neither.

Every wrapper has the same two-line shape:

func encodeBlocks(dst, src []byte, alphabet *[64]byte) int {
	if !haveAVX2 {
		return 0
	}
	return encodeAVX2(dst, src, alphabet)
}

decodeBlocks and validBlocks use the same pattern, returning 0 without AVX2. Zero means “consumed nothing,” and the Go caller takes over. The three kernel declarations carry //go:noescape, which tells escape analysis that their pointer arguments do not escape through the assembly call. That can help callers keep eligible backing storage on the stack, but it is not by itself a guarantee that an allocation will be stack-based.

The !purego half of the constraint removes this file from a purego build. The same snapshot also has an arm64 implementation: codec_arm64.go selects NEON kernels at compile time, with 48-byte encode and 64-byte decode thresholds. Go is also adding architecture-specific SIMD APIs; this archsimd overview covers that separate approach.

Skipping the zeroing with runtime.mallocgc

The most version-sensitive part is base64/alloc_linkname.go:

//go:build (amd64 || arm64) && !purego && !moego_zeroalloc

package base64

import "unsafe"

//go:linkname mallocgc runtime.mallocgc
func mallocgc(size uintptr, typ unsafe.Pointer, needzero bool) unsafe.Pointer

func makeNoZero(n int) []byte {
	if n == 0 {
		return []byte{}
	}
	return unsafe.Slice((*byte)(mallocgc(uintptr(n), nil, false)), n)
}

make([]byte, n) must return zeroed bytes. Passing needzero=false lets the runtime avoid clearing a reused pointer-free allocation before code immediately overwrites it, removing work that is unnecessary when every byte is initialized before publication.

The file’s comments state both caveats. On the linkname: “Go keeps this linkname target stable because widely used packages depend on it. typ is nil, so the memory is treated as pointer-free.” On the caller’s obligation: “Callers must write all n bytes, or clear the unwritten ones, before the slice becomes visible.”

The nil typ marks the allocation as pointer-free, so the garbage collector does not scan its contents. That is suitable for bytes. It would be invalid for values containing pointers: the collector would not see those references and could reclaim the objects they should keep alive. Garbage collection in Go explains why pointer metadata matters.

The important companion is base64/alloc_zero.go, which carries the inverse constraint:

//go:build (!amd64 && !arm64) || purego || moego_zeroalloc

package base64

func makeNoZero(n int) []byte { return make([]byte, n) }

Building with -tags moego_zeroalloc selects an ordinary allocation without changing source. That escape hatch matters because runtime.mallocgc is an internal runtime detail even though the Go runtime currently preserves its signature for packages that access it through linkname.

EncodeToString is the consumer visible in the excerpt: it allocates with makeNoZero(e.EncodedLen(len(src))), encodes into the buffer, then hands it to ownedString. The doc comment advertises the result as “an immutable encoding of src using one allocation.”

Bounds-check hints and copy-on-configure

Encode indexes _ = dst[e.EncodedLen(len(src))-1] before entering the fast path. The one- and two-byte tail branches similarly establish the highest index they will need. Besides producing the documented panic for a short destination, these checks give the compiler facts it can use to eliminate redundant bounds checks on later writes.

Configuration uses a value receiver:

func (e Encoding) WithPadding(padding rune) *Encoding {
	e.stdlib = e.stdlib.WithPadding(padding)
	e.padding = padding
	return &e
}

The receiver is Encoding, not *Encoding. The method updates a copy and returns its address; Strict() does the same. This mirrors the standard library’s configuration API and leaves the original encoding unchanged. The configured values are then safe to share between concurrent callers.

Compatibility is handled with an alias rather than a lookalike type:

type CorruptInputError = std.CorruptInputError

Code doing err.(base64.CorruptInputError) continues to use the standard library’s error type after an import-path change. A basic use looks like this:

package main

import (
	"fmt"

	"github.com/MoeForge/moego/base64"
)

func main() {
	payload := []byte("some media bytes that are long enough to hit the SIMD path")

	// One allocation, per the EncodeToString doc comment.
	s := base64.StdEncoding.EncodeToString(payload)
	fmt.Println(s)

	// AppendEncode grows dst with slices.Grow and encodes in place.
	buf := make([]byte, 0, 128)
	buf = base64.RawURLEncoding.AppendEncode(buf, payload)
	fmt.Println(string(buf))
}

AppendEncode grows through slices.Grow from the standard slices package, so a destination with enough spare capacity can reuse its backing array.

Differential testing against a reference implementation

Hand-written assembly needs a reference implementation. The repository’s bench/json/cmd directory contains two utilities for comparing JSON behavior. They target the separate ojson package, but parts of the harness design transfer to binary codecs too.

cmd/oracle/main.go imports gjson and sjson and runs hardcoded case groups selected by os.Args[1]. The three in the excerpt are ins, del, and val, each a dense table of edge cases: empty documents, duplicate keys, escaped dots in paths, negative array indexes. Results print through a res helper that includes math.Float64bits(r.Num), so float comparisons happen bit-for-bit instead of through whatever %v decides to do today.

cmd/gq/main.go is the open-ended sibling. One JSON array of strings per line on stdin, first element naming the operation: get, set, setraw, del, esc, lines, less, methods, and about two dozen more listed in the header comment. That header also says to “run it under timeout and ulimit,” which reads like the input comes from a fuzzer.

One useful detail is that reference-library panics become output rather than terminating the utility:

func safe(f func() string) (s string) {
	defer func() {
		if e := recover(); e != nil {
			s = fmt.Sprintf("PANIC(%v)", e)
		}
	}()
	return f()
}

The named return plus recover preserves the case and records what the reference did. In the repository’s generated differential tests, a reference panic is not treated as matching behavior: exact comparison is skipped, then ojson is still checked to ensure that it does not panic.

And the scanner is sized for inputs a generator will actually produce:

sc := bufio.NewScanner(os.Stdin)
sc.Buffer(make([]byte, 1<<20), 1<<26)

bufio.Scanner defaults to a maximum token size of 64 KiB. This utility supplies a 1 MiB initial buffer and a 64 MiB ceiling so it can accept much larger JSONL cases. It should also check sc.Err() after the loop; as written, an oversized token or input error stops scanning without being reported.

The get case also checks gjson.Get against gjson.GetBytes on the same input, appending " BYTES " and the second result only when the two disagree. The consistency check is encoded in the output, so an external comparison can spot the divergence without a separate assertion.

The same principle applies to the SIMD codec: fallback design prevents unsupported instructions from breaking normal execution, while differential and fuzz tests verify that optimized and reference paths return the same bytes and errors.