A case for Go code generation: testify

Discussion(s): lobste.rs

If you’ve been using Go for a while, you’re probably familiar with the testing library stretchr/testify. It makes it easy to write tests and provides several assertion functions, such as Equal, Contains, Greater, and many more.

Assertion functions behaves differently depending on the scope. For example, when called from the assert or require package. The former logs the error and continue, while the latter stops the test immediately. Both packages offer the same function signatures. Assertions are also called as methods from a suite struct, which wraps the *testing.T pointer. And each assertion has a formatting counterpart with a *f suffix and additional (msg string, params ...interface{}) parameters at the end.

This leaves us with three dimensions of assertion functions: Static functions, suite methods and formatted. A total of eight different versions of each function:

This is an interesting case for code generation. It seems like we can implement all functions as wrapper to a canonical assertion implementation. It’s mostly repetitive or boilerplate code that would be otherwise tedious to write by hand. And, unsurprisingly, that’s exactly what the package does.

The rest of this post is a brain-dump as I go through the code, trying to understand how testify works internally, and how they organized their code to use code generation.

Example: Equal function

Let’s first look at the Equal function, and how the behavior differs on assert and require:

func Test_RequireVsAssert(t *testing.T) {
	assert.Equal(t, "a", "b") // assertion fails but tests continue
	require.Equal(t, "a", "b") // assertion fails and test stops
	panic("This panic shouldn't happen")
}

The underlying implementation of .Equal is the same. Comparing two values and providing a text output should not change on assert or require implementations. What changes here is the flow of the test. It either stops, or records the error and moves on.

func Equal(t TestingT, expected, actual interface{}, msgAndArgs ...interface{}) bool {
	if h, ok := t.(tHelper); ok {
		h.Helper()
	}
	if err := validateEqualArgs(expected, actual); err != nil {
		return Fail(t, fmt.Sprintf("Invalid operation: %#v == %#v (%s)",
			expected, actual, err), msgAndArgs...)
	}

	if !ObjectsAreEqual(expected, actual) {
		diff := diff(expected, actual)
		expected, actual = formatUnequalValues(expected, actual)
		return Fail(t, fmt.Sprintf("Not equal: \n"+
			"expected: %s\n"+
			"actual  : %s%s", expected, actual, diff), msgAndArgs...)
	}

	return true
}

The assert.Equal function where the canonical implementation of Equal.

I’ll spare the details about how the comparison works beyond the *ObjectsAreEqual* function. Let’s focus on the surrounding code, and how it’s invoked elsewhere in the code-base.

An important thing to note here is that the assert.* functions return a bool. This is leveraged by their require.* counterpart.

require.Equal looks exactly what you would expect:

func Equal(t TestingT, expected interface{}, actual interface{}, msgAndArgs ...interface{}) {
	if h, ok := t.(tHelper); ok {
		h.Helper()
	}
	if assert.Equal(t, expected, actual, msgAndArgs...) {
		return
	}
	t.FailNow()
}

It wraps the assert.Equal function and calls t.FailNow() in case the assertion failed (returned false).

The same is true for every single assertion function. Rather than writing the boilerplate function signatures for each assertion, let’s see how they used templates to generate function bodies.

Assert function templates

require package

Every function in the require package has the same boilerplate code. It must (1) call assert.* function; and (2) fail the test if the assertion returns false.

{{.Comment}}
func {{.DocInfo.Name}}(t TestingT, {{.Params}}) {
	if h, ok := t.(tHelper); ok { h.Helper() }
	if assert.{{.DocInfo.Name}}(t, {{.ForwardedParams}}) { return }
	t.FailNow()
}

Note that there is also a *Helper()* function invoked here. They used for internal testing purposes, so we can ignore this for the sake of the length of this blog post.

Suite methods

Since these assertions are methods in the suite struct, they need a receiver, which will be of type *Assertion on both .Assert() and .Require() structs.

{{.CommentWithoutT "a"}}
func (a *Assertions) {{.DocInfo.Name}}({{.Params}}) {
	if h, ok := a.t.(tHelper); ok { h.Helper() }
	{{.DocInfo.Name}}(a.t, {{.ForwardedParams}})
}

format functions

They only used format functions templates for assert package. All other formats are generated based on the assert package functions, including the format prefixed ones. This is why you will only find a formatting template in the assert package.

{{.CommentFormat}}
func {{.DocInfo.Name}}f(t TestingT, {{.ParamsFormat}}) bool {
	if h, ok := t.(tHelper); ok { h.Helper() }
	return {{.DocInfo.Name}}(t, {{.ForwardedParamsFormat}})
}

Note the f suffix at the end of the function name declaration.

Combining these three templates, you can generate all the function layouts provided by testify:

1. Use *f template to generate format functions based on the canonical assert implementations.
2. Use require template to generate wrapper functions assert functions.
3. Use suite method template to generate functions with *Assertions receiver that wraps calls to assert.* and require.* functions.

Code generator

There is a single generator file under _codegen folder, which is used by all function types. It gets called from each package using flag parameters. For example, the command used to generate require.* functions:

//go:generate sh -c "cd ../_codegen && go build && cd - && ../_codegen/_codegen -output-package=require -template=require.go.tmpl -include-format-funcs"

Code generation scripts are rarely simple. They usually require a large context in your mind to reason about them. By providing flags to the generator, they’ve abstracted the problem of providing a specific generation for each format. Every template receives the same data structure to determine how the parameters are forwarded. This is important to keep the generator agnostic.

There is much more to how the code generation script works, but I don’t think it’s useful to walk through that logic here. The most important part to understand their approach is to look at this struct:

type testFunc struct {
	CurrentPkg string
	DocInfo    *doc.Func
	TypeInfo   *types.Func
}

Parsing the pkg input file (which by default points to assert) outputs a slice of testFunc. Each one of these elements is used to generate one counterpart function based on a template. The struct offers some helper functions to build arguments and params:

func (f *testFunc) Qualifier(p *types.Package) string
func (f *testFunc) Params() string
func (f *testFunc) ForwardedParams() string
func (f *testFunc) ParamsFormat() string
func (f *testFunc) ForwardedParamsFormat() string
func (f *testFunc) Comment() string
func (f *testFunc) CommentFormat() string
func (f *testFunc) CommentWithoutT(receiver string) string

You can easily guess what they do based on the names.

The code generation is complex, but it’s a hidden complexity that doesn’t creep into the rest of the library code. Once the parsing code is built, it’s unlikely that it needs to be changed.

An alternative approach

Code generation is fun, but more often than not when I ask myself if it’s really needed the answer is no. Code generations avoids writing many lines of boilerplate code. But it leaves a complex generator function, often untested, that uses a messy ast logic.

Whenever I feel the need to use code generation, I ask myself if the problem I’m trying to solve is an artificial problem. If it’s a real problem, can it be solved in other ways? Let’s apply this reasoning to the testify package:

There are three main reasons code generation is used here. The same assert functions can be offered with different flavors:

1. Static functions vs methods: providing a function or a method on struct that wraps the *testing.T is an artificial problem. An opinionated approach would be to provide only methods.
2. With formatting and without formatting: this seems like a legitimate problem, but there are other solution rather than offering a different function suffix.
3. Fail fast: also a legitimate feature, but can be solved with simpler constructs than providing different package scoped functions.

The reason why formatting functions (2) were added seems to originate from a go vet error in code-bases using formatting arguments in testify messages https://github.com/stretchr/testify/issues/339.

As a thought exercise, here’s an alternative that can be built without needing code generation:

func Test_CheckValuesAreSame(t *testing.T) {
  assert.With(t).Equal("a", "b") // assertion fails but tests continue
  assert.With(t).Must().Equal("a", "b") // assertion fails and test stops

  panic("This panic shouldn't happen")
}

func Test_FormatTesting(t *testing.T) {
  assert.With(t).Messagef("%s should be equal %s", "a", "b").
	  Equal("a", "b")
}

If we accept that (1) is an artificial problem and provide only methods, you could design assertions using a builder pattern. Assertions belongs to an assertion struct which has configure functions to that set control flow behavior.

Here’s a prototype of an assertion struct that complies to that interface and holds formatting and flow properties as part of the test context:

type Assertion interface { 
	// ... redacted for brevity
}

func With(t *testing.T) Assertion {
	return &assertImpl{wrappedT: t}
}

type assertImpl struct {
	wrappedT *testing.T
	stopOnFailure bool
	failedMessage string
}

func (a *assertImpl) Must() Assertion {
	a.stopOnFailure = true
	return a
}

func (a *assertImpl) Message(msg string) Assertion {
	a.failedMessage = msg
	return a
}

func (a *assertImpl) Messagef(msg string, args ...interface{}) Assertion {
	a.failedMessage = fmt.Sprintf(msg, args...)
	return a
}

func (a *assertImpl) Equal(sideA, sideB interface{}) {
	// Equal logic using "github.com/google/go-cmp/cmp"
  // But this can be substituted by the custom equal logic found in testify.
	if !cmp.Equal(sideA, sideB) {
		diff := cmp.Diff(sideA, sideB)
		a.wrappedT.Errorf("Test failed: %s\n%s", a.failedMessage, diff)
		if a.stopOnFailure {
			a.wrappedT.FailNow()
		}
	}
}

It’s worth mentioning that this is not a serious implementation, this is more of a thought exercise on how this could be structured. There are missing features here like providing support for **testing.B* calls, and likely other use cases I can’t see right now.

Trade-offs

I think the biggest trade-off here is: complexity shifted from the testify package to the client test code. Writing tests using testify is extremely simple. I’ve rarely opened their documentation after the first few times I wrote tests. Which is a unique experience compared with other test libraries. For example with JavaScript’s chai, where often I forgot the order and idiomatic ways I should write assertions.

This is a great trait of testify. Good libraries are the ones you don’t have to think too much about them, and they just work.

The code generation path leaves a burden for the library developer to maintain, but it might simplify the package API, which makes adoption faster. I think testify developers probably made the right call here in making this simpler for the user, even if it risks complicating the maintainability of their tool.

After all, it’s better to have all complexity in a single place than a little complexity everywhere.