I was recently debugging an issue in some integration-style Go tests which made me realize that I didn't have a very deep understanding of how parallelism works when using go test. I knew that there were ways that Go could parallelize tests, and I thought it had something to do with using t.Parallel() inline in each of my test functions.

Because I feel the concurrency behavior of go test is non-obvious, and for posterity so I don't forget in the future, I wanted to write something up here to document my understanding of how go test parallelization works as of Go 1.19.

Testing a single package

Let's start simple and assume we're testing a single package. Let's assume this package has no dependencies. By default, unit tests in your package will run sequentially. In the below example, TestFirstTest will run first, followed by TestSecondTest. We'll run these using go test -v.

package main

import (
  "fmt"
  "testing"
)

func TestFirstTest(t *testing.T) {
  time.Sleep(1 * time.Second)
  fmt.Println("1")
}

func TestSecondTest(t *testing.T) {
  fmt.Println("2")
}

=== RUN   TestFirstTest
1
--- PASS: TestFirstTest (1.00s)
=== RUN   TestSecondTest
2
--- PASS: TestSecondTest (0.00s)
PASS
ok  	test-concurrency	1.180s

Simple enough. We first see the output of 1, followed by the output of 2.

You can also add t.Parallel() to your tests to make them run in parallel. First go test will run all tests sequentially, and each time it encounters a call to t.Parallel(), it will PAUSE execution of the test (noted in the test output). Once all sequential tests have completed, it will resume execution (CONT) of all tests which invoked t.Parallel(), in parallel.

package main

import (
	"fmt"
	"testing"
	"time"
)

func TestFirstTest(t *testing.T) {
	fmt.Println("1 start")
	t.Parallel()
	time.Sleep(1 * time.Second)
	fmt.Println("1 end")
}

func TestSecondTest(t *testing.T) {
	t.Parallel()
	fmt.Println("2")
}

func TestThirdTest(t *testing.T) {
	fmt.Println("3")
}

=== RUN   TestFirstTest
1 start
=== PAUSE TestFirstTest
=== RUN   TestSecondTest
=== PAUSE TestSecondTest
=== RUN   TestThirdTest
3
--- PASS: TestThirdTest (0.00s)
=== CONT  TestFirstTest
=== CONT  TestSecondTest
2
--- PASS: TestSecondTest (0.00s)
1 end
--- PASS: TestFirstTest (1.00s)
PASS
ok  	test-concurrency	1.160s

Here we first see the output of the sequential tests, including 1 start from TestFirstTest before it invokes t.Parallel(). After TestFirstTest and TestThirdTest have run in sequence, TestSecondTest (2) and the remainder of TestFirstTest (1 end) are executed in parallel. Neat.

Testing Multiple Packages

The above is simple enough for tests in a single package, but most codebases consist of more than a single package. Let's define a hypothetical project with two packages: a and b. Let's define a test in each of them and run go test ./... -v.

package a

import (
	"testing"
	"time"
)

func TestA(t *testing.T) {
  time.Sleep(1 * time.Second)
  fmt.Println("a")
}

package b

import (
	"testing"
)

func TestB(t *testing.T) {
	fmt.Println("b")
}

=== RUN   TestA
a
--- PASS: TestA (1.00s)
PASS
ok  	test-concurrency/a	1.256s
=== RUN   TestB
b
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	0.068s

Based on the output, it looks like these tests ran sequentially. We see the output a before the output b despite the fact that TestA utilizes time.Sleep before printing its output.

But in fact, these tests are being run in parallel. Confusingly, the output is buffered when running go test across multiple packages. The logs are batched until each test is completed. We can see this more clearly if we print some timestamp instead. i.e. fmt.Println("a", time.Now().Second()).

=== RUN   TestA
a 6
--- PASS: TestA (1.00s)
PASS
ok  	test-concurrency/a	1.369s
=== RUN   TestB
b 5
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	0.441s

Printing the current second makes it clear that TestB logs a timestamp one second before TestA. This is because these tests are actually being run in parallel. (As an aside there is an open issue here in golang/go to address the buffered nature of go test across multiple packages).

These tests are not using t.Parallel(), so why are they being run in parallel? Well, by default Go runs tests in multiple packages in parallel. The t.Parallel() function we mentioned above only indicates whether tests within the same package are run in parallel. To be more specific, go test accepts a flag called -p, or -parallel which is documented in go help testflags.

	-parallel n
	    Allow parallel execution of test functions that call t.Parallel, and
	    fuzz targets that call t.Parallel when running the seed corpus.
	    The value of this flag is the maximum number of tests to run
	    simultaneously.
	    While fuzzing, the value of this flag is the maximum number of
	    subprocesses that may call the fuzz function simultaneously, regardless of
	    whether T.Parallel is called.
	    By default, -parallel is set to the value of GOMAXPROCS.
	    Setting -parallel to values higher than GOMAXPROCS may cause degraded
	    performance due to CPU contention, especially when fuzzing.
	    Note that -parallel only applies within a single test binary.
	    The 'go test' command may run tests for different packages
	    in parallel as well, according to the setting of the -p flag
	    (see 'go help build').

This -parallel flag allows us to control the parallelism of tests which use t.Parallel(), but critically it also specifies the parallelism across multiple packages. The documentation says "By default, -parallel is set to the value of GOMAXPROCS."; and GOMAXPROCS defaults to the number of CPU cores on the machine. According to the docs:

GOMAXPROCS sets the maximum number of CPUs that can be executing simultaneously and returns the previous setting. It defaults to the value of runtime.NumCPU.

So putting it all together, running go test ./... will use GOMAXPROCS as the default value of -parallel, GOMAXPROCS defaults to the number of cores on the machine, and therefore go test ./... will run all package tests in parallel across all cores on the machine.

Multiple Packages and package initialization

I want to touch on one other non-obvious feature of go test, which is how imported packages are initialized in tests. In order to demonstrate this, we need to add a third package into the mix which we'll call c. Let's suppose that both a and b are importing the package c as part of their import chain. We'll modify our TestA and TestB functions to print a global value exported from c. Finally, we'll add an init() function to the c package. Remember that init is a special package-level function that is called once after the imported packages have been initialized.

package a

import (
	"fmt"
	"test-concurrency/c"
	"testing"
)

func TestA(t *testing.T) {
	fmt.Println("a", c.Global)
}

package b

import (
	"fmt"
	"test-concurrency/c"
	"testing"
)

func TestB(t *testing.T) {
	fmt.Println("b", c.Global)
}

package c

import "time"

var Global int

func init() {
	Global = time.Now().Nanosecond()
}

=== RUN   TestA
a 455538000
--- PASS: TestA (0.00s)
PASS
ok  	test-concurrency/a	(cached)
=== RUN   TestB
b 381870000
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	(cached)
?   	test-concurrency/c	[no test files]

You might expect that the output above would be b 455538000 instead of b 381870000 given that the init function should only run once, however (and this is the non-obvious part), both a and b actually import and initialize a separate instance of the c package. 🤔

The reason for this is that, under the hood, go test is actually compiling and running separate binaries for each of your packages' tests. These binaries include the test code, the code being tested, and any dependencies. In fact, if you run go test ./... -work, you can access the temporary directory used for this process. Within this directory, you will find an executable for each of your packages.

This helps explain why references exported from c are not shared between the a and b package tests, and the init function of c is invoked twice. In the case of simple unit testing, this is generally not problematic, but can lead to unexpected behavior if the tests are integration-style tests which rely on package-level locks to manage system-level resources like files, ports, etc.

Understanding these nuances around Go testing will help you write better and faster test suites. Now, go forth and improve your coverage. 🫡