bryce.is/writing/code

3 Go Gotchas

2015-11-01T10:00:00-00:00

Lately I've been doing more work in the Go programming language. Today I thought I would share three "gotchas" that caught me off guard, or otherwise produced results that I would not have expected in my work with Go.

1. The range clause

The range clause is very convenient. It allows you to iterate over a slice or map with two variables which represent the index and the value of each item. For example:

for index, value := range mySlice {
    fmt.Println("index: " + index)
    fmt.Println("value: " + value)
}

However, something notable is happening under the hood. Let's see more complex example:

type Foo struct {
    bar string
}

func main() {
    list := []Foo{
        {"A"},
        {"B"},
        {"C"},
    }

    list2 := make([]*Foo, len(list))
    for i, value := range list {
        list2[i] = &value
    }

    fmt.Println(list[0], list[1], list[2])
    fmt.Println(list2[0], list2[1], list2[2])
}

In this example we are doing a few things.

We are creating a slice of Foo structs called list.
We are defining a second slice of pointers to Foo structs called list2.
We iterate through each struct in list in order to assign its pointer to the corresponding index in list2.

So therefore you might expect the output of the above code to be the following:

{A} {B} {C}
&{A} &{B} &{C}

However, this is not what is happening. Let's take a look at the output of this code:

{A} {B} {C}
&{C} &{C} &{C}

The first line is as expected. These are the structs we initially created in list, but the second line is unexpected. It looks like we are printing out a pointer to the last struct in the list three times. But why is this happening ?

The culprit is the range clause.

for i, value := range list {
    list2[i] = &value
}

Here's the problem: Go uses a copy of the value instead of the value itself within a range clause. So when we take the pointer of value, we're actually taking the pointer of a copy of the value. This copy gets reused throughout the range clause, which leaves our list2 slice full of three references to the same pointer (the copy pointer).

According to the Go reference manual, "The iteration values are assigned to the respective iteration variables as in an assignment statement." So effectively you can imagine the above range clause to be the same as writing:

var value Foo
for var i := 0; i < len(list); i++ {
    value = list[i]
    list2[i] = &value
}

In order to produce the output we expect, we should use the index to take a pointer to the actual value, instead of the copy.

for i, _ := range list {
    list2[i] = &list[i]
}

2. The append built-in function

Slices are primitive types in Go. In other languages you might reach for an Array where in Go you would reach for a Slice. Here's an example of how we might add a value to the end of an integer slice.

list := []int{0,1,2}
list = append(list, 3)
fmt.Println(list) // [0 1 2 3]

On the surface it seems to be similar to a push() array method, but slices are not quite arrays, and the built-in append function surprised me with its behavior under the hood. Have a look at the example below:

func main() {
    a := []byte("foo")
    b := append(a, []byte("bar")...)
    c := append(a, []byte("baz")...)
        
    fmt.Println(string(a), string(b), string(c))
}

In this example we define a slice a of bytes with an initial value of ["foo"]. Next we append another slice ["bar"] to our initial slice a, and again we append another slice ["baz"] to our initial slice a. The output of the above snippet is:

foo foobaz foobaz

Whaaat? 😮 Shouldn't it be foo foobar foobaz ?

In order to understand what's going on here we have to understand what a slice really is. A slice is a descriptor which consists of three components:

A pointer to an underlying array - that is, one allocated by Go which you don't have direct access to.
The capacity of said underlying array.
The effective length of the slice.

So what's really happening ? Go will reuse the same underlying array in append() if it can do so without resizing the underlying array. So all three of these structs are referencing the exact same array in memory. The only practical difference is their length value which in the case of a is 3, and in the case of b and c is 6.

Keep in mind, Go will only reuse the same underlying array if the length of the newly created slice is less than or equal to the capacity of the initial slice.

It's important to understand what is happening under the hood when using some of the built-in slice functions. For more info, Robe Pike wrote a very helpful blog post which goes into a lot of useful details around slices.

3. Variable shadowing

When people first look at Go code for the first time, one of the first things they ask is "What is that := operator for?". Well, as you know, it's a shorthand variable declaration operator. It's used for both declaring and setting the value of a variable. The type is implicitly declared. This is very convenient, but can lead to some problems if you aren't careful. Specifically I'm talking about variable shadowing. This is something that has bitten me in particular, because I have been programming almost exclusively in es5 javascript for the last two years where there is no block-level scoping.

Check this out:

func main() {
    list := []string{"a", "b", "c"}
    for {
        list, err := repeat(list)
        if err != nil {
            panic(err)
        }
        fmt.Println(list)
        break
    }
    fmt.Println(list)
}

func repeat(list []string) ([]string, error) {
    if len(list) == 0 {
        return nil, errors.New("Nothing to repeat!")
    }
    list = append(list, list...)
    return list, nil
}

This is somewhat of a contrived example, so bear with me because I think it illustrates the point well. The run down of this snippet is as follows:

We create a slice of strings, list.
We enter a for loop.
The for loop calls a function repeat() which returns a new slice, and an error.
We break out of the for loop, and we print the value of list.

You might expect the output here to be:

[a b c a b c]
[a b c a b c]

but in fact, it is:

[a b c a b c]
[a b c]

Because we use the shorthand variable declaration operator, we are actually redeclaring the list variable inside of the scope of the for loop. This is awkward, because the second variable err is a new variable which we want to declare. We can fix this by changing the first few lines within the for block to:

var err error
list, err = duplicate(list)

One way to catch these problems early is by taking advantage of the go vet tool. It has an option -shadow which can help to detect issues with variable shadowing.

There is a good blog post called The Golang Beartrap that talks more in detail about variable shadowing.

Summary

Go is a great language, and once you understand its quirks it can be a real pleasure to work in. I really appreciate the fantastic tooling, and readability of the language. Good luck, and happy gophering! 😄

Getting started with mdb_v8

2016-02-06T10:00:00-00:00

Did you know you can inspect your node application's memory without instrumenting your code? It's possible using a tool called mdb_v8. With mdb_v8 we can produce a core dump of a running node process, and then later inspect the memory at the time the core dump was generated. This leads to some powerful postmortem debugging capabilities.

In this post I'm going to walk you through setting up mdb_v8 locally so that you can inspect your node application's memory. Recently I went through this exercise myself and realized there was no good guide to do so. I decided to document the process and post it here.

The steps were going work through are as follows:

Generate a core dump of our running node application
Install a Solaris derivative in a VM
Install mdb_v8, and run it against the core dump

Be aware that this guide assumes a couple things about your workflow. First, that you are using a version of node at 0.10 or greater. At the time of this writing, all previous versions are no longer supported. Second, that your node service is running on a system where gcore is available, such as linux.

A Solaris derivative is required to run mdb_v8. There are two post mortem debuggers that don't require Solaris, but they aren't as fully featured. These are llnode and lldb_v8. Alternatively, if instrumenting your code isn't an issue you may want to look into node-heapdump.

What is mdb_v8 anyways ?

We can't talk about mdb_v8 without first talking about mdb. mdb is a low-level debugger developed by Sun for the Solaris operating system. What makes it unique is that is extremely extensible. It is possible to extend the tool with custom modules ("dmods") for different contexts. This is exactly what Joyent has done with their mdb_v8 module. The mdb_v8 module adds a new set of commands instrumented specifically for the v8 javascript engine and node.js. It includes commands that allow you to inspect the application state, javascript frames, and javascript objects on the heap at the time the core dump was generated.

It's worth emphasizing here that mdb only works on Solaris derivatives. That is -- it won't run OS X, Windows, or Linux. More on this in a moment.

Generating a core dump

A core dump is a file which consists of the working memory of an application. It also typically contains some other metadata such as process registers. In linux generating a core dump of our node.js application manually is pretty straight forward. We just run the following command:

> gcore -o filename pid

This will produce a core dump file.

You can also set up your node service to save a core dump when it crashes by using the --abort_on_uncaught_exception flag.

Installing mdb_v8

So mdb_v8 sounds great, right ? You will be able to inspect your application state, and find out where all of your sneaky memory leaks are hiding. Well, that's true, but as I mentioned above mdb_v8 only works on Solaris.

Therefore, we will use VirtualBox to install a Solaris derivative (OmniOS).

Setting up VirtualBox with OmniOS

If you don't already have VirtualBox installed, you can proceed to the downloads page to download the correct binary for your system and install it.

:arrow_down: Download VirtualBox

After you've installed VirtualBox you will download a Solaris derivative. Here we will use OmniOS. Download the current/stable release under "From CD/iso"

:arrow_down: Download OmniOS

Once you've downloaded the image you can move back over to VirtualBox.

Select "Oracle Solaris 11", and continue with everything else as the default.

Once the VM is running, start it up and choose the OmniOS image that you downloaded above.

Proceed through the installation process choosing your language, timezone, etc. Keep in mind that VirtualBox has a sandbox auto-resizing hard drive. It's fine to install the image on the "whole disk". You can also ignore any warnings about kvm: No hardware support.

After you've installed OmniOS, proceed to "Devices -> Optical drives -> Remove disk from virtual drive" in VirtualBox to unmount the disk. Finally, reboot the VM. You will be prompted to login once OmniOS starts up. The default login in OmniOS is root with no password.

Utilizing mdb_v8

If you've made it this far you should be staring at an empty shell. There are still a few remaining steps before we can begin using mdb_v8. Those steps are as follows:

We need to be able to SSH into our VM from our local machine
We need to install the mdb_v8.so module for mdb
We need to SCP our core dump, and node.js binary to the virtual machine

SSHing into our VM

The first issue is that our VM doesn't have access to the local network. We need to enable this option in VirtualBox. We want to give our VM an internal IP that we can reach via SSH. We can do this by right clicking our VM in VirtualBox, and heading to the settings page. Here we will change our networking adapter from NAT to Bridged Adapter.

The next step is that we need to enable DHCP in OmniOS. This way our VM can be assigned an IP address on the network. We do that by using a tool called ipadm to create an IP interface.

> ipadm create-if e1000g0
> ipadm create-addr -T dhcp e1000g0/v4

Now that we have created our IP interface we can configure our DNS resolution. Run these commands to add a nameserver.

> echo 'nameserver 8.8.8.8' >> /etc/resolv.conf
> cp /etc/nsswitch.dns /etc/nsswitch.conf

Hurray. Now we have network connectivity. Now let's get our new local IP on the network by running ifconfig.

> ifconfig e1000g0

This will show us our local network IP address (directly after the inet).

Even though we have the network IP address, we still won't be able to SSH into the machine. Why ? Well because by default OmniOS (and many distributions) don't allow you to SSH in as the root user. This is typically a security precaution. However, because we are just doing this to run mdb_v8 locally, we can enable this functionality.

To allow us to SSH in as root we can edit the sshd_config

> vim /etc/ssh/sshd_config

and scroll down to the line that says PermitRootLogin no and change it to PermitRootLogin yes. After this we want to restart the SSH daemon.

> sudo svcadm restart SSH

Now we should be able to SSH into our VM. Nice! On to the next step.

Installing mdb_v8

Luckily, OmniOS comes with mdb installed by default, but it does not come with the mdb_v8 module. We need to manually install this. To do this we can simply use curl to download the binary.

Let's first create a working directory.

> mkdir ~/mdbv8
> cd ~/mdbv8

You can first curl Joyent's manta service to find the latest version of mdb_v8.

> curl https://us-east.manta.joyent.com/Joyent_Dev/public/mdb_v8/latest | cut -c 27-

Run the following command below (replace {version} below with version you got from the above curl i.e. "v1.1.2")

> curl -O https://us-east.manta.joyent.com/Joyent_Dev/public/mdb_v8/{version}/mdb_v8_amd64.so

Great. Now we have everything we need to run mdb_v8.

Putting it all together

The last and final step is to move over both our core dump file(s), and our node binary. (Don't have a core dump file ? See the beginning of this post for a snippet of how to use gcore.)

What's all this about a node binary ? Well, mdb_v8 will only work against the node binary that was running your node application at the time that the core dump was generated. You can download it directly from the node.js website provided you know the exact version and build. The easiest way is to simply grab the node binary off of the machine where you took the core dump. Typically you can find the node binary by running which node, but if you're using something like nvm it may require a little bit more digging.

Provided you have downloaded the node binary and your core dump on your local machine, you can now SCP these files onto your VM.

> scp ./node root@192.168.1.82:~/mdbv8
> scp ./my-dump.gcore root@192.168.1.82:~/mdbv8

Now we can start the mdb_v8 debugger against our core dump and node binary.

> mdb ./node ./my-dump.gcore

This will drop us into the mdb shell. All we need to do now is to load the mdb_v8 module by executing:

::load /root/mdbv8/mdb_v8_amd64.so

💥 Now we can proceed to run any of the mdb_v8 commands against our core dump.

I like to start my mdb_v8 search by listing all js objects in memory. Try it yourself by typing.

::findjsobjects

For a list of useful mdb_v8 commands I recommend the official mdb_v8 usage guide.

I also found this gist by tjfontaine to be pretty helpful.

Summary

In this guide I've walked you through generating core dumps, installing solaris locally, and setting up mdb_v8. Hopefully you will find that mdb_v8 is a very useful node.js debugging tool to have in your back pocket for postmortem debugging. Good luck plugging all your leaks! 🚱

Debugging using system calls in Mac OS X

2016-07-30T11:00:00-00:00

Today I want to talk about an issue I was having with UDP sockets in Go, and how I learned more about my program by making use of two tools that ship with Mac OS X called dtruss, and DTrace.

I do most of my programming on a Mac, and I have been working on a project written in Go which makes use of UDP sockets. When trying to add some unit tests to my code I decided to create one around reconnections. In my test I created two sockets on localhost with two different ports, read/wrote from them, closed them, and then repeated this several hundred times. Eventually I would receive a "bind: address already in use" error upon the creation a socket. I wondered, why was this happening if I was closing the sockets ?

Understanding UDP in Go

In Go we can use the net.ListenPacket method to start listening for UDP packets on a given network address. This function returns a net.PacketConn interface.

According to the Go documentation for PacketConn, the Close method should either close the connection, or return an error:

type PacketConn interface {
...
    // Close closes the connection.
    // Any blocked ReadFrom or WriteTo operations will be unblocked and return errors.
    Close() error
...
}

If this were true then why was I getting these EADDRINUSE errors ? Time to investigate 🔎 But first a brief recap on system calls.

System calls

Systems calls are the way your program interfaces with your operating system. In unix-based x86 systems specifically your compiled code will contain a software interrupt which is passed a value corresponding to a function in the kernel. In our case we were interested in two system calls; bind and close. bind assigns a file descriptor to a network address. close deallocates a file descriptor.

How can we see when our program is making system calls like bind and close ?

dtruss

dtruss is a tool that ships with Mac OS X. It was written by Brendan Gregg. It is similar to strace on linux systems. You can use it to see which system calls a program is making to the kernel. This means we can see if a program is attempting to close a socket when we expect it to, or if in fact the Go standard library is lying to us. After running dtruss you will see something like this:

$ sudo dtruss -a -t bind go test -run TestReconnect
PID/THRD         RELATIVE  ELAPSD    CPU SYSCALL(args)               = return

82830/0x978a1c:   1357371      99     95 close(0x5)      = 0 0
82830/0x978a20:   1378137      20     16 socket(0x2, 0x2, 0x0)       = 5 0
82830/0x978a20:   1378139       4      0 fcntl(0x5, 0x2, 0x1)        = 0 0
82830/0x978a20:   1378142       2      0 fcntl(0x5, 0x3, 0x0)        = 2 0
82830/0x978a20:   1378144       4      1 fcntl(0x5, 0x4, 0x6)        = 0 0
82830/0x978a20:   1378148       6      2 setsockopt(0x5, 0xFFFF, 0x20)       = 0 0
82830/0x978a20:   1378161       8      4 bind(0x5, 0xC821AA576C, 0x10)       = -1 Err#48

So here we can see a bunch of interesting information.

Which threads are making which syscalls
How long they are taking
The arguments they're called with
The return value

One interesting thing here is that on the last line at the very end we see Err#48. This is our "bind: address already in use" error. In this particular trace we can see on the first line it looks like we close the 0x5 socket with no errors, and in the last line it says 0x5 is already in use when we try to bind it. Confusingly the 0x5 just refers to the file descriptor, and not the actual network address. The network address is stored in the second argument 0xC821AA576C above. This is a pointer to the sockaddr struct passed to the kernel, which when dereferenced would look something like this:

struct sockaddr {
   sa_family_t sa_family;
   char        sa_data[14]; // 14 bytes of protocol address
}

The part we're interested in is the sa_data field.

So we need to dereference the pointer somehow. How is this possible ? Enter DTrace!

DTrace

Dtrace is a powerful debugging tool originally built by Sun Microsystems for Solaris. It is also available on Mac OS X. The amazing thing about Dtrace is that it ships with its own scripting language called D. With these scripts you can write tracing programs which will execute upon entry/return of specified system calls. So using this we can actually write some tracing around bind to dereference the sockaddr struct, and ultimately find out which port is being bound.

dtrace -c "go test -run TestReconnect" -n 'bind:entry { trace(arg0); trace(copyinstr(arg1)); }' -f bind

This command is saying, "Execute my Go program, but whenever you see a bind system call invocation, run this D script." In the script itself I'm tracing the first two arguments. With the second argument I am tracing the result of a function called copyinstr. In D, the copyinstr function will dereference a pointer in user space and output the result as a string.

Running DTrace with these arguments gives me something like this:

 0    358                       bind:entry                 5
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 10 02 0b b9 7f 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        ...

At the end of the first line we see the 5 file descriptor being used like before. On the next lines we see the output of copyinstr(arg1). This is the dereferenced pointer to the sockaddr struct. It looks like mostly unused memory. If we look at 3rd and 4th bytes in the above output we see the port number of the socket which is 0x0bb9 (3001). Now I can see that file descriptor 0x5 is binding to port 3001. Whew! 😅 Using this tool I was able to see that in fact an existing file descriptor was still bound to the same port when the subsequent bind call was made.

Conclusion

After digging through the Go standard library I eventually found my answer. Go documentation does mention that any blocked ReadFrom or WriteTo calls will be unblocked and return errors. What it fails to mention is that until that happens, Go will not close the socket. Meaning that even after calling Close() on a net.PacketConn there is no guarantee that the socket has actually been closed. This is due to a mutex around the underlying file descriptor in Go. The solution in my case was to use a semaphore in my code to ensure that all ReadFrom and WriteTo calls had returned before attempting to open a new socket on the same port.

The important thing about this exercise for me was being able to know for certain that I understood what my program was doing, and what my operating system was doing. This means I could be confident that the problem was sitting in between these layers in the stack.

Happy debugging! 😄

go test and parallelism

2023-02-13T09:30:00-00:00

I was recently debugging an issue in some integration-style Go tests which made me realize that I didn't have a very deep understanding of how parallelism works when using go test. I knew that there were ways that Go could parallelize tests, and I thought it had something to do with using t.Parallel() inline in each of my test functions.

Because I feel the concurrency behavior of go test is non-obvious, and for posterity so I don't forget in the future, I wanted to write something up here to document my understanding of how go test parallelization works as of Go 1.19.

Testing a single package

Let's start simple and assume we're testing a single package. Let's assume this package has no dependencies. By default, unit tests in your package will run sequentially. In the below example, TestFirstTest will run first, followed by TestSecondTest. We'll run these using go test -v.

package main

import (
  "fmt"
  "testing"
)

func TestFirstTest(t *testing.T) {
  time.Sleep(1 * time.Second)
  fmt.Println("1")
}

func TestSecondTest(t *testing.T) {
  fmt.Println("2")
}

=== RUN   TestFirstTest
1
--- PASS: TestFirstTest (1.00s)
=== RUN   TestSecondTest
2
--- PASS: TestSecondTest (0.00s)
PASS
ok  	test-concurrency	1.180s

Simple enough. We first see the output of 1, followed by the output of 2.

You can also add t.Parallel() to your tests to make them run in parallel. First go test will run all tests sequentially, and each time it encounters a call to t.Parallel(), it will PAUSE execution of the test (noted in the test output). Once all sequential tests have completed, it will resume execution (CONT) of all tests which invoked t.Parallel(), in parallel.

package main

import (
	"fmt"
	"testing"
	"time"
)

func TestFirstTest(t *testing.T) {
	fmt.Println("1 start")
	t.Parallel()
	time.Sleep(1 * time.Second)
	fmt.Println("1 end")
}

func TestSecondTest(t *testing.T) {
	t.Parallel()
	fmt.Println("2")
}

func TestThirdTest(t *testing.T) {
	fmt.Println("3")
}

=== RUN   TestFirstTest
1 start
=== PAUSE TestFirstTest
=== RUN   TestSecondTest
=== PAUSE TestSecondTest
=== RUN   TestThirdTest
3
--- PASS: TestThirdTest (0.00s)
=== CONT  TestFirstTest
=== CONT  TestSecondTest
2
--- PASS: TestSecondTest (0.00s)
1 end
--- PASS: TestFirstTest (1.00s)
PASS
ok  	test-concurrency	1.160s

Here we first see the output of the sequential tests, including 1 start from TestFirstTest before it invokes t.Parallel(). After TestFirstTest and TestThirdTest have run in sequence, TestSecondTest (2) and the remainder of TestFirstTest (1 end) are executed in parallel. Neat.

Testing Multiple Packages

The above is simple enough for tests in a single package, but most codebases consist of more than a single package. Let's define a hypothetical project with two packages: a and b. Let's define a test in each of them and run go test ./... -v.

package a

import (
	"testing"
	"time"
)

func TestA(t *testing.T) {
  time.Sleep(1 * time.Second)
  fmt.Println("a")
}

package b

import (
	"testing"
)

func TestB(t *testing.T) {
	fmt.Println("b")
}

=== RUN   TestA
a
--- PASS: TestA (1.00s)
PASS
ok  	test-concurrency/a	1.256s
=== RUN   TestB
b
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	0.068s

Based on the output, it looks like these tests ran sequentially. We see the output a before the output b despite the fact that TestA utilizes time.Sleep before printing its output.

But in fact, these tests are being run in parallel. Confusingly, the output is buffered when running go test across multiple packages. The logs are batched until each test is completed. We can see this more clearly if we print some timestamp instead. i.e. fmt.Println("a", time.Now().Second()).

=== RUN   TestA
a 6
--- PASS: TestA (1.00s)
PASS
ok  	test-concurrency/a	1.369s
=== RUN   TestB
b 5
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	0.441s

Printing the current second makes it clear that TestB logs a timestamp one second before TestA. This is because these tests are actually being run in parallel. (As an aside there is an open issue here in golang/go to address the buffered nature of go test across multiple packages).

These tests are not using t.Parallel(), so why are they being run in parallel? Well, by default Go runs tests in multiple packages in parallel. The t.Parallel() function we mentioned above only indicates whether tests within the same package are run in parallel. To be more specific, go test accepts a flag called -p, or -parallel which is documented in go help testflags.

	-parallel n
	    Allow parallel execution of test functions that call t.Parallel, and
	    fuzz targets that call t.Parallel when running the seed corpus.
	    The value of this flag is the maximum number of tests to run
	    simultaneously.
	    While fuzzing, the value of this flag is the maximum number of
	    subprocesses that may call the fuzz function simultaneously, regardless of
	    whether T.Parallel is called.
	    By default, -parallel is set to the value of GOMAXPROCS.
	    Setting -parallel to values higher than GOMAXPROCS may cause degraded
	    performance due to CPU contention, especially when fuzzing.
	    Note that -parallel only applies within a single test binary.
	    The 'go test' command may run tests for different packages
	    in parallel as well, according to the setting of the -p flag
	    (see 'go help build').

This -parallel flag allows us to control the parallelism of tests which use t.Parallel(), but critically it also specifies the parallelism across multiple packages. The documentation says "By default, -parallel is set to the value of GOMAXPROCS."; and GOMAXPROCS defaults to the number of CPU cores on the machine. According to the docs:

GOMAXPROCS sets the maximum number of CPUs that can be executing simultaneously and returns the previous setting. It defaults to the value of runtime.NumCPU.

So putting it all together, running go test ./... will use GOMAXPROCS as the default value of -parallel, GOMAXPROCS defaults to the number of cores on the machine, and therefore go test ./... will run all package tests in parallel across all cores on the machine.

Multiple Packages and package initialization

I want to touch on one other non-obvious feature of go test, which is how imported packages are initialized in tests. In order to demonstrate this, we need to add a third package into the mix which we'll call c. Let's suppose that both a and b are importing the package c as part of their import chain. We'll modify our TestA and TestB functions to print a global value exported from c. Finally, we'll add an init() function to the c package. Remember that init is a special package-level function that is called once after the imported packages have been initialized.

package a

import (
	"fmt"
	"test-concurrency/c"
	"testing"
)

func TestA(t *testing.T) {
	fmt.Println("a", c.Global)
}

package b

import (
	"fmt"
	"test-concurrency/c"
	"testing"
)

func TestB(t *testing.T) {
	fmt.Println("b", c.Global)
}

package c

import "time"

var Global int

func init() {
	Global = time.Now().Nanosecond()
}

=== RUN   TestA
a 455538000
--- PASS: TestA (0.00s)
PASS
ok  	test-concurrency/a	(cached)
=== RUN   TestB
b 381870000
--- PASS: TestB (0.00s)
PASS
ok  	test-concurrency/b	(cached)
?   	test-concurrency/c	[no test files]

You might expect that the output above would be b 455538000 instead of b 381870000 given that the init function should only run once, however (and this is the non-obvious part), both a and b actually import and initialize a separate instance of the c package. 🤔

The reason for this is that, under the hood, go test is actually compiling and running separate binaries for each of your packages' tests. These binaries include the test code, the code being tested, and any dependencies. In fact, if you run go test ./... -work, you can access the temporary directory used for this process. Within this directory, you will find an executable for each of your packages.

This helps explain why references exported from c are not shared between the a and b package tests, and the init function of c is invoked twice. In the case of simple unit testing, this is generally not problematic, but can lead to unexpected behavior if the tests are integration-style tests which rely on package-level locks to manage system-level resources like files, ports, etc.

Understanding these nuances around Go testing will help you write better and faster test suites. Now, go forth and improve your coverage. 🫡

fully nix-pilled

2024-06-21T05:00:00-00:00

I recently bought a new MacBook, and instead of manually installing everything for the nth time, I decided to go down the path of using Nix and nix-darwin to codify my dev machine configuration. If you're curious, let's dive in.

What is Nix?

Nix is a few things which I would break down into three categories:

A configuration and build system referred to as "Nix".
A configuration language, confusingly also referred to as "Nix".
A package manager wherein the official packages are referred to as "nixpkgs".

First and foremost, Nix is a configuration system for describing declaratively how your system should be configured. This includes things like packages, applications, and shell configuration, but can even include OS settings like your preferred scroll direction (natural scrolling disabled of course, for the sophisticated among us). If you've ever used something like Terraform or CloudFormation to set up your infrastructure, then in this context you might think of Nix as being like Terraform for your personal machine. When you "build" a version of your system configuration, Nix produces a "generation", which conceptually can be thought of as a deterministic snapshot of your system given the current configuration.

Second, the name "Nix" is also used (confusingly) to refer to the underlying configuration language itself that the configuration files are written in. This language is a declarative scripting language which is used to configure your system. In my experience it was relatively easy to get started with. The documentation is great, and there are a number of existing examples to work from.

Finally, Nix is a package manager. It has an official repository of packages called "nixpkgs" which you can browse here. When you define your configuration, you'll likely specify packages which will be sourced from nixpkgs. For example, things like git, or zsh, or python.

What is nix-darwin?

As my personal computer is running macOS, I am also using nix-darwin, which is a set of modules specifically for configuring macOS. It allows you (among other things) to modify operating system settings like the dock, hot corners, dark mode, etc. Not every single item within "System Settings" is supported yet, but it gets you most of the way there.

Conclusion

Ok, so that's a high-level overview of Nix, but why bother with any of that? For myself, I'd break it down as follows:

Pros

Improved auditability. I can more easily see at a glance what is installed on my system. No more forgetting about some random package I installed 6 months ago and never use anymore!
If something goes wrong, rollbacks with nix are super easy. If I break something critical, I can always roll back to a previous version of my system.
Peace of mind that I could blow away my entire machine and start over quickly if needed.

Cons

There was some initial investment required to get my system set up in the way I like. But I only needed to do this once!
There was some learning curve in ramping up on Nix the language.
GUI apps on macOS are not well supported, so you still need to use homebrew for these. You can at least configure which homebrew casks should be installed on the system via nix-darwin.
Ideally, every single macOS system setting would be configurable with nix-darwin, but that's not true (yet).

Using Nix for my dev machine has been working great for me so far, and I'm now fully nix-pilled. You can check out my configuration here.

We could go deeper down the rabbit hole and talk in more detail about how Nix works under the hood. Or we could speak to more of the Nix ecosystem like home-manager, flakes, or NixOS; but I'll stop here for now. 🙃

autotidy

2026-02-04T09:00:00-00:00

I recently released a new open source project called autotidy. It's a file-system organizer ala Hazel, Maid, and Organize.

What does it do exactly? Well, it allows you to define rules that should be applied when a directory changes. For example, I use it to organize my Downloads folder. When I download a file I often want it to automatically end up in a certain place. For example, audio files that begin with the name "New Recording" are voice memos. Images that begin with "ChatGPT" or "Gemini_" are images generated with AI models.

With autotidy I can sort these files into better homes by defining rules like this:

rules:
  - name: Sort AI images
    locations: ~/Downloads
    filters:
      - any:
        - name: Gemini_*
        - name: ChatGPT*
      - extension: [png, jpeg]
    actions:
      - move: ~/images/slop

Why build it?

I built autotidy to solve this problem for myself after finding that none of the alternatives met my goals. I needed something that met the following requirements:

Works cross-platform (Linux & MacOS specifically)
Free & Open Source
Declarative and configuration-based. No coding required.
Installable and configurable via Nix (how I manage my systems)
Automatic. i.e. Doesn't require me to manually run a one-off command to invoke my rules.
(Optional) No runtime dependencies (i.e. No Ruby, Node, Python etc)

To my surprise, the alternatives I found all had their own shortcomings.

Hazel: MacOS only, proprietary/commerical.

Maid: Requires Ruby runtime. Requires writing Ruby to define rules.

Organize: Requires Python runtime. No built-in file watching. Requires rules to be manually run (or via cron, etc).

This is not a critique of these tools or their developers, just that my requirements differed. They are all fantastic in their own ways, and I borrowed ideas from each of them.

Technical details

I wrote autotidy in Go. I debated writing it in Rust, but I am less familiar with Rust and I wanted to knock something out relatively quickly. There is nothing overly complex about how autotidy works. It reads a configuration file, builds rules in memory, watches files for changes, and then applies the rules to the appropriate files. But there were some things that I stumbled across when making it that I thought might be interesting to mention.

Watching for changes with fsnotify

Watching directories for changes is a bit of a tedious problem, especially if you want to do so in a way that is cross-platform. On Linux this is done with inotify, on macOS kqueue, and on Windows ReadDirectoryChangesW. Instead of re-implementing the wheel here I decided to use fsnotify which provides a standard interface around all of these implementations. There are still some cross-platform details that differ, but they are relatively few and manageable.

I also stumbled across some interesting issues along the way. Like for example, what happens if a directory that is being watched is deleted? or what happens if the user defines a rule for a directory which doesn't exist until later? To solve this problem I defined what I call "Missing Roots", locations that don't exist now, but may exist in the future. When a location that is defined in the config doesn't exist, I recursively move up the file tree until I find a directory that does exist, and I watch this directory for changes that result in getting closing to the target directory. This works well, but it comes with some other problems. Like for example, what happens if the Missing Root is located at /a, and the user in quick-succession creates /a/b/c/d/location? Our watch of /a may notice the creation of /a/b, but by the time it begins watching this directory /a/b/c may have already been created. This necessitates a bit of logic to ensure that created directories are not missed. Further, because file changes can be erratic, and we want to maintain performance, we need to debounce our operations by some small period with each change.

fsnotify does not tell us what new files created are. To determine whether the file is a directory (to look for the Missing Root, or to recurse if the rule is a recursive rule) we need to ask the Operating System what type of file it is. In order to maintain performance I ended up using readdir if the number of changed files is greater than some threshold, rather than stating every file, which could be slower if there are dozens or hundreds of file changes at once.

afero for file-system abstraction

I used afero as a way to abstract file system operations. This was useful in unit tests where I wanted to operate on files in memory rather than on the file-system itself. It had the additional benefit of allowing me to build "dry-run" functionality, which is the default when running one-off rules via autotidy run. Using afero meant I could simulate the result of running rules by running them in memory by using a Copy-On-Write implementation, and then print information about the resulting changes to the terminal without actual making them.

Packaging and distribution

One of the biggest challenges with building a cross-platform headless program like this is how to distribute it. This is arguably one advantage of using a runtime like Python, Ruby, or Node. Users can simply use pip or npm to install the software. While I could have used go install for the CLI, it wouldn't be sufficient for a daemon which needs to run in the background and launch at system startup. On linux this means adding a unit to systemd, on MacOS launchd, and on Windows the registry.

I explored a few different approaches here for varying platforms.

Homebrew

Homebrew was most simple, and is done by just creating a tap repository and instructing the user to brew install from this repository.

brew install prettymuchbryce/tap/autotidy
brew services start autotidy
autotidy status

The brew services start line is required for persistent background processes.

Nix

Nix was also relatively simple. You simply add a flake.nix file to the repository and instruct the user to add it as an input to their configuration. This was important to me as it's the one I personally use.

Other Linux

I was unsure how to tackle other linux distributions aside from Nix. For new projects, pursuing official inclusion in big package managers like yum or apt seems inappropriate. I ended up crafting a one-liner to run an install script, as well as creating deb/rpm packages with nfpm.

I'm not crazy about curl piping scripts into a shell for many reasons, so I am not totally happy with this solution.

Windows

Windows is similar to the above. There is a powershell script that installs and sets up the service.

I have the least confidence in this platform, and while I have done cursory manual and automated testing, I don't use Windows regularly enough to be sure that I have adhered to all of the standard idioms.

Testing

In addition to manual testing, I setup installation-related integration tests for each platform. At the very least this helps me catch any issues related to install before shipping new versions.

Working with Claude

Since it's 2026, and therefore I am compelled to mention AI in this post, I should say that I made heavy use of Claude Code while building this project. It is not "vibe-coded" in the sense that I threw prompts at the wall and hoped for the best. I found myself using Claude Code to help answer questions, perform menial tasks (like moving files around, or setting up boilerplate), as well as writing additional unit test cases. Against common ai-influencer wisdom, I strived to review every line of code and in doing so found a healthy amount of mistakes and bugs that were missed by the agent. I am still trying to find the right balance here of using agents to move faster, while also retaining my own strong opinions about architecture and maintainability.

Summary

This was a nice little side project to build, and I had fun with it. I am not sure if the core problem is a common one, or if my solution is differentiated enough to lure users away from the existing solutions. Ultimately I saw this as an opportunity to solve my own problem. I'd like to get in the habit of doing more things like this in the future. Thanks for reading!