Yesterday I described an approach to writing lexers in Go that pushed the actual lexing into a separate goroutine, so that it could run as straight-line code that simply consumed input and produced a stream of tokens (which were sent to a channel). Effectively we’re using a goroutine to implement what would be a generator in some other languages. But because we’re using goroutines and channels, there’s something important we need to do : we need to make sure the lexer goroutine is run to completion, so that the goroutine will actually finish.

Right now you may be saying ’well of course the lexer will always be run to the end of the input, that’s what the parser does’. But not so fast ; what happens if the parser runs into a parse error because of a syntax error or the like ? The natural thing to do in the parser is to immediately error out without looking at any further tokens from the lexer, which means that the actual lexer goroutine will stall as it sits there trying to send the next token into its communication channel, a channel that will never be read from because it’s been abandoned by the parser.

The answer here is that the parser must do something to explicitly run the lexer to completion or otherwise cause it to exit, even if the tokens the lexer are producing will never be used. In some environments having the lexer process all of the remaining input is okay because it will always be small (and thus fast), but if you’re lexing large bodies of text you’ll want to arrange some sort of explicit termination signal via another channel or something.

This is an important way in which goroutines and channels aren’t a perfect imitation of generators. In typical languages with generators, abandoning a generator results in it getting cleaned up via garbage collection ; you can just walk away without doing anything special. In Go with goroutines, this isn’t the case ; you need to consider goroutine termination conditions and generally make sure it always happens.

You might think that this is a silly bug and of course anyone who uses goroutines like this will handle it as a matter of course. If so, I regret to inform you that I didn’t come up with this realization on my own ; instead Rob Pike taught it to me with his bugfix to Go’s standard text/template module. If Rob Pike can initially overlook this issue in his own code in the standard library, anyone can.