Forwards and Backwords Pipes in Go

In the Go language, an often-asked question is “how do I tell if all the goroutines are done?”  I asked myself that recently, while porting a measurement library to use channels (pipes).  As the channels went to goroutines, how would I know when they were done and that I could exit?

The common answer I found was to use a semaphore, a sync.WaitGroup, increment it before calling a goroutine and decrement it at the end of the goroutine. The main program could then wait for it to drop to zero and only then exit. Exiting any earlier would have killed the goroutines, and therefore discard any work they hadn’t finished.

However, some of the elegant examples in books and talks didn’t include waitgroups. For example, Lexical Scanning in Go by Rob Pike didn’t use anything special, and neither did a tiny lexer and parser I had copied from his talk.

Instead my code started a lexer/parser goroutine and then looped, consuming the output,  something like:

go run(l) // Runs the lexer/parser, sends to l.Pipe
for tok = range l.Pipe {
        slice = append(slice, tok)
        if tok.Typ == token.EOF {
 return slice

The idiom here is to make the last step in the pipeline part of the mainline, and consume the material created in the pipe. In effect I’m build the pipeline protruding “backwards” from main(), instead of “forwards” and needing to have the main program wait for a semaphore.

I can’t always structure library code this way: in a number of cases I have needed to have the pipe proceed “forwards” from the mainline and I indeed use sync.WaitGroup there.

However, when I can, I like the idea of writing go this(); go that(); go theOtherThing() and then reading the results in the mainline. It’s elegant.


2 thoughts on “Forwards and Backwords Pipes in Go

  1. Hello David. Your blog provides interesting reading even for a programming-challenged person such as I am.. And, despite the coding talk, the substance is probably more comprehensible than the substance of my blog.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s