Go for gophers
GopherCon closing keynote
25 April 2014
Andrew Gerrand
Google, Inc.
Andrew Gerrand
Google, Inc.
A video of this talk was recorded at GopherCon in Denver.
2I joined Google and the Go team in February 2010.
Had to re-think some of my preconceptions about programming.
Let me share what I have learned since.
3I used to think about classes and types.
Go resists this:
It instead emphasizes interfaces.
5Go interfaces are small.
type Stringer interface { String() string }
A Stringer
can pretty print itself.
Anything that implements String
is a Stringer
.
An io.Reader
value emits a stream of binary data.
type Reader interface { Read([]byte) (int, error) }
Like a UNIX pipe.
7// ByteReader implements an io.Reader that emits a stream of its byte value. type ByteReader byte func (b ByteReader) Read(buf []byte) (int, error) { for i := range buf { buf[i] = byte(b) } return len(buf), nil }
type LogReader struct { io.Reader } func (r LogReader) Read(b []byte) (int, error) { n, err := r.Reader.Read(b) log.Printf("read %d bytes, error: %v", n, err) return n, err }
Wrapping a ByteReader
with a LogReader
:
// +build ignore,OMIT
package main
import (
"fmt"
"io"
"log"
)
// ByteReader implements an io.Reader that emits a stream of its byte value.
type ByteReader byte
func (b ByteReader) Read(buf []byte) (int, error) {
for i := range buf {
buf[i] = byte(b)
}
return len(buf), nil
}
type LogReader struct {
io.Reader
}
func (r LogReader) Read(b []byte) (int, error) {
n, err := r.Reader.Read(b)
log.Printf("read %d bytes, error: %v", n, err)
return n, err
}
// STOP OMIT
func main() {
r := LogReader{ByteReader('A')} b := make([]byte, 10) r.Read(b) fmt.Printf("b: %q", b)
}
By wrapping we compose interface values.
9Wrapping wrappers to build chains:
var r io.Reader = ByteReader('A') r = io.LimitReader(r, 1e6) r = LogReader{r} io.Copy(ioutil.Discard, r)
More succinctly:
// +build ignore,OMIT
package main
import (
"io"
"io/ioutil"
"log"
)
// ByteReader implements an io.Reader that emits a stream of its byte value.
type ByteReader byte
func (b ByteReader) Read(buf []byte) (int, error) {
for i := range buf {
buf[i] = byte(b)
}
return len(buf), nil
}
type LogReader struct {
io.Reader
}
func (r LogReader) Read(b []byte) (int, error) {
n, err := r.Reader.Read(b)
log.Printf("read %d bytes, error: %v", n, err)
return n, err
}
func main() {
// START OMIT
var r io.Reader = ByteReader('A')
r = io.LimitReader(r, 1e6)
r = LogReader{r}
io.Copy(ioutil.Discard, r)
// STOP OMIT
return
io.Copy(ioutil.Discard, LogReader{io.LimitReader(ByteReader('A'), 1e6)})
}
Implement complex behavior by composing small pieces.
10Interfaces separate data from behavior.
With interfaces, functions can operate on behavior:
// Copy copies from src to dst until either EOF is reached // on src or an error occurs. It returns the number of bytes // copied and the first error encountered while copying, if any. func Copy(dst Writer, src Reader) (written int64, err error) {
// +build ignore,OMIT
package main
import (
"io"
"io/ioutil"
"log"
)
// ByteReader implements an io.Reader that emits a stream of its byte value.
type ByteReader byte
func (b ByteReader) Read(buf []byte) (int, error) {
for i := range buf {
buf[i] = byte(b)
}
return len(buf), nil
}
type LogReader struct {
io.Reader
}
func (r LogReader) Read(b []byte) (int, error) {
n, err := r.Reader.Read(b)
log.Printf("read %d bytes, error: %v", n, err)
return n, err
}
func main() {
// START OMIT
var r io.Reader = ByteReader('A')
r = io.LimitReader(r, 1e6)
r = LogReader{r}
io.Copy(ioutil.Discard, r)
// STOP OMIT
return
io.Copy(ioutil.Discard, LogReader{io.LimitReader(ByteReader('A'), 1e6)})
}
Copy
can't know about the underlying data structures.
sort.Interface
describes the operations required to sort a collection:
type Interface interface { Len() int Less(i, j int) bool Swap(i, j int) }
IntSlice
can sort a slice of ints:
type IntSlice []int func (p IntSlice) Len() int { return len(p) } func (p IntSlice) Less(i, j int) bool { return p[i] < p[j] } func (p IntSlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
sort.Sort
uses can sort a []int
with IntSlice
:
// +build ignore,OMIT
package main
import (
"fmt"
"sort"
)
type IntSlice []int
func (p IntSlice) Len() int { return len(p) }
func (p IntSlice) Less(i, j int) bool { return p[i] < p[j] }
func (p IntSlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
func main() {
s := []int{7, 5, 3, 11, 2} sort.Sort(IntSlice(s)) fmt.Println(s)
}
The Organ
type describes a body part and can print itself:
// +build ignore,OMIT
package main
import "fmt"
type Organ struct { Name string Weight Grams } func (o *Organ) String() string { return fmt.Sprintf("%v (%v)", o.Name, o.Weight) } type Grams int func (g Grams) String() string { return fmt.Sprintf("%dg", int(g)) } func main() { s := []*Organ{{"brain", 1340}, {"heart", 290}, {"liver", 1494}, {"pancreas", 131}, {"spleen", 162}} for _, o := range s { fmt.Println(o) } }
The Organs
type knows how to describe and mutate a slice of organs:
type Organs []*Organ func (s Organs) Len() int { return len(s) } func (s Organs) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
The ByName
and ByWeight
types embed Organs
to sort by different fields:
type ByName struct{ Organs } func (s ByName) Less(i, j int) bool { return s.Organs[i].Name < s.Organs[j].Name } type ByWeight struct{ Organs } func (s ByWeight) Less(i, j int) bool { return s.Organs[i].Weight < s.Organs[j].Weight }
With embedding we compose types.
14
To sort a []*Organ
, wrap it with ByName
or ByWeight
and pass it to sort.Sort
:
// +build ignore,OMIT
package main
import (
"fmt"
"sort"
)
type Organ struct {
Name string
Weight Grams
}
func (o *Organ) String() string { return fmt.Sprintf("%v (%v)", o.Name, o.Weight) }
type Grams int
func (g Grams) String() string { return fmt.Sprintf("%dg", int(g)) }
// PART1 OMIT
type Organs []*Organ
func (s Organs) Len() int { return len(s) }
func (s Organs) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
// PART2 OMIT
type ByName struct{ Organs }
func (s ByName) Less(i, j int) bool { return s.Organs[i].Name < s.Organs[j].Name }
type ByWeight struct{ Organs }
func (s ByWeight) Less(i, j int) bool { return s.Organs[i].Weight < s.Organs[j].Weight }
// PART3 OMIT
func main() {
s := []*Organ{ {"brain", 1340}, {"heart", 290}, {"liver", 1494}, {"pancreas", 131}, {"spleen", 162}, } sort.Sort(ByWeight{s}) printOrgans("Organs by weight", s) sort.Sort(ByName{s}) printOrgans("Organs by name", s)
}
func printOrgans(t string, s []*Organ) {
fmt.Printf("%s:\n", t)
for _, o := range s {
fmt.Printf(" %v\n", o)
}
}
The Reverse
function takes a sort.Interface
and
returns a sort.Interface
with an inverted Less
method:
func Reverse(data sort.Interface) sort.Interface { return &reverse{data} } type reverse struct{ sort.Interface } func (r reverse) Less(i, j int) bool { return r.Interface.Less(j, i) }
To sort the organs in descending order, compose our sort types with Reverse
:
// +build ignore,OMIT
package main
import (
"fmt"
"sort"
)
type Organ struct {
Name string
Weight Grams
}
func (o *Organ) String() string { return fmt.Sprintf("%v (%v)", o.Name, o.Weight) }
type Grams int
func (g Grams) String() string { return fmt.Sprintf("%dg", int(g)) }
type Organs []*Organ
func (s Organs) Len() int { return len(s) }
func (s Organs) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
type ByName struct{ Organs }
func (s ByName) Less(i, j int) bool { return s.Organs[i].Name < s.Organs[j].Name }
type ByWeight struct{ Organs }
func (s ByWeight) Less(i, j int) bool { return s.Organs[i].Weight < s.Organs[j].Weight }
func main() {
s := []*Organ{
{"brain", 1340},
{"heart", 290},
{"liver", 1494},
{"pancreas", 131},
{"spleen", 162},
}
sort.Sort(Reverse(ByWeight{s})) printOrgans("Organs by weight (descending)", s) sort.Sort(Reverse(ByName{s})) printOrgans("Organs by name (descending)", s)
}
func printOrgans(t string, s []*Organ) {
fmt.Printf("%s:\n", t)
for _, o := range s {
fmt.Printf(" %v\n", o)
}
}
func Reverse(data sort.Interface) sort.Interface {
return &reverse{data}
}
type reverse struct{ sort.Interface }
func (r reverse) Less(i, j int) bool {
return r.Interface.Less(j, i) // HL
}
These are not just cool tricks.
This is how we structure programs in Go.
17Sigourney is a modular audio synthesizer I wrote in Go.
Audio is generated by a chain of Processors
:
type Processor interface { Process(buffer []Sample) }
Roshi is a time-series event store written by Peter Bourgon. It provides this API:
Insert(key, timestamp, value) Delete(key, timestamp, value) Select(key, offset, limit) []TimestampValue
The same API is implemented by the farm
and cluster
parts of the system.
An elegant design that exhibits composition.
(github.com/soundcloud/roshi)
Interfaces are the generic programming mechanism.
This gives all Go code a familiar shape.
Less is more.
20It's all about composition.
Interfaces—by design and convention—encourage us to write composable code.
21
Interfaces types are just types
and interface values are just values.
They are orthogonal to the rest of the language.
22Interfaces separate data from behavior. (Classes conflate them.)
type HandlerFunc func(ResponseWriter, *Request) func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) { f(w, r) }
Think about composition.
Better to have many small simple things than one big complex thing.
Also: what I thought of as small is pretty big.
Some repetition in the small is okay when it benefits "the large".
24
My first exposure to concurrency was in C, Java, and Python.
Later: event-driven models in Python and JavaScript.
When I saw Go I saw:
"The performance of an event-driven model without callback hell."
But I had questions: "Why can't I wait on or kill a goroutine?"
26Goroutines provide concurrent execution.
Channels express the communication and synchronization of independent processes.
Select enables computation on channel operations.
The binary tree comparison exercise from the Go Tour.
"Implement a function
func Same(t1, t2 *tree.Tree) bool
that compares the contents of two binary trees."
type Tree struct { Left, Right *Tree Value int }
A simple depth-first tree traversal:
// +build ignore,OMIT
package main
import (
"fmt"
"code.google.com/p/go-tour/tree"
)
func Walk(t *tree.Tree) { if t.Left != nil { Walk(t.Left) } fmt.Println(t.Value) if t.Right != nil { Walk(t.Right) } } func main() { Walk(tree.New(1)) }
A concurrent walker:
func Walk(root *tree.Tree) chan int { ch := make(chan int) go func() { walk(root, ch) close(ch) }() return ch } func walk(t *tree.Tree, ch chan int) { if t.Left != nil { walk(t.Left, ch) } ch <- t.Value if t.Right != nil { walk(t.Right, ch) } }
Walking two trees concurrently:
// +build ignore,OMIT
package main
import (
"fmt"
"code.google.com/p/go-tour/tree"
)
func Walk(root *tree.Tree) chan int {
ch := make(chan int)
go func() {
walk(root, ch)
close(ch)
}()
return ch
}
func walk(t *tree.Tree, ch chan int) {
if t.Left != nil {
walk(t.Left, ch)
}
ch <- t.Value
if t.Right != nil {
walk(t.Right, ch)
}
}
// STOP OMIT
func Same(t1, t2 *tree.Tree) bool { w1, w2 := Walk(t1), Walk(t2) for { v1, ok1 := <-w1 v2, ok2 := <-w2 if v1 != v2 || ok1 != ok2 { return false } if !ok1 { return true } } } func main() { fmt.Println(Same(tree.New(3), tree.New(3))) fmt.Println(Same(tree.New(1), tree.New(2))) }
func Same(t1, t2 *tree.Tree) bool { w1, w2 := Walk(t1), Walk(t2) for { v1, ok1 := w1.Next() v2, ok2 := w2.Next() if v1 != v2 || ok1 != ok2 { return false } if !ok1 { return true } } }
The Walk
function has nearly the same signature:
func Walk(root *tree.Tree) *Walker {
func (w *Walker) Next() (int, bool) {
(We call Next
instead of the channel receive.)
But the implementation is much more complex:
func Walk(root *tree.Tree) *Walker { return &Walker{stack: []*frame{{t: root}}} } type Walker struct { stack []*frame } type frame struct { t *tree.Tree pc int } func (w *Walker) Next() (int, bool) { if len(w.stack) == 0 { return 0, false } // continued next slide ...
f := w.stack[len(w.stack)-1] if f.pc == 0 { f.pc++ if l := f.t.Left; l != nil { w.stack = append(w.stack, &frame{t: l}) return w.Next() } } if f.pc == 1 { f.pc++ return f.t.Value, true } if f.pc == 2 { f.pc++ if r := f.t.Right; r != nil { w.stack = append(w.stack, &frame{t: r}) return w.Next() } } w.stack = w.stack[:len(w.stack)-1] return w.Next() }
func Walk(root *tree.Tree) chan int { ch := make(chan int) go func() { walk(root, ch) close(ch) }() return ch } func walk(t *tree.Tree, ch chan int) { if t.Left != nil { walk(t.Left, ch) } ch <- t.Value if t.Right != nil { walk(t.Right, ch) } }
But there's a problem: when an inequality is found,
a goroutine might be left blocked sending to ch
.
Add a quit
channel to the walker so we can stop it mid-stride.
func Walk(root *tree.Tree, quit chan struct{}) chan int { ch := make(chan int) go func() { walk(root, ch, quit) close(ch) }() return ch } func walk(t *tree.Tree, ch chan int, quit chan struct{}) { if t.Left != nil { walk(t.Left, ch, quit) } select { case ch <- t.Value: case <-quit: return } if t.Right != nil { walk(t.Right, ch, quit) } }
Create a quit
channel and pass it to each walker.
By closing quit
when the Same
exits, any running walkers are terminated.
func Same(t1, t2 *tree.Tree) bool { quit := make(chan struct{}) defer close(quit) w1, w2 := Walk(t1, quit), Walk(t2, quit) for { v1, ok1 := <-w1 v2, ok2 := <-w2 if v1 != v2 || ok1 != ok2 { return false } if !ok1 { return true } } }
Goroutines are invisible to Go code. They can't be killed or waited on.
You have to build that yourself.
There's a reason:
As soon as Go code knows in which thread it runs you get thread-locality.
Thread-locality defeats the concurrency model.
38
The model makes concurrent code easy to read and write.
(Makes concurrency is accessible.)
This encourages the decomposition of independent computations.
39The simplicity of the concurrency model makes it flexible.
Channels are just values; they fit right into the type system.
Goroutines are invisible to Go code; this gives you concurrency anywhere.
Less is more.
40Concurrency is not just for doing more things faster.
It's for writing better code.
41At first, Go syntax felt a bit inflexible and verbose.
It affords few of the conveniences to which I was accustomed.
For instance:
Favor readability above all.
Offer enough sugar to be productive, but not too much.
44
Getters and setters turn assignments and reads into function calls.
This leads to surprising hidden behavior.
In Go, just write (and call) the methods.
The control flow cannot be obscured.
45Map/filter/reduce/zip are useful in Python.
a = [1, 2, 3, 4] b = map(lambda x: x+1, a)
In Go, you just write the loops.
a := []int{1, 2, 3, 4} b := make([]int, len(a)) for i, x := range a { b[i] = x+1 }
This is a little more verbose,
but makes the performance characteristics obvious.
It's easy code to write, and you get more control.
46Go functions can't have optional arguments.
Instead, use variations of the function:
func NewWriter(w io.Writer) *Writer func NewWriterLevel(w io.Writer, level int) (*Writer, error)
Or an options struct:
func New(o *Options) (*Jar, error) type Options struct { PublicSuffixList PublicSuffixList }
Or a variadic list of options.
Create many small simple things, not one big complex thing.
47The language resists convoluted code.
With obvious control flow, it's easy to navigate unfamiliar code.
Instead we create more small things that are easy to document and understand.
So Go code is easy to read.
(And with gofmt, it's easy to write readable code.)
48I was often too clever for my own good.
I appreciate the consistency, clarity, and transparency of Go code.
I sometimes miss the conveniences, but rarely.
49I had previously used exceptions to handle errors.
Go's error handling model felt verbose by comparison.
I was immediately tired of typing this:
if err != nil { return err }
Go codifies errors with the built-in error
interface:
type error interface { Error() string }
Error values are used just like any other value.
func doSomething() error err := doSomething() if err != nil { log.Println("An error occurred:", err) }
Error handling code is just code.
(Started as a convention (os.Error
). We made it built in for Go 1.)
Error handling is important.
Go makes error handling as important as any other code.
53
Errors are just values; they fit easily into the rest of the language
(interfaces, channels, and so on).
Result: Go code handles errors correctly and elegantly.
54We use the same language for errors as everything else.
Lack of hidden control flow (throw/try/catch/finally) improves readability.
Less is more.
55To write good code we must think about errors.
Exceptions make it easy to avoid thinking about errors.
(Errors shouldn't be "exceptional!")
Go encourages us to consider every error condition.
My Go programs are far more robust than my programs in other languages.
I don't miss exceptions at all.
56
I found the capital-letter-visibility rule weird;
"Let me use my own naming scheme!"
I didn't like "package per directory";
"Let me use my own structure!"
I was disappointed by lack of monkey patching.
58Go packages are a name space for types, functions, variables, and constants.
59
Visibility is at the package level.
Names are "exported" when they begin with a capital letter.
package zip func NewReader(r io.ReaderAt, size int64) (*Reader, error) // exported type Reader struct { // exported File []*File // exported Comment string // exported r io.ReaderAt // unexported } func (f *File) Open() (rc io.ReadCloser, err error) // exported func (f *File) findBodyOffset() (int64, error) // unexported func readDirectoryHeader(f *File, r io.Reader) error // unexported
Good for readability: easy to see whether a name is part of the public interface.
Good for design: couples naming decisions with interface decisions.
Packages can be spread across multiple files.
Permits shared private implementation and informal code organization.
Packages files must live in a directory unique to the package.
The path to that directory determines the package's import path.
The build system locates dependencies from the source alone.
61Go forbids modifying package declarations from outside the package.
But we can get similar behavior using global variables:
package flag var Usage = func() { fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0]) PrintDefaults() }
Or registration functions:
package http func Handle(pattern string, handler Handler)
This gives the flexibility of monkey patching but on the package author's terms.
(This depends on Go's initialization semantics.)
62The loose organization of packages lets us write and refactor code quickly.
But packages encourage the programmer to consider the public interface.
This leads to good names and simpler interfaces.
With the source as the single source of truth,
there are no makefiles to get out of sync.
(This design enables great tools like pkg.go.dev and goimports.)
Predictable semantics make packages easy to read, understand, and use.
63
Go's package system taught me to prioritize the consumer of my code.
(Even if that consumer is me.)
It also stopped me from doing gross stuff.
Packages are rigid where it matters, and loose where it doesn't.
It just feels right.
Probably my favorite part of the language.
64
Godoc reads documentation from Go source code, like pydoc
or javadoc
.
But unlike those two, it doesn't support complex formatting or other meta data.
Why?
Godoc comments precede the declaration of an exported identifier:
// Join concatenates the elements of a to create a single string. // The separator string sep is placed between elements in the resulting string. func Join(a []string, sep string) string {
It extracts the comments and presents them:
$ godoc strings Join func Join(a []string, sep string) string Join concatenates the elements of a to create a single string. The separator string sep is placed between elements in the resulting string.
Also integrated with the testing framework to provide testable example functions.
func ExampleJoin() { s := []string{"foo", "bar", "baz"} fmt.Println(strings.Join(s, ", ")) // Output: foo, bar, baz }
Godoc wants you to write good comments, so the source looks great:
// ValidMove reports whether the specified move is valid. func ValidMove(from, to Position) bool
Javadoc just wants to produce pretty documentation, so the source is hideous:
/** * Validates a chess move. * * @param fromPos position from which a piece is being moved * @param toPos position to which a piece is being moved * @return true if the move is valid, otherwise false */ boolean isValidMove(Position fromPos, Position toPos)
(Also a grep for "ValidMove"
will return the first line of documentation.)
Godoc taught me to write documentation as I code.
Writing documentation improves the code I write.
70There are many more examples.
The overriding theme:
Those decisions make the language—and Go code—better.
Sometimes you have to live with the language a while to see it.
71Be articulate:
New features can weaken existing features.
Features multiply complexity.
Complexity defeats orthogonality.
Orthogonality is vital: it enables composition.
74Don't solve problems by building a thing.
Instead, combine simple tools and compose them.
75Invest the time to find the simple solution.
77These lessons were all things I already "knew".
Go helped me internalize them.
Go made me a better programmer.
78Let's build small, simple, and beautiful things together.