{"id":1634,"date":"2013-11-03T17:24:21","date_gmt":"2013-11-03T17:24:21","guid":{"rendered":"http:\/\/www.syslog.cl.cam.ac.uk\/?p=1634"},"modified":"2013-11-04T12:18:00","modified_gmt":"2013-11-04T12:18:00","slug":"liveblog-from-programming-languages-and-operating-systems-2013","status":"publish","type":"post","link":"https:\/\/www.syslog.cl.cam.ac.uk\/2013\/11\/03\/liveblog-from-programming-languages-and-operating-systems-2013\/","title":{"rendered":"Liveblog from Programming Languages and Operating Systems 2013"},"content":{"rendered":"

\"The<\/a>I'm here in the Nemacolin Woodlands<\/a> at SOSP 2013 with a huge gaggle of the SRG (both old and new), in a room packed to capacity for the PLOS 2013 workshop! \u00c2\u00a0I'm co-chairing it with Tim Harris, so I'll be liveblogging talks when I'm not coordinating sessions. \u00c2\u00a0The keynote is from Russ Cox from Google, and you can find all the papers on the ACM digital library<\/a>.<\/p>\n

<\/p>\n

Russ Cox: Using Go to build distributed systems<\/h2>\n

Go is an open-source programming language that makes it easy to build simple, reliable and efficient software. Started in 2007, open sourced in Nov 2009, and\u00c2\u00a0has been open source development since then. \u00c2\u00a0Started as an answer to software problems at Google: multicore, networked systems, massive clusters, and working at scale: 10^7 LoC, 10^3 programmers, 10^6(+) machines.\"\"<\/a><\/p>\n

Go is a simple but fun language. Start with C and remove the complex parts -- add interfaces, concurrency and GC\/closures\/reflection\/strings.<\/p>\n

\"Less is exponentially more\".<\/p><\/blockquote>\n

Engineering: fast compilation<\/h3>\n

The joke is that Go started when waiting for a large C++ program to compile.<\/p>\n

\r\npackage main\r\n  import \"fmt\"\r\n  func main () {\r\n    fmt.Printf(\"\")\r\n  }\r\n<\/code><\/pre>\n
import \"fmt\" is guaranteed to read exactly one file.\u00c2\u00a0 Metadata in that package describes what else may be needed in the future.<\/p>\n
Q from audience: what doesnt scale with 1000 includes instead of one?
\nA: the compilation time doesnt scale -- not runtime issues.<\/p>\n
\r\n$ go get github.com\/golang\/glog\r\n\r\npackage main\r\n  import (\r\n    \"flag\"\r\n    \"github.com\/golang\/glog\"\r\n  }\r\n\r\nfunc main () {\r\n  flag.Set(\"logtostderr\", \"true\")\r\n  glog.InfoF(\"hello world\")\r\n  }\r\n<\/code><\/pre>\n
Still guaranteed to read exact one file, but the import namespace is decentralized.<\/p>\n
Engineering: Program rewrites<\/h3>\n
There's a mechanical mandated format that go requires for syntax.<\/p>\n
\r\n$ gofmt -r 'glog.InfoF -> glog.Errorf' hello1.go\r\n<\/code><\/pre>\n
Prints the same program but with the Info line from before into Error.\u00c2\u00a0 Very useful for refactoring large code bases.<\/p>\n
Engineering: garbage collection<\/h3>\n
In C\/C++, too much programming and API design is about mem management. Go has garbage collection only.\u00c2\u00a0 Fundamental for interfaces.<\/p>\n
Go lets you limit allocation by controlling memory layout -- for example, a ring buffer.<\/p>\n
The GC is still an area of active research. Design decision: cannot reuse Java GC directly due to an early design decision to permit interior pointers, as are foreign pointers.\u00c2\u00a0 It is implemented as a mark and sweep language.\u00c2\u00a0 It's idiomatic in Go to plan your memory layout upfront, unlike (for example) in Java where you are encouraged to treat garbage collection as \"free\".<\/p>\n
Interfaces<\/h3>\n
An interface defines a set of methods.<\/p>\n
\r\npackage io\r\ntype Writer interface {\r\nWrite (data []byte) (n int, err error)\r\n}\r\n<\/code><\/pre>\n
A type implements the interface by implementing the methods (structural typing).<\/p>\n
An implementation of an interface can be assigned to a variable of that interface type.<\/p>\n
Reader is an obvious counterpart, and this lets you write a function Copy that operates purely over Reader and Writer interfaces.<\/p>\n
A more complex adapter is Adapters.<\/p>\n
\r\nfunc MultiWriter(writers ...Writer) Writer\r\n<\/code><\/pre>\n
creates a writer that duplicates it writes to all the provided writers, like Unix tee.<\/p>\n
Interfaces tend to be fairly small, but one bigger example is in the Networking library.\u00c2\u00a0 Errors are explicitly returned.<\/p>\n
\r\nfunc NewClient(conn net.Conn, host string) (*Client, error)\r\n<\/code><\/pre>\n
You can build other things such as SSL on top of this interface.<\/p>\n
Interface lessons<\/h3>\n
There's no dependence between the interface and implementation, unlike in Java where interfacing is very explicit.\u00c2\u00a0 Expressive composition, easy testing and avoids overdesign and the rigid hierarchy of inheritance-based OO.<\/p>\n
Interfaces are the source of all generality the Go.<\/p>\n
Concurrency vs Parallelism<\/h3>\n
Concurrency is about dealing with a lot of things at once. Parallelism is about doing a lot of things at once.\u00c2\u00a0 Concurrency is about structure, and parallelism is about execution.<\/p>\n
Examples of concurrent: keyboard, display, disk drivers in an OS.<\/p>\n
Parallel: vector dot product, matrix multiply which can often be sped up almost linearly with cores.<\/p>\n
Concurrency enables parallelism, but is useful on its own since modern programs must deal with many things at once.<\/p>\n
Go provides two important concepts: a goroutine is a thread of control within the program, with its own local variables and stack.\u00c2\u00a0 They are cheap and easy to create.<\/p>\n
Channels provide a mechanism for goroutines to communicate via typed messages between goroutines.<\/p>\n
\r\npackage main\r\nimport \"fmt\"\r\n  func main() {\r\n  c := make(chan string)\r\n  go func() {\r\n    c <- \"Hello\"\r\n    c <- \"World\"\r\n  } ()\r\n  fmt.Println(<-c, <-c)\r\n}\r\n<\/code><\/pre>\n
The func() runs concurrently with the main() which consumes those messages in the Println.<\/p>\n
The model is taken from Tony Hoare's CSP and is orthogonal to the rest of the language and can keep a familiar model for computation, and focus on composition instead.\u00c2\u00a0 You can switch to CSP primitives only when you want to worry about computation.\u00c2\u00a0 On the other hand, something like Erlang requires you to learn a new method of working (actors) and is less easy to move from C\/C++.<\/p>\n
Caveat: they're not purely memory safe and sharing is legal.\u00c2\u00a0 Passing a pointer over a channel is idiomatic, and its been working well in practise.<\/p>\n\n
Sequential network address resolution given a work list.<\/p>\n

\nfor _, w := range worklist {
\nw.addrs, w.err = LookupHost(w.host)
\n}
\n<\/code><\/p>\n
This is a simple one that does it step by step.\u00c2\u00a0 To do it in parallel, lets use channels<\/p>\n

\ndone := make(chan bool, len(worklist))<\/code><\/p>\n
for _, w := range worklist {
\ngo func (w *Work) {
\nw.addrs, w.err = LookupHost(w.host)
\ndone <- true
\n} (w)
\n}<\/p>\n
for i:= ; i<len(worklist)l;i++) {
\n<- done
\n}<\/p>\n
Channels are being used here to build a pattern, which can be put in the stdlib.\u00c2\u00a0 This is actually a sync.WaitGroup from the standard library. Parallelism can also be bounded by building a channel to keep track of channel limits.<\/p>\n
If you are using mutexes, you are certainly creating bugs.\u00c2\u00a0 Channels seem easier for people to get right in practise.<\/p>\n
Example: replicated storage with read\/write quorums.<\/b>\u00c2\u00a0 (Eventually consistent)<\/p>\n

\nconst (
\nF = 2
\nN = 5 \/\/ >= 2F + 1
\nReadQuorum = F + 1
\nWriteQuorum = N - F
\n)
\n<\/code><\/p>\n
(Shows an example of this using channels and writes)<\/p>\n
Q: How are slow messages cancellable?<\/p>\n
A: the channel itself will be GCed since the writer stuffs the request into the channel. However cancellation is a common pattern in distributed systems and so a more advanced version can return a channel to be used for interruptions (Which can in turn by watched by a network RPC engine and invoked if stuff takes too long).<\/p>\n
Select allows choosing between multiple channel ops.\u00c2\u00a0 For example a chat program<\/p>\n
\r\nfor {\r\n  select {\r\n  case event := <- ui:\r\n  \/\/process user interface event\r\n  case msg := <- server:\r\n  \/\/ process server message\r\n case t := <- tick:\r\n  \/\/ time has elapsed\r\n  }\r\n}\r\n<\/code><\/pre>\n
Concurrency lessons<\/h3>\n
This is a *key* feature for building distributed systems, and supported by closures and garbage collection.\u00c2\u00a0 Message passing inside the program, and also outside the program.\u00c2\u00a0 This helps you to not change your thinking when moving from within a single program to a distributed system (which has to be message passing).<\/p>\n
Most importantly, do not communicate by sharing memory and instead, share memory by communicating.<\/p>\n
Q: are closures deep copy?
\nA: no, they are capturing references to variables.<\/p>\n
Q: what about cache coherence across domains?
\nA: being worked on, but not there yet.<\/p>\n
Type systems<\/h3>\n
What the type system chooses not to express is as important as what it does.<\/p>\n
Interfaces are most powerful when they are the least restrictive.\u00c2\u00a0 A Write for example, doesnt have a flush call, just a write.\u00c2\u00a0 Restrictions that could bifurcate interfaces that Go doesnt have are:<\/p>\n