«

»

Jul 28

Go language for python programmers

Preface

I’ve been programming in python for over 5 years now, and I love the language a lot. I would look for any opportunity to accomplish my coding tasks using python, as opposed to learning new languages. PyQt4 for user interfaces, Django for web design, etc… And when python just isn’t an option, I have done my share of php, javascript, objective-c and so on.
Recently I started thinking that I should probably expand my programming knowledge a bit to make myself more marketable. While python is pretty key in the world of Visual Effects pipelines… so is C++. Having previously focused on being a Compositor and not a programmer, I had always told myself that if I needed to learn C++, then I was going too far in that direction. But now that I have been working in pipeline development for over 2.5 years, it has become clear that I really should explore the world of compiled languages.

Grabbing some books on C+, I dove in. Yuk. Why is it so freaking boring? It really does suck trying to learn C++ after having been so spoiled with python for so long. Its like taking a Aborigine and trying to turn him into a snooty English gentleman. Hmm…Is that right? Well whatever. Its completely disorienting. So many things that I never had to think about, like type declarations, memory management, pointers… But I kept on reading and learning.

Then I came across a post on Google+, by someone that actually works at Google, mentioning a language being developed in-house. So I started reading about Go @ http://golang.org/

The Go Language

From the standpoint of a python programmer, Go feels like it sits right between C/C++ and python. You get the simpler syntax, but with the speed of a compiled language. Because Go has garbage collection, memory management isn’t a concern. I was never used to the code/compile/test/repeat pattern before, but it compiles so fast and is so easy to set up a project that it feels pretty natural. Go doesn’t require you to have header files and declare everything in advance, so banging out a simple program is quite fast and only slightly more overhead than writing a python script. Just have to add the step of compiling it. As far as the library, so far I have found everything pretty useful. And it seems that every time I search for a Go binding for something, I find one. I will go into more detail on that in a bit.

One thing that will make python programmers feel more at home is the type inference while creating a variable. While you can do something like this in Go:  var myString string = "Foo" you can also type the same thing like this:  myString := "Foo"
:= operator lets you initialize a new variable and makes the compiler figure out what type it should be, based on the return type of the right hand side.

Pointers are still kind of strange for python programmers, but Go is a lot more flexible about using them. You don’t always have to explicitly dereference them like   (*myPointer).myFunction(). It just does it for you when you access member functions and attributes:  myPointer.myFunction().

Having no type inheritance is also a bit different as well. Instead of creating base abstract classes, and subclassing them, you only have structs… no classes. But you share functionality by using interfaces. An interface is just a definition of methods. If any object implements those methods, its consider that type of interface. This is something I have yet to really get into, since my current first project is more of a cmd program, rather than a pkg library. I’m sure I could be using interfaces already, but it hasn’t quite felt natural enough to incorporate as of yet.

A pretty crazy aspect of Go is its native support for concurrency using what they call “Goroutines”. The closest way I have been able to compare it to my experience in python is while using ZeroMQ for messaging. ZeroMQ promotes not only using its library for messaging, but also to replace issues with threads and locks. It has similar concepts in promoting concurrency. You divide your program up into its components and instead of sharing data structures between them, you communicate over channels (sockets in ZeroMQ). When you fire off a Goroutine, you aren’t waiting for it anymore. It can run and do its thing and you keep on going. You can then send data back and forth with channels, and even use them just for signaling, like saying “ok now exit”.

Actual Usage

I’ve been writing a message server so far in python, using the Tornado web server, along with some socket.io bindings called TornadIO, and also ZeroMQ for internal communication. So far its been working pretty well, but there are a lot of complex layers, with ZeroMQ sort of riding on top of Tornados ioloop. I decided to try and rewrite this server in Go. Turns out Go has a lot of built-in support for doing exactly what this python server was doing. The whole web-socket server functionality is part of the standard http library module. I quickly found Go bindings for socket.io and I was on my way.

It was quite fast to get the server to the point of doing global messages, but now I had to think about implementing the support for channel subscriptions. My first instinct was to go grab ZeroMQ and its bindings again, or to use Redis for the messaging, but then I was thinking “Shouldn’t I be able to do all this with Go’s concurrency?”. One thing that really helped me out was how fast everyone responds on the golang-nuts discussion group. It was quickly pointed out by more than one person that I should definitely be able to accomplish the internal message routing purely in Go. And they were right. I just set up a “dispatcher” function and run it in a loop as a goroutine, and then pass messages in and out of it. The dispatcher manages its data structure, and no other part of the code accesses it directly.

So far, this Go server is turning out great, and I’m excited by the fact that its compiled and faster. I don’t have to distribute my source code now :-)

Performance

* Disclaimer: These numbers are just comparisons between what I built in python vs Go. I think the point is to reflect what I naturally came up with on my first pass at using Go, vs applying years of python experience. I’m dead sure my Go code isn’t written as efficiently as someone with more experience, which I think makes it even more interesting of a comparison.

Go comes with built in testing and benchmarking functionality. What I built was a client test that connects to a running server and rapid fires messages. It times how long it takes for a 150 byte message to be sent out, flow through the server, and come back to that client as been delivered. The gotest utility that is used to run the test code will run the test, and if it ran too fast to calculate timings, it will repeat the test over and over with larger iterations. When I first got my Go server working to where a client would send a message and it would just get broadcasted right back out to everyone, I ran a benchmark. Here are the results of my tests…

TEST TIME PER MSG NET RESULT
Python (Tornado, …) 835314 ns/op -
Go (barebones messaging) 107091 ns/op 7.8x faster
Go (1-to-1 python port) 159823 ns/op 5.2x faster
Go (weekly.12-01-2011) 76230 ns/op 11x faster

 

The second Go test was after I finished implementing the same 1-to-1 feature set of the python version. I had thought the numbers would be dramatically impacted after adding the overhead of all the internal message routing, but the Go server still came out almost 5x faster than the python version. And this is from my first attempt at writing a Go program! I bet once I get a lot more solid with the language I can optimize this code a lot more.

Conclusion

I’m pretty hooked on the language. I feel its the perfect option for a python programmer that wants some speed increases and simple concurrency, without having to learn something as intense as C++. Go is supposed to get faster and faster as they improve things like the goroutines, channels, and the garbage collector, so its a great time to jump in and start learning. Its really helped me understand more formal concepts that will probably make learning C++ even easier once I decide to go back to learning it :-)

 

4 comments

Skip to comment form

  1. Greg Worley

    I’m just about to watch your video on Python in Maya, and came across your blog, and this article. I’m using Go in every project that I can and especially appreciate how great the mailing list is. In fact, watching you video is going to be a refresher in Python for me.

    1. Justin Israel

      Thanks for checkin out my video, Greg! And yea, that Go community is really top notch. I only have two projects in Go right now so I am still learning, and I regularly follow the message board to gain as much info as possible. The dev team is very involved in the community.

  2. Ralph Caraveo

    Hi Justin,

    I know this article is a little old but I wanted to ask a question about Go vs the Python/0MQ approach. While it seams a lot can be said about Go with it’s built-in concurrency support using goroutines and channels, I’m still not sure if it completely replaces certain really nice functionality that exists in 0MQ. For example, in 0MQ, you can create a client and server and you don’t have to worry about building the low level socket communication between them. With 0MQ, you just bind to a port for the server and your client can just connect to a port and the plumbing magically works using message based content vs stream based. Also, it doesn’t matter the order of which one connects first, and if one goes offline, it will connect back up when the process is restarted. Also, 0MQ has the support for network topologys like PUB/SUB, REQ/REPLY and other patterns.

    While I would love to fully ditch 0MQ, do you know if using Go alone can fully replaces some of these concepts that 0MQ has support for? If so, that would be great and then I can truly see that Go can stand on it’s own.

    Thanks,

    -Ralph

    1. Justin Israel

      Go doesn’t natively have all of the 1-to-1 functionality of 0MQ in its stdlib. They even have 0MQ bindings for Go, to continue using it exactly the same way. I would have to say though, that Go has quite a lot of “ready to go” networking like starting an http server in about 5 lines of code.
      You can still totally keep using 0MQ regardless of the language.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>