«

»

Jul 25

Python 2.7.2 serializer speed comparisons

In a recent python project where I was sending multiple messages per second of data over a basic socket, I had initially just grabbed the cPickle module to get the prototype proof-of-concept functioning properly. cPickle is awesome for easily serializing more complex python objects like custom classes, even though in my case I am only sending basic types.

My messages were dicts with some nested dicts, lists, floats, and string values. Roughly 500-1000 bytes. cPickle was doing just fine, but there came a point where I wanted to investigate the areas that could be tightened up. The first thing I realized was that I had forgotten to encode cPickle in the binary format (the default is ascii). That saved me quite a bit of time. But then I casually searched online to see if any json options might be better since my data is pretty primitive anyways.

I found UltraJSON, which is a pure C json parsing library for python, and ran some tests. There are benchmarks on the project page for ujson, as well as other articles on the internet, but I just wanted to post up my own results using a mixed type data container. ujson came out extremely fast: faster than binary cPickle and msgpack, in the encoding test. Although in the decoding test, msgpack appeared to be fastest, followed by binary cPickle, and then ujson coming in 3rd

This test included the following:

Here is my Python 2.7.2 test script using timeit for each encode and decode step.

4 comments

Skip to comment form

  1. schlamar

    You should have included “msgpack”, it is really fast. See my benchmark results.

    1. Justin Israel

      I’ve just redone the entire test, including msgpack, and including specific versions of the packages used in the test.

  2. Andrei

    Do not forget to mention the package versions!!!

    1. Justin Israel

      I will re-run these tests against specific versions, and properly document the version numbers.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>