Tuesday, March 17, 2009

More on benchmarking java serialization tools

The serialization benchmarking discussed in previous posts is getting to be more interesting. Thanks to all who looked at the code, contributed, suggested and pointed bugs. Tree major contributions are from cowtowncoder who fixed the stax code, Chris Pettitt who added the json code and David Bernard for the xstream and java externalizable. Most of the code is at the google code svn repository.
The charts are scaled and some are chopped. So if you’re interested in exact numbers, here they are:

Library, Object create, Serializaton, Deserialization, Serilized Size
java , 113.23390, 17305.80500, 72637.29300, 845
xstream default , 116.40035, 119932.61000, 171796.68850, 931
json , 112.58555, 3324.76450, 5318.12600, 310
stax , 113.05025, 6172.06000, 9566.96200, 406
java (externalizable) , 99.76580, 6250.40100, 18970.58100, 315
thrift , 174.72665, 4635.35750, 5133.24450, 314
scala , 66.10890, 27047.10850, 155413.44000, 1473
protobuf , 250.37140, 3849.69050, 2416.94800, 217
xstream with conv , 115.22810, 13492.50250, 47056.58750, 321

Serialize size (bytes), less is better.
May very a lot depending on number of repetitions in lists, usage of number compacting in protobuf, strings vs numerics and more. Interesting point is Scala and Java which holds the name of the classes in the serialized form. I.e. longer class names = larger serialized form. In Scala its worse since the Scala compiler creates more implicit classes then java.

Deserialization in nanoseconds. The most expensive operation. Note that the xstream and Scala lines got trimmed.

Serialization (nanoseconds), way faster then deserialization.

Object creation, not so meaningful since it takes in average 100 nano to create an object. The surprise comes from protobuf which takes a very long time to create an object. Its the only point in this set of benchmarks where it didn't perform as well as thrift. Scala (and to a lesser point - java) on the other hand is fast, seems like its a good language to handle in memory data structures but when coming to serialization you might want to check the alternatives.

17 comments:

Ophir March 18, 2009 at 6:48 AM  

Very interesting, thanks for the input. I'm wondering how Cajo (https://cajo.dev.java.net/) would compare.

David Bernard March 18, 2009 at 8:14 AM  

Just a quick note, result of the bench is important. But don't forget when you choose a technologie, the size of the code, developer need to write to serialize/deserialize "business" object (protobuf message aren't). If you have lot of message it could be a long task.

Eishay Smith March 18, 2009 at 8:42 AM  

@ofir you are welcome to write the code and compare, though I don't see how is Cajo relevant to this benchmark.
@david i agree. One first need to see it serialization is a bottleneck at all. If its not and will never be then premature optimization is the source of all evil. If it is a bottleneck then the performance may vary a lot with different data sets (size, type etc.) and jvm profile.

Ophir March 19, 2009 at 3:07 AM  

@eishay At it's core Cajo is about RPC, which means a lot of serialization/deserialization of object representations. I'm evaluating several remoting alternatives and your entry is very instrumental doing that.

David Bernard March 19, 2009 at 8:27 AM  

More results (with new lib)

, Object create, Serializaton, Deserialization, Serialized Size
java (externalizable) , 73.65100, 4683.54400, 14538.46300, 315
xstream(xpp) , 97.98270, 75685.54700, 135505.52950, 931
json , 87.99550, 2847.90250, 4306.42750, 310
protobuf , 189.38740, 3038.74250, 1979.13000, 217
xstream(stax) with conv , 87.61310, 9067.81750, 18113.96850, 325
java , 86.94640, 12254.38150, 57572.16250, 845
sbinary , 55.59485, 2846.53100, 2758.94450, 190
xstream(xpp) with conv , 87.63940, 10858.89200, 36172.06400, 310
thrift , 142.58605, 3683.40700, 3812.90900, 314
javolution xmlformat , 87.79790, 4487.80100, 10278.79150, 345
stax , 87.28780, 4323.57150, 7692.04550, 406
scala , 68.03025, 31674.93900, 150840.46500, 1950

cowtowncoder March 20, 2009 at 11:31 PM  

I added some more tests; xml/stax with Aalto, as well as Fast Infoset (binary xml) via its stax api implementation.

Eishay Smith March 21, 2009 at 12:05 AM  

Cool!
I'll post the results in a dedicated page, its growing too big for this blog :-)

cowtowncoder March 22, 2009 at 8:01 PM  

Yes, separate page(s) make sense. Perhaps at the project page?

One more thing to possibly add: Hessian is one more simple thing to add, and something that is quite fast as well based on tests I did. It's a binary (java) object serialization protocol.
I could cobble a test together quite easily I think.

As to Cajo, I also do not quite see how that would fit. It seems like a big framework, and nothing suggest it uses a separate module serialization/deserialization component. From description I'd guess it could use plain old Java serialization.

Eishay Smith March 22, 2009 at 8:11 PM  

@cowtowncoder the main page is on the project's wiki at http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking?ts=1237772203&updated=Benchmarking
Project commiters can update it and/or add more pages. For public discussion lets use the google group from now on at http://groups.google.com/group/java-serialization-benchmarking
Adding Hessian sounds like a good idea.

Bryan Duxbury May 2, 2009 at 6:03 PM  

I assume you are using Binary Protocol for Thrift. If you try the newer Compact Protocol, it will serialize down probably as small as protobuf.

Also, if you are using the released version of Thrift from sometime last year, the current trunk version (which is very stable, release coming soon!) has had a fair amount of performance work done on it.

Eishay Smith May 2, 2009 at 6:11 PM  

Thanks, I'll check it out once it'll be released!

Anonymous December 19, 2009 at 5:49 AM  

Hello,

I have also written, but not publicized, a benchmark to test serialization. I found this project and I was however very surprised by the published results, which do not match mine.

I was mostly interested in the differences between avro/reflection, java, hessian. I was also surprised to see that text serializers (xml, json) are better then hessian or java.

Taking a look at the source. I was wondering why not use an existing benchmark framework such as p-unit or japex.

I also think that each benchmark should be running in its own JVM, so as to make sure that all tests are run under the same conditions.
I am not sure that the warm-up methods are enough.

The single test also includes the creation of the serializer. This does not take into account that serializers may have optimizations, so that the first call to the serializer will take much longer then subsequent calls.
So there should be tests for different use cases:

- single serialize/deserialize (for example used in deep clone)
- single serializer/deserializer for multiple operations (for example used in client/server communication)
- etc... any suggestions ?

The test is conducted with only one type of data. However different serializers excel in different domains. Some are better for graph structures, some for flat data, some depend on the length of the attribute names, package names etc., some optimize better with repeated objects, etc.

The test does not differentiate between the features and requirements of the different serializers. Some require a schema or stub, some require code generation, some are schema tolerant, some require the transmission/storage of the schema etc.

I hope these comments will help create a better benchmark.

-- Ron
http://yajsw.sourceforge.net

Anonymous December 19, 2009 at 5:50 AM  

Hello,

I have also written, but not publicized, a benchmark to test serialization. I found this project and I was however very surprised by the published results, which do not match mine.

I was mostly interested in the differences between avro/reflection, java, hessian. I was also surprised to see that text serializers (xml, json) are better then hessian or java.

Taking a look at the source. I was wondering why not use an existing benchmark framework such as p-unit or japex.

I also think that each benchmark should be running in its own JVM, so as to make sure that all tests are run under the same conditions.
I am not sure that the warm-up methods are enough.

The single test also includes the creation of the serializer. This does not take into account that serializers may have optimizations, so that the first call to the serializer will take much longer then subsequent calls.
So there should be tests for different use cases:

- single serialize/deserialize (for example used in deep clone)
- single serializer/deserializer for multiple operations (for example used in client/server communication)
- etc... any suggestions ?

The test is conducted with only one type of data. However different serializers excel in different domains. Some are better for graph structures, some for flat data, some depend on the length of the attribute names, package names etc., some optimize better with repeated objects, etc.

The test does not differentiate between the features and requirements of the different serializers. Some require a schema or stub, some require code generation, some are schema tolerant, some require the transmission/storage of the schema etc.

I hope these comments will help create a better benchmark.

-- Ron
http://yajsw.sourceforge.net

cowtowncoder December 20, 2009 at 9:20 PM  

Hi Ron! These are valid issues, and most are known to developers. And new contributions for improvements are happily accepted I think (I have contributed some additions, others too, it's a work in progress, as always).

One thing that may help explain results is the small size of the test document -- this heavily penalizes implementations that have significant per-document overhead. Seems like this is the case for Hessian. I was also surprised to see its poor performance, as I too have benchmarked it to be quite fast on other tests. But on the other hand, PB and Thrift do quite well, so there is something odd with Hessian here (I contributed hessian driver so it should not have any obvious flaws -- but if there are, I would love to find them).

Also: serializers for json, xml, are the fastest available. There are many implementations that are significantly slower; so another reason may be that perhaps some binary codecs are not quite as heavily optimized.

At any rate I think that binary codecs would show better relative performance for larger documents. Default Java serialization possibly a little bit too; but that depends on ratio of class metadata to actual data (JDK serialization has to add per-class metadata for each included type -- when there are multiple instances, overhead is less).

Eishay Smith December 20, 2009 at 9:50 PM  

Thanks Ron!
> I was however very surprised by the published results
In the project wiki I tried to emphasize that the sample set may be misleading and some serializers work better with some datasets and not others. See the maling list for more discussions about it. The data type (integer, floating point, characters, enums), their lengths and the data structures (single value, list, map) are very important when deciding on the serializer to use. At kaChing for example we have a pluggable serializer module and we switch between JSON and Protobuf according to the value types.

> I was also surprised to see that text serializers (xml, json) are better then hessian or java.
When dealing with small diverse data structure (mix of few strings and numerics) they are much better. Less so if you have a large list of the same type.

> I was wondering why not use an existing benchmark framework such as p-unit or japex.
No reason. Contributions for integration will be welcomed.

> I also think that each benchmark should be running in its own JVM, so as to make sure that all tests are run under the same conditions.
They do.

> I am not sure that the warm-up methods are enough.
Played with more warm-up time and this one was the optimal. Please let me know if you have better warm-up properties.

> I hope these comments will help create a better benchmark.
They might, joining the project and contributing is more effective :-)

Harsha January 22, 2011 at 9:15 PM  

http://harshcreationz.blogspot.com/2011/01/serialization-and-deserialization-from.html
Here you can deserialize object from another project

Javin Paul April 16, 2011 at 5:51 AM  

Nice article , you have indeed covered topic in details with benchmarking result and graphics.

Thanks
Javin
Top 10 Java Serialization Interview Question

Creative Commons License This work by Eishay Smith is licensed under a Creative Commons Attribution 3.0 Unported License.