Tuesday, November 18, 2008

Protobuf with option optimize for SPEED

Jon Skeet pointed out to me that I missed to include
option optimize_for = SPEED
Thanks Jon!

I added it and it does make a lot of difference. I wonder why its not the default. It appears that without the flag protobuf is using Java introspection which is very expensive. With the speed optimization protobuf is faster then thrift, though not by far.
The serialization speed differences between them are probably not meaningful for transactions that take few miliseconds to perform.




18 comments:

Jon Skeet November 19, 2008 at 1:04 AM  

As far as I can tell, this post is still using the same graphs as the previous one. Am I missing something, or is this a copy and paste error?

Jon Skeet November 19, 2008 at 7:08 AM  

Doh - ignore previous comment. I'd missed the extra line, and was still just looking at the protobuf one...

cowtowncoder March 10, 2009 at 6:15 PM  

One thing to note: Stax reference implementation is NOT a good Stax implementation. Mostly because it is buggy, but it is also not as fast as alternatives, such as:

* Woodstox, [http://woodstox.codehaus.org[
* Sun sjsxp, [http://sjsxp.dev.java.net] (bundled with JDK 1.6)
* Aalto, [http://www.cowtowncoder.com/hatchery/aalto/index.html]

Of these, Aalto is the fastest, then Woodstox then Sjsxp.

So it would be nice to see how upgrading to a good stax implementation would change results for xml.

Eishay Smith March 10, 2009 at 6:38 PM  

Used Woodstox for the benchmark.

cowtowncoder March 10, 2009 at 7:44 PM  

Ok thanks. I was mistaken then, just saw a link was to home page of ref impl and assumed that was the impl used.
Are sources for the test available? Differences are somewhat larger than what I have seen with other tests, so I am curios. Plus might be nice to see how json does and so on.

Eishay Smith March 10, 2009 at 9:09 PM  

I looked again on the benchmark and it appears that I did after all used the reference implementation (sorry). The benchmark code is here and it includes protobuf, pojo, scala, thrift and stax (more information in the original post). No json there, but if you wish you can extend it to include it and other stax implementation.

cowtowncoder March 10, 2009 at 10:10 PM  

Ok. :-)

One quick comment on code for StaxSerializer: code looks good, but one very expensive thing there is the call to XMLInputFactory.newInstance() (and XMLOutputFactory.newInstance()): this because they do extensive class-path based introspection. Construction of the factories themselves is not expensive, it's just the dynamic discovery part. Same problem occurs with JAXP API when creating DOM and SAX parsers too.
Simplest thing to do is to just create input and output factory in constructor.

Anyway, otherwise test is probably quite fair, but I mention this because I happened to notice on one system I built, profiler pointed to it as using more than half of the time for xml processing.

Code for Thrift should be useful, I have some performance tests and have been thinking of including Thrift along with PB.

Eishay Smith March 10, 2009 at 10:52 PM  

Code for thrift is actually there under the gen-javabean. I left it there and not under src since the thrift compiler (unlike protobuf) does not let you choose the destination directory of the generated code. Ok, I could have written a small script that copies the files but its ugly and another con for thrift.
I agree about the factory code, its not fair. I see that FactoryFinder does use some cache but only for configurations, definitely not as I expected it to. I might do another iteration on the benchmarking, but I already made the decision the original one was made for (i.e. using protobuf).

cowtowncoder March 10, 2009 at 11:11 PM  

Gotcha. Just for fun I did compile the code, replaced stax-RI with Woodstox, and changed factory instantiation.
Turns out this decreased Stax numbers by factor of 10 (!) for serialization, and 7 for deserialization.
After this the only other thing I added was "parser.close()" after parsing, which increased throughput by 50% (closing of readers/writers allows for releasing and reusing resources).
And if one uses Aalto, numbers change a bit more (see below), downwards.

So I am not trying to change your mind on choice, I just hope that readers will get a reasonable understanding of relative speeds -- with all the modifications, PB is still a bit faster for deserialization than Thrift or XML/Stax.
And obviously produces compact data.

For what it's worth, here are numbers I got on my system:

---
using Woodstox as Stax impl:

warming up...
Starting
,Object create, Serializaton, Deserialization, Serilized Size
thrift, 1295.77960, 23697.43200, 22989.80000, 314
stax, 996.85880, 31536.41900, 41365.32500, 406
protobuf, 2033.16400, 26118.13100, 15469.00900, 217
java, 975.85050, 76045.71900, 259198.09900, 845
scala, 652.98820, 118169.24200, 547530.91400, 1473

using Aalto as Stax impl:

warming up...
Starting
,Object create, Serializaton, Deserialization, Serilized Size
thrift, 1304.13260, 23069.41900, 24145.53400, 314
protobuf, 2081.43830, 26319.83200, 15060.01900, 217
java, 973.97880, 75996.27200, 260578.72200, 845
scala, 655.33490, 118616.22600, 548926.90300, 1473
stax, 1003.50770, 17027.86700, 27728.39200, 406

Eishay Smith March 10, 2009 at 11:37 PM  

Wow, that's great! We do use Woodstox in few place, I'll definitely check out Aalto to replace it.
Would you mind sharing the code? You can send a patch or I can give you svn access if you'll want to extend it even more.

cowtowncoder March 11, 2009 at 9:59 AM  

No problem: changes were really small.
I can send diffs from home, or submit (can't read it right now).

But quick summary is:

- Add member fields for XMLInputFactory
- Assign on constructor, using XMLxxxFactory.newInstance
- Use this factory to create reader/writer
- After deserialization, call parser.close()

No changes are needed to switch Stax implementation, just add in jars (woodstox 4 needs core, stax2-api; aalto similarly its own and stax2-api).
I did add debug statements to verify correct factories are used.

Also I could submit Ant build.xml, just wrote a really simple one. I hope I can use my gmail account with google-code?

Eishay Smith March 11, 2009 at 10:03 AM  

Thanks!
Send your gmail address to eishay [at] gmail.com and I'll give you write access

cowtowncoder March 13, 2009 at 1:48 PM  

Done. Code should be up to date, might be easier to run too (added build.xml, wrapper script). In case anyone wants to (re-)run results. :-)

(Stax numbers are radically different after change)

Eishay Smith March 13, 2009 at 4:14 PM  

Thanks cowtowncoder. Fyi, I added Chris to the commiters list. He adds a json serializer to the benchmark. Is there a problem running serializers for the three stax implementations we talked about, all with the same jvm? Would be nice to compare them. Running is separate might not do them justice.

cowtowncoder March 13, 2009 at 4:39 PM  

It is possible to run 3 separate ones, just need to force specific factories to use for each. Could just pass input/output factory classes or instances to serializer constructor.

Cool to have json serializer(s) as well.

David Bernard March 17, 2009 at 6:38 AM  

Hi,

I added (I could send the patch)
* xstream (default conf)
* xstream with dedicated converter (like protobuf, and other already have)

And because every sample have dedicated converter/mapping rules, I think it's important to also provide them for java serialization so I added "Externalizable"

, Object create, Serializaton, Deserialization, Serilized Size
java , 113.23390, 17305.80500, 72637.29300, 845
xstream default , 116.40035, 119932.61000, 171796.68850, 931
json , 112.58555, 3324.76450, 5318.12600, 310
stax , 113.05025, 6172.06000, 9566.96200, 406
java (externalizable) , 99.76580, 6250.40100, 18970.58100, 315
thrift , 174.72665, 4635.35750, 5133.24450, 314
scala , 66.10890, 27047.10850, 155413.44000, 1473
protobuf , 250.37140, 3849.69050, 2416.94800, 217
xstream with conv , 115.22810, 13492.50250, 47056.58750, 321

Eishay Smith March 17, 2009 at 9:28 AM  

Thanks David!
If you'll send your gmail account to eishay[at]gmail.com I'll give you svn write access.

cowtowncoder March 17, 2009 at 10:28 AM  

Neat. One minor thing we could add is name of impl used (for stax, json). I can do that.

Creative Commons License This work by Eishay Smith is licensed under a Creative Commons Attribution 3.0 Unported License.