Comments on Wondering around: Protobuf with option optimize for SPEED

Neat. One minor thing we could add is name of impl...

2009-03-17T10:28:00.000-07:00

Neat. One minor thing we could add is name of impl used (for stax, json). I can do that.

Thanks David!
If you'll send your gmail account...

2009-03-17T09:28:00.000-07:00

Thanks David!
If you'll send your gmail account to eishay[at]gmail.com I'll give you svn write access.

Hi,

I added (I could send the patch)
* x...

2009-03-17T06:38:00.000-07:00

Hi,

I added (I could send the patch)
* xstream (default conf)
* xstream with dedicated converter (like protobuf, and other already have)

And because every sample have dedicated converter/mapping rules, I think it's important to also provide them for java serialization so I added "Externalizable"

, Object create, Serializaton, Deserialization, Serilized Size
java , 113.23390, 17305.80500, 72637.29300, 845
xstream default , 116.40035, 119932.61000, 171796.68850, 931
json , 112.58555, 3324.76450, 5318.12600, 310
stax , 113.05025, 6172.06000, 9566.96200, 406
java (externalizable) , 99.76580, 6250.40100, 18970.58100, 315
thrift , 174.72665, 4635.35750, 5133.24450, 314
scala , 66.10890, 27047.10850, 155413.44000, 1473
protobuf , 250.37140, 3849.69050, 2416.94800, 217
xstream with conv , 115.22810, 13492.50250, 47056.58750, 321

It is possible to run 3 separate ones, just need t...

2009-03-13T16:39:00.000-07:00

It is possible to run 3 separate ones, just need to force specific factories to use for each. Could just pass input/output factory classes or instances to serializer constructor.

Cool to have json serializer(s) as well.

Thanks cowtowncoder. Fyi, I added Chris to the com...

2009-03-13T16:14:00.000-07:00

Thanks cowtowncoder. Fyi, I added Chris to the commiters list. He adds a json serializer to the benchmark. Is there a problem running serializers for the three stax implementations we talked about, all with the same jvm? Would be nice to compare them. Running is separate might not do them justice.

Done. Code should be up to date, might be easier t...

2009-03-13T13:48:00.000-07:00

Done. Code should be up to date, might be easier to run too (added build.xml, wrapper script). In case anyone wants to (re-)run results. :-)

(Stax numbers are radically different after change)

Thanks!
Send your gmail address to eishay [at] ...

2009-03-11T10:03:00.000-07:00

Thanks!
Send your gmail address to eishay [at] gmail.com and I'll give you write access

No problem: changes were really small.
I can se...

2009-03-11T09:59:00.000-07:00

No problem: changes were really small.
I can send diffs from home, or submit (can't read it right now).

But quick summary is:

- Add member fields for XMLInputFactory
- Assign on constructor, using XMLxxxFactory.newInstance
- Use this factory to create reader/writer
- After deserialization, call parser.close()

No changes are needed to switch Stax implementation, just add in jars (woodstox 4 needs core, stax2-api; aalto similarly its own and stax2-api).
I did add debug statements to verify correct factories are used.

Also I could submit Ant build.xml, just wrote a really simple one. I hope I can use my gmail account with google-code?

Wow, that's great! We do use Woodstox in few place...

2009-03-10T23:37:00.000-07:00

Wow, that's great! We do use Woodstox in few place, I'll definitely check out Aalto to replace it.
Would you mind sharing the code? You can send a patch or I can give you svn access if you'll want to extend it even more.

Gotcha. Just for fun I did compile the code, repla...

2009-03-10T23:11:00.000-07:00

Gotcha. Just for fun I did compile the code, replaced stax-RI with Woodstox, and changed factory instantiation.
Turns out this decreased Stax numbers by factor of 10 (!) for serialization, and 7 for deserialization.
After this the only other thing I added was "parser.close()" after parsing, which increased throughput by 50% (closing of readers/writers allows for releasing and reusing resources).
And if one uses Aalto, numbers change a bit more (see below), downwards.

So I am not trying to change your mind on choice, I just hope that readers will get a reasonable understanding of relative speeds -- with all the modifications, PB is still a bit faster for deserialization than Thrift or XML/Stax.
And obviously produces compact data.

For what it's worth, here are numbers I got on my system:

---
using Woodstox as Stax impl:

warming up...
Starting
,Object create, Serializaton, Deserialization, Serilized Size
thrift, 1295.77960, 23697.43200, 22989.80000, 314
stax, 996.85880, 31536.41900, 41365.32500, 406
protobuf, 2033.16400, 26118.13100, 15469.00900, 217
java, 975.85050, 76045.71900, 259198.09900, 845
scala, 652.98820, 118169.24200, 547530.91400, 1473

using Aalto as Stax impl:

warming up...
Starting
,Object create, Serializaton, Deserialization, Serilized Size
thrift, 1304.13260, 23069.41900, 24145.53400, 314
protobuf, 2081.43830, 26319.83200, 15060.01900, 217
java, 973.97880, 75996.27200, 260578.72200, 845
scala, 655.33490, 118616.22600, 548926.90300, 1473
stax, 1003.50770, 17027.86700, 27728.39200, 406

Code for thrift is actually there under the
2009-03-10T22:52:00.000-07:00
Code for thrift is actually there under the gen-javabean. I left it there and not under src since the thrift compiler (unlike protobuf) does not let you choose the destination directory of the generated code. Ok, I could have written a small script that copies the files but its ugly and another con for thrift.
I agree about the factory code, its not fair. I see that FactoryFinder does use some cache but only for configurations, definitely not as I expected it to. I might do another iteration on the benchmarking, but I already made the decision the original one was made for (i.e. using protobuf).

Ok. :-)

One quick comment on code for StaxS...

2009-03-10T22:10:00.000-07:00

Ok. :-)

One quick comment on code for StaxSerializer: code looks good, but one very expensive thing there is the call to XMLInputFactory.newInstance() (and XMLOutputFactory.newInstance()): this because they do extensive class-path based introspection. Construction of the factories themselves is not expensive, it's just the dynamic discovery part. Same problem occurs with JAXP API when creating DOM and SAX parsers too.
Simplest thing to do is to just create input and output factory in constructor.

Anyway, otherwise test is probably quite fair, but I mention this because I happened to notice on one system I built, profiler pointed to it as using more than half of the time for xml processing.

Code for Thrift should be useful, I have some performance tests and have been thinking of including Thrift along with PB.

I looked again on the benchmark and it appears tha...

2009-03-10T21:09:00.000-07:00

I looked again on the benchmark and it appears that I did after all used the reference implementation (sorry). The benchmark code is here and it includes protobuf, pojo, scala, thrift and stax (more information in the original post). No json there, but if you wish you can extend it to include it and other stax implementation.

Ok thanks. I was mistaken then, just saw a link wa...

2009-03-10T19:44:00.000-07:00

Ok thanks. I was mistaken then, just saw a link was to home page of ref impl and assumed that was the impl used.
Are sources for the test available? Differences are somewhat larger than what I have seen with other tests, so I am curios. Plus might be nice to see how json does and so on.

Used Woodstox for the benchmark.

2009-03-10T18:38:00.000-07:00

Used Woodstox for the benchmark.

One thing to note: Stax reference implementation i...

2009-03-10T18:15:00.000-07:00

One thing to note: Stax reference implementation is NOT a good Stax implementation. Mostly because it is buggy, but it is also not as fast as alternatives, such as:

* Woodstox, [http://woodstox.codehaus.org[
* Sun sjsxp, [http://sjsxp.dev.java.net] (bundled with JDK 1.6)
* Aalto, [http://www.cowtowncoder.com/hatchery/aalto/index.html]

Of these, Aalto is the fastest, then Woodstox then Sjsxp.

So it would be nice to see how upgrading to a good stax implementation would change results for xml.

Doh - ignore previous comment. I'd missed the extr...

2008-11-19T07:08:00.000-08:00

Doh - ignore previous comment. I'd missed the extra line, and was still just looking at the protobuf one...

As far as I can tell, this post is still using the...

2008-11-19T01:04:00.000-08:00

As far as I can tell, this post is still using the same graphs as the previous one. Am I missing something, or is this a copy and paste error?

Comments on Wondering around: Protobuf with option optimize for SPEED

Neat. One minor thing we could add is name of impl...

Thanks David!If you'll send your gmail account...

Hi,I added (I could send the patch)* x...

It is possible to run 3 separate ones, just need t...

Thanks cowtowncoder. Fyi, I added Chris to the com...

Done. Code should be up to date, might be easier t...

Thanks!Send your gmail address to eishay [at] ...

No problem: changes were really small.I can se...

Wow, that's great! We do use Woodstox in few place...

Gotcha. Just for fun I did compile the code, repla...

Ok. :-)One quick comment on code for StaxS...

I looked again on the benchmark and it appears tha...

Ok thanks. I was mistaken then, just saw a link wa...

Used Woodstox for the benchmark.

One thing to note: Stax reference implementation i...

Doh - ignore previous comment. I'd missed the extr...

As far as I can tell, this post is still using the...

Thanks David!
If you'll send your gmail account...

Hi,

I added (I could send the patch)
* x...

Thanks!
Send your gmail address to eishay [at] ...

No problem: changes were really small.
I can se...

Ok. :-)

One quick comment on code for StaxS...