Thursday, November 13, 2008

protobuf java serialization pains

The implementation in the last post is actually pretty ugly since its not generic and one needs to create a new serialization wrapper to every protobuf object.
The natural solution would be to use generics and get it over with, but it seems that protobuf makes it difficult to do.
The key problem with using the generics approach is that the method "parseFrom" that serialize the generated protobuf object is not declared in an interface or the GeneratedMessage class it inherits from. This means that one has to have the class at hand to do the deserialization, but that's the whole point! I do want to have a general serializer for and protocolbuf object.

So here is my solution, looks a bit hacky with the introspection and byte writing, but it works fine.

* Manually serializing Protobuf objects
* The serialize form first has an integer SIZE which is the
* size of the test of the serialized protobuf.
* After the integer there are SIZE bites of the protobuf serialized object
class ProtobufSerializer implements Externalizable{
* Object to serialize
private transient T _proto;
private transient String _className;
public ProtobufSerializer(){}
public ProtobufSerializer(T proto){
_proto = proto;
if(null != _proto)_className = _proto.getClass().getName();
public T get(){return _proto;}

* If the first byte is the size of zero, the object is null
public void readExternal (ObjectInput in)throws IOException, ClassNotFoundException{
int size = in.readInt();
if(0 == size)return;
byte[] array = new byte[size];
in.readFully(array, 0, size);
_className = new String(array);
size = in.readInt();
array = new byte[size];
in.readFully(array, 0, size);
Class clazz = getClass().getClassLoader().loadClass(_className);
Method parseMethod = clazz.getMethod("parseFrom", array.getClass());
_proto = (T)parseMethod.invoke(clazz, array);
catch (Exception e){
throw new IOException("could not load class " + _className);

* If the the object is null then the int zero is written to the stream
public void writeExternal (ObjectOutput out)
throws IOException{
if(null == _proto){
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] array = baos.toByteArray();
And here is how to use it:

private ProtobufSerializer _mediaHolder;
public NewsMediaContent getNewsMediaContent ()
return null == _mediaHolder ? null : _mediaHolder.get();
public void setNewsMediaContent (NewsMediaContent media)
_mediaHolder = new ProtobufSerializer(media);
By the way, I noticed that Hadoop has a similar issue with Thrift.


chacha November 23, 2009 at 9:30 PM  

hi there,

the eample there looks ok. (i dont have qualms for it being hacky for now). thats working for me too. but when i use it with extensions in the proto files it gives expetion at parseMethod.invoke point. And f**kin strangely IllegalArgumentException
this.proto = (T) parseMethod.invoke(clazz, new Object[] { array })

do you have some extension example handy. I donmt think at the time of writing protobuf registry needs to be set up. While at the time of merging it needs to be given and i am giving it and its working fine. Any pointers?

chacha November 24, 2009 at 12:42 AM  

BTW. Nice protobuf experience all around ..... good going buddy... catch you #koolak82 on twi....

Creative Commons License This work by Eishay Smith is licensed under a Creative Commons Attribution 3.0 Unported License.