Reply
Explorer
Posts: 9
Registered: ‎08-28-2014

Spark integration with Avro(In CDH 5.1.0)

Hi All, How to load Avro message to Spark? I have avro stream message that I want to load in Spark and do groupBykey operation as per my requirement. My work:- sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); sparkConf.set("spark.kryo.registrator", "xyz.PlatformKryoRegistrator"); public class PlatformKryoRegistrator implements KryoRegistrator { public void registerClasses(Kryo kryo) { kryo.register(ResourceMessage.class, new FieldSerializer(kryo, ResourceMessage.class)); } } ///ResourceMessage class gets generated from avro tools public class ResourceMessage extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord { ......... } Problem is that it is not able to register custom PlatformKryoRegistrator class.I am not sure what I am doing wrong. Please suggest me. Thanks Nishi

Explorer
Posts: 9
Registered: ‎08-28-2014

Re: Spark integration with Avro(In CDH 5.1.0)

Hi All,

How to load Avro message to Spark? I have avro stream message that I want to load in Spark and do groupBykey operation as per my requirement.

 

My work:-

 

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

sparkConf.set("spark.kryo.registrator", "xyz.PlatformKryoRegistrator");

 

public class PlatformKryoRegistrator implements KryoRegistrator {

public void registerClasses(Kryo kryo) {

  kryo.register(ResourceMessage.class, new FieldSerializer(kryo, ResourceMessage.class)); }

}

 

///ResourceMessage class gets generated from avro tools

public class ResourceMessage extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord { ......... }

 

Problem is that it is not able to register custom PlatformKryoRegistrator class.I am not sure what I am doing wrong.