Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I specify a Java class in the kafka bolt Producer Config within Storm?

avatar
Rising Star

Kafka 0.9 requires that the "key.serializer" and "value.serializer" items in ProducerConfig be java classes, not a string containing the name of a java class. See https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/P...

However, if try to do that like this:

Properties props = new Properties();props.put("bootstrap.servers", topoProperties.getProperty("bootstrap.servers"));
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);

conf.put(KafkaBolt.KAFKA_BROKER_PROPERTIES, props);

Storm fails to start with this error:

java.lang.IllegalArgumentException: Topology conf is not json-serializable

See: https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/StormSubmitter.java#...

Seems like kafka-bolt's prepare method will have to transform a string into a Java class. Before I go down that path, I was wondering if anyone else had run into this problem and if there's a workaround?

Thanks! -Aaron

1 ACCEPTED SOLUTION

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
1 REPLY 1

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login