SAM Kafka Topology Spouts Log Error

[#1]

Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[traffic]' is invalid

Valid topic that exists.

[root@princeton10 bin]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper princeton10.field.hortonworks.com:2181 --list __consumer_offsets ambari_kafka_service_check gps3 gps4 traffic

ava.lang.NullPointerException: null value in entry: id=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:135) at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:206) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.convertValue(AvroStreamsSnapshotDeserializer.java:68) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:49) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:39) at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:149) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:335) at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:316) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:236) at org.apache.storm.daemon.executor$fn__10364$fn__10379$fn__10410.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

[#2 · Accepted solution]

It seems that there are two issues here. One is "Unable to get offset lags for kafka ......" error that you see in storm ui for SAM topology in question. Reason for that is https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/... It does not work in 3.0.0.0. You need the fixed line of code in https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/or... You can apply that one line of change on HDF-3.0.0.0 for storm-kafka-client and rebuild it, install it in maven local on SAM server under the version used by mavenDeps string for Kafka source. You can figure out the version by running

find . -name kafka-source-topology-component.json and checking the mavenDeps file there.

There also seems to be another issue which is an exception thrown from the spout. HDF-3.0.0.0 has a bug where nullable avro fields are not allowed in your source topic messages. You can use a new topic and schema with non nullable fields only and make sure that producer puts some sentinel value for nulls. The other option is to apply the fix at https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c

[#3]

the HDF-3.0-maint is 404 for me

[#4]

Hi @pshah, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

When i'm trying to run the storm code, it's executing successfully. But on storm UI, under "Topology spouts lag error" it gives out the below error, "Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[samplenftest]' is invalid"

Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/...

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/or...

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.

[#5]

There was bad messages in my Kafka queue that did not have schemas and were not in valid AVRO. This is the cause. Stop Kafka. Stop Storm. Stop SAM.

Clean out your topics, delete the topic and recreate.

Send a valid AVRO message with a valid subscription.

[#6]

Options: You can upgrade.

Have non-null messages.

Or apply a hotfix. https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c You can contact support to assist with upgrade or hotfix

[#7]

Hi @Timothy Spann, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

When i'm trying to run the storm code, it's executing successfully. But on storm UI, under "Topology spouts lag error" it gives out the below error, "Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[samplenftest]' is invalid"

Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/...

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/or...

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.