Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SAM Kafka Topology Spouts Log Error

avatar
Master Guru

Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[traffic]' is invalid

Valid topic that exists.

[root@princeton10 bin]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper princeton10.field.hortonworks.com:2181 --list __consumer_offsets ambari_kafka_service_check gps3 gps4 traffic

ava.lang.NullPointerException: null value in entry: id=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:135) at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:206) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.convertValue(AvroStreamsSnapshotDeserializer.java:68) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:49) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:39) at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:149) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:335) at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:316) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:236) at org.apache.storm.daemon.executor$fn__10364$fn__10379$fn__10410.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

1 ACCEPTED SOLUTION

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Rising Star
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Guru

the HDF-3.0-maint is 404 for me

avatar
Explorer

Hi @pshah, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

When i'm trying to run the storm code, it's executing successfully. But on storm UI, under "Topology spouts lag error" it gives out the below error, "Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[samplenftest]' is invalid"


Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/...

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/or...

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.

avatar
Master Guru

There was bad messages in my Kafka queue that did not have schemas and were not in valid AVRO. This is the cause. Stop Kafka. Stop Storm. Stop SAM.

Clean out your topics, delete the topic and recreate.

Send a valid AVRO message with a valid subscription.

avatar
Master Guru

Options: You can upgrade.

Have non-null messages.

Or apply a hotfix. https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c You can contact support to assist with upgrade or hotfix

avatar
Explorer

Hi @Timothy Spann, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

When i'm trying to run the storm code, it's executing successfully. But on storm UI, under "Topology spouts lag error" it gives out the below error, "Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[samplenftest]' is invalid"

Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/...

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/or...

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.