question Re: SAM Kafka Topology Spouts Log Error in Support Questions

SAM Kafka Topology Spouts Log Error

TimothySpann — Sat, 12 Aug 2017 04:12:26 GMT

Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[traffic]' is invalid

Valid topic that exists.

[root@princeton10 bin]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper princeton10.field.hortonworks.com:2181 --list __consumer_offsets ambari_kafka_service_check gps3 gps4 traffic

ava.lang.NullPointerException: null value in entry: id=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:135) at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:206) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.convertValue(AvroStreamsSnapshotDeserializer.java:68) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:49) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:39) at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:149) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:335) at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:316) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:236) at org.apache.storm.daemon.executor$fn__10364$fn__10379$fn__10410.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

Re: SAM Kafka Topology Spouts Log Error

pshah — Mon, 14 Aug 2017 08:35:12 GMT

It seems that there are two issues here. One is "Unable to get offset lags for kafka ......" error that you see in storm ui for SAM topology in question. Reason for that is https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L59 It does not work in 3.0.0.0. You need the fixed line of code in https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L62 You can apply that one line of change on HDF-3.0.0.0 for storm-kafka-client and rebuild it, install it in maven local on SAM server under the version used by mavenDeps string for Kafka source. You can figure out the version by running

find . -name kafka-source-topology-component.json and checking the mavenDeps file there.

There also seems to be another issue which is an exception thrown from the spout. HDF-3.0.0.0 has a bug where nullable avro fields are not allowed in your source topic messages. You can use a new topic and schema with non nullable fields only and make sure that producer puts some sentinel value for nulls. The other option is to apply the fix at https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c

Re: SAM Kafka Topology Spouts Log Error

TimothySpann — Mon, 14 Aug 2017 20:17:33 GMT

the HDF-3.0-maint is 404 for me

Re: SAM Kafka Topology Spouts Log Error

deep_it2007 — Tue, 06 Feb 2018 22:17:17 GMT

Hi @pshah, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

When i'm trying to run the storm code, it's executing successfully. But on storm UI, under "Topology spouts lag error" it gives out the below error, "Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[samplenftest]' is invalid"

Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L59

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L62

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.

Re: SAM Kafka Topology Spouts Log Error

deep_it2007 — Tue, 06 Feb 2018 22:17:27 GMT

Hi @Timothy Spann, i'm facing the similar issue on my HDP cluster. My cluster details (HDP 2.6.0.3, Kafka 0.10.1, Storm 1.1.0)

Also, the URLs mentioned here in the post to fix the issue, are not accessible (gives 404 error)

https://github.com/hortonworks/storm/blob/HDF-3.0.0.0/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L59

https://github.com/hortonworks/storm/blob/HDF-3.0-maint/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/NamedSubscription.java#L62

Can you please help me out on how to resolve this issue.

Regards,

Sandeep.

Re: SAM Kafka Topology Spouts Log Error

TimothySpann — Tue, 06 Feb 2018 22:20:46 GMT

There was bad messages in my Kafka queue that did not have schemas and were not in valid AVRO. This is the cause. Stop Kafka. Stop Storm. Stop SAM.

Clean out your topics, delete the topic and recreate.

Send a valid AVRO message with a valid subscription.

Re: SAM Kafka Topology Spouts Log Error

TimothySpann — Tue, 06 Feb 2018 22:21:53 GMT

Options: You can upgrade.

Have non-null messages.

Or apply a hotfix. https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c You can contact support to assist with upgrade or hotfix