Hi @Gitanjali Bare, Could you please share the complete error you are getting? Thanks! ... View more

Hi @Surendra Shringi, May I know if you are connecting to a Kerberized Kafka or unsecure? For template , kindly refer: https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html Its an old article but it covers the basics in detail. Also, is Kafka in the same HDF cluster or are you using a separate Kafka cluster? Thanks! ... View more

@Surendra Shringi Hi, Could you please try passing the FQDN of the kafka broker in the 'Kafka Brokers property of the processor? Also may I know what version of Kafka are you trying to publish to? Thanks! ... View more

Hi @Vadim Dzyuban Thank you for detailed explanation. It was my mistake that I missed the part where you mentioned sasl mechanism as PLAIN and not GSSAPI. In that case you do not need to pass the "a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT" Regarding this error: "Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup" Could you please provide complete error stack for more details on this error? Thank you! ... View more

@Vadim DzyubanHi, Yes, the ServiceName is required in the Jaas when you are connecting to the secure Kafka (Kerberos enabled), for reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/secure-kafka-config-options.html If you do not have Kerberos on Kafka then you need not to use : a1.sources.s1.kafka.consumer.security.protocol=SASL_PLAINTEXT Instead you can just use: a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT But I am little confused when you mentioned "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT" Can you verify the listeners property in your Kafka broker and let me know what security protocol is it listening to? Thank you! ... View more

@dhieru singh No problem at all, I am glad it helped! ... View more

Hi @nisrine elloumi Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing. ... View more

@Vadim Dzyuban Hi, The property kafka.consumer.auto.offset.reset comes into picture when there is no initial offset in Kafka or if the current offset does not exist any more on the server. As per the flume docs: https://flume.apache.org/FlumeUserGuide.html One workaround that I can think of is changing the kafka.consumer.group.id and restart the agent. Kindly let me know if that helps. Thank you! ... View more

@nisrine elloumi 1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations. If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html 2. Kindly run the following command and provide the output: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181 Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka). Thank you! ... View more

@nisrine elloumi Hi, Could you please verify the Kafka broker and port that you have configured in atlas? And since you have tagged kerberos I assume this is kerberized env, in that case could you please also check the security.protocol in atlas properties to connect to secure Kafka. Thank you! ... View more

Thank you @sohan kanamarlapudi. Did you set partition.assignment.strategy to null or empty string in your properties file which is being read by your spark application? Possible values for this property is range or roundrobin and default is [org.apache.kafka.clients.consumer.RangeAssignor] Reference: https://kafka.apache.org/0100/documentation.html#newconsumerconfigs Is it possible for you to share the code snippet where you have configured the Kafka consumer? (Kindly omit any sensitive information) Thanks! ... View more

@Eric RichardsonHi, Could you please share the port set in Listeners property in Kafka's server.properties file. Also I would recommend to use FQDN of broker instead of localhost in producer/consumer command. Thank you! ... View more

@sohan kanamarlapudi Hi, May I know what version of Spark and Kafka are you using? Thanks! ... View more

@Mahesh Glad it worked! Thanks! ... View more

@Mahesh Hi, Did the suggestion help? Thanks! ... View more

@Pankaj Goel Hi, Please refer section 'Bolts' of https://storm.apache.org/releases/1.2.2/Concepts.html and let me know if you have any question. Thanks! ... View more

@L Nin I am glad it worked! Thanks! ... View more

@Erkan ŞİRİN Can you please try running the following command: /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server hadooptest01.datalonga.com:6667 --topic erkan_deneme --new-consumer --from-beginning ... View more

@Erkan ŞİRİN May I know what Kafka version you are on? Old Consumer API used to use zookeeper to store offsets but in recent versions we do have option to enable dual commit to commit offsets to Kafka as well as zookeeper. Also could you please share your server.properties for a quick review. Thanks! ... View more

@sangeetha sivakumar Did it help? ... View more

@sangeetha sivakumar As far as I know the byte capacity should be 80% of the total amount of heap space available to the process. The following link might help you if you need to get more details on this: http://flume.apache.org/FlumeUserGuide.html#memory-channel Thanks! ... View more

@sangeetha sivakumar Hi, May be you can try to match file with regex pattern to read specific files: filter.pattern For the memory issue, have you tried increasing the byte capacity of the channel? Currently I can see its: FtpAgent.channels.MemChannel.byteCapacity =6912212 Thanks! ... View more

@Kennon Rodrigues Hi, 1. I would recommend to run with FQDNs instead of the IP in --bootstrap-server 2. Are you facing the same issue with any test topic that you create? Could you please also describe ' __consumer_offsets' topic? 3. You can also turn on client side debugging by changing the log level to DEBUG in tools-log4j.properties file : log4j.rootLogger=DEBUG, stderr Thanks! ... View more

@Swaapnika Guntaka Could you please make sure what port is set in the listener property of Kafka? Also Make sure producer is producing messages or not. You can wait for a while after producing messages to see whether its timing out or not. Thanks! ... View more