Hi @Gitanjali Bare, Could you please share the complete error you are getting? Thanks! ... View more

Hi @Surendra Shringi, May I know if you are connecting to a Kerberized Kafka or unsecure? For template , kindly refer: https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html Its an old article but it covers the basics in detail. Also, is Kafka in the same HDF cluster or are you using a separate Kafka cluster? Thanks! ... View more

@Surendra Shringi Hi, Could you please try passing the FQDN of the kafka broker in the 'Kafka Brokers property of the processor? Also may I know what version of Kafka are you trying to publish to? Thanks! ... View more

Hi @Vadim Dzyuban Thank you for detailed explanation. It was my mistake that I missed the part where you mentioned sasl mechanism as PLAIN and not GSSAPI. In that case you do not need to pass the "a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT" Regarding this error: "Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup" Could you please provide complete error stack for more details on this error? Thank you! ... View more

@Vadim DzyubanHi, Yes, the ServiceName is required in the Jaas when you are connecting to the secure Kafka (Kerberos enabled), for reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/secure-kafka-config-options.html If you do not have Kerberos on Kafka then you need not to use : a1.sources.s1.kafka.consumer.security.protocol=SASL_PLAINTEXT Instead you can just use: a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT But I am little confused when you mentioned "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT" Can you verify the listeners property in your Kafka broker and let me know what security protocol is it listening to? Thank you! ... View more

@dhieru singh No problem at all, I am glad it helped! ... View more

Hi @nisrine elloumi Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing. ... View more

@Vadim Dzyuban Hi, The property kafka.consumer.auto.offset.reset comes into picture when there is no initial offset in Kafka or if the current offset does not exist any more on the server. As per the flume docs: https://flume.apache.org/FlumeUserGuide.html One workaround that I can think of is changing the kafka.consumer.group.id and restart the agent. Kindly let me know if that helps. Thank you! ... View more

@nisrine elloumi 1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations. If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html 2. Kindly run the following command and provide the output: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181 Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka). Thank you! ... View more