Member since
06-14-2016
69
Posts
28
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5818 | 07-30-2018 06:45 PM | |
3817 | 06-22-2018 10:28 PM | |
881 | 06-20-2018 04:29 AM | |
827 | 06-20-2018 04:24 AM | |
1962 | 06-15-2018 08:24 PM |
09-25-2018
10:49 PM
Hi @Gitanjali Bare, Could you please share the complete error you are getting? Thanks!
... View more
08-17-2018
06:28 AM
Hi @Surendra Shringi, May I know if you are connecting to a Kerberized Kafka or unsecure? For template , kindly refer: https://community.hortonworks.com/articles/57262/integrating-apache-nifi-and-apache-kafka.html Its an old article but it covers the basics in detail. Also, is Kafka in the same HDF cluster or are you using a separate Kafka cluster? Thanks!
... View more
08-16-2018
06:26 PM
@Surendra Shringi Hi, Could you please try passing the FQDN of the kafka broker in the 'Kafka Brokers property of the processor? Also may I know what version of Kafka are you trying to publish to? Thanks!
... View more
08-04-2018
08:02 PM
Hi @Vadim
Dzyuban
Thank you for detailed explanation. It was my mistake that I missed the part where you mentioned sasl mechanism as PLAIN and not GSSAPI. In that case you do not need to pass the "a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT" Regarding this error: "Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup" Could you please provide complete error stack for more details on this error? Thank you!
... View more
08-03-2018
08:35 PM
@Vadim
DzyubanHi, Yes, the ServiceName is required in the Jaas when you are connecting to the secure Kafka (Kerberos enabled), for reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/secure-kafka-config-options.html If you do not have Kerberos on Kafka then you need not to use : a1.sources.s1.kafka.consumer.security.protocol=SASL_PLAINTEXT Instead you can just use: a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT But I am little confused when you mentioned "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT" Can you verify the listeners property in your Kafka broker and let me know what security protocol is it listening to? Thank you!
... View more
07-30-2018
06:54 PM
@dhieru singh No problem at all, I am glad it helped!
... View more
07-30-2018
06:47 PM
Hi @nisrine elloumi Yes, as I suspected in the first point that it could be authorization issue. Thank you for sharing.
... View more
07-30-2018
06:45 PM
@dhieru singh
Hi,
Below are the options that you can use with ./kafka-consumer-groups.sh --reset-offsets: For a specific topic you can use --topic option instead of --all-topics. Please let me know if that answers your question. Thank you!
... View more
07-30-2018
06:34 PM
@Vadim
Dzyuban
Hi, The property kafka.consumer.auto.offset.reset comes into picture when there is no initial offset in Kafka or if the current offset does not exist any more on the server. As per the flume docs: https://flume.apache.org/FlumeUserGuide.html One workaround that I can think of is changing the kafka.consumer.group.id and restart the agent. Kindly let me know if that helps. Thank you!
... View more
07-27-2018
07:53 PM
@nisrine elloumi 1. May I know if you have Ranger enabled or not? and whether the user 'atlas' has proper permissions? If Ranger is enabled, please verify that policy is enabled for both topics: ATLAS_HOOK and ATLAS_ENTITIES. If Ranger is not enabled then atlas user should be provided ACLs for read/write operations. If you are using Ranger, kindly refer the section 'Configure Atlas Security' of the following doc for detailed explanation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-governance/content/ch_hdp_data_governance_install_atlas_ambari.html 2. Kindly run the following command and provide the output: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper <zk_host>:2181 Also please verify the broker name in atlas properties is correct ( as configured in listeners property in kafka). Thank you!
... View more