Re: How to Kafka topic clone.

Nilesh · ‎02-01-2019

Hello, How can we create kafka topics directly into hdfs.

Nilesh · ‎02-01-2019

Hello, How can we create kafka topics directly into hdfs.

Nilesh · ‎02-01-2019

Hi Sandeep, Even we format namenode again we can loose the data. My question is why only namenode format why not datanodes.

Nilesh · ‎12-15-2018

Hello, Why does not datanodes formatted during formatting the namenode server. Thank you.

Nilesh · ‎04-04-2018

Hi, I will be grateful if someone can answer the following questions on Kafka. 1. What is inter.broker.protocol.version and what it does. 2. What is log.message.format.version and what is does 3. How log.message.format and inter.broker.protocol relate each other. 4. When and on what basis need to the change log.message.format and inter.broker.protocol version. 5. What changes are required on consumer and producer side after changed the version. Thanks, Nilesh

Nilesh · ‎04-02-2018

@Geoffrey Shelton Okot Thank you for your reply. I have some following queries, 1. What is inter.broker.protocol.version and what it does. 2. How log.message.format and inter.broker.protocol relate each other. 3. Could you please explain in detail the use of log.message.format.version. 4. When and on what basis need to the change log.message.format and inter.broker.protocol version. 5. What changes are required on consumer and producer side after changed the version. Thanks, Nilesh

Nilesh · ‎04-02-2018

Hi, 1. What is log.message.format.version in Kafka 2. What is the use of log.message.format.version in Kafka 3.What could be the version of log.message.format Thanks, Nilesh

Nilesh · ‎01-25-2018

@Sandeep Nemuri I tried this URL. This document is not updated

Nilesh · ‎01-25-2018

What does inter.broker.protocol.version in Kafka?

Nilesh · ‎01-16-2018

@Harald Berghoff Thank you for the reply. How to check this in zookeeper.

Nilesh · ‎01-16-2018

Team, I am receiving following an error while executing kafka mirror-maker command. Kerberos is configured on source and target cluster. kafka@hdpensrv$ /usr/hdp/current/kafka-broker/bin/kafka-mirror-maker.sh --consumer.config /tmp/sourceClusterConsumer.properties --producer.config /tmp/targetClusterProducer.properties --whitelist="topic_test" [2018-01-16 06:21:45,493] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188) at kafka.tools.MirrorMaker$MirrorMakerProducer.<init>(MirrorMaker.scala:626) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:213) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277) ... 4 more

Nilesh · ‎11-30-2017

Hi, How producer and consumer identify the leader in Kafka. Thank you.

Nilesh · ‎11-29-2017

Hi, Is there any command to check Kafka version. Thank you.

Nilesh · ‎11-27-2017

Hi @ Abdelkrim Hadjidj Thank you for the reply. My question is I am ingesting same data again and again on the same topic then what will happen? Will it overwrite the data or will make another copy of data or it will just ignore and notify? Thank you.

Nilesh · ‎11-22-2017

Team, What will be happened if we ingest same data, in particular, Kafka topic again? Will it overwrite the data or will make another copy of data or it will just ignore and notify.

Nilesh · ‎11-01-2017

Hi, We are trying to integrate Nifi and Ranger with LDAP. Ranger is successfully integrated with LDAP and i am able to login with the AD user. But when we are trying to integrate LDAP with NIFI using Ranger we facing issues. Will elaborate it using two scenarios. 1) When we select user as "{USER}" than we can login to Nifi UI using AD users and it also gets logged in with Anonymous users. Please find below screenshot. capture1.png capture2.png 2) If we remove "{USER}" from select user and put AD user for eg "apatank". Than we am unable to login and it gives me "Insufficient Permissions" error. Please find below screenshot. capture3.png capture4.png How can we remove anonymous user "{USER}" and use AD users in select users option in Ranger Ploicy and login to NIFI UI? Please help us in resolving above issue. Regards, Avdhoot

Nilesh · ‎10-28-2017

Thank you for sharing the URL. I am looking for topic cloning within the same cluster. Can we do this.

Nilesh · ‎10-28-2017

Can we delete these files?

Nilesh · ‎10-27-2017

Hi, I found following log files in /var/log/log_hadoop/log/kafka/audit/hdfs/spool/archive path and these are taking lot of space. What are these files? Can I delete these files? How to control it? 11G spool_kafka_20171022-0406.23.log 11G spool_kafka_20171023-0406.30.log 11G spool_kafka_20171024-0406.30.log 11G spool_kafka_20171025-0406.31.log 11G spool_kafka_20171026-0406.32.log

Nilesh · ‎10-27-2017

Can we clone single Kafka topic? If yes then how?

Nilesh · ‎10-16-2017

Hi Experts, We have 3 nodes of Kafka broker cluster setup along with 3 nodes of the zookeeper. Following are some queries while creating topics in the cluster. I am running following command to create topic ./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic portfolio_break_stat Here are my questions, 1. If I put replication-factor 1 then on which node it will create the topic. It will create on all nodes or only one node. If only one node how Kafka decide it? 2. If I did not mention replication-factor then how the topic is created on the node and which node it will pick up?

Nilesh · ‎04-17-2017

Team, What is use of ambari-qa account. Why it is require in hdp cluster. Can It replace with "root" account during hdp installation via ambari.

Nilesh · ‎04-11-2017

Thanks Jay :)

Nilesh · ‎04-11-2017

Team, How do I download Ambari-Server 2.5.0 and agent RPM file.

Nilesh · ‎03-29-2017

@Mike Riggs We have already having Kafka setup but no one knows flow of Kafka then in this case how do I identified from where Kafka taking the data and where it store in hadoop.

Nilesh · ‎03-27-2017

Dear experts, How to identify Kafka source and destination (Hdfs,Hbase) path

Nilesh · ‎09-08-2016

Dear Team, If i used linux based encryption on hadoop cluster then is it encrypt hdfs data.

Nilesh · ‎04-29-2016

@Ravi Mutyala Does it compressed the output.

Nilesh · ‎04-27-2016

@Ravi Mutyala Does sqoop import support orc file. Where is compressed parameter.

Nilesh · ‎04-26-2016

Hi Team, How to import oracle tables in hive with compressed format via sqoop command. Where to configure compression. i.e MapReduce or HDFS or Hive. Should I configure compression for Mapreduce ,HDFS and Hive. Which is the best compression formats supported by Hive.

Online	Offline
Last Visited	‎04-12-2018 06:26 AM

Date Registered	‎04-04-2018 01:30 AM
Last Visited	‎04-12-2018 06:26 AM
Total Messages Posted	80
Total Kudos Received	32

Re: How to Kafka topic clone.

How to create Kafka topics directly on HDFS

How to create Kafka topics directly on HDFS

Re: Why not datanode format

Why not datanode format

Kafka Basics

Re: log.message.format.version in Kafka

log.message.format.version in Kafka

Re: Kafka | MirrorMaker | Kerberos

inter.broker.protocol.version in kafka

Re: How producer and consumer identify the leader ...

Kafka | MirrorMaker | Kerberos

How producer and consumer identify the leader in K...

How to check Kafka version

Re: Ingest same data in Kafka topic

Ingest same data in Kafka topic

Insufficient Permissions error while login to Nifi...

Re: How to Kafka topic clone.

Re: Kafka spool log files

Kafka spool log files

How to Kafka topic clone.

Kafka topic | created on which node

Ambari-qa user || Ambari Installation

Re: Ambari-Server and Agent RPM Download

Ambari-Server and Agent RPM Download

Re: Kafka | Source and Destination Path

Kafka | Source and Destination Path

HDFS Rest Encryption vs Linux based Encryption

Re: Sqoop Import to Hive with Compression

Re: Sqoop Import to Hive with Compression

Sqoop Import to Hive with Compression