@mike_bronson7  To achieve your goal for the 2 issues you will need to edit server.properties of Kafka to add the following line.   auto.leader.rebalance.enable = false   Then run the below assuming you are having a zookeeper quorum of host1;host2,host3   bin/kafka-preferred-replica-election.sh --zookeeper host1:2181,host2:2181,host3:2181/kafka   This should balance your partitions you can validate with  bin/kafka-topics.sh --zookeeper host1:2181,host2:2181,host3:2181/kafka --describe For the second issue with the lost  broker, you need to create a new broker and update the broker.id with the previous broker's id which was not gone or not recoverable then run $ kafka-preferred-replica-election.sh to balance the topics. ... View more

@hanu  Can you be precise what platform CDH/CDP or HDP and it's version also confirm whether it's kerberized or not? The more info you give the better     ... View more

@rampradeep_    All servers in a cluster should be managed by CM or Ambari etc. In the case of CDH 6.3.3 you will use the Cloudera manager to add gateway aka client roles to the remote server so that these gateway/Client or edge node interchangeably is centrally managed by CM which deploys the client software like YARN,zk,hdfs Clients/gateways depending on the language. If you decide to install any client manually then you will have to manually CORE/SITE/MAPRED-SITE.xml's. These files will be overridden if CM managed else it's a vanilla setup quite a headache to manage. ... View more

@Yuriy_but    The answer is very simple you have logging in as admin user in hue and admin has no HDFS home directory. there are 2-ways  delegate the HDFS home directory creation to HUE by checking Create a home directory in HUE Users--->AddSync LDAP User username=admin [search] Distinguished Name = unchecked Create home directory= checked or as the HDFS user $ hdfs dfs -mkdir /user/admin Change permissions $ hdfs dfs -chown admin /user/admin Now when you log in HUE you should get any issues please let me know ... View more

thank you very much Mr. Shelton ... View more

@Yuriy_but  Absolutely that's not normal. There are a couple of thinks to check the Bad disk or corruption. Force checkpoint and restart the DN to see the behavior. Ensure the NN has enough memory allocated. Remove it from the cluster and rejoin afresh after reformating. Check the NIC and connectivity. Run the balancer to equally distribute the date.     ... View more

I'm using Cloudera QuickStart VM 5.13 and I installed their Kafka version. Listing Kafka Topics: /usr/bin/kafka-topics --list --zookeeper quickstart.cloudera:2181 Creating Kafka Topic: /usr/bin/kafka-topics --create --zookeeper quickstart.cloudera:2181 --replication-factor 1 --partitions 3 --topic myFirstTopic Start a Producer: /usr/bin/kafka-console-producer --broker-list quickstart.cloudera:9092 --topic myFirstTopic Start a Consumer: /usr/bin/kafka-console-consumer --bootstrap-server quickstart.cloudera:9092 --topic myFirstTopic --from-beginning   Notes for your issue: Replications need to be less than or equal to the number of brokers. I changed offsets.topic.replication.factor in Kafka configuration from Cloudera Manager and I set it to 1 (Cause I have 1 broker)  You can delete brokers from Zookeeper as shown in the link below. Then restart Kafka to recreate these brokers. Zookeeper CLI: Accessing Zookeeper CLI: /usr/bin/zookeeper-client List All: ls / Output: [cluster, controller, brokers, zookeeper, admin, isr_change_notification, log_dir_event_notification, ngdata, controller_epoch, solr, consumers, latest_producer_id_block, config, hbase] List Kafka Brokers: ls /brokers Output: [ids, topics, seqid] List Kafka Topics in Zookeeper: ls /brokers/topics Output: [myFirstTopic, __consumer_offsets]  Delete a Path in Zookeeper: rmr /brokers   ... View more

The following map rule is wrong:   RULE:[2:\$1@\$0](rm@MY_REALM)s/.*/rm/   the user for the ResourceManager is not "rm" but "yarn" and this should be the replacement value. This is the same as for the hadoop.security.auth_to_local in Hadoop/HDFS configuration. ... View more

@ni4ni  mentions, There is no standby/secondary NN. ... View more