hi all       We are performing now the change hostname configuration on production cluster according to the document - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_changing_host_names.html   The last stage is talking about – "in case NameNode HA enabled , then need to run the following command on one of the name node"     hdfs zkfc -formatZK -force     since we have active name node and standby name node we assume that our namenode is HA enable ?     but we want to understand  what are the risks when doing the following cli on one of the namenode   hdfs zkfc -formatZK -force   is the below command is safety to run without risks ? ... View more

We have HDP cluster version `2.6.5` and ambari `2.6.1` version Cluster include 3 masters machines , and 211 data-nodes machines ( workers machines ) , all machines are `rhel 7.2` version Example master1.sys77.com , master2.sys77.com , master3.sys77.com … And data nodes machines as worker01.sys77.com , worker02.sys77.com ----> worker211.sys77.com Now we want to change the domain name to `bigdata.com` instead of `sys77.com` What is the procedure to replace the `domain name` (`sys77.com`) for Hadoop cluster ? ( HDP cluster with ambari ) ... View more

ZooKeeper server creates snapshot and log files, but never deletes them. So we need to care about the retention policy. How to decide on the right amount of remaining Zookeeper snapshot files?   Need to say that ZooKeeper server itself only needs the latest complete fuzzy snapshot and the log files from the start of that snapshot. But since ZooKeeper creates a backup of snapshot file, how many ZooKeeper snapshot backups do we need to retain?   Sometimes snapshots can be corrupted, so the backup of snapshot files should take this into consideration. In our ZooKeeper server we saw that snapshot backup is created each day. Example of snapshot file from my ZooKeeper server:     -rw-r--r-- 1 ZooKeeper hadoop 458138861 Aug 10 07:12 snapshot.19000329d1 -rw-r--r-- 1 ZooKeeper hadoop 458138266 Aug 10 07:13 snapshot.19000329de -rw-r--r-- 1 ZooKeeper hadoop 458143590 Aug 10 09:24 snapshot.1900032d7a -rw-r--r-- 1 ZooKeeper hadoop 458142983 Aug 10 09:25 snapshot.1900032d84 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 11 03:42 snapshot.1900034b74 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 12 01:51 snapshot.1900036fa3 -rw-r--r-- 1 ZooKeeper hadoop 458138079 Aug 12 03:03 snapshot.1900037196 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 12 03:08 snapshot.19000371c8 -rw-r--r-- 1 ZooKeeper hadoop 458138432 Aug 12 03:09 snapshot.19000371de -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 12 12:02 snapshot.1900038053 -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 12 18:04 snapshot.1900038a39 -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 13 13:01 snapshot.190003a923 -rw-r--r-- 1 ZooKeeper hadoop 2 Aug 13 13:01 currentEpoch -rw-r--r-- 1 ZooKeeper hadoop 67108880 Aug 13 21:17 log.190002d2ce ... View more

we set `retention bytes` value - `104857600` for topic - `topic_test` [root@confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600 WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic "topic_test". Now we verify the `retention bytes` from the zookeeper: [root@confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_test Connecting to confluent1:2181 {"version":1,"config":{"retention.bytes":"104857600"}} cZxid = 0xb30a00000038 WATCHER:: WatchedEvent state:SyncConnected type:None path:null ctime = Mon Jun 29 11:42:30 GMT 2020 mZxid = 0xb31100008978 mtime = Wed Jul 22 19:22:20 GMT 2020 pZxid = 0xb30a00000038 cversion = 0 dataVersion = 7 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 54 numChildren = 0 now we performed `reboot` to the kafka confluent01 machine after machines started and kafka service start successfully , we checked again the `retention-bytes` from zookeeper: but now ( after machine reboot ) we can see that the `retention bytes` isn't configured in zookeeper [root@confluent01 ~]#zookeeper-shell confluent01:2181 get /config/topics/topic_test Connecting to confluent1:2181 WATCHER:: WatchedEvent state:SyncConnected type:None path:null  no retention bytes value {"version":1,"config":{}} cZxid = 0xb30a00000038 ctime = Mon Jun 29 11:42:30 GMT 2020 mZxid = 0xb3110000779b mtime = Wed Jul 22 14:09:19 GMT 2020 pZxid = 0xb30a00000038 cversion = 0 dataVersion = 2 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 25 numChildren = 0 **the question is** - how to remain the `retention bytes` even after restart of kafka machine ?   ***NOTE - we not want to use the retention bytes from `server.properties` because we set different retention bytes to each topic*** ... View more