@christophe VALMIR Usually, after the restart give the process a minute or 2 for the processes to pick up. Please don't forget to vote a helpful answer as the issue is resolved,that way other HCC users can quickly find the solution when they encounter the same issue. HTH ... View more

@Pavel Orekhov You can change the log level while logged on as hdfs with the below steps, you don't need to restart the namenode Get the current log level $ hadoop daemonlog -getlevel {namenode_host}:50070 BlockStateChange Output Connecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange Submitted Log Name: BlockStateChange Log Class: org.apache.commons.logging.impl.Log4JLogger Effective Level: INFO Change to DEBUG $ hadoop daemonlog -setlevel {namenode_host}:50070 BlockStateChange DEBUG Output Connecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange&level=DEBUG Submitted Log Name: BlockStateChange Log Class: org.apache.commons.logging.impl.Log4JLogger Submitted Level: DEBUG Setting Level to DEBUG ... Effective Level: DEBUG Validate DEBUG mode $ hadoop daemonlog -getlevel {namenode_host}:50070 BlockStateChange Output Connecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange Submitted Log Name: BlockStateChange Log Class: org.apache.commons.logging.impl.Log4JLogger Effective Level: DEBUG You should be able to notice the logging level in namenode.log has been updated, without restarting the service. After finishing your diagnostics you can reset the logging level back to INFO Reset to INFO $ hadoop daemonlog -setlevel {namenode_host}:50070 BlockStateChange INFO Output Connecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange&level=INFO Submitted Log Name: BlockStateChange Log Class: org.apache.commons.logging.impl.Log4JLogger Submitted Level: INFO Setting Level to INFO ... Effective Level: INFO Validate INFO $ hadoop daemonlog -getlevel {namenode_host}:50070 BlockStateChange Output Connecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange Submitted Log Name: BlockStateChange Log Class: org.apache.commons.logging.impl.Log4JLogger Effective Level: INFO There you go If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer. ... View more

@Richard Wheeler if you left the sandbox idling then for sure it MUST be the logs generated in /var/logs/{component}/ the HDP components continually generate logs with the components statuses and on the sandbox, it's mount on / # du -a /var/log/ | sort -n -r | head -n 20 Sample output 3363560 /var/log/ 1966344 /var/log/kafka 494300 /var/log/ambari-metrics-collector 267092 /var/log/hadoop 265560 /var/log/hadoop/hdfs 171528 /var/log/hadoop/hdfs/hadoop-hdfs-namenode-test.tarta.se.log 159432 /var/log/ambari-agent 98756 /var/log/ambari-infra-solr 81932 /var/log/ambari-metrics-collector/ambari-metrics-collector.log.3 81932 /var/log/ambari-metrics-collector/ambari-metrics-collector.log.2 81932 /var/log/ambari-metrics-collector/ambari-metrics-collector.log.1 81928 /var/log/ambari-metrics-collector/ambari-metrics-collector.log.4 81924 /var/log/ambari-metrics-collector/ambari-metrics-collector.log.5 69956 /var/log/oozie 49116 /var/log/hbase 40056 /var/log/ranger 39136 /var/log/ranger/admin 36420 /var/log/hive 36232 /var/log/hadoop-yarn 36176 /var/log/hbase/hbase-hbase-regionserver-test.tarta.se.log So you will need to delete the old files to regain some space. You can also run discretely a script in the cron ! ... View more

@Michael Bronson Apache Kafka uses Zookeeper to select a controller, Zookeeper tracks the status of Kafka cluster nodes and also plays a vital role for serving many other purposes, such as leader detection, configuration management, synchronization, detecting when a new node joins or leaves the cluster, etc.and maintain cluster membership by storing configuration, including the list of topics in the cluster. In order to remain part of the Kafka cluster, each broker has to send keep-alive to Zookeeper in regular intervals. This is something every Zookeeper client does by default. If the broker doesn't heartbeat Zookeeper every zookeeper.session.timeout.ms milliseconds (6000 by default), Zookeeper will assume the broker is dead. This will cause leader election for all partitions that had a leader on that broker. If this broker happened to be the controller, you will also see a new controller elected. In a Kafka cluster, service discovery helps the brokers find each other and know who’s in the cluster; and consensus helps the brokers elect a cluster controller, know what partitions exist, where they are, if they’re a leader of a partition or if they’re a follower and need to replicate, and so on. A controller is not too complex it is a normal broker that simply has an additional responsibility. That means it still leads partitions, has writes/reads going through it and replicates data. The most important part of that additional responsibility is keeping track of nodes in the cluster and appropriately handling nodes that leave, join or fail. This includes rebalancing partitions and assigning new partition leaders. There is always exactly one controller broker in a Kafka cluster. HTH ... View more

@ram sriram The below command sets your replication factor to 1 for all new files you will create, with a potential data loss unless you are running HDP 3.x which has a new HDFS algorithm EC erasure coding $ hdfs dfs -setrep -w 1 -R / As responded above the changes only affect new files you will create. After changing the replication factor you won't see any hdfs size changes until the trash time interval which was set on 360 minutes configured by the hdfs trash interval has been reached fs.trash.interval Once the NameNode metadata has been updated, it is the DataNodes which would actually do the operation. There could be some delay, but space is definitely reclaimed. HTH ... View more