Hi, I want to rebalance cluster, but I am not sure how this threshold value is calculated? What threshold I should keep to finish it quickly? I am not looking for a good balance right now. But one of our datanode is now over 80% used and we do want to reduce it to atleast 60% or 70% Can you please tell me what threshold we should keep and how I can monitor HDFS balancing is finished or not in ambari? ... View more

How to calculate number of total containers in cluster? Does it depend on Memory? or VCores ? How can I get the exact number of containers which are available in my cluster.? ... View more

Hi, We received below alert "NameNode RPC Latency" - I checked hadoop-hdfs/namenode logs, but I couldnt find anything wrong. Why this alert gets triggered? and is there any other logs I can check for this alert? ... View more

Hi, We currently have 8 Datanodes which has two hdfs disk mounted on each of the datanodes. One of the disk from the datanode is full. On this node HDFS(nodemanager) service was not coming up as it had below error Upon checking into articles found out that we can setup- "DataNode failed disk tolerance" value to 1 and ignore this volume as its 100% full, But I would like to understand how I can cleanup the data from this disk? I tried doing rebalance HDFS but which threshold value I should use? StorageLocation [DISK]file:/hdfs/data1/hadoop/hdfs/data/ org.apache.hadoop.util.DiskChecker$DiskErrorException: Error checking directory /hdfs/data1/hadoop/hdfs/data /dev/xvdl 50G 49G 0 100% /hdfs/data1 /dev/xvdk 50G 27G 21G 56% /hdfs/data0 ... View more

Hi, We currently have 13 EBS disks volumes in our one datanode, we have around 10 such datanodes each disk of the data node is of 1.5 TB used. We want to copy 1.5TB of volume to new disk. copying 100GB take about an hour or more. so 1.5TB will take so many hours to copy. Is there any faster way to copy the data? I am using rsync command to copy to new disk. ... View more

Hi, I am seeing that suddenly causing CPU spike earlier it was using 10% CPU now if I run top command I do see that Ambari infra solr causing 60% CPU spike.. and I see GC logs are almost 2GB Can someone tell me why there is so much spike ? Thanks ... View more

Recently we had issue with hiveserver2. After increasing the hive.server2.thrift.max.worker.threads to 1000 the issue seems to have resolved. How can we analyze the hiveserver jstack/threadump? What is the maximum value of hive.server2.thrift.max.worker.threads can have? Do we need to consider memory or anything while increasing threads? ... View more

Hi I do see lot of under replicated blocks in Ambari UI. Block Errors: 0 corrupt replica / 0 missing / 26463 under replicated What should I do in this case? Can someone help me how to fix this? Thanks ... View more

Hi, I have installed spark2 on the server. I do see some jobs are in the running status in resource manager and not getting completed. the user and name of those job is : hive Thrift JDBC/ODBC Server hive org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 Which are these jobs and why its not getting completed? Is it expected that after installation of spart2 services such as Spark2 History Server Livy for Spark2 Server Spark2 Thrift Server that these jobs will run? Kindly help, Thanks ... View more

Oozie jobs are not running currently. I tried to restart mysqldb and Oozie server, upon checking in the logs found out that in Ozziee-error log E0810: Job state is not [RUNNING]. Skipping ActionStart Execution, Error Code: E0810 An Exception occured while talking to the database: Access denied for user 'oozie'@'<host>' (using password: YES) java.sql.SQLException: Access denied for user 'oozie'@'<host>' (using password: YES) In Oozie Web UI I see Job is stuck in PREP status. Can you please help? ... View more