Member since
04-27-2016
26
Posts
6
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
651 | 09-30-2020 10:30 AM | |
5880 | 07-14-2020 12:40 PM | |
1452 | 07-14-2020 12:27 PM | |
2823 | 05-10-2017 03:45 PM | |
977 | 04-24-2017 09:14 PM |
09-30-2020
11:07 AM
@vincentD Please look at DATANODE logs to check for any FATAL/ERRORs before the shutdown. That could shed some light on the root cause of DN failure.
... View more
09-30-2020
10:30 AM
@Koffi Heap plays a vital role on overall HDFS performance. So you might need to confirm the usage is legitimate and you have adequate heap configured for Name node and Data nodes. Please refer below documentation. https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/configuring-namenode-heap-size.html
... View more
07-14-2020
12:40 PM
1 Kudo
@mike_bronson7 Are you trying to copy local files from your remote machine to destination HDFS cluster? You could use distcp if it is between hdfs. Please refer below documentation. https://docs.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_data_cluster_migrate.html For the local files in remote machine, you could SCP the files to any of the cluster node which has hdfs client installed and the do a "-copyFromLocal" or "-put" to push that to HDFS. Hope this helps.
... View more
07-14-2020
12:27 PM
1 Kudo
@mike_bronson7 It is recommended to have minimum 3 data nodes in the cluster to accommodate 3 healthy replicas of a block as the default replication factor is 3. HDFS will not write replicas of same blocks on the same data node. In this scenario there will be under replicated blocks and 2 healthy replicas will be placed on the available 2 data nodes.
... View more
05-19-2017
01:55 PM
@ARUN Please refer the below URLs. https://community.hortonworks.com/content/supportkb/49134/how-to-stop-start-a-ambari-service-component-using.html https://community.hortonworks.com/questions/6592/is-there-any-way-to-do-a-fully-automated-rolling-r.html
... View more
05-12-2017
04:38 PM
Hi @PJ - Please verify your AMS heap settings are configured properly as per the below documentation. https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-operations/content/ams_general_guidelines.html - Do a cleanup on "hbase.tmp.dir". a) Shutdown AMS. b) Move "hbase.tmp.dir". c) Start AMS Please monitor AMS and check if you are getting same exceptions.
... View more
05-12-2017
02:18 PM
@hardik desai Could you please paste the exception you are getting on HDFS /YARN logs at the time of starting the services? What is the value configured for the properties "hadoop.security.authentication" & "hadoop.security.authorization"?
... View more
05-11-2017
03:31 PM
@hardik desai Could you please confirm if both your name nodes are up and running as expected after enabling kerberos? Please check if Ambari is able to recognize and display active/standby state of your name nodes(If NN HA configured).
... View more
05-11-2017
02:35 PM
1 Kudo
From the logs and HDFS usage outputs it confirms the growth is related to HBASE snapshots. We can confirm this by checking if all the snapshots having same time stamp.
"list_snapshots" command from hbase shell will provide an output like below. hbase> list_snapshots
SYSTEM.CATALOG-ru-20160512 SYSTEM.CATALOG (Thu May 12 01:47:24 +0000 2016)
SYSTEM.FUNCTION-ru-20160512 SYSTEM.FUNCTION (Thu May 12 01:47:24 +0000 2016)
SYSTEM.SEQUENCE-ru-20160512 SYSTEM.SEQUENCE (Thu May 12 01:47:24 +0000 2016)
SYSTEM.STATS-ru-20160512 SYSTEM.STATS (Thu May 12 01:47:32 +0000 2016)
US_1-ru-20160512 US_1 (Thu May 12 01:47:32 +0000 2016)
ambarismoketest-ru-20160512 ambarismoketest (Thu May 12 01:47:32 +0000 2016)
dev.hadoop-ru-20160512 dev.hadoop(Thu May 12 01:47:33 +0000 2016)
prod.hadoop-ru-20160512 prod.hadoop (Thu May 12 01:47:35 +0000 2016)
compact.daily-ru-20160512 compact.daily (Thu May 12 01:47:43 +0000 2016)
compact.hourly-ru-20160512 compact.hourly (Thu May 12 01:47:43 +0000 2016)
test-ru-20160512 test (Thu May 12 01:47:43 +0000 2016)
We can confirm the timestamp of these snapshots from "hdfs dfs -ls -R /apps/hbase/ " out put as well. drwxr-xr-x - hbase hdfs 0 2016-05-12 01:58 /apps/hbase/data/.hbase-snapshot
drwxr-xr-x - hbase hdfs 0 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.CATALOG-ru-20160512
-rw-r--r-- 3 hbase hdfs 55 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.CATALOG-ru-20160512/.snapshotinfo
-rw-r--r-- 3 hbase hdfs 972 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.CATALOG-ru-20160512/data.manifest
drwxr-xr-x - hbase hdfs 0 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.FUNCTION-ru-20160512
-rw-r--r-- 3 hbase hdfs 57 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.FUNCTION-ru-20160512/.snapshotinfo
-rw-r--r-- 3 hbase hdfs 1064 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.FUNCTION-ru-20160512/data.manifest
drwxr-xr-x - hbase hdfs 0 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.SEQUENCE-ru-20160512
-rw-r--r-- 3 hbase hdfs 57 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.SEQUENCE-ru-20160512/.snapshotinfo
-rw-r--r-- 3 hbase hdfs 16813 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.SEQUENCE-ru-20160512/data.manifest
drwxr-xr-x - hbase hdfs 0 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.STATS-ru-20160512
-rw-r--r-- 3 hbase hdfs 51 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.STATS-ru-20160512/.snapshotinfo
-rw-r--r-- 3 hbase hdfs 928 2016-05-12 01:47 /apps/hbase/data/.hbase-snapshot/SYSTEM.STATS-ru-20160512/data.manifest
Hbase snapshots will be created as part of HDP Upgrade process. During upgrade process “snapshot_all” command will be triggered from “hbase_upgrade.py” script. Hence we can see all the snapshots have the same time stamp. Initially the snapshots will be a reference to the original table. When we run jobs after upgrade or insert data in to Hbase tables, these snapshots will expand with the delta to maintaining its original state. This could cause the gradual increase of snapshot size and hence HDFS size.
It is safe to delete the HBASE snapshots since they are just a reference to the original HBASE table. Deletion of the snapshots will clear up respective archive files too. Please remember not to delete archive directly or we will corrupt snapshot.
... View more
Labels:
05-10-2017
09:47 PM
This issue is caused due to the size of "~/.beeline/history" file in user's home directory. While we run big queries, it will flood the history file and slow down beeline on start up and shutdown. To resolve this issue, please move "~/.beeline/history" file and retry beeline.
In the lower versions (below 2.1.0), we can have a cron job to delete "history" file periodically. This issue has been fixed in latest Hive versions by adding "--maxHistoryRows". Please check the below JIRA for more details.
https://issues.apache.org/jira/browse/HIVE-15166
... View more
Labels: