Member since
12-16-2015
198
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1325 | 09-01-2020 04:15 AM |
07-06-2021
10:26 AM
@diplompils It is not necessary that file is lost if you are getting the output as false for recoverLease command. Usually file can't be deleted until it has lease acquired and not explicitly deleted using rm command. You can try below- hdfs debug recoverLease -path <file> -retries 10 Or you may check - https://issues.apache.org/jira/browse/HDFS-8576
... View more
05-12-2021
03:24 AM
1 Kudo
@PrernaU Ambari 2.7.5 is only fully compatible with HDP 3.1.5, while other HDP 3.1.x or 3.0.x versions are partially compatible for upgrade only. Please see.
... View more
05-12-2021
03:22 AM
1 Kudo
@fpaezalban Ambari 2.7.5 is only fully compatible with HDP 3.1.5, while other HDP 3.1.x or 3.0.x versions are partially compatible for upgrade only. Please see.
... View more
05-12-2021
03:20 AM
1 Kudo
@Jans What is the exact HDP version? Ambari 2.7.5 is only fully compatible with HDP 3.1.5, while other HDP 3.1.x version are compatible only for upgrade.
... View more
05-10-2021
03:22 AM
@PrernaU I hope you haven't changed any config in HDFS. Please compare pre & Post upgrade configs using ambari UI. Considering the exception below, I have seen this issue once due to memory issue on a single DataNode. ERROR: Cannot set priority of datanode process 45359 i.e. The available memory was 12GB while the DN heap was set to 16GB & due to this the DN JVM was failing to start. /proc/meminfo:
MemTotal: 131407744 kB
MemFree: 2180792 kB
MemAvailable: 12004080 kB You can try checking which process is using RAM by running below command and try to reduce the RAM utilization and start the DN process #ps aux --sort -rss There could be something else but probably some host level crunch is causing the JVM to not get started properly. Please check DN.out and DN.log file once for more details.
... View more
09-01-2020
08:40 AM
There are several options available to achieve this use case. The easiest and the best approach would be HBase snapshots method to transfer the data.
Note: All actions need to be performed as the HBase user only, to ensure correct permissions.
On source cluster: #hbase shell> snapshot ‘Test’_Table’, ‘Test_Table_SS’
#hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats
#hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot 'Test_Table_SS' -copy-to hdfs://<Destination_NN_hostname>:8020/hbase -mappers 16 -bandwidth 200
On destination cluster: #hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats
#hbase shell> clone_snapshot ‘Test_Table_SS’, ‘Test_Table’
#hbase shell> major_compact 'Test_Table'
Once done, you can choose to delete the snapshots on both ms05 and as05: #hbase shell> delete_snapshot 'Test_Table_SS'
But If you plan to use CopyTable , this will not work without additional configuration.
Communication between an older client and a newer server is not guaranteed, there is currently a workaround to allow this to work by adding the following property to your client configuration:
On the client being used to launch the CopyTable, you can do either:
Command Line:
-Dhbase.meta.replicas.use=true
OR
hbase-site.xml:
hbase.meta.replicas.use
true
... View more
09-01-2020
04:15 AM
1 Kudo
There are several options available for this. If you plan to use CopyTable, this will not work without additional configuration. Communication between an older client and newer server is not guaranteed, there is currently a workaround to allow this to work by adding the following property to your client configuration. On the client being used to launch the CopyTable, you can do either: Command Line: -Dhbase.meta.replicas.use=true OR hbase-site.xml: hbase.meta.replicas.use true Your other option would be to use snapshots to transfer the data. Note- If this helps, please don't forget to click on "Accept as Solution".
... View more
03-27-2020
10:12 AM
@dineshc Please specify where the mentioned workaround property needs to be added. Ams-site.xml or Ams-Hbase-site.xml?
... View more
02-06-2020
09:12 AM
@josh_nicholson NOTE: For Kerberized Cluster use the value of "zookeeper.znode.parent" may be "/ams-hbase-secure" so we can connect to it as following: /usr/hdp/2.5.0.0-1245/phoenix/bin/sqlline.py c6403.ambari.apache.org:61181:/ams-hbase-secure
... View more
01-06-2020
08:57 AM
@aengineer Also noticed from one comment in HDFS-8789 that " balancer doesn't support anything other than the default placement policy (BlockPlacementPolicyDefault). " HDFS-14053 says ability for NN to re-replicate blocks based on policy change is fixed in hadoop 3.3.0 [not sure if it's hadoop version or not, though NN version doesn't make sense], while HDFS-14637 supports above statement until UD get enable.
... View more
01-06-2020
08:08 AM
@aengineer HDFS-14637 and HDFS-8789 appear to contradict each other as HDFS-14637 says that after changing the network topology or placement policy on a cluster and restarting the namenode, the namenode will scan all blocks on the cluster at startup, and check if they meet the current placement policy. If they do not, they are added to the replication queue and the namenode will arrange for them to be replicated to ensure the placement policy is used. It would be good to get some clarity on this.
... View more
01-06-2020
03:09 AM
@aengineer In continuation of Smarak's concern, do we have any further info on this? JIRA HDFS-8789 is still in unresolved state, so looks change in block placement policy is still not applicable to existing blocks. Any thoughts?
... View more
01-02-2020
08:27 AM
1 Kudo
@o20021106 You can check if you are missing ZK quorum property under hive config [hive-site.xml]. hbase.zookeeper.quorum= Because of missing quorum hive is trying to connect to default hbase znode [/hbase] and not getting anything cause it is not even existing.
... View more
07-19-2018
02:29 PM
Hi Bala, Why this secure process is for DN only ? No as such process runs for NN.
... View more
07-06-2018
01:39 PM
Will it be possible to create namespace on the custom hdfs path ?
... View more
09-25-2017
09:00 PM
Can we use test_alert_disk_space.sh file at place of test_alert_disk_space.py ?
... View more