Member since
11-05-2017
20
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3221 | 02-15-2018 05:51 AM |
02-06-2019
02:24 PM
@Kartik Ramalingam Thank you for your wonderful and helpful post! Ranger authorization is still incorrect in the post. Ranger authorization is already enabled in initial topology. However, Step 7 must describe it to disable Ranger authorization by modifying the parameter value from XASecurePDPKnox to AclsAuthz. Also, Step 7 example needs to be corrected. Regards, Sakhuja
... View more
02-21-2018
02:21 PM
In my case, it was a PRIVATE_IP variable in Profile. Once removed, it started working.
... View more
02-21-2018
12:37 PM
@wbu Thank you for the post but could you please help me understand that how you have created HORTONWORKS.COM (REALM) and "hadoopadmin" principal on mac for which you have generated a ticket using principal's password? I am using "kadmin -l" to init a new REALM "EXAMPLE.COM" in line with cluster REALM and also the username "hadoopadmin" but when I try adding a REALM using "init -r <realm name>", I get:
kadmin: create_random_entry(krbtgt/EXAMPLE.COM@EXAMPLE.COM): randkey failed: Principal does not exist
init -r <realm name>
Or if I try adding a principal "add -r hadoopadmin@EXAMPLE.COM", I get:
kadmin: adding hadoopadmin@EXAMPLE.COM: Principal does not exist
vi /Library/Preferences/edu.mit.Kerberos OR vi /etc/krb5.conf
.example.com = "EXAMPLE.COM"
example.com = "EXAMPLE.COM"
[libdefaults]
default_realm = "EXAMPLE.COM"
dns_fallback = "yes"
noaddresses = "TRUE"
[realms]
EXAMPLE.COM = {
admin_server = "ad.example.com"
default_domain = "example.com"
kdc = "ad.example.com"
}
As far as I understand, on mac machine following steps must be performed before doing the above given steps:
1. Create vi /etc/krb5.conf
2. Create a new REALM "EXAMPLE.COM" (same as Hadoop cluster Kerberos REALM)
2. Create a new user principal "hadoopadmin" (same as Hadoop cluster Kerberos principal used to access the services)
3. Then only I can create a ticket (kinit) with the same password used in Step 2 while creating the user principal
Regards,
... View more
02-19-2018
05:30 AM
@pdarvasi Finally found the solution to this issue. Here are my findings: When we started HDP using cloudbreak, HDP default configuration had calculated non-HDFS reserved storage "dfs.du.datanode.reserved" (approx 3.5 %) on total disk for the lowest storage configured for a datanode (among the compute config groups) which had three drives and one drive was in TBs. Our default configuration to store data on datanode "dfs.datanode.data.dir" was pointing to a drive with lowest capacity (around 3 % of overall DN storage). This 3 % < 3.5 % had made HDFS capacity as 0% and our existing datanode storage had some supporting directories and files in KBs which had resulted in marking negative KB capacity of the datanode. To fix the downscaling issue, either, we need to lower down non hdfs reserved capacity (lower than 3 %) or point our datanode to higher disk capacity (greater than 3.5 %) I had tried this and it worked. No more changing WASB URI, therefore, keeping it as a default storage. However, I am thankful to you for making suggestions.
... View more
02-19-2018
05:29 AM
@pdarvasi Finally found the solution to this issue. Here are my findings: When we started HDP using cloudbreak, HDP default configuration had calculated non-HDFS reserved storage "dfs.du.datanode.reserved" (approx 3.5 %) on total disk for the lowest storage configured for a datanode (among the compute config groups) which had three drives and one drive was in TBs. Our default configuration to store data on datanode "dfs.datanode.data.dir" was pointing to a drive with lowest capacity (around 3 % of overall DN storage). This 3 % < 3.5 % had made HDFS capacity as 0% and our existing datanode storage had some supporting directories and files in KBs which had resulted in marking negative KB capacity of the datanode. To fix the downscaling issue, either, we need to lower down non hdfs reserved capacity (lower than 3 %) or point our datanode to higher disk capacity (greater than 3.5 %) I had tried this and it worked. No more changing WASB URI, therefore, keeping it as a default storage. However, I am thankful to you for making suggestions.
... View more
02-15-2018
05:51 AM
Thank you @pdarvasi for answering it in another post: https://community.hortonworks.com/questions/171210/hortonworks-cloudbreak-default-hdfs-as-azure-wasb.html
... View more
02-14-2018
01:52 PM
@pdarvasi Thank you so much for answering and this is what I was looking for. Let me see how we can make these working.
... View more
02-14-2018
12:25 PM
Thank you for your quick response. I don't have any plan to keep the data on local HDFS and therefore, using Azure WASB for all storage (no worry of data loss). Redundancy will be covered by WASB and no plans to keep data on local HDFS. So, when I am using default storage as WASB then why I should add some capacity to local HDFS for decommissioning?
... View more
02-14-2018
11:41 AM
Configured Azure WASB storage as a default HDFS location through Cloudbreak, which had made Hadoop local HDFS capacity as 0 in Ambari (100 % utilized). I have default replication as 1 but now when I am trying to decommission a node, datanode tries to rebalance some 28KB of data to another available datanode. However, our HDFS has 0 capacity and therefore, decommissioning fails with below given error: New node(s) could not be removed from the cluster. Reason Trying to move '28672' bytes worth of data to nodes with '0' bytes of capacity is not allowed Getting the information on cluster shows that default local HDFS is still used for some KB space which is getting rebalanced whereas available capacity is 0: "CapacityRemaining" : 0,
"CapacityTotal" : 0,
"CapacityUsed" : 131072,
"DeadNodes" : "{}",
"DecomNodes" : "{}",
"HeapMemoryMax" : 1060372480,
"HeapMemoryUsed" : 147668152,
"NonDfsUsedSpace" : 0,
"NonHeapMemoryMax" : -1,
"NonHeapMemoryUsed" : 75319744,
"PercentRemaining" : 0.0,
"PercentUsed" : 100.0,
"Safemode" : "",
"StartTime" : 1518241019502,
"TotalFiles" : 1,
"UpgradeFinalized" : true,
There is an ambari decommissioned jar used by Cloudbreak to check, if HDFS is running out of space. Is there a way to change this jar? if (remainingSpace < safetyUsedSpace) {
throw new BadRequestException(
String.format("Trying to move '%s' bytes worth of data to nodes with '%s' bytes of capacity is not allowed", usedSpace, remainingSpace)
);
} Reference link: https://github.com/hortonworks/cloudbreak/blob/1.16.4/core/src/main/java/com/sequenceiq/cloudbreak/service/cluster/flow/AmbariDecommissioner.java#L314-L321
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Hortonworks Cloudbreak
02-13-2018
08:21 AM
@pdarvasi I am still stuck at the same issue. Is it possible to edit the jar "AmbariDecommissioner.java"? If yes, where I can find on cloudbreak? Thanks for your help in advance!
... View more