Member since
01-18-2016
156
Posts
28
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
926 | 04-06-2018 09:24 PM | |
810 | 05-02-2017 10:43 PM | |
2369 | 01-24-2017 08:21 PM | |
16755 | 12-05-2016 10:35 PM | |
3511 | 11-30-2016 10:33 PM |
06-19-2018
08:57 PM
Thanks, Pardeep. To make it 500x faster, do 500 files per call to the hadoop command. By changing the second line above, we can do this instead: $ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files
# Now using xargs -n 500 (or --max-args 500)
$ cat /tmp/under_replicated_files |xargs -n 500 hdfs dfs -setrep 1 /tmp/under_replicated_files<br>
... View more
04-06-2018
09:24 PM
1 Kudo
The code for creating principals in AD is here: ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ADKerberosOperationHandler.java
... View more
02-20-2018
06:03 PM
@Rachna Dhand I know you must be way past this issue, but -- You have to install the packages on all NodeManager nodes as root so they are available to all users. Maybe this will help someone else in the future.
... View more
09-27-2017
06:48 PM
Thanks @Karthik Narayanan. We do not see an option for cross role permission. Would it have another name? They did grant "Assumerole" but it is actually the same account so I'm not sure why it would need to assume a role in the first place.
... View more
09-26-2017
10:48 PM
In Nifi with the PutS3Object, we get an error using AWS Credentials Provider service. AwsCredentialsProviderControlerService is configured to use IAM roles as follows: Use Default Credentials = True Use Anonymous Credentials = False Assume Role ARN = arn:aws:iam::ahjhdiauisjkk:role/role-test Assume Role Session Name = nifitest (*arbitary name*) Session time = 3600 No other values are set in the AwsCredentialsProviderControlerService We are using IAM roles because of organizational policies. The error is loosely transcribe here (it may contain typos): 13:40:46 EDT - All Nodes - ERROR
PutS3Object[id=asdfasdfasdfasdf] Failed to put StandardFlowFileRecord[uuid=xxxxxxxx,claim=StandardContentClaim[resourceClaim=StandardResourceClaim[id=11111111, container=default,section=1], offset=0,length=222222],offset=1,name=test3,size=33333] to Amazon S2 due to com.amazonaws.services.securitytokenmodel.AWSSEcurityToeknServiceException: User: arn:aws:sts::7777777:assumed-role/role-hdf-node/i-03333330000 is not authorized to perform: sts:Assumerole on resource: arn:aws:sts::7777777:role/role-hdf-node (Service: AWSSecurityToeknService;Status Code: 403; Error Code: AccessDenied; Request ID: aaaaaaaaaaaa)
com.amazonaws.services.securitytokenmodel.AWSSEcurityToeknServiceException: User: arn:aws:sts::7777777:assumed-role/role-hdf-node/i-03333330000 is not authorized to perform: sts:Assumerole on resource: arn:aws:sts::7777777:role/role-hdf-node (Service: AWSSecurityToeknService;Status Code: 403; Error Code: AccessDenied; Request ID: aaaaaaaaaaaa) Thanks for any help. Jim
... View more
Labels:
- Labels:
-
Apache NiFi
07-18-2017
03:09 PM
Thank you @vperiasamy. We resolved the issue. I talked to Centrify Support and based on the way this environment was configured, authentication was not possible with the uid and Linux dn. The company's Centrify admins agreed to change the uid of users to match the Windows ID. We then configured authentication to go against the Windows identity, which happens to match the uid in Linux. It's a long story about the use-case, but this was by far the cleanest solution due to the fact that they use both Linux and Windows to access the cluster (we avoided manual mapping of identities this way). The article you shared is very good information.
... View more
06-29-2017
08:33 PM
Hi @Michael Szymczak - any thoughts on this?
... View more
06-29-2017
08:31 PM
I am trying to configure LDAP authentication with Ambari and other components (e.g. Zeppelin, Ranger and Knox). This is a complex environment with Centrify where users have multiple identities which resolve to a single ID in Linux. On Linux, my id is "u1101". I can execute "id u1101" or "id james-jones" and both result in the same Linux ID -- e.g. uid=60012(u1101) gid=5001(user). On a Windows system, only james-jones works. There are two domains: company.com and corp.company.int. I want to configure LDAP authentication using the Linux ID rather than the windows sAMAccountName, but it only works with the sAMAccountName. I have configured the correct settings in Ambari for userObjectClass, baseDn, userAttributeName, primaryUrl, managerDn, etc. I don't know if the problem is with Centrify or an LDAP server hosting the Linux accounts. I decided to conduct a test, and it seems that using dn bind account is prohibited in one case but not the other. Since Ambari LDAP authentication essentially does two queries, one with the bind user to find the person logging in, and then a second query with the person's DN and the password they entered, I tested two ldapsearch queries. One with the Windows DN and the second using the Linux DN. Using ldapsearch, this query succeeds using my Windows domain DN ldapsearch -D "CN=1101737,OU=NY,OU=my_company,OU=CO_Users,DC=corp,DC=company,DC=int" -W \
-b "dc=corp,dc=company,dc=int" \
"sAMAccountName=james-jones"
This query fails on authentication using my UNIX domain DN ldapsearch -D "CN=james-jones@corp.company.int,CN=Users,CN=Global,CN=Zones,OU=Centrify,OU=Enterprise Systems,DC=corp,DC=company,DC=int" -W \
-b "DC=corp,DC=company,DC=int" \
"sAMAccountName=james-jones" There are two different LDIFs for these two ids (u1101 and james-jones): sAMAccountName=james-jones
objectClass=person
dn=CN=1101737,OU=NY,OU=my_company,OU=CO_Users,DC=corp,DC=company,DC=int
distinguishedName==CN=1101737,OU=NY,OU=my_company,OU=CO_Users,DC=corp,DC=company,DC=int The other looks like non-AD LDAP and I think openLdap may be behind it with some centrify magic going on. uid=u1101
objectClass=posixAccount
dn=CN=james-jones@corp.company.int,CN=Users,CN=Global,CN=Zones,OU=Centrify,OU=Enterprise Systems,DC=corp,DC=company,DC=int
distinguishedName=CN=james-jones@corp.company.int,CN=Users,CN=Global,CN=Zones,OU=Centrify,OU=Enterprise Systems,DC=corp,DC=company,DC=int The reason this matters is that when I use Ambari views, my account needs to match the Unix accounts. Hopefully that made sense. It's a bit long and complicated. Any help is very appreciated.
... View more
Labels:
- Labels:
-
Apache Ambari
05-30-2017
06:25 PM
1 Kudo
Well, I can't say about running Solr in YARN, but Solr can now be installed and managed by Ambari 2.5 and later. I don't think it was available in 2.4. Check out this link: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_solr-search-installation/content/ch_hdp-search-install-ambari.html
... View more
05-26-2017
04:09 PM
1 Kudo
Ambari 2.5.0.3 complains about openblas-Rblas x86_64 0.2.19-4.el7 installed when installing HDP 2.6. This was in the checks done during service installation. What version of R is HDP happy with? If we remove openblasRblas, it will also remove R 3.3.3 and we want to replace it with the correct version.
... View more
Labels: