Member since
12-08-2015
34
Posts
16
Kudos Received
0
Solutions
05-10-2016
04:56 AM
@Rajkumar Singh Thank you for your reply. I had refered to this article before posting this question. As I couldn't manage heap memory using recommended CMS Configuration mentioned in the article, I'm investigating more into the use of G1 in HadoopFramework.
... View more
05-09-2016
10:53 AM
I am running ResourceManagerin the following environment, 2CPU/64GBMEM/4TB OS Linux6.5, JDK 1.8 ResourceManager sometimes takes much time to Full GC. ・ORACLE tutorials.Getting started with the G1 GarbageCollector http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html ・ORACLE G1GC turning https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/g1_gc_tuning.html#sthref61 As per my current undetstanding, the possible fix forthis could be G1GC. I have gone through few documents.However, I am unable to understand it in detail. Does anyone know recommended values to be configured for G1GC inResource-Manager? Which key will be used for G1GC,and how will each key be used?
... View more
Labels:
03-22-2016
09:47 AM
1 Kudo
I can not get the enabled policies in the list of policies for Hive. When I type "status=true", I can get the enabled policies. I looked up source code and found the cause of this problem. https://github.com/hortonworks/ranger-release/blob/HDP-2.4.0.0-tag/security-admin/src/main/java/org/apache/ranger/common/ServiceUtil.java#L731-L732 AppConstants.STATUS_DISABLED and AppConstants.STATUS_ENABLED are used for the statusEnum. However, these values don't exist. I think that RangerCommonEnums.STATUS_DISABLED and RangerCommonEnums.STATUS_ENABLED are proper values. Please check the source code of Ranger.
... View more
Labels:
02-29-2016
02:11 AM
1 Kudo
@Ancil McBarnett @Neeraj Sabharwal Thanks for the good information. I modified /usr/hdp/<version>/ranger-admin/ews/webapp/WEB-INF/log4j.xml with the same way and made sure that I colud do what I want to do.
... View more
02-29-2016
02:04 AM
1 Kudo
@Neeraj Sabharwal Sorry my reply is late. I have just accepted the answer. Thank you so much for your kindness.
... View more
02-04-2016
04:55 AM
1 Kudo
@Neeraj Sabharwal Thank you for your prompt reply. If I don't have any other means, I will set the cron. However, I think it is better that the Ranger removes log fles with log4j or tomcat configs and I would like the Ambari to enable to change the dulation of the rotation for the ranger-admin.
... View more
02-04-2016
02:22 AM
1 Kudo
I have used the Ranger for several days. I found that there were a number of access files and that files was not made rotate. $ ls /var/log/ranger/admin/access_log.2015-1* | head
/var/log/ranger/admin/access_log.2015-11-27.15.log
/var/log/ranger/admin/access_log.2015-11-27.16.log
/var/log/ranger/admin/access_log.2015-11-27.17.log
/var/log/ranger/admin/access_log.2015-11-27.18.log
/var/log/ranger/admin/access_log.2015-11-27.19.log
/var/log/ranger/admin/access_log.2015-11-27.20.log
/var/log/ranger/admin/access_log.2015-11-27.21.log
/var/log/ranger/admin/access_log.2015-11-27.22.log
/var/log/ranger/admin/access_log.2015-11-27.23.log
/var/log/ranger/admin/access_log.2015-11-28.00.log
$ ls /var/log/ranger/admin/access_log.2015-1* | wc -l
500 I searched the conf file included the access log setting, but I couldn't find. How do I control the access log of the ranger-admin?
... View more
Labels:
02-04-2016
12:24 AM
@Artem Ervits So Sorry for replying late. I understood that our environment was unusual. I'm about to write the small script. This script will get users and groups list from our LDAP server and make available for the Ranger to read. The Ranger will synchronize users and groups list to the formated list.
... View more
01-27-2016
11:47 AM
1 Kudo
@Neeraj Sabharwal Thank you for your prompt reply. I'm happy now.
... View more
01-27-2016
10:27 AM
1 Kudo
I built the hadoop cluster included the Ranger service with the Ambari. When I checked the usersync.log, I needed the sudo command. On the other hands, the logs for the ranger-admin allow anyone to read. Why is the access mode of the usersync.log 600 ?
... View more
Labels:
01-27-2016
01:01 AM
@Neeraj Sabharwal I'm looking forward to that version 🙂
... View more
01-27-2016
12:34 AM
@Neeraj Sabharwal Thank you for your kindness. I'm sorry for the delay in responding. As you pointed out, I couldn't work "Remove the Hive Metastore password entry from the Hive configuration" with the Ambari. When the Hive restarts with the Ambari, the hive-site.xml will be removed.
... View more
01-25-2016
05:10 PM
Thank you for the reference. I will try to do it. Thanks a lot!!
... View more
01-25-2016
04:35 PM
@Neeraj Sabharwal Thank you for your prompt reply. I alse see that the password is encrypted on Ambari. On the hive-site.xml in the hive metastore server(/etc/hive/conf/conf.server/hive-site.xml), I can find the password for "javax.jdo.option.ConnectionPassword" that is plane text.
... View more
01-25-2016
10:23 AM
1 Kudo
Hive users can read the hive-site.xml and javax.jdo.option.ConnectionPassword because the password is stored in plane text. Can I make it encrypted ?
... View more
Labels:
01-21-2016
04:19 AM
@Artem Ervits Thanks to you, I could comminicate with Enis. Thank you so much.
... View more
01-21-2016
04:15 AM
Thank you so much for your kindness. I'll read jira pages and comment if there is something I want to say.
... View more
01-21-2016
02:23 AM
@Neeraj Sabharwal Thank you for the details. I could understand clearly.
... View more
01-21-2016
02:21 AM
@Enis Thank you for your reply and detail. I underrstood that the list command requires ADMIN or CREATE and it is need to fix this in HBase if I get the list or description of the tables by READ permission. How do you think that there are many accounts who can excecute hbase shell commands on HBase? I think users would like to know the table name and get the list of tables by the list command. I also think that the administrator does not want to give a lot of users ADMIN or CREATE permissions. For this reason, I thought READ permisson was better for the list command.
... View more
01-20-2016
03:56 PM
@Neeraj Sabharwal Thank you for your reply and the link. I understand that read permission enables me to just read column family at the table scope not for reading tables. Is it right?
... View more
01-20-2016
03:41 PM
@Artem Ervits Thank you for your reply. I got read permission on all tables, but I couldn't show tables by list command.
... View more
01-16-2016
12:23 PM
Thank you very much for your reply and very helpful solutions. I'd rather not manage both a repository HDFS and Hive if I can avoid it. However, we manage Hadoop resources by the YARN queue assigned to each user. For this reason I would like to keep "run as end user instead of hive"(hive.server2.enable.doAs=true).
... View more
01-15-2016
07:58 AM
1 Kudo
When I allow user1 to read the col1 column in the table on Hive, I will add the following policy to Hive service in Ranger. However, this is not enough in case "Run as end user instead of Hive user = true". I have to add the policy to HDFS service in Ranger. The following table shows the policies at each ACL layer. In this case, user1 can access to the entire table data by hdfs command or hive command without hiveserver2. I think that Ranger support column based ACL in case when "Run as end user instead of Hive user" is true.
... View more
Labels:
12-16-2015
01:40 AM
Thank you for telling me the url to the Microsoft documentation, but I couldn't find this page. I would like to tell you more information, but I don't know why our AD is like this. So sorry.
... View more
12-15-2015
05:40 AM
1 Kudo
I tried looking up the source code for Ranger Usersync. getUserGroups method in LdapUserGroupBuilder class. groupSearchResultEnum = ldapContext.search(
groupSearchBase,
extendedGroupSearchFilter,
new Object[]{userInfo.getUserFullName()},
groupSearchControls); userInfo.getUserFullName() returns "uid=user1,ou=user,dc=TEST". I suspect this is too long and userInfo.getUserName() method is more appropriate.
... View more
12-14-2015
10:09 AM
1 Kudo
Thank you for telling me the other document. I had tried changing some settings, but I have not solubed yet.
... View more
12-14-2015
08:06 AM
1 Kudo
@Ali Bajwa After restarting Ranger Usersync, I got the below logs. LdapUserGroupBuilder initialization completed with -
ldapAuthenticationMechanism: simple,
searchBase: dc=TEST,
userSearchBase: ou=user,dc=TEST,
userSearchFilter: ,
extendedUserSearchFilter: (objectclass=user),
userNameAttribute: uid,
userSearchAttributes: [uid, memberof, ismemberof],
userGroupNameAttributeSet: [memberof, ismemberof],
groupSearchEnabled: true,
groupSearchBase: ou=group,dc=TEST,
groupSearchFilter: ,
extendedGroupSearchFilter: (&(objectclass=group)(member={0})),
extendedAllGroupsSearchFilter: (&(objectclass=group)),
groupMemberAttributeName: member,
groupNameAttribute: cn, Using LDAP type is AD.
... View more
12-14-2015
07:03 AM
Thank you for your reply and telling me the document. I tried setting "memberof, ismemberof" to "ranger.usersync.ldap.user.groupnameattribute" and restarted Ranger. However, I could not resolve this problem and the logs showed the same content. I will make sure the settings again.
... View more
12-14-2015
06:06 AM
1 Kudo
I'm trying to sync users and groups from LDAP into Ranger using Ranger Usersync. How do I associate the groupname in the group info and the gid in the user and the group info? In my LDAP server, the user info has the gid, but does not have the groupname. I tried LDAP Usersync, but I can't get groupnames. Ranger UI only displaied the gid. I had tried after setting ranger.usersync.group.searchenabled to true, but I could not get groupnames again. The usersync.log showed the folloing logs: INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 1, userName: user1, groupList: []
INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 2, userName: user2, groupList: []
・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・・
INFO LdapUserGroupBuilder [UnixUserSyncThread] - computed groups for user: user1, groups: []
ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: org/apache/commons/httpclient/URIException, for user: user1, groups: []
INFO LdapUserGroupBuilder [UnixUserSyncThread] - computed groups for user: user2, groups: []ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: org/apache/commons/httpclient/URIException, for user: user2, groups: []
Settings of usersync was as follows : ranger.usersync.source.impl.class = org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder
ranger.usersync.ldap.user.searchscope = sub
ranger.usersync.ldap.user.searchfilter = (space)
ranger.usersync.ldap.user.searchbase = ou=account,dc=TEST
ranger.usersync.ldap.user.objectclass = user
ranger.usersync.group.memberattributename = member
ranger.usersync.group.nameattribute = cn
ranger.usersync.group.objectclass = group
ranger.usersync.group.searchbase = ou=group,dc=TEST
ranger.usersync.group.searchenabled = true
ranger.usersync.group.searchfilter = (space)
ranger.usersync.group.searchscope = (space)
ranger.usersync.ldap.searchBase = dc=TEST
ranger.usersync.ldap.user.groupnameattribute = gidNumber
ranger.usersync.ldap.user.nameattribute = uid The user and group setting is as follows: ・User dn: uid=user1,ou=user,dc=TEST
uid: user1
objectClass: user
uidNumber: 10
gidNumber: 50100
cn: user1
・Group dn: cn=group1,ou=group,dc=TEST
cn: group1
objectClass: group
gidNumber: 50100
member: user1
Please let me know what I should check. Version HDP 2.3.0.0
Ranger 0.5.0.2.3
I have bad English, so I apologize if I say something strange. Thanks.
... View more
Labels: