Hi, As per my knowledge (please correct me if I am wrong), the Datanodes sends the block report to both Active and Standby Namenodes. The job of Active NN is to write to the Journal Nodes and the job of Standby namenode is to read from Journal nodes. Now why does Standby namenode need to read from Journal nodes when the Datanodes (slaves) are already sending the block reports to it? ... View more

Hi, AT times, when I restart a Region Server, the RPS and Num. Regions won't catch up at all until and unless I do a rolling restart of all the Region Servers. Why does this happen? HDP version: 3.0.1 Hbase Version: 2.0.2 I could not see any ERROR snippet in the RS logs post restart. However, there are 2 WARNs: BTW, Major Compaction was triggered for a table and during that time, I restarted a Region Server. 2019-05-13 22:42:44,298 WARN  [regionserver/prd-hdpcyclops015:16020] throttle.CompactionThroughputControllerFactory: org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController is deprecated, please use org.apache.hadoop.hbase.regionserver.throttle.PressureAwareCompactionThroughputController instead 2019-05-13 22:42:44,332 WARN  [regionserver/prd-hdpcyclops015:16020] regionserver.SecureBulkLoadManager: hbase.bulkload.staging.dir  is deprecated. Bulkload staging directory is hdfs://cyclops/apps/hbase/data/staging 2019-05-13 22:42:44,404 INFO  [SplitLogWorker-prd-hdpcyclops015:16020] regionserver.SplitLogWorker: SplitLogWorker prd-hdpcyclops015.XXXXXX.nm1,16020,1557767558749 starting 2019-05-13 22:42:44,413 INFO  [regionserver/prd-hdpcyclops015:16020] regionserver.HeapMemoryManager: Starting, tuneOn=false 2019-05-13 22:42:44,421 INFO  [regionserver/prd-hdpcyclops015:16020] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 11796, initial count 0 2019-05-13 22:42:44,423 INFO  [regionserver/prd-hdpcyclops015:16020] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 13107, initial count 0 Can anyone please assist me here in resolving this issue? ... View more

My cluster was hung. Was unable to add hosts or perform any basic activities in Ambari like restart of a service. Was constantly seeing the WARN snippet in Ambari Server logs: Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1 Here's a small hack to resolve the issue: 1. Check the cluster id in your backend Ambari DB. Mine is MySQL. select * from clusterstate; 2. The same value found in step 1 should be there in Stage table's "cluster_id" columns select stage_id, request_id, cluster_id from stage; 3. If there are values as -1 please update it to the correct value found in step 1. Example: UPDATE stage SET cluster_id='2' WHERE request_id IN (383,384,388,389); 4. Restart Ambari-Server ambari-server restart 5. Post this check by restarting any service like Grafana or any small service which not does impact the Hadoop service. If it proceeds, the cluster is now stable and you will be able to add nodes. 6. If issue persists, the perform the following in your backend Ambari DB. SELECT * FROM host_role_command WHERE status='PENDING'; 7. If you get any output, you need to update the status to "ABORTED". UPDATE host_role_command SET status='ABORTED' WHERE status='PENDING'; 8. Restart Ambari-Server ambari-server restart Validate the health of Ambari by restarting Grafana or any small service which not does impact the Hadoop service. If everything is good, proceed by adding the nodes. ... View more

This is just a knowledge sharing article. I had faced this issue in production and took me a day to resolve it. The workaround that I'm sharing will help you in getting your table back online in "Enabled" without deleting Zookeeper Hbase table znode or any data as a matter of fact.. Here are steps to resolve it. 1. Run a "get" command against hbase:meta for the affected table hbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' COLUMN                                   CELL  table:state                             timestamp=1551456805377, value=\x08\x02 2. Notice the above "value". Its pointing to \x08\x02 which is wrong. The value should either be \x08\x00 (Enabled) or \x08\x01 (Disabled) 3. Edit the value manually. hbase(main):003:0> put 'hbase:meta','<AFFECTED_TABLE_NAME>','table:state',"\b\0" Click here for more information on "Control Characters" 4. Verify the same. hbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' The "value" now should be \x08\x00 Post this, again run disable <table_name> and enable <table_name> just for the love of sanity check in hbase shell and you are done with the issue. ... View more

Hi, I've setup Hbase cross cluster replication between 2 clusters. After running some stress test by running the following command - "sudo -su hbase hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 1" which will do random inserts. The count on both the cluster's table matched i.e 906856. However we have to verify if the replication is consistent on both cluster. To do that, I followed Hortonwork's document and ran the command. The output is shown below ROWS_SCANNED=906856 RPC_CALLS=9070 RPC_RETRIES=0 org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication$Verifier$Counters BADROWS=3 CONTENT_DIFFERENT_ROWS=3 GOODROWS=906853 The number of rows scanned is correct 906856 which is total count of table. But there are 3 bad rows. Same result was given when ran in another cluster as well. With this result I can say that there is problem with Quality and not with Quantity. The main question now is: How to find out and view the actual 3 Bad Rows in the table ? Regards, Shesh Kumar ... View more

Hi, My cluster is Kerberized and I'm planning to integrate Ranger with FreeIPA. My Auth Method is LDAP in Ranger and Sync source is LDAP/AD. I've provided the values for required fields, users are also seen in usersync.log file as well but its not syncing with Ranger. Here's the sample of the Usersync log error. Kindly check attachment for full logs. 21 Dec 2018 12:05:08 INFO UnixAuthenticationService [main] - Starting User Sync Service! 21 Dec 2018 12:05:08 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 21 Dec 2018 12:05:08 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 21 Dec 2018 12:05:08 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created 21 Dec 2018 12:05:08 INFO UserGroupSyncConfig [UnixUserSyncThread] - Sleep Time Between Cycle can not be lower than [3600000] millisec. resetting to min value. 21 Dec 2018 12:05:08 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder 21 Dec 2018 12:05:08 INFO PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync@XXXXXX.COM and keytab = /etc/security/keytabs/ranger_user_sync.keytab 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to build Group List : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildGroupList(PolicyMgrUserGroupBuilder.java:590) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$000(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$1.run(PolicyMgrUserGroupBuilder.java:215) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$1.run(PolicyMgrUserGroupBuilder.java:211) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildUserGroupInfo(PolicyMgrUserGroupBuilder.java:211) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.init(PolicyMgrUserGroupBuilder.java:198) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:51) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 14 more 21 Dec 2018 12:05:09 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 21 Dec 2018 12:05:09 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created 21 Dec 2018 12:05:09 INFO UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldap://prd-freeipa001-prd-clouddb077.XXXXXX.nm1:389, ldapBindDn: cn=Directory Manager, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: cn=users,cn=accounts,dc=XXXXXX,dc=com, userSearchBase: [cn=users,cn=accounts,dc=XXXXXX,dc=com], userSearchScope: 2, userObjectClass: person, userSearchFilter: , extendedUserSearchFilter: (objectclass=person), userNameAttribute: uid, userSearchAttributes: [uid], userGroupNameAttributeSet: [uid], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: false, groupSearchBase: [cn=groups,cn=accounts,dc=XXXXXX,dc=com], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: , extendedGroupSearchFilter: (&(objectclass=group)(|(memberof={0})(memberof={1}))), extendedAllGroupsSearchFilter: (&(objectclass=group)), groupMemberAttributeName: memberof, groupNameAttribute: cn, groupSearchAttributes: [cn, memberof], groupUserMapSyncEnabled: false, groupSearchFirstEnabled: false, userSearchEnabled: true, ldapReferral: follow 21 Dec 2018 12:05:09 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: admin, groupName: admin 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 1, userName: admin, groupList: [admin] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: shesh.kumar, groupName: shesh.kumar 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 2, userName: shesh.kumar, groupList: [shesh.kumar] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: ambari-server, groupName: ambari-server 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 3, userName: ambari-server, groupList: [ambari-server] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: ambari-server-lucy, groupName: ambari-server-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 4, userName: ambari-server-lucy, groupList: [ambari-server-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: hbase-lucy, groupName: hbase-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 5, userName: hbase-lucy, groupList: [hbase-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: hdfs-lucy, groupName: hdfs-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 6, userName: hdfs-lucy, groupList: [hdfs-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: ambari-qa-lucy, groupName: ambari-qa-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 7, userName: ambari-qa-lucy, groupList: [ambari-qa-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: spark-lucy, groupName: spark-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 8, userName: spark-lucy, groupList: [spark-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: yarn-ats-lucy, groupName: yarn-ats-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 9, userName: yarn-ats-lucy, groupList: [yarn-ats-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: zeppelin-lucy, groupName: zeppelin-lucy 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 10, userName: zeppelin-lucy, groupList: [zeppelin-lucy] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: animesh.chandra, groupName: animesh.chandra 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 11, userName: animesh.chandra, groupList: [animesh.chandra] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: saurabh.deshpande, groupName: saurabh.deshpande 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 12, userName: saurabh.deshpande, groupList: [saurabh.deshpande] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_digigold_mysql_ingestion, groupName: svc_digigold_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 13, userName: svc_digigold_mysql_ingestion, groupList: [svc_digigold_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: shoyeb.ibuse, groupName: shoyeb.ibuse 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 14, userName: shoyeb.ibuse, groupList: [shoyeb.ibuse] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_referral_mysql_ingestion, groupName: svc_referral_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 15, userName: svc_referral_mysql_ingestion, groupList: [svc_referral_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_offerengine_mysql_ingestion, groupName: svc_offerengine_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 16, userName: svc_offerengine_mysql_ingestion, groupList: [svc_offerengine_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_symphony_mysql_ingestion, groupName: svc_symphony_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 17, userName: svc_symphony_mysql_ingestion, groupList: [svc_symphony_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_wallet_mysql_ingestion, groupName: svc_wallet_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 18, userName: svc_wallet_mysql_ingestion, groupList: [svc_wallet_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_nexus_mysql_ingestion, groupName: svc_nexus_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 19, userName: svc_nexus_mysql_ingestion, groupList: [svc_nexus_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_users_mysql_ingestion, groupName: svc_users_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 20, userName: svc_users_mysql_ingestion, groupList: [svc_users_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_iris_mysql_ingestion, groupName: svc_iris_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 21, userName: svc_iris_mysql_ingestion, groupList: [svc_iris_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_merchant_mysql_ingestion, groupName: svc_merchant_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 22, userName: svc_merchant_mysql_ingestion, groupList: [svc_merchant_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: svc_payment_mysql_ingestion, groupName: svc_payment_mysql_ingestion 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 23, userName: svc_payment_mysql_ingestion, groupList: [svc_payment_mysql_ingestion] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: sachidanand.sinha, groupName: sachidanand.sinha 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 24, userName: sachidanand.sinha, groupList: [sachidanand.sinha] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: airflow, groupName: airflow 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 25, userName: airflow, groupList: [airflow] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - longGroupName: rangerusersync, groupName: rangerusersync 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Updating user count: 26, userName: rangerusersync, groupList: [rangerusersync] 21 Dec 2018 12:05:09 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() completed with user count: 26 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:09 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: admin, groups: [admin] 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:09 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: ambari-server-lucy, groups: [ambari-server-lucy] 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:09 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:09 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: svc_nexus_mysql_ingestion, groups: [svc_nexus_mysql_ingestion] 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:10 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: animesh.chandra, groups: [animesh.chandra] 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:10 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: yarn-ats-lucy, groups: [yarn-ats-lucy] 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 21 Dec 2018 12:05:10 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: shesh.kumar, groups: [shesh.kumar] 21 Dec 2018 12:05:10 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) at com.google.gson.Gson.fromJson(Gson.java:803) at com.google.gson.Gson.fromJson(Gson.java:768) at com.google.gson.Gson.fromJson(Gson.java:717) at com.google.gson.Gson.fromJson(Gson.java:689) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getMUser(PolicyMgrUserGroupBuilder.java:1231) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$800(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1195) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$5.run(PolicyMgrUserGroupBuilder.java:1191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addMUser(PolicyMgrUserGroupBuilder.java:1191) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addOrUpdateUser(PolicyMgrUserGroupBuilder.java:365) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.stream.JsonReader.beginObject(JsonReader.java:374) at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:165) ... 15 more # hadoop org.apache.hadoop.security.HadoopKerberosName rangerusersync@REALM.COM Name: rangerusersync@REALM.COM to rangerusersync rangerusersync@prd-xxxx120:~$ kinit -kt /etc/security/keytabs/ranger_user_sync.keytab rangerusersync@REALM.COM rangerusersync@prd-xxxx120:~$ klist Ticket cache: FILE:/tmp/krb5cc_1000 Default principal: rangerusersync@REALM.COM Valid starting Expires Service principal 12/21/18 12:10:26 12/22/18 12:10:26 krbtgt/REALM.COM@REALM.COM renew until 12/28/18 12:10:26 Please help me in resolving the issue. Thanks, Shesh Kumar Attachment: usersync-123.txt ... View more

Hi, I want to know if we can enable replication in Hbase such that the Hbase data in Cluster A is replicated to Cluster B. And the Hbase data of Cluster B is replicated to Cluster A. In short, "cross cluster replication". I want this feature for all the existing tables present in Hbase (both Cluster A and Cluster B). It should be active replication. For instance, any column family is added or populated or new table is created in Hbase, it should immediately replicate the same to to a different cluster. Security: None! the 2 clusters are plain, vanilla cluster. Can this be configured in Ambari? If yes, kindly guide me. Your valuable inputs will be highly appreciated. Thanks, Shesh Kumar ... View more

Hi, We have kerberized our cluster and also integrated with LDAP account. When we restart the "Ranger Usersync", below error is observed and the sync is failing. 18 Dec 2018 12:17:23 INFO LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync/stg-agent001-stg-cloud009.XXXXX.nm2@XXXXXX.COM and keytab = /etc/security/keytabs/rangerusersync.service.keytab 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to build Group List : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.xxxxxxx.nm2:6080/service/xusers/groups/ returned a response status of 404 Not Found and 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder.getUsers() completed with user count: 0 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.xxxxxxx.nm2:6080/service/xusers/ugsync/auditinfo/ returned a response status of 404 Not Found Full logs: 18 Dec 2018 12:17:23 INFO LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync/stg-agent001-stg-cloud009.XXXXXX.nm2@XXXXXX.COM and keytab = /etc/security/keytabs/rangerusersync.service.keytab 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to build Group List : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.XXXXXX.nm2:6080/service/xusers/groups/ returned a response status of 404 Not Found at com.sun.jersey.api.client.WebResource.handle(WebResource.java:688) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.getAddedGroupInfo(LdapPolicyMgrUserGroupBuilder.java:252) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.access$000(LdapPolicyMgrUserGroupBuilder.java:68) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$1.run(LdapPolicyMgrUserGroupBuilder.java:206) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$1.run(LdapPolicyMgrUserGroupBuilder.java:202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.addGroupInfo(LdapPolicyMgrUserGroupBuilder.java:202) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.addOrUpdateGroup(LdapPolicyMgrUserGroupBuilder.java:180) at org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder.getGroups(LdapDeltaUserGroupBuilder.java:722) at org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder.updateSink(LdapDeltaUserGroupBuilder.java:343) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add addorUpdate group info 18 Dec 2018 12:17:24 ERROR LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder.getGroups() failed with exception: java.lang.Exception: Failed to add addorUpdate group info 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder.getGroups() group count: 1 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - User search is enabled and hence computing user membership. 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - extendedUserSearchFilter = (&(objectclass=person)(|(uSNChanged>=0)(modifyTimestamp>=19700101120000Z))) 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181208171657Zand currentDeltaSyncTime = 1544289417000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: admin 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181113102826Zand currentDeltaSyncTime = 1542104906000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: hadoopadmin 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181207085230Zand currentDeltaSyncTime = 1544172750000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: ambari-server-ldap 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181207085230Zand currentDeltaSyncTime = 1544172750000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: hdfs-ldap 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181207085230Zand currentDeltaSyncTime = 1544172750000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: ambari-qa-ldap 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181217183001Zand currentDeltaSyncTime = 1545071401000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: shesh.kumar 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181121070529Zand currentDeltaSyncTime = 1542783929000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: csantana 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181127123558Zand currentDeltaSyncTime = 1543278958000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: saurabh.deshpande 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181127124814Zand currentDeltaSyncTime = 1543279694000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: achanta.vamsi 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181207085231Zand currentDeltaSyncTime = 1544172751000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: yarn-ats-ldap 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181128112251Zand currentDeltaSyncTime = 1543404171000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: krishnanvr 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - timeStampVal = 20181218110959Zand currentDeltaSyncTime = 1545131399000 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - Updating user count: 0, userName: rangerusersync 18 Dec 2018 12:17:24 INFO LdapDeltaUserGroupBuilder [UnixUserSyncThread] - LdapDeltaUserGroupBuilder.getUsers() completed with user count: 0 18 Dec 2018 12:17:24 ERROR LdapPolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add User : com.sun.jersey.api.client.UniformInterfaceException: POST http://stg-agent001-stg-cloud009.XXXXXX.nm2:6080/service/xusers/ugsync/auditinfo/ returned a response status of 404 Not Found at com.sun.jersey.api.client.WebResource.handle(WebResource.java:688) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.getUserGroupAuditInfo(LdapPolicyMgrUserGroupBuilder.java:522) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.access$300(LdapPolicyMgrUserGroupBuilder.java:68) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$4.run(LdapPolicyMgrUserGroupBuilder.java:495) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder$4.run(LdapPolicyMgrUserGroupBuilder.java:491) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.addUserGroupAuditInfo(LdapPolicyMgrUserGroupBuilder.java:491) at org.apache.ranger.ldapusersync.process.LdapPolicyMgrUserGroupBuilder.postUserGroupAuditInfo(LdapPolicyMgrUserGroupBuilder.java:474) at org.apache.ranger.ldapusersync.process.LdapDeltaUserGroupBuilder.updateSink(LdapDeltaUserGroupBuilder.java:358) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) 18 Dec 2018 12:17:24 INFO UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink 18 Dec 2018 12:17:24 INFO UserGroupSync [UnixUserSyncThread] - Done initializing user/group source and sink I followed this link (Option 2) to add LDAP's crt file. Also Ranger is not SSL/TLS enabled service in my cluster (not even Ambari server in SSL enabled) Any idea why I'm hitting "404" error ??? Please provide any technical assistance. It will be highly appreciated. HDP Version: HDP-3.0.1.0 Thanks, Shesh Kumar ... View more

Hi, I'm seeing this error in RS logs. Can someone help me resolve the issue? 2018-12-12 18:28:54,407 INFO [B.defaultRpcServer.handler=12,queue=0,port=16020] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x3d630f8a): could not load 1093156401_BP-853897652-10.84.192.246-1489729943941 due to InvalidToken exception. org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/cyclops-edges/865c0549943a300f09f5dfcd63fbaa67/s/1979e37ca9294d24bd16cd580fb663ab 2018-12-12 18:09:57,326 INFO [sync.2] wal.FSHLog: Slow sync cost: 105 ms, current pipeline: [DatanodeInfoWithStorage[10.84.197.254:50010,DS-159866b7-1b97-469f-9a22-4b03b3dbbe56,DISK], DatanodeInfoWithStorage[10.84.192.255:50010,DS-26463076-ed2d-4883-9013-2870ce87f281,DISK], DatanodeInfoWithStorage[10.84.192.76:50010,DS-b1b894fc-d1e2-4ccf-8cc7-7ccd649cd507,DISK]] 2018-12-12 18:09:57,391 WARN [B.defaultRpcServer.handler=21,queue=0,port=16020] hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530 at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140) at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:456) at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:424) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:816) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:695) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181) at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118) at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92) at org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1625) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:441) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:642) at java.lang.Thread.run(Thread.java:745) 2018-12-12 18:09:57,391 WARN [B.defaultRpcServer.handler=21,queue=0,port=16020] hdfs.DFSClient: Connection failure: Failed to connect to /10.84.192.246:50010 for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604 for block BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530:java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530 java.io.IOException: Got error, status message opReadBlock BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-853897652-10.84.192.246-1489729943941:blk_1093248459_19508530, for OP_READ_BLOCK, self=/10.84.197.254:13978, remote=/10.84.192.246:50010, for file /apps/hbase/data/data/default/cyclops-audits-dedup/e44738e4889089bcecf58b66878a7501/l/457245b1ea724f7f80bab245ea6c0604, for pool BP-853897652-10.84.192.246-1489729943941 block 1093248459_19508530 at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140) The issue that we are facing is very slow Hbase compaction. HDP Version: HDP-2.4.3.0-227 HBase: 1.1.2.2.4 Thanks, Shesh ... View more

Hi, I've setup a HDP 3.0.1 cluster which is Kerberized. However, the user sync is not happening with sync source as "Unix", min user id as "500". Below is the error observed in the logs: 04 Dec 2018 14:49:06 INFO UnixAuthenticationService [main] - Starting User Sync Service! 04 Dec 2018 14:49:06 WARN UnixUserGroupBuilder [UnixUserSyncThread] - DEPRECATED: Unix backend is configured to use /etc/passwd and /etc/group files directly instead of standard system mechanisms. 04 Dec 2018 14:49:06 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder 04 Dec 2018 14:49:06 INFO PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync/prd-lucy110.XXXXX.nm1@XXXXXX.COM and keytab = /etc/security/keytabs/rangerusersync.service.keytab 04 Dec 2018 14:49:07 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to build Group List : com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 1 at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176) 04 Dec 2018 14:49:07 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 04 Dec 2018 14:49:07 ERROR UnixUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: jatin, groups: [jatin, dev] 04 Dec 2018 14:49:07 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to add portal user 04 Dec 2018 14:49:07 ERROR UnixUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: Failed to add portal user, for user: suraj.ghosh, groups: [suraj.ghosh, dev] In the same machine I added a user "rangerusersync". Ran the python script "updatepolicymgrpassword.py" and provided same username and password. But still fails! Please see the attached screenshot of Ranger Audit UI (Usersync tab). Is this normal ? Also attached full logs. Please check :- usersync-log.txt Note:- Ambari server is TLS (SSL) enabled. But not Ranger. Can anyone please help me in resolving this issue? It would be highly appreciated. Thanks, Shesh Kumar ... View more

Hi. We are running HDP 3.0.1 which is Kerberized. We are not able to sync the 'linux' users to Ranger. Below is the error observed in usersync log file. 03 Dec 2018 18:03:40INFO UserGroupSync [UnixUserSyncThread] - Begin: update user/group from source==>sink 03 Dec 2018 18:03:40 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to communicate Ranger Admin : com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) at com.sun.jersey.api.client.Client.handle(Client.java:652) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.tryUploadEntityWithCred(PolicyMgrUserGroupBuilder.java:895) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.cookieBasedUploadEntity(PolicyMgrUserGroupBuilder.java:1248) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getUserGroupAuditInfo(PolicyMgrUserGroupBuilder.java:1688) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.access$1000(PolicyMgrUserGroupBuilder.java:79) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$8.run(PolicyMgrUserGroupBuilder.java:1660) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder$8.run(PolicyMgrUserGroupBuilder.java:1656) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.addUserGroupAuditInfo(PolicyMgrUserGroupBuilder.java:1656) at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.postUserGroupAuditInfo(PolicyMgrUserGroupBuilder.java:1615) at org.apache.ranger.unixusersync.process.UnixUserGroupBuilder.updateSink(UnixUserGroupBuilder.java:186) at org.apache.ranger.usergroupsync.UserGroupSync.syncUserGroup(UserGroupSync.java:107) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:85) at java.lang.Thread.run(Thread.java:745) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Menlo; color: #000000} span.s1 {font-variant-ligatures: no-common-ligatures} at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler$1$1.getOutputStream(URLConnectionClientHandler.java:238) at com.sun.jersey.api.client.CommittingOutputStream.commitStream(CommittingOutputStream.java:117) at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:89) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at java.io.BufferedWriter.flush(BufferedWriter.java:254) at com.sun.jersey.core.util.ReaderWriter.writeToAsString(ReaderWriter.java:191) at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:128) at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(StringProvider.java:88) p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Menlo; color: #000000} span.s1 {font-variant-ligatures: no-common-ligatures} at com.sun.jersey.core.impl.provider.entity.StringProvider.writeTo(StringProvider.java:58) at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:217) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ... 18 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ... 46 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ... 52 more 03 Dec 2018 18:03:40INFO UserGroupSync [UnixUserSyncThread] - End: update user/group from source==>sink 03 Dec 2018 18:04:40INFO UserGroupSync [UnixUserSyncThread] - Begin: update user/group from source==>sink 03 Dec 2018 18:04:40 ERROR PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Failed to communicate Ranger Admin : com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) at com.sun.jersey.api.client.Client.handle(Client.java:652) I've also followed below HW Support KB but still no luck. https://community.hortonworks.com/content/supportkb/49025/ranger-usersync-fails-with-unable-to-find-valid-ce.html Any technical help will be highly appreciated. Thanks, ... View more

Hi, We are running a self-signed certificate Ambari cluster (with HTTPS) and we also enabled the cluster with FreeIPA+Kerberos. Ambari URL: https://xxxx.xxxx.nm1:8443 (its not .com) HDP: 3.0.1 (Latest) After successfully integrating FreeIPA+Kerberos with Ambari cluster, we are unable to access few important GUIs such as Namenode UI, Resource Manager UI and Oozie UI. The error we are getting below is this: HTTP ERROR 401 Problem accessing /index.html. Reason:Authentication required I've tried all possible scenarios to debug this error like running the following command in my MAC terminal but its of no use. defaults write com.google.Chrome AuthServerWhitelist "*.REALM_NAME.COM" defaults write com.google.Chrome AuthNegotiateDelegateWhitelist "*.REALM_NAME.COM" I ran the same above command in Google Chrome console (option+command+j in MAC) and got this error: Uncaught SyntaxError: Unexpected identifier The following Keytabs are present in /etc/security/keytabs : kerberos.service_check.113018.keytab ambari.server.keytab spnego.service.keytab yarn-ats.hbase-regionserver.service.keytab yarn-ats.hbase-master.service.keytab smokeuser.headless.keytab oozie.service.keytab nn.service.keytab hive.service.keytab ams-monitor.keytab nm.service.keytab hive.llap.task.keytab hbase.headless.keytab spark.service.keytab spark.headless.keytab rm.service.keytab hdfs.headless.keytab ambari-infra-solr.service.keytab zk.service.keytab yarn.service.keytab yarn-ats.hbase-client.headless.keytab dn.service.keytab There is a valid ticket HDFS user as well but still unable to access the UI: hdfs@xxxxxxx:/etc/security/keytabs$ klist Ticket cache: FILE:/tmp/krb5cc_1213 Default principal: nn/xxxxxx.xxxxxx.nm1@REALM.COM Valid starting Expires Service principal 11/30/18 16:13:31 12/01/18 16:13:31 krbtgt/REALM.COM@REALM.COM renew until 12/07/18 16:13:31 I also tried using "spnego.service.keytab" but still no use: root@xxxxxxx102:/etc/security/keytabs# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: HTTP/xxxxxx102.xxxxx.nm1@REALM.COM Valid starting Expires Service principal 11/30/18 17:23:38 12/01/18 17:23:38 krbtgt/REALM.COM@REALM.COM renew until 12/07/18 17:23:38 Kindly provide your technical suggestions. It would be very helpful and highly appreciated. Should I disable the Kerberos HTTP authentication ? If yes, please guide me the same for NN, RM and Oozie URLs Thanks, Shesh Kumar ... View more

We are trying out FreeIPA and integrated the same to our Ambari Hadoop cluster (HDP v3.0.1). We are able to add users and provide them access to Hadoop with help of Kinit command. However, when deleting the users in FreeIPA GUI, the principal gets deleted. The deleted user's principal will not be there in the "kadmin" prompt when I do listprincs. But the user will still be having a valid ticket when he does "klist" and can access Hadoop even though the principal is removed. We cannot do "kdestory" manually. Typically, when users are removed in FreeIPA, the same users should not be able to access Hadoop as well. Can't FreeIPA handle kdestroy? Please provide your suggestions. Thanks, Shesh ... View more

Hi, I'm trying to configure SSL for HDFS Ranger Plugin (Self-Signed Certificate) by following this document: Configure the Ranger HDFS Plugin for SSL (HDP 3.0.1 version) The Step 6 in the link says -- "Select Advanced ranger-hdfs-policymgr-ssl and set the following properties" However, in my Ambari UI (HDP 3.0.1), I cannot see that option at all. Please see the screenshot (Attached after my signature). I've enabled the HDFS plugin as well. I've no idea how to proceed further. Any suggestions would be highly appreciated. Thanks, Shesh Kumar ... View more