Member since
10-20-2015
92
Posts
79
Kudos Received
9
Solutions
03-11-2020
06:10 AM
Thx, D! It’s works at Ranger v2.0 from new CDP Data Center, BareMetal version! Regards, Caseiro.
... View more
09-20-2019
04:21 AM
Hello @dvillarreal, Thank you for the script files, makes life easy! Just a small edit: The proc.psql file was missing one SQL statement @line 96: delete from x_user_module_perm where user_id = x_portal_user_id; We were facing the below issue while trying the commands from psql file manually: ERROR: update or delete on table "x_portal_user" violates foreign key constraint "x_user_module_perm_fk_userid" on table "x_user_module_perm"
DETAIL: Key (id)=(19) is still referenced from table "x_user_module_perm" Because of the above blocker, the script was not deleting the users from the DB. Once we added the missing SQL statement and ran the deleteRangerUser.sh we were able to use it with successful results.
... View more
05-17-2019
06:50 PM
1 Kudo
Customers have asked me about wanting to review ranger audit archive logs stored on HDFS as the UI only shows the Last 90 days of data using Solr infra. I decided to approach the problem using Zeppelin/Spark for a fun example. 1. Prerequisites - Zeppelin and Spark2 installed on your system. As well as ranger with ranger audit logs being stored in HDFS. Create a policy in ranger for HDFS to allow your zeppelin user to read and execute recursively for /ranger/audit directory. 2. Create your notebook in Zeppelin and create some code like the following example: %spark2.spark
// --Specify service and date if you wish
//val path = "/ranger/audit/hdfs/20190513/*.log"
// --Be brave and map the whole enchilada
val path = "/ranger/audit/*/*/*.log"
// --read in the json and drop any malformed json
val rauditDF = spark.read.option("mode", "DROPMALFORMED").json(path)
// --print the schema to review and show me top 20 lines.
rauditDF.printSchema()
rauditDF.show(20,false)
// --Do some spark sql on the data and look for denials
println("sparksql--------------------")
rauditDF.createOrReplaceTempView(viewName="audit")
var readAccessDF = spark.sql("SELECT reqUser, repo, access, action, evtTime, policy, resource, reason, enforcer, result FROM audit where result='0'").withColumn("new_result", when(col("result") === "1","Allowed").otherwise("Denied"))
readAccessDF.show(20,false) 3. Output should look something like path: String = /ranger/audit/*/*/*.log
rauditDF: org.apache.spark.sql.DataFrame = [access: string, action: string ... 23 more fields]
root
|-- access: string (nullable = true)
|-- action: string (nullable = true)
|-- additional_info: string (nullable = true)
|-- agentHost: string (nullable = true)
|-- cliIP: string (nullable = true)
|-- cliType: string (nullable = true)
|-- cluster_name: string (nullable = true)
|-- enforcer: string (nullable = true)
|-- event_count: long (nullable = true)
|-- event_dur_ms: long (nullable = true)
|-- evtTime: string (nullable = true)
|-- id: string (nullable = true)
|-- logType: string (nullable = true)
|-- policy: long (nullable = true)
|-- reason: string (nullable = true)
|-- repo: string (nullable = true)
|-- repoType: long (nullable = true)
|-- reqData: string (nullable = true)
|-- reqUser: string (nullable = true)
|-- resType: string (nullable = true)
|-- resource: string (nullable = true)
|-- result: long (nullable = true)
|-- seq_num: long (nullable = true)
|-- sess: string (nullable = true)
|-- tags: array (nullable = true)
| |-- element: string (containsNull = true)
sql
readAccessDF: org.apache.spark.sql.DataFrame = [reqUser: string, repo: string ... 9 more fields]
+--------+------------+------------+-------+-----------------------+------+-------------------------------------------------------------------------------------+----------------------------------+----------+------+----------+
|reqUser |repo |access |action |evtTime |policy|resource |reason |enforcer |result|new_result|
+--------+------------+------------+-------+-----------------------+------+-------------------------------------------------------------------------------------+----------------------------------+----------+------+----------+
|dav |c3205_hadoop|READ_EXECUTE|execute|2019-05-13 22:07:23.971|-1 |/ranger/audit/hdfs |/ranger/audit/hdfs |hadoop-acl|0 |Denied |
|zeppelin|c3205_hadoop|READ_EXECUTE|execute|2019-05-13 22:10:47.288|-1 |/ranger/audit/hdfs |/ranger/audit/hdfs |hadoop-acl|0 |Denied |
|dav |c3205_hadoop|EXECUTE |execute|2019-05-13 23:57:49.410|-1 |/ranger/audit/hiveServer2/20190513/hiveServer2_ranger_audit_c3205-node3.hwx.local.log|/ranger/audit/hiveServer2/20190513|hadoop-acl|0 |Denied |
|zeppelin|c3205_hive |USE |_any |2019-05-13 23:42:50.643|-1 |null |null |ranger-acl|0 |Denied |
|zeppelin|c3205_hive |USE |_any |2019-05-13 23:43:08.732|-1 |default |null |ranger-acl|0 |Denied |
|dav |c3205_hive |USE |_any |2019-05-13 23:48:37.603|-1 |null |null |ranger-acl|0 |Denied |
+--------+------------+------------+-------+-----------------------+------+-------------------------------------------------------------------------------------+----------------------------------+----------+------+----------+ 4. You can proceed to run sql as well on the audit view information using sql if you so desire. 5. You may need to fine tune your spark interpreter in zeppelin to meet your needs like SPARK_DRIVER_MEMORY, spark.executor.cores, spark.executor.instances, & spark.executor.memory. It helped to see what was happening by tailing the zeppelin log for spark. tailf zeppelin-interpreter-spark2-spark-zeppelin-cluster1.hwx.log
... View more
Labels:
10-15-2018
03:23 PM
My pleasure! @Jasper
... View more
11-16-2017
12:39 AM
Yes, for each configured OU in group search base, ranger usersync computes the nested groups. Group hierarchy level is applied for each OU independently. Few points to note: 1. If the directory server contains more levels of nested groups than the ones configured in the usersync group hierarchy levels, then usersync limits the nested group computation based on the usersync configuration 2. If the directory server contains less levels of nested groups than the ones configured in the usersync group hierarchy levels, then usersync limits the nested group computation based on the directory server nested group levels 3. Nested groups are computed only for the groups that are part of the group search base. For example, group search base is configured as "ou=groups,dc=test,dc=com;ou=groups2,dc=test,dc=test,dc=com" and if a group (grp1) that is part of the one of these configured OU has a member group (grp2) that is not part of any of the configured OUs, then grp2 is ignored from group computation. 4. Nested group computation is supported with Incremental sync as well as Full sync.
... View more
03-06-2017
09:11 PM
@badr bakkou This would probably be best answered if you submitted as a new question. Provide the gateway.log & gateway-audit.log outputs, topology, and lastly the configuration string you are using with its associated output. Best regards, David
... View more
06-19-2018
08:14 PM
How to assign privileges to a group when it is created?
... View more
02-09-2017
08:16 PM
13 Kudos
AD admins may be busy and you may happen to know the ambari admin principal for enabling Kerberos. How would you go about adding a principal for AD with this information and add it to your kerberos keytab? Below is one way to do it. Thanks to @Robert Levas for collaborating with me on this.
1. Create LDIF file ad_user.ldif. (Make sure there are no spaces at the ends of each of these lines)
dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
distinguishedName: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM
cn: HTTP/loadbalancerhost
userAccountControl: 514
accountExpires: 0
userPrincipalName: HTTP/loadbalancerhost@HOST.COM
servicePrincipalName: HTTP/loadbalancerhost
dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=host,DC=com
changetype: modify
replace: unicodePwd
unicodePwd::IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA==
dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM
changetype: modify
replace: userAccountControl
userAccountControl: 66048
Do not have spaces at the ends of the above lines or you will get an error like the following:
ldap_add: No such attribute (16)
additional info: 00000057: LdapErr: DSID-0C090D8A, comment: Error in attribute conversion operation, data 0, v2580
2. Create unicode Password for the above principal with the password hadoopRocks123!. Replace unicodePWD field in step 1:
[root@host1 ~]# echo -n '"hadoopRocks123!"' | iconv -f UTF8 -t UTF16LE | base64 -w 0
IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA==
3. Add the account to AD:
[root@host1 ~]# ldapadd -x -H ldaps://sme-2012-ad.support.com:636 -D "test1@host.com" -W -f add_user.ldif
Enter LDAP Password:
adding new entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM"
modifying entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=com"
modifying entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM"
4. Test the account with kinit:
[root@host1 ~]# kinit HTTP/loadbalancerhost@HOST.COM
Password for HTTP/loadbalancerhost@HOST.COM:
[root@host1 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/loadbalancerhost@HOST.COM
Valid starting Expires Service principal
02/09/17 19:02:33 02/10/17 19:02:33 krbtgt/HOST.COM@HOST.COM
renew until 02/09/17 19:02:33
5. Take it one step further if you need to add the principal to a keytab file
[root@host1 ~]# ktutil
ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e aes128-cts-hmac-sha1-96
Password for HTTP/loadbalancerhost@HOST.COM:
ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e aes256-cts-hmac-sha1-96
Password for HTTP/loadbalancerhost@HOST.COM:
ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e arcfour-hmac-md5-exp
Password for HTTP/loadbalancerhost@HOST.COM:
ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e des3-cbc-sha1
Password for HTTP/loadbalancerhost@HOST.COM:
ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e des-cbc-md5
Password for HTTP/loadbalancerhost@HOST.COM:
ktutil: write_kt spenego.service.keytab
ktutil: exit
[root@host1 ~]# klist -ket spenego.service.keytab
Keytab name: FILE:lb.service.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (aes128-cts-hmac-sha1-96)
1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (aes256-cts-hmac-sha1-96)
1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (arcfour-hmac-exp)
1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (des3-cbc-sha1)
1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (des-cbc-md5)
... View more
Labels:
12-27-2016
07:51 PM
3 Kudos
PROBLEM: Some users may be associated to many groups causing a very long list of groups to be passed through the Rest APIs headers in Ranger and KMS. ERROR: error log from /var/log/ranger/kms/kms.log 2016-12-01 14:04:12,048 INFO Http11Processor - Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Request header is too large
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396)
at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2016-12-01 14:04:12,074 INFO Http11Processor - Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Request header is too large
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396)
at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
ROOT CAUSE: Rest API calls being passed with large headersizes in this case users with large amount of groups that exceed the webservers maxHttpHeaderSize. SOLUTION:
In Ambari go to Ranger Admin->config->Advanced Tab->Custom ranger-admin-site->Add Property. Put ranger.service.http.connector.property.maxHttpHeaderSize in Key field and provide the required value for maxHttpHeaderSize attribute in Value field.
Save the changes and then go to Ranger KMS->config->Advanced Tab->Custom ranger-kms-site->Add Property. Put ranger.service.http.connector.property.maxHttpHeaderSize in Key field and provide the required value for maxHttpHeaderSize attribute in Value field.
Save the changes and restart all Ranger and Ranger KMS services.
... View more
Labels:
12-25-2016
08:09 PM
SYMPTOM: Knox logs are filling up disk space ROOT CAUSE:
Kerberos debug is turned on by default causing the gateway.out file to grow rapidly. RESOLUTION:
To turn off kerberos debug logging.
1. Go to Ambari. KNOX -> Configs-> Advanced gateway-site
2. Change parameter sun.security.krb5.debug from true to false.
3. Restart Knox.
... View more
Labels: