1. HA provider for webhdfs is needed in your topology. <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param> </provider> 2. The namenode service url value should contain your name service ID. (This can be found in your hdfs-default.xml under parameter dfs.internal.nameservices) <service> <role>NAMENODE</role> <url>hdfs://chupa</url> </service> 3. Make sure webhdfs url for each namenode is added in your WEBHDFS service area. <service> <role>WEBHDFS</role> <url>http://chupa1.openstacklocal:50070/webhdfs</url> <url>http://chupa2.openstacklocal:50070/webhdfs</url> </service> 4. Here is a working topology using the knox default demo LDAP. <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapRealm.userDnTemplate</name> <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> </param> <param> <name>main.ldapRealm.contextFactory.url</name> <value>ldap://chupa1.openstacklocal:33389</value> </param> <param> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> <value>simple</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param> </provider> </gateway> <service> <role>NAMENODE</role> <url>hdfs://chupa</url> </service> <service> <role>JOBTRACKER</role> <url>rpc://chupa3.openstacklocal:8050</url> </service> <service> <role>WEBHDFS</role> <url>http://chupa1.openstacklocal:50070/webhdfs</url> <url>http://chupa2.openstacklocal:50070/webhdfs</url> </service> <service> <role>WEBHCAT</role> <url>http://chupa2.openstacklocal:50111/templeton</url> </service> <service> <role>OOZIE</role> <url>http://chupa2.openstacklocal:11000/oozie</url> </service> <service> <role>WEBHBASE</role> <url>http://chupa1.openstacklocal:8080</url> </service> <service> <role>HIVE</role> <url>http://chupa2.openstacklocal:10001/cliservice</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://chupa3.openstacklocal:8088/ws</url> </service> <service> <role>RANGERUI</role> <url>http://chupa3.openstacklocal:6080</url> </service> </topology> 5. If you would like to test that it is working you can issue the following command to manually failover the cluster and test. hdfs haadmin -failover nn1 nn2 6. Test with Knox connection string to webhdfs. curl -vik -u admin:admin-password 'https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS' ... View more

1. Create User [root@chupa1 ~]# curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Users/user_name":"dav","Users/password":"pass","Users/active":"true","Users/admin":"false"}' http://localhost:8080/api/v1/users * About to connect() to localhost port 8080 (#0) * Trying ::1... connected * Connected to localhost (::1) port 8080 (#0) * Server auth using Basic with user 'admin' > POST /api/v1/users HTTP/1.1 > Authorization: Basic YWRtaW46YWRtaW4= > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > X-Requested-By: ambari > Content-Length: 93 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 201 Created HTTP/1.1 201 Created 2. Create Group [root@chupa1 ~]# curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Groups/group_name":"davgroup"}' http://localhost:8080/api/v1/groups * About to connect() to localhost port 8080 (#0) * Trying ::1... connected * Connected to localhost (::1) port 8080 (#0) * Server auth using Basic with user 'admin' > POST /api/v1/groups HTTP/1.1 > Authorization: Basic YWRtaW46YWRtaW4= > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > X-Requested-By: ambari > Content-Length: 32 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 201 Created HTTP/1.1 201 Created 3. Map user to Group [root@chupa1 ~]# curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"MemberInfo/user_name":"dav", "MemberInfo/group_name":"davgroup"}' http://localhost:8080/api/v1/groups/davgroup/members * About to connect() to localhost port 8080 (#0) * Trying ::1... connected * Connected to localhost (::1) port 8080 (#0) * Server auth using Basic with user 'admin' > POST /api/v1/groups/davgroup/members HTTP/1.1 > Authorization: Basic YWRtaW46YWRtaW4= > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > X-Requested-By: ambari > Content-Length: 66 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 201 Created HTTP/1.1 201 Created ... View more

AD admins may be busy and you may happen to know the ambari admin principal for enabling Kerberos. How would you go about adding a principal for AD with this information and add it to your kerberos keytab? Below is one way to do it. Thanks to @Robert Levas for collaborating with me on this. 1. Create LDIF file ad_user.ldif. (Make sure there are no spaces at the ends of each of these lines) dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM changetype: add objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user distinguishedName: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM cn: HTTP/loadbalancerhost userAccountControl: 514 accountExpires: 0 userPrincipalName: HTTP/loadbalancerhost@HOST.COM servicePrincipalName: HTTP/loadbalancerhost dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=host,DC=com changetype: modify replace: unicodePwd unicodePwd::IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA== dn: CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM changetype: modify replace: userAccountControl userAccountControl: 66048 Do not have spaces at the ends of the above lines or you will get an error like the following: ldap_add: No such attribute (16) additional info: 00000057: LdapErr: DSID-0C090D8A, comment: Error in attribute conversion operation, data 0, v2580 2. Create unicode Password for the above principal with the password hadoopRocks123!. Replace unicodePWD field in step 1: [root@host1 ~]# echo -n '"hadoopRocks123!"' | iconv -f UTF8 -t UTF16LE | base64 -w 0 IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA== 3. Add the account to AD: [root@host1 ~]# ldapadd -x -H ldaps://sme-2012-ad.support.com:636 -D "test1@host.com" -W -f add_user.ldif Enter LDAP Password: adding new entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM" modifying entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=com" modifying entry "CN=HTTP/loadbalancerhost,OU=dav,OU=hortonworks,DC=HOST,DC=COM" 4. Test the account with kinit: [root@host1 ~]# kinit HTTP/loadbalancerhost@HOST.COM Password for HTTP/loadbalancerhost@HOST.COM: [root@host1 ~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: HTTP/loadbalancerhost@HOST.COM Valid starting Expires Service principal 02/09/17 19:02:33 02/10/17 19:02:33 krbtgt/HOST.COM@HOST.COM renew until 02/09/17 19:02:33 5. Take it one step further if you need to add the principal to a keytab file [root@host1 ~]# ktutil ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e aes128-cts-hmac-sha1-96 Password for HTTP/loadbalancerhost@HOST.COM: ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e aes256-cts-hmac-sha1-96 Password for HTTP/loadbalancerhost@HOST.COM: ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e arcfour-hmac-md5-exp Password for HTTP/loadbalancerhost@HOST.COM: ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e des3-cbc-sha1 Password for HTTP/loadbalancerhost@HOST.COM: ktutil: add_entry -password -p HTTP/loadbalancerhost@HOST.COM -k 1 -e des-cbc-md5 Password for HTTP/loadbalancerhost@HOST.COM: ktutil: write_kt spenego.service.keytab ktutil: exit [root@host1 ~]# klist -ket spenego.service.keytab Keytab name: FILE:lb.service.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (aes128-cts-hmac-sha1-96) 1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (aes256-cts-hmac-sha1-96) 1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (arcfour-hmac-exp) 1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (des3-cbc-sha1) 1 01/18/17 03:12:38 HTTP/loadbalancerhost@HOST.COM (des-cbc-md5)   ... View more

PROBLEM: Some users may be associated to many groups causing a very long list of groups to be passed through the Rest APIs headers in Ranger and KMS. ERROR: error log from /var/log/ranger/kms/kms.log 2016-12-01 14:04:12,048 INFO Http11Processor - Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396) at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) 2016-12-01 14:04:12,074 INFO Http11Processor - Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Request header is too large at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:515) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) at org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:396) at org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:271) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1007) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) ROOT CAUSE: Rest API calls being passed with large headersizes in this case users with large amount of groups that exceed the webservers maxHttpHeaderSize. SOLUTION: In Ambari go to Ranger Admin->config->Advanced Tab->Custom ranger-admin-site->Add Property. Put ranger.service.http.connector.property.maxHttpHeaderSize in Key field and provide the required value for maxHttpHeaderSize attribute in Value field. Save the changes and then go to Ranger KMS->config->Advanced Tab->Custom ranger-kms-site->Add Property. Put ranger.service.http.connector.property.maxHttpHeaderSize in Key field and provide the required value for maxHttpHeaderSize attribute in Value field. Save the changes and restart all Ranger and Ranger KMS services. ... View more

SYMPTOM: Knox logs are filling up disk space ROOT CAUSE: Kerberos debug is turned on by default causing the gateway.out file to grow rapidly. RESOLUTION: To turn off kerberos debug logging. 1. Go to Ambari. KNOX -> Configs-> Advanced gateway-site 2. Change parameter sun.security.krb5.debug from true to false. 3. Restart Knox. ... View more

SYMPTOM: Hbase is giving an error KeyValue size too large when inserting large data sizes. ERROR: 0java.lang.IllegalArgumentException: KeyValue size too large at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1521) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:147) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:134) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:105) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1050) at org.apache.hadoop.hbase.rest.RowResource.update(RowResource.java:229) at org.apache.hadoop.hbase.rest.RowResource.put(RowResource.java:318) at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) at ROOT CAUSE: hbase.client.keyvalue.maxsize is set too low RESOLUTION: Set hbase.client.keyvalue.maxsize=0 This will allow the client key value to be allowed. Just be careful with this as too large of a keyvalue >1-2gb could have performance implications. ... View more

Below are some examples of how you would achieve this: Case1: Restrict to users in a single group – In this example, only the users who are members of “scientist” group are only allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccountName={0})(memberof=cn=scientist,ou=groups,dc=hwqe,dc=hortonworks,dc=com)) Case2: Restrict to users in multiple groups – In this example, only the users who are members of either “scientist” group OR “analyst” group are allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccountName={0})(|((memberof=cn=scientist,ou=groups,dc=hwqe,dc=hortonworks,dc=com)(memberof=cn=analyst,ou=groups,dc=hwqe,dc=hortonworks,dc=com)))) Case3: Restrict to given list of users – In this example, only the users whose cn (or common name) starts with “sam r” are allowed to login to ranger admin. User search filter parameter would look something like this: (&(sAMAccountName={0})(cn=sam r*)) ... View more

Problem: Time to live for collection data was never set on Solr Cloud server and caused the disk and nodes to fill up with too many documents due to ranger audit. Solution: 1. Delete the collection through Solr api because the ranger archive was stored in HDFS anyways. (The following cleared up disk space) http://<solr_host>:<solr_port>/solr/admin/collections?action=DELETE&name=collection 2. Set the configuration in solr to have a time to live. a. Download each of the following configs from zookeeper: schema.xml, solrconfig.xml, and managed-schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/solrconfig.xml >/tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/schema.xml >/tmp/schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/managed-schema >/tmp/managed-schema b. Added the following to solrconfig <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor class="solr.DefaultValueUpdateProcessorFactory"> <str name="fieldName">_ttl_</str> <str name="value">+90DAYS</str> </processor> <processor class="solr.processor.DocExpirationUpdateProcessorFactory"> <int name="autoDeletePeriodSeconds">86400</int> <str name="ttlFieldName">_ttl_</str> <str name="expirationFieldName">_expire_at_</str> </processor> <processor class="solr.FirstFieldValueUpdateProcessorFactory"> <str name="fieldName">_expire_at_</str> </processor> c. Added the following to schema.xml and managed-schema.xm <field name="_expire_at_" type="tdate" multiValued="false" stored="true" docValues="true"/> <field name="_ttl_" type="string" multiValued="false" indexed="true" stored="true"/> d. Uploaded each edited file to zookeeper. /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/solrconfig.xml /tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/schema.xml /tmp/schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/managed-schema /tmp/managed-schema 3.Make sure on each node the ranger_audit replicas were removed from the solr directories in the local filesystem. 4. Lastly we issue the create command http://<host>:8983/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2 MORE INFO: Check out more Solr General info and Ambari-infra TTL info here: https://community.hortonworks.com/articles/63853/solr-ttl-auto-purging-solr-documents-ranger-audits.html ... View more