SYMPTOM: Hbase is giving an error KeyValue size too large when inserting large data sizes. ERROR: 0java.lang.IllegalArgumentException: KeyValue size too large at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1521) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:147) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doMutate(BufferedMutatorImpl.java:134) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:105) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1050) at org.apache.hadoop.hbase.rest.RowResource.update(RowResource.java:229) at org.apache.hadoop.hbase.rest.RowResource.put(RowResource.java:318) at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134) at ROOT CAUSE: hbase.client.keyvalue.maxsize is set too low RESOLUTION: Set hbase.client.keyvalue.maxsize=0 This will allow the client key value to be allowed. Just be careful with this as too large of a keyvalue >1-2gb could have performance implications. ... View more

Problem: Time to live for collection data was never set on Solr Cloud server and caused the disk and nodes to fill up with too many documents due to ranger audit. Solution: 1. Delete the collection through Solr api because the ranger archive was stored in HDFS anyways. (The following cleared up disk space) http://<solr_host>:<solr_port>/solr/admin/collections?action=DELETE&name=collection 2. Set the configuration in solr to have a time to live. a. Download each of the following configs from zookeeper: schema.xml, solrconfig.xml, and managed-schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/solrconfig.xml >/tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/schema.xml >/tmp/schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd get /ranger_audits/configs/ranger_audits/managed-schema >/tmp/managed-schema b. Added the following to solrconfig <updateRequestProcessorChain name="add-unknown-fields-to-the-schema"> <processor class="solr.DefaultValueUpdateProcessorFactory"> <str name="fieldName">_ttl_</str> <str name="value">+90DAYS</str> </processor> <processor class="solr.processor.DocExpirationUpdateProcessorFactory"> <int name="autoDeletePeriodSeconds">86400</int> <str name="ttlFieldName">_ttl_</str> <str name="expirationFieldName">_expire_at_</str> </processor> <processor class="solr.FirstFieldValueUpdateProcessorFactory"> <str name="fieldName">_expire_at_</str> </processor> c. Added the following to schema.xml and managed-schema.xm <field name="_expire_at_" type="tdate" multiValued="false" stored="true" docValues="true"/> <field name="_ttl_" type="string" multiValued="false" indexed="true" stored="true"/> d. Uploaded each edited file to zookeeper. /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/solrconfig.xml /tmp/solrconfig.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/schema.xml /tmp/schema.xml /opt/hostname-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost <zookeeper host>:<zookeeper port> -cmd putfile ranger_audits/configs/ranger_audits/managed-schema /tmp/managed-schema 3.Make sure on each node the ranger_audit replicas were removed from the solr directories in the local filesystem. 4. Lastly we issue the create command http://<host>:8983/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2 MORE INFO: Check out more Solr General info and Ambari-infra TTL info here: https://community.hortonworks.com/articles/63853/solr-ttl-auto-purging-solr-documents-ranger-audits.html ... View more

SYMPTOM: Specifying a valid group in Ranger Knox policy results in a 403 authorization error. ERROR: The result is a 403 forbidden error. ROOT CAUSE: Most likely the cause of this issue is that the topology is not setup in Knox for ldap groups to be passed to Ranger from the Knox plugin. RESOLUTION: Make sure the following values are present and correct in the topology: <!-- changes needed for group sync--> <param> <name>main.ldapRealm.authorizationEnabled</name> <value>true</value> </param> <param> <name>main.ldapRealm.groupSearchBase</name> <value>OU=MyUsers,DC=AD-HDP,DC=COM</value> </param> <param> <name>main.ldapRealm.groupObjectClass</name> <value>group</value> </param> <param> <name>main.ldapRealm.groupIdAttribute</name> <value>cn</value> </param> ALTERNATIVE SOLUTION: Instead of getting the above LDAP group settings working open up Knox Authorization to everyone by using the 'Public' group value on the Knox policy and then do authorization at the other service level policies like HDFS, HIVE, HBASE, etc. DEBUG TECHNIQUES This knox log setting should show you what is getting passed to RANGER from the KNOX Plugin. Modify the gateway-log4j.properties like below, restart Knox and review the ranger Knox plugin log in the file ranger.knoxagent.log #Ranger Knox Plugin debug ranger.knoxagent.logger=DEBUG,console,KNOXAGENT ranger.knoxagent.log.file=ranger.knoxagent.log log4j.logger.org.apache.ranger=${ranger.knoxagent.logger} log4j.additivity.org.apache.ranger=false log4j.appender.KNOXAGENT =org.apache.log4j.DailyRollingFileAppender log4j.appender.KNOXAGENT.File=${app.log.dir}/${ranger.knoxagent.log.file} log4j.appender.KNOXAGENT.layout=org.apache.log4j.PatternLayout log4j.appender.KNOXAGENT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n %L log4j.appender.KNOXAGENT.DatePattern=.yyyy-MM-dd ... View more

SYMPTOM: KMS gets 500 error when decrypting files when being accessed from another one way trust realm. ERRORS: Command line error: [root@support ~]$ hdfs dfs -cat /zone_encr3/abc1.txt cat: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 500, message: Internal Server Error KMS Stack Trace error: 2016-05-04 15:44:21,677 ERROR [webservices-driver] - Servlet.service() for servlet [webservices-driver] in context with path [/kms] threw exception org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user06@HDP.COM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519) at org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) ROOT CAUSE: There is a one way trust from another realm and auth to local rules are not given to the KMS configuration. RESOLUTION: hadoop.kms.authentication.kerberos.name.rules property needs to have the auth to local rules. By default this property is set to DEFAULT. If you replace this value with the value from auth_to_local (core-site.xml) and restart ranger KMS service then we will see a user from another realm is able to decrypt the file successfully. Copy the rules from core-site.xml (auth_to_local property) to "hadoop.kms.authentication.kerberos.name.rules" in KMS and restart KMS service. Note:- When you paste the rules from auth_to_local to "hadoop.kms.authentication.kerberos.name.rules", the rules are pasted with space separated values instead of newline. This is fine. ... View more

1. Make sure webhbase is running: [root@ambari-server]# curl http://localhost:60080/version rest 0.0.3 [JVM: Oracle Corporation 1.8.0_60-25.60-b23] [OS: Linux 2.6.32-504.el6.x86_64 amd64] [Server: jetty/6.1.26.hwx] [Jersey: 1.9] 2. If the webhdfs REST daemon is not running then you will need to start it. [root@ambari-server]# /usr/hdp/current/hbase-client/bin/hbase-daemon.sh start rest -p 60080 starting rest, logging to /var/log/hbase/hbase-root-rest-ambari-server.support.com.out [root@ambari-server security]# ps -ef | grep 60080 root 19147 1 0 20:17 pts/1 00:00:00 bash /usr/hdp/current/hbase-client/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-client/bin/../conf foreground_start rest -p 60080 root 19161 19147 28 20:17 pts/1 00:00:03 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_rest -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.3.4.0-3485 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_client_jaas.conf -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201605312017 -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-root-rest-ambari-server.support.com.log -Dhbase.home.dir=/usr/hdp/current/hbase-client/bin/.. -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.rest.RESTServer -p 60080 start 3. Test the webhbase call through knox. (https://cwiki.apache.org/confluence/display/KNOX/Examples+HBase) [root@ambari-slave2 topologies]# curl -ik -u admin:admin-password -H "Accept: text/xml" -X GET 'https://localhost:8443/gateway/default/hbase/version' HTTP/1.1 200 OK Set-Cookie: JSESSIONID=vs8e1emsk00h1va2vk5aeq2b9;Path=/gateway/default;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Content-Type: text/xml Content-Length: 192 Server: Jetty(8.1.14.v20131031) <?xml version="1.0" standalone="yes"?><Version JVM="Oracle Corporation 1.8.0_60-25.60-b23" REST="0.0.3" OS="Linux 2.6.32-504.el6.x86_64 amd64" Server="jetty/6.1.26.hwx" Jersey="1.9"></Version> ... View more

1. Username and password input fields in ranger should be the ldap user and password authenticating through knox. 2. Configure the admin topology to meet your configuration needs by default it points to the demo ldap server on knox server. 3. To test manually, login to the URL directly using ldap credentials from the ranger server. https://<knoxhost>:8443/gateway/admin/api/v1/topologies 4. Make sure the ranger server trusts the knox certificate being presented; if it doesn't; you will need to add the knox gateway trust certificate to the ranger/java truststore. 5. Currently knox service lookups don't work in ranger and a bug has been filed to fix this. Currently, as of HDP 2.4, only topology lookup in ranger works. ... View more

Issue: In 2.3.4 Ranger Hive Repo Test Connection not working with HS2 HTTP TransportMode configuration. Workaround: 1) Move httpclient-4.2.5.jar httpcore-4.2.5.jar from /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib to a tmp folder. 2) fetch the httpclient-4.4.jar and httpcore-4.4.jar from /usr/hdp/current/hive-server2/lib/ and place it in /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib folder 3) Restart ranger admin and configure hive repo with http config and test the HTTP connection and lookup. ... View more