Greetings, We've successfully upgraded our 5 node cluster to CDH 6.3, everything went better than expected, so we're generally happy 🙂 However, after the upgrade we are no longer able to access our HBase tables from Hue. Even in 5.15 it needed a bit of tinkering, but it did work after adding the following to Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini:   [hbase] hbase_conf_dir=/etc/hbase/conf thrift_transport=buffered I tried removing the config snippet, but it didn't help. The logs (hbase-cmf-hbase-HBASETHRIFTSERVER-dev-edge.log.out) say that: ERROR org.apache.hadoop.hbase.thrift.ThriftHttpServlet: Kerberos Authentication failed Caused by: org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Kerberos authentication failed Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)  The cluster is Kerberized, TLS/SSL is enabled as well. The following configuration options could be of interest: hbase.security.authentication=kerberos hbase.rpc.protection=privacy hbase.master.kerberos.principal=hbase/_HOST@DEVMH.BI hbase.thrift.kerberos.principal=HTTP/_HOST@DEVMH.BI hadoop.security.authorization=true hbase.rest.ssl.enabled=true hbase.thrift.security.qop=auth-conf  Please let me know if you need further details. Any and all help is appreciated 🙂 Thanks! Kind regards, Gyuszi ... View more

Greetings,   We're encountering a rather strange issue. We are running HBase as part of a healthy CDH 5.15 cluster. Because of configuration changes we have to restart HBase every now and then.   Every time, no matter what, HBase fails to start after a graceful shutdown. The error is always the same, kinit fails. I have to manually regenerate the HBase keytabs from cloudera manager, afterwards HBase starts right away.   I looked around online but failed to find any mention of such behavior. Other services handle restarts without hiccups, no need to regenerate other keytabs.   Please let me know if additional details are needed to further debug this issue.   Thank you for your help in advance!   Kind regards, Julius ... View more

Hi, We are exploring the features in the brand new HDP 3.0 release and in the process of doing so we stumbled upon a curious situation. Our goal is to achieve HA for a handful of services, but the Yarn Timeline Service is causing us some trouble as it can't be set-up in a highly available fashion "out of the box". We decided to disable the timeline service and see how it affects other components. In the Advanced Yarn settings we unchecked "yarn.timeline-service.enabled", restarted the required services, and ran some tests. (Please note, the only configuration change we performed is removing the tick-mark from the yarn.timeline-service.enabled checkbox, as seen in the attached image.) With this configuration our Hbase, Oozie, and Flink tests all finished without an issue. Hive however refused to cooperate, complaining about the missing Timeline service. The full error message is: ERROR : Failed to execute tez graph. org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1533717105751_0001 to YARN : java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x at org.apache.tez.client.TezClient.start(TezClient.java:407) ~[tez-api-0.9.1.3.0.0.0-1634.jar:0.9.1.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:516) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:451) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:373) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:368) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:195) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2668) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2339) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2015) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1713) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1707) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329) ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_112] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_112] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1533717105751_0001 to YARN : java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:304) ~[hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar:?] at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77) ~[tez-api-0.9.1.3.0.0.0-1634.jar:0.9.1.3.0.0.0-1634] at org.apache.tez.client.TezClient.start(TezClient.java:402) ~[tez-api-0.9.1.3.0.0.0-1634.jar:0.9.1.3.0.0.0-1634] ... 28 more ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask INFO : Completed executing command(queryId=hive_20180808083450_cac490e9-225b-4ea9-a1b0-22933fa27869); Time taken: 0.634 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1) On HDP 2.6 disabling "yarn.timeline-service.enabled" produced a different result - we were able to run our tests using Beeline without an issue. Is this a bug? Is this the expected behavior with the configuration changes and the components in HDP 3.0? We were expecting similar behavior to HDP 2.6, that is, on HDP 3.0 disabling the timeline service in the Yarn configuration should not have had a negative effect on our test scenario. Thank you for your responses in advance! PS: TeZ advanced settings has one parametre referencing the Yarn timeline, yarn.timeline-service.enabled, it's value is false and does not seem to be affected by modifying the same setting in the Yarn advanced configuration options, ie. it stay false even if we enable the option in the Yarn settings. ... View more