Member since
12-18-2015
22
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2741 | 05-19-2016 08:18 AM |
05-19-2016
10:20 AM
I have a kerberized cluster with Ambari 2.2.1.0 and HDP 2.3.2. I've kerberized Ambari too and I'm trying to use the Hive view. I can browse hive and create tables but an insert statement gives the following TEZ/YARN error: main : run as user is admin
main : requested yarn user is admin
User admin not found
Failing this attempt.
Failing the application.at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:718)at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:207)at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:257)at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:140)at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1655)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1414)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:74 There is a YARN application submitted by user admin with 2 failed attempts but no logs! I modified the hive view configuration substituting ${username} with hive and it fails again, the "User admin not found" text is not there but this time there are plenty of logs in the YARN application submitted this time with by user hive. The error in YARN ist: ...
[ERROR] [main] |web.WebUIService|: Tez UI History URL is not set
...
...
org.apache.tez.dag.api.TezReflectionException: Unable to load class: org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService
...
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
-
Apache Tez
05-19-2016
08:18 AM
@emaxwell
Basically, the problem was that I secured ambari-server with a kerberos-key ambari-server@REAM. As ambari-server runs from root, I had to create root@REALM and re-secure ambari-server with it, now it works! Thanks! In the documentation, it should explicitly explained that the kerberos-key need to have the same name of the linux-user running ambari-server.
... View more
05-18-2016
08:13 AM
I have successfully kerberized a cluster running with Ambari 2.2.1.0 and HDP 2.3.2. All services are running without alerts. I have successfully kerberized ambari-server according to the instructions creating an ambari-server@REALM kerberos user. In the File Explorer view settings, auth=KERBEROS;proxyuser=ambari-server is entered. In the custom core-site settings, the following proxyuser-settings are present (everything set to "*"): hadoop.proxyuser.HTTP.groups
hadoop.proxyuser.HTTP.hosts
hadoop.proxyuser.ambari-server.groups
hadoop.proxyuser.ambari-server.hosts
hadoop.proxyuser.falcon.groups
hadoop.proxyuser.falcon.hosts
hadoop.proxyuser.hbase.groups
hadoop.proxyuser.hbase.hosts
hadoop.proxyuser.hcat.groups
hadoop.proxyuser.hcat.hosts
hadoop.proxyuser.hdfs.groups
hadoop.proxyuser.hdfs.hosts
hadoop.proxyuser.hive.groups
hadoop.proxyuser.hive.hosts
hadoop.proxyuser.hue.groups
hadoop.proxyuser.hue.hosts
hadoop.proxyuser.knox.groups
hadoop.proxyuser.knox.hosts
hadoop.proxyuser.oozie.groups
hadoop.proxyuser.oozie.hosts
hadoop.proxyuser.root.groups
hadoop.proxyuser.root.hosts
hadoop.proxyuser.yarn.groups
hadoop.proxyuser.yarn.hosts
However, neither the File Explorer view nor WebHDFS on URL http://<namenode>:50070/explorer.html#/ are accessible. The File View in Ambari shows the following error: 500 Usernames not matched: name=root != expected=ambari-server Which setting is missing?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
05-11-2016
09:45 AM
I had to unkerberize and rekerberize the cluster, now it works!
... View more
05-11-2016
09:24 AM
@Robert Levas @Pranay Vyas
It was definitively a DNS problem: Kerberos can't use the DNS, it can resolve names only over /etc/hosts Many thanks!
... View more
05-11-2016
07:31 AM
@Robert Levas @Pranay Vyas Name resolution works over a DNS server, but Kerberos seems to ignore it. Adding IP/Hosts to the /etc/hosts file seems to help, so thank you for the tip! However, this doesn't solve the problem but generate a different error message: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.authorize.AuthorizationException: User nm/msas6502i.msg.de@HDP23CLUSTER (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.yarn.server.api.ResourceTrackerPB, expected client Kerberos principal is nm/10.100.233.13@HDP23CLUSTER
... View more
05-10-2016
04:53 PM
@Robert Levas The command kinit -kt /etc/security/keytabs/rm.service.keytab rm/<res_mgr_host>@hdp23cluster works only on the RM-node, maybe because of the missing keytab. After copying the rm.service.keytab on all nodes the command works in the console, but the node manager fails again with the same error "Server has invalid Kerberos principal".
... View more
05-10-2016
04:09 PM
@Robert Levas Thank you for the tip, after recreating and re-configuring everything with an uppercase realm, it still doesn't work. However, I have noticed that the keytab rm.service.keytab is present on the RM host, but not in the other hosts. Should the keytab be present on every host? If yes, than the automatic deployment of the keytabs doesn't work well. Keytabs on the non-RM node: dn.service.keytab
hbase.headless.keytab
hdfs.headless.keytab
knox.service.keytab
nfs.service.keytab
nm.service.keytab
nn.service.keytab
smokeuser.headless.keytab
spark.headless.keytab
spnego.service.keytab
zk.service.keytab
Keytabs on the RM node: dn.service.keytab
hbase.headless.keytab
hdfs.headless.keytab
hive.service.keytab
jhs.service.keytab
nfs.service.keytab
nm.service.keytab
nn.service.keytab
oozie.service.keytab
rm.service.keytab
sbetp.headless.keytab
smokeuser.headless.keytab
spark.headless.keytab
spnego.service.keytab
yarn.service.keytab
zk.service.keytab
... View more
05-10-2016
01:45 PM
@Robert Levas: The realm ist in lower case also in kerberos. Should I indeed enter it in uppercase in Ambari?
... View more
05-10-2016
01:43 PM
@Sagar Shimpi: the user exists.
... View more