Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie issue in kerberized cluster

Oozie issue in kerberized cluster

Explorer

Hello All,

 

I am in of desperate help.  I can't get oozie to start after enabling kerberos for oozie.  Oozie will start fine with following config:

<property>
<name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
<value>false</value>
</property>
<property>
<name>local.realm</name>
<value>localhost</value>
</property>
<property>
<name>oozie.authentication.type</name>
<value>simple</value>
</property>

 But, once I change config to following I can't get oozie to start and i am getting following error:

2015-06-11 12:27:33,625 INFO org.apache.oozie.service.HadoopAccessorService: SERVER[enk1hdfs1.hdfs.net] USER[-] GROUP[-] Oozie Kerberos Authentication [enabled]
2015-06-11 12:27:34,293 FATAL org.apache.oozie.service.Services: SERVER[enk1hdfs1.hdfs.net] USER[-] GROUP[-] E0100: Could not initialize service [org.apache.oozie.service.Hadoo         pAccessorService], Login failure for oozie/enk1hdfs1.hdfs.net@HDFS.NET from keytab /etc/oozie/oozie-http.keytab
org.apache.oozie.service.ServiceException: E0100: Could not initialize service [org.apache.oozie.service.HadoopAccessorService], Login failure for oozie/enk1hdfs1.hdfs.net@HDFS         .NET from keytab /etc/oozie/oozie-http.keytab
        at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:182)
        at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:127)
        at org.apache.oozie.service.HadoopAccessorService.init(HadoopAccessorService.java:98)
        at org.apache.oozie.service.Services.setServiceInternal(Services.java:383)
        at org.apache.oozie.service.Services.setService(Services.java:369)
        at org.apache.oozie.service.Services.loadServices(Services.java:302)
        at org.apache.oozie.service.Services.init(Services.java:210)
        at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:45)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.io.IOException: Login failure for oozie/enk1hdfs1.hdfs.net@HDFS.NET from keytab /etc/oozie/oozie-http.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:947)
        at org.apache.oozie.service.HadoopAccessorService.kerberosInit(HadoopAccessorService.java:174)
        ... 31 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:938)
        ... 32 more
2015-06-11 12:27:34,310 INFO org.apache.oozie.service.Services: SERVER[enk1hdfs1.hdfs.net] Shutdown

OOZIE Kerberos config: 

<property>
<name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
<value>true</value>
</property>
<property>
<name>local.realm</name>
<value>HDFS</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.keytab.file</name>
<value>/etc/oozie/oozie-http.keytab </value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.kerberos.principal</name>
<value>oozie/enk1hdfs1.hdfs.net@HDFS.NET</value>
</property>
<property>
<name>oozie.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>oozie.authentication.kerberos.principal</name>
<value>HTTP/enk1hdfs1.hdfs.net@HDFS.NET</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
<value/>
<description/>
</property>
<property>
<name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
<value/>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>

  kerberos principals from keytab file - 

 

[hdfs@enk1hdfs1 ~]$ klist -e -k -t /etc/oozie/conf/oozie-http.keytab
Keytab name: FILE:/etc/oozie/conf/oozie-http.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (aes256-cts-hmac-sha1-96)
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (aes128-cts-hmac-sha1-96)
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (des3-cbc-sha1)
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (arcfour-hmac)
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (des-hmac-sha1)
   7 06/09/15 14:38:54 oozie/enk1hdfs1.hdfs.net@HDFS.NET (des-cbc-md5)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (aes256-cts-hmac-sha1-96)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (aes128-cts-hmac-sha1-96)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (des3-cbc-sha1)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (arcfour-hmac)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (des-hmac-sha1)
   7 06/09/15 14:38:54 HTTP/enk1hdfs1.hdfs.net@HDFS.NET (des-cbc-md5)
[hdfs@enk1hdfs1 ~]$ su -

any help would be greatly appreciated.. 

5 REPLIES 5

Re: Oozie issue in kerberized cluster

Explorer

OK...I have figured it out, but now new issue poping up when accessing web console

 

first how i fixed it...

<property>
<name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
<value>true</value>
</property>
<property>
<name>local.realm</name>
<value>HDFS.NET</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.keytab.file</name>
<value>/etc/oozie/conf/oozie.keytab</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.kerberos.principal</name>
<value>oozie/enk1hdfs1.hdfs.net@HDFS.NET</value>
</property>
<property>
<name>oozie.authentication.type</name>
<value>kerberos</value>
</property>
<property>
<name>oozie.authentication.kerberos.principal</name>
<value>HTTP/enk1hdfs1.hdfs.net@HDFS.NET</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
<value/>
<description/>
</property>
<property>
<name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
<value/>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
    <name>oozie.authentication.kerberos.keytab</name>
    <value>/etc/oozie/conf/oozie.keytab</value>
    <description>
        Location of the keytab file with the credentials for the principal.
        Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
    </description>
</property>

 

after I have added oozie.authentication.kerberos.keytab same keytab service is starting..

 

Now, new issue.  when I click web UI I am getting this:

 

HTTP Status 403 - GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)

type Status report

message GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)

description Access to the specified resource has been forbidden.

Apache Tomcat/6.0.43
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:520)
        at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:159)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
        at sun.security.jgss.GSSHeader.<init>(GSSHeader.java:97)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:306)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:366)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
        ... 17 more

 

Re: Oozie issue in kerberized cluster

Explorer
bump? any help would be great...
Highlighted

Re: Oozie issue in kerberized cluster

Expert Contributor

In order to access the Oozie web UI after enabling kerberos, you have to have a valid kerberos ticket on the client host where your browser is.  If you are using AD for your KDC, then you should be OK there.  If not, then you need the MIT kerberos client.  Windows client is here:

 

http://web.mit.edu/kerberos/dist/#kfw-4.0

 

Once you have the client installed and pulled a valid ticket from HDFS.NET, then you need to configure Firefox(it's easier than chrome or IE).  Enter "about:config" as the URL and then edit these settings:

 


network.negotiate-auth.trusted-uris = fqdn of the Oozie Host

network.auth-use-sspi = false(if using windows and MIT otherwise dont' touch)

 

 

Re: Oozie issue in kerberized cluster

Contributor

Thanks for your reply,

I already added hostname(FQDN)  in network.negotiate-auth.trusted-uris config in firefox but no use.

 

Regards,

Rakesh

Re: Oozie issue in kerberized cluster

Contributor

Thanks for your reply,

I already added hostname(FQDN)  in network.negotiate-auth.trusted-uris config in firefox but no use.

 

Regards,

Rakes

Don't have an account?
Coming from Hortonworks? Activate your account here