I got this Error : Invalid name provided (Mechanism level: KrbException: Cannot locate default realm) while submit workflow to oozie

[#1]

In my project, I was using the Oozie Client API. But for Kerberised authentication, I was using AuthOozieClient to authenticate a user to access Oozie, but I getting error

org.ietf.jgss.GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm) at sun.security.jgss.krb5.Krb5NameElement.getInstance(Krb5NameElement.java:129) ~[na:1.8.0_144] at sun.security.jgss.krb5.Krb5MechFactory.getNameElement(Krb5MechFactory.java:95) ~[na:1.8.0_144] at sun.security.jgss.GSSManagerImpl.getNameElement(GSSManagerImpl.java:203) ~[na:1.8.0_144] at sun.security.jgss.GSSNameImpl.getElement(GSSNameImpl.java:477) ~[na:1.8.0_144] at sun.security.jgss.GSSNameImpl.init(GSSNameImpl.java:201) ~[na:1.8.0_144] at sun.security.jgss.GSSNameImpl.<init>(GSSNameImpl.java:170) ~[na:1.8.0_144] at sun.security.jgss.GSSNameImpl.<init>(GSSNameImpl.java:151) ~[na:1.8.0_144] at sun.security.jgss.GSSManagerImpl.createName(GSSManagerImpl.java:128) ~[na:1.8.0_144] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:297) ~[hadoop-auth-2.7.3.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) ~[hadoop-auth-2.7.3.jar:na] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_144] at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_144] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) ~[hadoop-auth-2.7.3.jar:na] at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) ~[hadoop-auth-2.7.3.jar:na] at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) ~[hadoop-auth-2.7.3.jar:na] at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:128) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient$1.doExecute(OozieClient.java:461) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.retry.ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.createRetryableConnection(OozieClient.java:458) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.getSupportedProtocolVersions(OozieClient.java:338) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:298) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:413) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:76) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:509) ~[oozie-client-4.1.0.jar:4.1.0] at org.apache.oozie.client.OozieClient.getCoordJobInfo(OozieClient.java:1246) ~[oozie-client-4.1.0.jar:4.1.0] at com.oi.hermes.scheduler.service.DashboardManagerService.getOozieCoordWorkflows(DashboardManagerService.java:67) ~[classes/:na] at com.oi.hermes.scheduler.controller.HermesJobController.getAllWorkFlowInfo(HermesJobController.java:106) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455) [tomcat-embed-core-8.5.14.jar:8.5.14] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.14.jar:8.5.14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.14.jar:8.5.14] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

Here is my Java code :

public List<CoordinatorExecutions> getOozieCoordWorkflows(String scheduleId) throws OozieClientException { InsertScheduleCoordinatorInfo scheduleCoordinatiorInfo = new InsertScheduleCoordinatorInfo(); scheduleCoordinatiorInfo = scheduleWorkflowInfoRepository.findOne(UUID.fromString(scheduleId)); String coordIdString = scheduleCoordinatiorInfo.getCoordinatorId(); AuthOozieClient wc=new AuthOozieClient(oozieUrl); CoordinatorJob coordinatorJob = wc.getCoordJobInfo(coordIdString); WorkflowJob workflowJob; List<CoordinatorAction> action = coordinatorJob.getActions(); CoordinatorExecutions coordinatorExecutions = null; List<CoordinatorExecutions> executionsList = new ArrayList<CoordinatorExecutions>(); for (CoordinatorAction coordinatorAction : action) { coordinatorExecutions = new CoordinatorExecutions(); coordinatorExecutions.setId(coordinatorAction.getId()); coordinatorExecutions.setStatus(coordinatorAction.getStatus().toString()); LOGGER.info(coordinatorAction.getExternalId()); workflowJob = wc.getJobInfo(coordinatorAction.getExternalId()); coordinatorExecutions.setCreatedTime(workflowJob.getStartTime().toString()); if (workflowJob.getEndTime() != null) { coordinatorExecutions.setLastModifiedTime(workflowJob.getEndTime().toString()); } else { coordinatorExecutions.setLastModifiedTime(""); } executionsList.add(coordinatorExecutions); } return executionsList; }

[#2]

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

[#3]

Actually, I want it to handle by the application. So I don't want to run kinit command.

so I added one class to project on GitHub. It will create a config file with given properties. and return back HTTP response with Authentication. So go through it and let me know its useful or not. if yes how?. https://github.com/dineshjadhav-oi/Oozie-User-Authentication

[#4]

I ran the kinit command before launch application but it will shows following error:

IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, GSSException: No valid credent ials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)

[#5]

@Dinesh Jadhav

Can you paste the content of the following file you can scramble the REALM and other sensitive info:

- kinit command 
- krb5.conf
- kdc.conf
- kadm5.acl

Make sure that you have a local copy of krb5.conf on all hosts and that the kadmin is up and running

[#6]

Sorry I cant provide you conf files. So if you send me to sample conf files then I can compare them with my config files.

[#7]

@Geoffrey Shelton Okot

Can you please send me sample configuration files.

[#8]

Hi @Geoffrey Shelton Okot

I attached a text file which has all config files here - kerberos-config.txt. So you can check it and let me know.

[#9]

@Dinesh Jadhav

There were a couple of errors in the kdc.conf,krb5.conf and kadm5.acl.

Please see the attached files. I would first ask you to backup your current kdc,krb5 and kadm5.acl files.

I have tried to separate the config files with --------

Please let me know if you have well understood if you need clarifications don't hesitate

[#10]

@Geoffrey Shelton Okot

I do the changes in conf files and run that code again, but it will give the same error. Can you give me working example?. Which connect to the oozie and get data from it with kerberos authentication. I think something is missing in the code, So I don't know what!

[#11]

@Dinesh Jadhav

Okay lets first eliminate the Kerberos error can you attach your current files below

1. - krb5.conf
2. - kdc.conf
3. - kadm5.acl

Can you also run as root on the kdc server and see if you get any output

# kadmin.local 

Then look at this oozie config for kerberos