unable to submit oozie job when using azure datalake storage gen2

arindam1 — Thu, 17 Oct 2019 17:10:16 GMT

Hi,

I am getting a 500 error when submitting the job.

In oozie log, the error is:

2019-10-17 13:14:49,070 ERROR org.apache.oozie.servlet.V1JobsServlet: SERVER[poc-01.cdh-cluster.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST <a href="<a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a>" target="_blank"><a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a</a>>] error, java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2505) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3203) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3235) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478) at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625) at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:623) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:594) at org.apache.oozie.servlet.V1JobsServlet.existsOnDFS(V1JobsServlet.java:211) at org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:162) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:308) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.oozie.servlet.HttpResponseHeaderFilter.doFilter(HttpResponseHeaderFilter.java:68) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2503) ... 53 more 2019-10-17 13:14:49,073 WARN org.eclipse.jetty.servlet.ServletHandler: SERVER[poc-01.cdh-cluster.internal] /oozie/v2/jobs java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2505) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3203) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3235) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478) at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625) at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:623) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:594) at org.apache.oozie.servlet.V1JobsServlet.existsOnDFS(V1JobsServlet.java:211) at org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:162) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:308) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.oozie.servlet.HttpResponseHeaderFilter.doFilter(HttpResponseHeaderFilter.java:68) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2503) ... 53 more

Re: unable to submit oozie job when using azure datalake storage gen2

arindam1 — Fri, 18 Oct 2019 21:02:26 GMT

This solved the issue:

# ls /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/oozie/libext/
hadoop-azure-3.0.0-cdh6.3.0.jar hadoop-azure-datalake-3.0.0-cdh6.3.0.jar hadoop-azure-datalake.jar hadoop-azure.jar mysql-connector-java.jar

question unable to submit oozie job when using azure datalake storage gen2 in Support Questions

unable to submit oozie job when using azure datalake storage gen2

Re: unable to submit oozie job when using azure datalake storage gen2