Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!
Solved Go to solution

unable to submit oozie job when using azure datalake storage gen2

Labels:
avatar
author-rank arindam1
Explorer

Created on ‎10-17-2019 06:16 AM - last edited on ‎10-17-2019 10:10 AM by Guru Guru

Hi,

I am getting a 500 error when submitting the job.

 

In oozie log, the error is:

 

2019-10-17 13:14:49,070 ERROR org.apache.oozie.servlet.V1JobsServlet: SERVER[poc-01.cdh-cluster.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST <a href="<a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a>" target="_blank"><a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a</a>>] error, java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2505)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3203)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3235)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
	at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
	at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:623)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:594)
	at org.apache.oozie.servlet.V1JobsServlet.existsOnDFS(V1JobsServlet.java:211)
	at org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:162)
	at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:308)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
	at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
	at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.oozie.servlet.HttpResponseHeaderFilter.doFilter(HttpResponseHeaderFilter.java:68)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2503)
	... 53 more
2019-10-17 13:14:49,073 WARN org.eclipse.jetty.servlet.ServletHandler: SERVER[poc-01.cdh-cluster.internal] /oozie/v2/jobs
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2505)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3203)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3235)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
	at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
	at org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:623)
	at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:594)
	at org.apache.oozie.servlet.V1JobsServlet.existsOnDFS(V1JobsServlet.java:211)
	at org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:162)
	at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:308)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780)
	at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
	at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.oozie.servlet.HttpResponseHeaderFilter.doFilter(HttpResponseHeaderFilter.java:68)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2503)
	... 53 more

 

3,521 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank arindam1
Explorer

Created ‎10-18-2019 02:02 PM

This solved the issue:

 

# ls /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/oozie/libext/
hadoop-azure-3.0.0-cdh6.3.0.jar hadoop-azure-datalake-3.0.0-cdh6.3.0.jar hadoop-azure-datalake.jar hadoop-azure.jar mysql-connector-java.jar

 

View solution in original post

3,488 Views
0 Kudos
1 REPLY 1

avatar
author-rank arindam1
Explorer

Created ‎10-18-2019 02:02 PM

This solved the issue:

 

# ls /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/oozie/libext/
hadoop-azure-3.0.0-cdh6.3.0.jar hadoop-azure-datalake-3.0.0-cdh6.3.0.jar hadoop-azure-datalake.jar hadoop-azure.jar mysql-connector-java.jar

 

3,489 Views
0 Kudos
Announcements
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
Community Announcements
June 2026 Community Highlights
View More Announcements