Reply
New Contributor
Posts: 1
Registered: ‎05-06-2014

WebHCat and CDH5

Hi,

 

We are upgrading our CDH4.5 cluster to CDH5 and I got the following problem with WebHCat. Job submission works but we got an exception while we query the status of a job:

 

WARN | 05 May 2014 21:35:24,979 | org.eclipse.jetty.servlet.ServletHandler | /templeton/v1/jobs/job_201404282258_0057
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobStatus.getJobFile()Ljava/lang/String;
at org.apache.hadoop.mapred.WebHCatJTShim23.getJobProfile(WebHCatJTShim23.java:47)
at org.apache.hive.hcatalog.templeton.StatusDelegator.makeStatus(StatusDelegator.java:76)
at org.apache.hive.hcatalog.templeton.StatusDelegator.makeStatus(StatusDelegator.java:97)
at org.apache.hive.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:55)
at org.apache.hive.hcatalog.templeton.Server.showJobId(Server.java:838)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at http://org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at http://org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:744)

 

 

I looked into JobStatus class and it looks like it is not extending org.apache.hadoop.mapreduce.JobStatus

https://github.com/cloudera/hadoop-common/blob/cdh5.0.0-release/hadoop-mapreduce1-project/src/mapred...

 

I also fund a different version of the same class that extends org.apache.hadoop.mapreduce.JobStatus

https://github.com/apache/hadoop-common/blob/release-2.4.0/hadoop-mapreduce-project/hadoop-mapreduce...

 

Our cluster is running on MRv1 so I'm wondering how we should configure the classpath to pick up the right version and be compatible with MRv1 on CDH5.

 

Thanks,

Francesco

 

 

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: WebHCat and CDH5

We do not recommend using WebHCatalog to submit jobs as it basically is attempting to duplicate Apache Oozie's functionality, which we test for to work perfectly and beyond WebHCatalog's capacity.

Please consider using Oozie instead of WebHCatalog for submitting jobs via an end-point. Oozie also provides a REST API, more on which is at http://blog.cloudera.com/blog/2013/06/how-to-use-the-apache-oozie-rest-api/.
Explorer
Posts: 10
Registered: ‎11-26-2015

Re: WebHCat and CDH5

Hi,

 

do you still recommend to use Oozie over WebHCat? I would like to connect SAP HANA with Hadoop and invoke MapReduce jobs. Unfortunately, I haven't had success till now.

 

Thanks,

Matthias

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.