Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

WebHCat and CDH5

WebHCat and CDH5

New Contributor

Hi,

 

We are upgrading our CDH4.5 cluster to CDH5 and I got the following problem with WebHCat. Job submission works but we got an exception while we query the status of a job:

 

WARN | 05 May 2014 21:35:24,979 | org.eclipse.jetty.servlet.ServletHandler | /templeton/v1/jobs/job_201404282258_0057
java.lang.NoSuchMethodError: org.apache.hadoop.mapred.JobStatus.getJobFile()Ljava/lang/String;
at org.apache.hadoop.mapred.WebHCatJTShim23.getJobProfile(WebHCatJTShim23.java:47)
at org.apache.hive.hcatalog.templeton.StatusDelegator.makeStatus(StatusDelegator.java:76)
at org.apache.hive.hcatalog.templeton.StatusDelegator.makeStatus(StatusDelegator.java:97)
at org.apache.hive.hcatalog.templeton.StatusDelegator.run(StatusDelegator.java:55)
at org.apache.hive.hcatalog.templeton.Server.showJobId(Server.java:838)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
at http://org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at http://org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:744)

 

 

I looked into JobStatus class and it looks like it is not extending org.apache.hadoop.mapreduce.JobStatus

https://github.com/cloudera/hadoop-common/blob/cdh5.0.0-release/hadoop-mapreduce1-project/src/mapred...

 

I also fund a different version of the same class that extends org.apache.hadoop.mapreduce.JobStatus

https://github.com/apache/hadoop-common/blob/release-2.4.0/hadoop-mapreduce-project/hadoop-mapreduce...

 

Our cluster is running on MRv1 so I'm wondering how we should configure the classpath to pick up the right version and be compatible with MRv1 on CDH5.

 

Thanks,

Francesco

 

 

2 REPLIES 2

Re: WebHCat and CDH5

Master Guru
We do not recommend using WebHCatalog to submit jobs as it basically is attempting to duplicate Apache Oozie's functionality, which we test for to work perfectly and beyond WebHCatalog's capacity.

Please consider using Oozie instead of WebHCatalog for submitting jobs via an end-point. Oozie also provides a REST API, more on which is at http://blog.cloudera.com/blog/2013/06/how-to-use-the-apache-oozie-rest-api/.

Re: WebHCat and CDH5

Hi,

 

do you still recommend to use Oozie over WebHCat? I would like to connect SAP HANA with Hadoop and invoke MapReduce jobs. Unfortunately, I haven't had success till now.

 

Thanks,

Matthias