About brajeshreddy

brajeshreddy · ‎06-07-2017

@yvora i have made the mentioned changes now the error has gone but no spark job is launched. There is no spark job in spark UI. hive> set hive.execution.engine=spark; hive> set spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-exec.jar ; hive> select count(*) from atlas.employee; Query ID = root_20170607172759_1ebd1bdc-cac5-4e20-a012-2242ddf4e753 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Spark Job = caa2546e-7379-48ed-b2b8-b0055f7e49ab Status: Failed FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask hive> exit; Hive.log 2017-06-07 17:28:48,769 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:48 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:49,771 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:49 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:50,773 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:50 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:51,762 INFO [RPC-Handler-3]: client.SparkClientImpl (SparkClientImpl.java:handle(522)) - Received result for caa2546e-7379-48ed-b2b8-b0055f7e49ab 2017-06-07 17:28:51,776 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:51 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING) 2017-06-07 17:28:52,691 ERROR [main]: status.SparkJobMonitor (SessionState.java:printError(989)) - Status: Failed 2017-06-07 17:28:52,691 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=SparkRunJob start=1496836708670 end=1496836732691 duration=24021 from=org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor> 2017-06-07 17:28:52,751 INFO [main]: hooks.ATSHook (ATSHook.java:<init>(91)) - Created ATS Hook 2017-06-07 17:28:52,751 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,752 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1496836732751 end=1496836732752 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,752 ERROR [main]: ql.Driver (SessionState.java:printError(989)) - FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask 2017-06-07 17:28:52,753 INFO [main]: ql.Driver (Driver.java:execute(1635)) - Resetting the caller context to 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=Driver.execute start=1496836681184 end=1496836732753 duration=51569 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,753 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1496836732753 end=1496836732753 duration=0 from=org.apache.hadoop.hive.ql.Driver> 2017-06-07 17:28:52,778 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/07 17:28:52 INFO Client: Application report for application_1496301296700_0065 (state: RUNNING)

brajeshreddy · ‎06-02-2017

Hi, All my hive jobs with spark as execution engine ( in fact all services that use spark for execution) are failing with below error. Hive version is 1.2.1000 and spark version is 1.6.2 i'm using hdp 2.5. 2017-06-02 11:48:10,090 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/02 11:48:10 INFO Client: Application report for application_1496301296700_0022 (state: RUNNING) 2017-06-02 11:48:11,092 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(569)) - 17/06/02 11:48:11 INFO Client: Application report for application_1496301296700_0022 (state: RUNNING) 2017-06-02 11:48:11,395 WARN [RPC-Handler-3]: rpc.RpcDispatcher (RpcDispatcher.java:handleError(142)) - Received error message:io.netty.handler.codec.DecoderException: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230) at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/Job at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656) at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at org.apache.hive.spark.client.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:96) at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327) ... 15 more Caused by: java.lang.ClassNotFoundException: org.apache.hive.spark.client.Job at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 39 more . 2017-06-02 11:48:11,398 WARN [RPC-Handler-3]: client.SparkClientImpl (SparkClientImpl.java:rpcClosed(116)) - Client RPC channel closed unexpectedly.

brajeshreddy · ‎06-01-2017

followed this link and it worked. http://www.ericlin.me/hive-metastore-migration-from-embedded-postgres-to-mysql

brajeshreddy · ‎06-01-2017

Hi, Im running hdp 2.5 with ambari 2.4.1. Using Postgres as my hive metastore. I would like to migrate hive metastore from postgres to mysql. Is there any document or link available for this??

brajeshreddy · ‎05-31-2017

This was resolved after copying tomcat-juli.jar and bootstrap.jar to eagle tomcat bin directory.

brajeshreddy · ‎05-30-2017

Im trying to integrate eagle with HDP 2.5. While starting eagle service, i see below exception in the log and it stops immediately. [root@server logs]# cat eagle-service.out Error: Could not find or load main class org.apache.catalina.startup.Bootstrap Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1024m; support was removed in 8.0 Error: Could not find or load main class org.apache.catalina.startup.Bootstrap [root@server logs]#

brajeshreddy · ‎05-29-2017

Hi @Jay SenSharma that solved the issue. Thanks for the support.

brajeshreddy · ‎05-29-2017

@Jay SenSharma Couldn't find much from the debugger tools. Curl has below details. [root@server ambari-server]# curl -u admin:@dm1n -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"\"}"}' http://192.168.1.9:8080/api/v1/persist HTTP/1.1 202 Accepted X-Frame-Options: DENY X-XSS-Protection: 1; mode=block Set-Cookie: AMBARISESSIONID=sgdgux8qy76z18nlimhl2bo59;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT User: admin Content-Type: text/plain Content-Length: 0 Server: Jetty(8.1.19.v20160209) [root@server ambari-server]#

brajeshreddy · ‎05-29-2017

Hi @Jay SenSharma I have removed the file app.js and reinstalled ambari server using yum. Now after reinstallation, app.js file is reflecting correct version. However after login, ambari UI is stuck in below screen.

brajeshreddy · ‎05-26-2017

I have reinstalled ambari-server but UI is still stuck and no file named set-ambari-version.sh is available in the cluster. [root@server db]# find / -name set-ambari-version.sh [root@server db]# date Fri May 26 15:24:23 IST 2017 [root@server db]#

Online	Offline
Last Visited	‎09-22-2023 12:46 PM

Member Since	‎03-23-2017 05:06 AM
Last Visited	‎09-22-2023 12:46 PM
Posts	110
Kudos received	2

Cloudera Community

Re: Migrate Hive Metastore from Postgres to MySQL

Re: Eagle service fails to start with class not fo...

Re: Ranger UI is not accessible with Context initi...

Re: atlas import-hive failed with exception

Re: Upgrade from hdp2.4 to 2.5 is failing as hive ...

Re: Spark on hive fails with java.lang.NoClassDefF...

Spark on hive fails with java.lang.NoClassDefFound...

Re: Migrate Hive Metastore from Postgres to MySQL

Migrate Hive Metastore from Postgres to MySQL

Re: Eagle service fails to start with class not fo...

Eagle service fails to start with class not found ...

Re: Ambari Server and Web Client versions do not m...

Re: Ambari Server and Web Client versions do not m...

Re: Ambari Server and Web Client versions do not m...

Re: Ambari Server and Web Client versions do not m...