Hi Team,   I have created a oozie workflow. When I try to execute it shows an error saying class not found. Please help me with this.   Regards, C ... View more

Hi Team,   Is there any API to get the count of open connection from all available hiveserver2 with its private IP.   Thanks in advance. ... View more

 Hi Team,   When I try to execute collect command in spark shell i'm getting issues. The Command that I executed: val logrdd = sc.textFile("/user/edu_qauser/sampletest.clean.log") logrdd.collect The sampletest.clean.log file size is 400MB. ERROR when I execute "logrdd.collect": [Stage 0:> (0 + 2) / 4]19/08/01 09:23:22 ERROR scheduler.TaskSetManager: Task 0 in sta ge 0.0 failed 4 times; aborting job org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 9, ip-x.x.x.x.ec2.internal, executor 2): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 40. To avoid this, increase spark.kryoserializer.buffer.max value. at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:315) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:367) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1430) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1417) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1417) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:797) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:797) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:797) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1645) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1600) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1589) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:623) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1930) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1943) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1956) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1970) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.collect(RDD.scala:934) ... 48 elided Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 40. To avoid this, increase spa rk.kryoserializer.buffer.max value. at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:315) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:367) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   When I change the file size to 100MB: The error I get is, [Stage 0:> (0 + 2) / 2]19/08/01 09:31:49 ERROR util.ResourceLeakDetector: LEAK: ByteBu f.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information. [Stage 0:> (0 + 2) / 2]Exception in thread "refresh progress" Exception in thread "LeaseRenewer:edu_qauser@nameservice1" java.lang.OutOfMemoryError: GC overhead limit exceeded at scala.Predef$.refArrayOps(Predef.scala:344) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:68) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.ArrayList.iterator(ArrayList.java:834) at org.apache.hadoop.hdfs.LeaseRenewer.clientsRunning(LeaseRenewer.java:241) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:491) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) Exception in thread "dispatcher-event-loop-12" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.spark.scheduler.TaskSetManager.dequeueTask(TaskSetManager.scala:356) at org.apache.spark.scheduler.TaskSetManager.resourceOffer(TaskSetManager.scala:431) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$org$apache$spark$scheduler$TaskSchedulerImpl$$resourceOfferSingleTaskSet$1.app ly$mcVI$sp(TaskSchedulerImpl.scala:274) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at org.apache.spark.scheduler.TaskSchedulerImpl.org$apache$spark$scheduler$TaskSchedulerImpl$$resourceOfferSingleTaskSet(TaskSchedulerI mpl.scala:269) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$4$$anonfun$apply$9.apply(TaskSchedulerImpl.scala:355) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$4$$anonfun$apply$9.apply(TaskSchedulerImpl.scala:353) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$4.apply(TaskSchedulerImpl.scala:353) at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$4.apply(TaskSchedulerImpl.scala:350) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:350) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint.org$apache$spark$scheduler$cluster$CoarseGrainedSche dulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:239) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedule rBackend.scala:142) at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/08/01 09:34:02 ERROR util.Utils: Uncaught exception in thread task-result-getter-0 java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "task-result-getter-0" java.lang.OutOfMemoryError: GC overhead limit exceeded   Please give a solution on what configuration should I change.   Thanks in advance. ... View more

Hi Team,   Few weeks before I had a dedicated Hiveserver and configured another Hiveserver in slavenode. But the problem is the traffic was directed only to the dedicated Hiveserver and not the other. So I have configured HAProxy in masternode which now acts as a loadbalancer and configured 3 dedicated Hiveserver. Now the traffic is being routed to the all the server. But the hive query that is being executed from hue is getting expired withing 4 to 5 Min. When I give one of the dedicated Hiveserver IP and Port in the hive loadbalancer configuration the query is there for an hour. But when I give the loadbalancer IP and Port in hive loadbalancer configuration thats when the hive query is geting expired soon. Is there some change that need to be done in haproxy config in order to make sure all the 3 dedicated Hiveserver can withstand query for a specified time. If so that will be much helpful.     Thanks in advance. ... View more

Hi,   I have launched a single node cluster with OS as Centos 7.5, 4vcpu, 16GB RAM in azure. But when I create a table and insert some values into the table in hive its gets to pending state with zero resources while seeing in resource manager. And also when I open up resource manager port to all. Some unknown jobs by dr.who is getting created.  No idea of what to do. Please help me with this 2 issue.     Thanks in advance. ... View more

Hi,   We are providing cloud lab for some of our students. lets say 700 users access us. When they execute multiple hive query in hue. while executing the query tends to get slow and also will get "502 Bad Gateway, a padding to disable MSIE and chrome friendly error page" and also I get "504 Gateway timeout". We have also added auto scaling. But the scaling is not done as the memory and core are not used. Even though I am facing this issue.   In " Java Heap Size of HiveServer2 in Bytes " is set to 10GB also "Max HiveServer2 Threads" is set to 500. When I check the JVM Heap Memory Usage it is in 4GB it is not even touching 7 or 8GB though I get error.   Please Give me a solution. This weekend is going to be crucial as 600 user are going to hit us. We have just hit by 200 to 300 users in this week. If I get a solution before that it will be much helpfull.   Thanks in advance. ... View more

Hi,   Normally for packages or parcel to add a new host it takes minimum 15 min, is there a way to reduce this timing.   Thank You, ... View more

Hi, I need an API to get hue active users count. There is a chart that shows the count, but I need to display via API. If there is one do share it please. ... View more

Hi, I wanted to get all spark job details like Job ID, Job Name, Query, Start Time, End Time.  Is there an API to get these details. If there is one, do tell me please asap. ... View more

Hi, I am trying to get Hive query and its details, But i can get its details not the full query. I need to get all the data by API or command, please give me an perfect solution asap.   Thanks, ... View more