Member since
02-28-2018
2
Posts
0
Kudos Received
0
Solutions
03-01-2018
08:36 AM
Hi @bkosaraju, I am using sc.stop() in the main where I am starting the sparkContext ,also ensuring that sc.stop is being used before sys.exit(). On the other hand, the resources are not overcommitted as I can see still YARN is 30% free. The first error I see is: 18/03/01 08:13:14 ERROR LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. Followed by Warnings almost every minute 18/03/01 08:13:14 WARN LiveListenerBus: Dropped 1 SparkListenerEvents since Thu Jan 01 00:00:00 UTC 1970
18/03/01 08:14:14 WARN LiveListenerBus: Dropped 4643 SparkListenerEvents since Thu Mar 01 08:13:14 UTC 2018
18/03/01 08:15:14 WARN LiveListenerBus: Dropped 4759 SparkListenerEvents since Thu Mar 01 08:14:14 UTC 2018
18/03/01 08:16:15 WARN LiveListenerBus: Dropped 4869 SparkListenerEvents since Thu Mar 01 08:15:14 UTC 2018
18/03/01 08:17:15 WARN LiveListenerBus: Dropped 4885 SparkListenerEvents since Thu Mar 01 08:16:15 UTC 2018
18/03/01 08:18:15 WARN LiveListenerBus: Dropped 4653 SparkListenerEvents since Thu Mar 01 08:17:15 UTC 2018
18/03/01 08:19:17 WARN LiveListenerBus: Dropped 4549 SparkListenerEvents since Thu Mar 01 08:18:15 UTC 2018
18/03/01 08:20:17 WARN LiveListenerBus: Dropped 4344 SparkListenerEvents since Thu Mar 01 08:19:17 UTC 2018
18/03/01 08:21:17 WARN LiveListenerBus: Dropped 4559 SparkListenerEvents since Thu Mar 01 08:20:17 UTC 2018
18/03/01 08:22:18 WARN LiveListenerBus: Dropped 4367 SparkListenerEvents since Thu Mar 01 08:21:17 UTC 2018
18/03/01 08:23:18 WARN LiveListenerBus: Dropped 4525 SparkListenerEvents since Thu Mar 01 08:22:18 UTC 2018
18/03/01 08:24:18 WARN LiveListenerBus: Dropped 4442 SparkListenerEvents since Thu Mar 01 08:23:18 UTC 2018
18/03/01 08:25:18 WARN LiveListenerBus: Dropped 4404 SparkListenerEvents since Thu Mar 01 08:24:18 UTC 2018 Not sure, if I am missing anything...
... View more
02-28-2018
12:38 PM
Scenario: HDinsight cluster with HDP 3.6 2 head node(8CPU/node,56GB/node) and 2 worker nodes(8CPU/node,56GB/node) Container Size: 512 MB There are around 800,000 files which needs to be merged as one single file which is estimated to be 2GB.
On an average, each file consists of 2kB data(single event/file)
We are loading events files(json) through spark using below flow. app submit command: /usr/bin/spark-submit --deploy-mode cluster --queue default --verbose --num-executors 25 --executor-memory 2G --executor-cores 2 --driver-memory 6G --conf spark.shuffle.consolidateFiles=true --conf spark.driver.maxResultSize=5G --conf spark.yarn.executor.memoryOverhead=52m --conf spark.yarn.driver.memoryOverhead=300m --jars /usr/lib/customhivelibs/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar --py-files /home/sshuser/smartRefinery_wo.properties PythonScriptProdInt_wo.py cont_log RDD=textFile(events_file) RDD.repartition(1).saveAsTextFile('merged_eventsFile') Issue:
Repartition is consuming around 50 mins to merge as a single file. Once this repartitioning is complete the application log throws below error continously for another hour(more than 6000 lines of error) 18/02/28 11:23:37 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(9,WrappedArray()) 18/02/28 11:23:37 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(8,WrappedArray()) 18/02/28 11:23:37 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(19,WrappedArray()) 18/02/28 11:23:38 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(1,WrappedArray()) 18/02/28 11:23:38 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(16,WrappedArray()) 18/02/28 11:23:38 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetri
csUpdate(11,WrappedArray()) Workaround tried:
Tried setting up the parameter spark.scheduler.listenerbus.eventqueue.size to 200000, but still recieving error Any help would be greatly appreciated!
... View more
Labels:
- Labels:
-
Apache Spark