Member since
12-12-2019
34
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2603 | 01-07-2020 01:29 AM | |
13784 | 12-26-2019 11:05 PM |
06-02-2022
06:19 AM
You should use "vacum" command first (this will delete transactions history in deltalog) After that both above statements should produce same results.
... View more
01-17-2020
04:36 AM
Hi Sudhnidra, Please take a look at: https://blog.cloudera.com/yarn-fairscheduler-preemption-deep-dive/ https://blog.cloudera.com/untangling-apache-hadoop-yarn-part-3-scheduler-concepts/ https://clouderatemp.wpengine.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/ What type of FairScheduler are you using: Steady FairShare Instantaneous FairShare What is the weight of the default queue you are submitting your apps to? Best, Lyubomir
... View more
01-07-2020
11:03 PM
Hi @Shelton , I was able to achieve my objective of running multiple Spark Sessions under a single Spark context using YARN capacity scheduler and Spark Fair Scheduling. However, the issue still remains with YARN fair scheduler. The second Spark job is still not running (with the same memory configuration) due to lack of resources. So, what additional parameters need to be set for YARN fair scheduler to achieve this? Please help me in fixing this issue. Thanks and Regards, Sudhindra
... View more
01-07-2020
01:29 AM
I have observed that by increasing the number of cores/executors and driver/executor memory, I was able to verify that around 6 tasks are running in parallel at a time. Thanks and Regards, Sudhindra
... View more
12-26-2019
11:05 PM
Hi All, The issue has got fixed. It was due to Spark Executor JVM Option being set incorrectly. Thanks and Regards, Sudhindra
... View more
12-12-2019
10:31 PM
Hi, I have tried the same steps in Ambari. I am using HDP 3.1.0.0-78. Here is what I have done: Step 1: Went to 'Yarn -> Configs -> Advanced -> Scheduler' and set the properties that you have given. Step 2: Restarted all the affected components in Yarn. However, I am getting an error java.io.IOException: Failed to initialize FairScheduler Caused by: org.xml.sax.SAXParseException; lineNumber: 19; columnNumber: 1; XML document structures must start and end within the same entity. Also, in "Scheduler" component in Yarn, I am only getting "Capacity Scheduler" related parameters. If I add "fair-scheduler.xml" in the same folder as that of "yarn-site.xml" and specify parameters only for "default" queue (Here I am not adding any new queue) only to change its scheduling mode to Fair, how to do it? Can you please guide me?
... View more