Hi,   How did you change the dns to local nodes using dnsmasq?   Could you please provide me the step by step. I have the same issue on my machine.   Thanks in advance! ... View more

Hello @regeamor   Thank you for posting the query with us.   Basically when you enable Dynamic allocation it gracefully remove the Idle containers which were idle for (60s as default) But when you lower the value will remove the executors frequently depending upon the executor's usage (just like the tasks getting allocated to those executors and the amount of tasks required)   https://spark.apache.org/docs/latest/job-scheduling.html#graceful-decommission-of-executors   Also, the above said behaviour is upto Spark (and yarn container pre-emption wont be aware of it) ... View more

Hello @yasmin,   Thanks for reaching us !   As per your reply it says that user who has triggered job, his user id is not listed on that particular node ? Is my understanding correct? ==> Yes you're right !   And your query regarding job slowness, we should consider the factors you mentioned along with below messages as well   ~~~ ExecutorLostFailure (executor 1480 exited caused by one of the running tasks) Reason: Stale executor after cluster manager re-registered.   I had 438 failures on 869 taks, that is a huge rate, another part has 873 out of 1236 ~~~   Here it seems the executors are getting lost and as a results tasks were dying.   Could you please check the yarn logs for the application (by using #yarn logs command in my previous reply) and see if there is any errors in executor logs. This will help us to see if there is any spcific reason for executor failures   Are you running  job in spark-client mode or cluster mode?   ... View more

Thanks Satz. ... View more

Hi @satz,   Logs that are comming in livy session is as follows,moreover information i want to provide you is,I am using spark with yarn.   18/12/24 11:30:53 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.184.153.88:4042 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/hamcrest-core-1.3-sources.jar at spark://10.184.153.88:50550/jars/hamcrest-core-1.3-sources.jar with timestamp 1545631253249 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/junit-4.12-sources.jar at spark://10.184.153.88:50550/jars/junit-4.12-sources.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/json-simple-1.1-sources.jar at spark://10.184.153.88:50550/jars/json-simple-1.1-sources.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/json-20131018-sources.jar at spark://10.184.153.88:50550/jars/json-20131018-sources.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/netty-all-4.0.37.Final.jar at spark://10.184.153.88:50550/jars/netty-all-4.0.37.Final.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/livy-rsc-0.5.0-incubating.jar at spark://10.184.153.88:50550/jars/livy-rsc-0.5.0-incubating.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/rsc-jars/livy-api-0.5.0-incubating.jar at spark://10.184.153.88:50550/jars/livy-api-0.5.0-incubating.jar with timestamp 1545631253250 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/repl_2.11-jars/commons-codec-1.9.jar at spark://10.184.153.88:50550/jars/commons-codec-1.9.jar with timestamp 1545631253251 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/repl_2.11-jars/livy-repl_2.11-0.5.0-incubating.jar at spark://10.184.153.88:50550/jars/livy-repl_2.11-0.5.0-incubating.jar with timestamp 1545631253251 18/12/24 11:30:53 INFO spark.SparkContext: Added JAR file:/scratch/livy-0.5.0-incubating-bin/repl_2.11-jars/livy-core_2.11-0.5.0-incubating.jar at spark://10.184.153.88:50550/jars/livy-core_2.11-0.5.0-incubating.jar with timestamp 1545631253251 18/12/24 11:30:53 INFO executor.Executor: Starting executor ID driver on host localhost 18/12/24 11:30:53 INFO executor.Executor: Using REPL class URI: spark://10.184.153.88:50550/classes 18/12/24 11:30:53 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52905. 18/12/24 11:30:53 INFO netty.NettyBlockTransferService: Server created on 10.184.153.88:52905 18/12/24 11:30:53 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 18/12/24 11:30:53 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.184.153.88, 52905, None) 18/12/24 11:30:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.184.153.88:52905 with 366.3 MB RAM, BlockManagerId(driver, 10.184.153.88, 52905, None) 18/12/24 11:30:53 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.184.153.88, 52905, None) 18/12/24 11:30:53 INFO storage.BlockManager: external shuffle service port = 7337 18/12/24 11:30:53 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.184.153.88, 52905, None) 18/12/24 11:30:53 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4282b62c{/metrics/json,null,AVAILABLE,@Spark} 18/12/24 11:30:54 INFO scheduler.EventLoggingListener: Logging events to hdfs://ofss2311699.in.oracle.com:8020/user/spark/applicationHistory/local-1545631253277 18/12/24 11:30:54 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.NavigatorAppListener 18/12/24 11:30:54 INFO driver.SparkEntries: Spark context finished initialization in 2460ms 18/12/24 11:30:54 INFO driver.SparkEntries: Created Spark session. 18/12/24 11:30:59 INFO driver.SparkEntries: Created SQLContext. 18/12/24 11:30:59 WARN spark.SparkContext: Using an existing SparkContext; some configuration may not take effect. 18/12/24 11:30:59 INFO spark.SparkContext: Added file /tmp/tmputrDLi/__livy__/new_pr_java.jar at file:/tmp/tmputrDLi/__livy__/new_pr_java.jar with timestamp 1545631259254 18/12/24 11:30:59 INFO util.Utils: Copying /tmp/tmputrDLi/__livy__/new_pr_java.jar to /tmp/spark-106fd442-bba9-4ff9-b058-6d54211a6543/userFiles-64dbb8bd-2b53-4888-a3e0-c86c9cc5403b/new_pr_java.jar 18/12/24 11:30:59 INFO spark.SparkContext: Added JAR hdfs://ofss2311699:8020/user/jars/new_pr_java.jar at hdfs://ofss2311699:8020/user/jars/new_pr_java.jar with timestamp 1545631259277 18/12/24 11:30:59 INFO driver.RSCDriver: Received bypass job request fa7e95e6-efba-4608-beb3-974ff9717fec 18/12/24 11:30:59 INFO driver.JobWrapper: Failed to run job fa7e95e6-efba-4608-beb3-974ff9717fec java.lang.NoClassDefFoundError: org/json/simple/parser/JSONParser at com.oracle.fsgbu.analytics.factory.ServiceImplFactory.extractBlock(ServiceImplFactory.java:109) at com.oracle.fsgbu.analytics.factory.ServiceImplFactory.getServiceImplementation(ServiceImplFactory.java:41) at com.oracle.fsgbu.analytics.endpoint.ServiceFacade.processRequest(ServiceFacade.java:28) at com.oracle.fsgbu.analytics.client.LivyServiceRequestWrapper.call(LivyServiceRequestWrapper.java:33) at com.oracle.fsgbu.analytics.client.LivyServiceRequestWrapper.call(LivyServiceRequestWrapper.java:1) at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:40) at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27) at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42) at org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.JSONParser at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 14 more Thanks, Upasana  ... View more

  Look, at first input-box i could set whole path to script - and that's all. Why we need file-property, and exec-property? ... View more

Yeah, I did, tks 😄 ... View more

Hello @Manyata111   Thanks for posting your query   Could you please check in your appliction log by running below command    #yarn logs -applicationId <applicationID> -appOwner <username_who_triggered_job>    kindly check if you are seeing any error messages  ... View more