@MarinaM, Did the response help resolve your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

@yanseoi, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

To save a DataFrame as a text file in PySpark, you need to convert it to an RDD first, or use DataFrame writer functions. Using DataFrame writer: df.write.format("text").save("path_to_output_directory") Converting to RDD and then using saveAsTextFile rdd = df.rdd.map(lambda row: str(row)) rdd.saveAsTextFile("path_to_output_directory") ... View more

@AK- Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks. ... View more

Hi Yuexin, you have been very helpful. Unfortunately, if I wanted to use "Dynamic Queue Scheduling" in CDP 717 at the moment, I would not have the guarantee to solve any problems via Cloudera support. In fact, it is not recommended to use it in production. Thank you very much ... View more

@ishika as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

Hi,   This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations.   Thanks AKR ... View more

Hi,   Check for the total no of applications in the Application history path, if the total no of files is more try to increase the heap size and look whether it works. Alternatively look for the spark history server logs too for any errors.   Thanks AKR ... View more

Hi Sara, I run vulnerability scans and our scanner picking up Spark 1.6 banner from the following path- for CVE-2018-8024 vulnerability, you did mention this vulnerability doesn't affect SPark 1.6 but didn't give detail reasons. This is where Qualys picksup the banner- /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/spark/conf/spark-env.sh: line 75: /usr/appl/cloudera/java/jdk1.8.0_162: is a directory Welcome to ____              __ / __/__  ___ _____/ /__ _\ \/ _ \/ _ `/ __/  '_/ /___/ .__/\_,_/_/ /_/\_\   version 1.6.0 /_/   Also we have 2 versions of SPARK running- do you really need version 1.6 to run v 2.3.0 Can you please help, advise.   ... View more

Depends on what you mean by 'storage locations'. If you mean "can other apps use HDFS?" then the answer is yes, as HDFS is an independent system unrelated to YARN and has its own access and control mechanisms not governed by a YARN scheduler. If you mean "can other apps use the scratch space on NM nodes" then the answer is no, as only local containers get to use that. If you're looking to strictly split both storage and compute, as opposed to just some form of compute, then it may be better to divide up the cluster entirely. ... View more