About Yuexin Zhang

VidyaSargur · ‎03-03-2024

@yanseoi, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.

krunal_lathiya · ‎01-02-2024

To save a DataFrame as a text file in PySpark, you need to convert it to an RDD first, or use DataFrame writer functions. Using DataFrame writer: df.write.format("text").save("path_to_output_directory") Converting to RDD and then using saveAsTextFile rdd = df.rdd.map(lambda row: str(row)) rdd.saveAsTextFile("path_to_output_directory")

DianaTorres · ‎10-30-2023

@AK- Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

HadoopHero · ‎06-14-2023

Hi Yuexin, you have been very helpful. Unfortunately, if I wanted to use "Dynamic Queue Scheduling" in CDP 717 at the moment, I would not have the guarantee to solve any problems via Cloudera support. In fact, it is not recommended to use it in production. Thank you very much

VidyaSargur · ‎03-29-2021

@ishika as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

AKR · ‎01-05-2020

Hi, This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations. Thanks AKR

AKR · ‎09-04-2019

Hi, Check for the total no of applications in the Application history path, if the total no of files is more try to increase the heap size and look whether it works. Alternatively look for the spark history server logs too for any errors. Thanks AKR

PatliGalli · ‎05-28-2019

Hi Sara, I run vulnerability scans and our scanner picking up Spark 1.6 banner from the following path- for CVE-2018-8024 vulnerability, you did mention this vulnerability doesn't affect SPark 1.6 but didn't give detail reasons. This is where Qualys picksup the banner- /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/spark/conf/spark-env.sh: line 75: /usr/appl/cloudera/java/jdk1.8.0_162: is a directory Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ Also we have 2 versions of SPARK running- do you really need version 1.6 to run v 2.3.0 Can you please help, advise.

Harsh J · ‎05-07-2019

Depends on what you mean by 'storage locations'. If you mean "can other apps use HDFS?" then the answer is yes, as HDFS is an independent system unrelated to YARN and has its own access and control mechanisms not governed by a YARN scheduler. If you mean "can other apps use the scratch space on NM nodes" then the answer is no, as only local containers get to use that. If you're looking to strictly split both storage and compute, as opposed to just some form of compute, then it may be better to divide up the cluster entirely.

eyalb · ‎05-07-2019

Hi MKay, As mentioned in my previous posts the Anaconda parcel for CDH comes only with Python 2.7 and I could find a free way to get a parcel with Python 3+. We ended up manually installing the different Python versions we needed by keeping different virtual envs for different Python versions. We executed the following procedure to install python 3.5: yum install python-pip curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" python get-pip.py pip install virtualenv yum install -y https://centos7.iuscommunity.org/ius-release.rpm yum install -y python35u python35u-libs python35u-devel python35u-pip mkdir -p /opt/venv35 cd /opt/venv35 virtualenv venv35 -p python3.5 source venv35/bin/activate Best, Eyal

Online	Offline
Last Visited	‎04-19-2024 04:40 AM

Member Since	‎03-01-2016 05:20 PM
Last Visited	‎04-19-2024 04:40 AM
Posts	590
Kudos received	9

Cloudera Community

Re: Incompatible Kafka Version

Re: CDS 3.3 support in CDP 7.1.7

Re: Time rules on capacity scheduler of YARN

Re: Is it possible to reserve whole nodes for excl...

Re: how to trigger sample spark and pyspark jobs ...

Re: Incompatible Kafka Version

Re: Pyspark issue AttributeError: 'DataFrame' obje...

Re: CDS 3.3 support in CDP 7.1.7

Re: Time rules on capacity scheduler of YARN

Re: Spark job fails in cluster mode.

Re: spark.yarn.executor.memoryOverhead

Re: Unable to see completed application in Spark 2...

Re: Removing Spark1.6

Re: Is it possible to reserve whole nodes for excl...

Re: Configuring CDH cluster with Python 3