About Yuexin Zhang

Yuexin Zhang · ‎06-13-2023

Hi There's similar feature in CDP as the time based scheduling rules in CDH, called "Dynamic Queue Scheduling". But it is Tech Preview in CDP 717 and GA in CDP 718, please refer to below docs: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/yarn-allocate-resources/topics/yarn-dynamic-queue-scheduling.html https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/yarn-allocate-resources/topics/yarn-dynamic-queue-scheduling.html Hope this helps.

Yuexin Zhang · ‎05-13-2019

To be more acturate, technically, CVE-2018-1334 is fixed in CDH 5.14.4. But there's a new issue been found with similar privilege escalation vulnerability, which is CVE-2018-11760. We fixed CVE-2018-11760 in CDH 5.15.1. So with CDH 5.15.1, you won't be affected by these two similar privilege escalation vulnerabilities.

Yuexin Zhang · ‎05-10-2019

CVE-2018-1334 is fixed in CDH 5.15.1 and higher version.

Yuexin Zhang · ‎04-28-2019

You may can achive the goal with YARN node label feature. See the detailed explaination on the following Hortonworks post: https://community.hortonworks.com/articles/72450/node-labels-configuration-on-yarn.html Current Cloudera CDH distribution does not officially support Node Label, we are working on release a uniform version of CDH + HDP. We will have a new Cloudera Data Platform (CDP) release later this year. If you are our subscruption customer, please feel free to contact cloudera support to enqury the state of this feature, also the state of the new CDP release. Thanks.

Yuexin Zhang · ‎04-28-2019

You can use the Spark Action in Oozie to submit any spark applications: https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SparkActionExtension.html#Spark_Action If you are more familar with spark-submit tool, you can try to use oozie shell action as well: https://archive.cloudera.com/cdh5/cdh/5/oozie/DG_ShellActionExtension.html You may need to make sure the spark gateway role is deployed on the oozie server and node manager nodes, so that the runtime env always have the depencies available.

Yuexin Zhang · ‎04-22-2019

The error message shows you don't have a valid leader for the partition you are accessing. In kafka, all read/writes should go through the leader of that partition. You should make sure the topic/partitions have healthy leader first, run: kafka-topics --describe --zookeeper <zk_url, put /chroot if you have any> --topic <topic_name>

Yuexin Zhang · ‎11-11-2018

You may need to increase: yarn.nodemanager.resource.memory-mb yarn.scheduler.maximum-allocation-mb the default one could be too small to launch a default spark executor container ( 1024MB + 512 overhead). You may also want to enable INFO logging for the spark shell to understand what exact error/warn it has: /etc/spark/conf/log4j.properties

Yuexin Zhang · ‎10-17-2018

Hi Continuum ships Anaconda parcel and Cloudera does not have control on which python version it installs. Please use the OS package management tool to install python 3.5 on the servers in the CDH cluster, once that is done, please follow this doc to set python for your pyspark job: https://www.cloudera.com/documentation/enterprise/5-8-x/topics/spark_python.html#spark_python__section_ark_lkn_25

Yuexin Zhang · ‎08-14-2018

Okay, since the process owner is cloudera-scm, one way to fix the issue is adding cloudera-scm user to 'spark' group on all nodes.

Yuexin Zhang · ‎08-14-2018

You may need to make sure the process owner of the Spark2 history server (by default it is spark user as well), belongs to the group "spark". So that the spark2 history server process would be able to read all the spark2 event log files. You can check the process owner with " ps -ef |grep java| grep SPARK2" on the node where spark2 history server runs on.

Online	Offline
Last Visited	‎09-17-2023 10:30 PM

Member Since	‎03-01-2016 05:20 PM
Last Visited	‎09-17-2023 10:30 PM
Posts	537
Kudos received	8

Cloudera Community

Cloudera Community

Re: Time rules on capacity scheduler of YARN

Re: Is it possible to reserve whole nodes for excl...

Re: how to trigger sample spark and pyspark jobs ...

Re: Pyspark issue AttributeError: 'DataFrame' obje...

Re: Solr DataImporter class not found

Re: Time rules on capacity scheduler of YARN

Re: Removing Spark1.6

Re: Removing Spark1.6

Re: Is it possible to reserve whole nodes for excl...

Re: how to trigger sample spark and pyspark jobs ...

Re: kakfa leader not available I have added 1 mor...

Re: Spark-shell in Yarn mode gets stuck

Re: Configuring CDH cluster with Python 3

Re: Unable to see completed application in Spark 2...

Re: Unable to see completed application in Spark 2...