About RangaReddy

RangaReddy · ‎10-24-2023

I think you need to verify the yarn and spark resources are configured properly. If yes then go and check from spark ui, it will show driver memory and executor memory. It is coming as expected then safely you can ignore it.

RangaReddy · ‎10-24-2023

Hi @adhishankarit Spark applications will run using spark engine and not a tez engine unlike hive. You no need to set any engine from spark side. If you want to run hive queries then you can set engines like Tez, Spark, MR

RangaReddy · ‎10-24-2023

I think you don't have sufficient resources to run the job for queue root.hdfs. Verify is there any pending running jobs/application in the root.hdfs queue from Resource Manager UI. If it is running kill those if it is not required. And also verify from spark side you have given less resource to test it.

Fawze · ‎10-19-2023

The node must have a NodeManager role to take part of the processing, Spark gateway, and Yarn Gateway

RangaReddy · ‎10-17-2023

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs. The below table provides you links to features released in each Spark 3 minor version: Spark Version Documentation Link Release Date Spark 3.0.0 https://spark.apache.org/docs/3.0.0/ 2020-06-16 Spark 3.0.1 https://spark.apache.org/docs/3.0.1/ 2020-11-05 Spark 3.0.2 https://spark.apache.org/docs/3.0.2/ 2021-02-19 Spark 3.0.3 https://spark.apache.org/docs/3.0.3/ 2022-06-17 Spark 3.1.1 https://spark.apache.org/docs/3.1.1/ 2021-03-02 Spark 3.1.2 https://spark.apache.org/docs/3.1.2/ 2022-06-17 Spark 3.1.3 https://spark.apache.org/docs/3.1.3/ 2022-06-17 Spark 3.2.0 https://spark.apache.org/docs/3.2.0/ 2021-10-13 Spark 3.2.1 https://spark.apache.org/docs/3.2.1/ 2022-06-17 Spark 3.2.2 https://spark.apache.org/docs/3.2.2/ 2022-07-15 Spark 3.2.3 https://spark.apache.org/docs/3.2.3/ 2022-11-28 Spark 3.2.4 https://spark.apache.org/docs/3.2.4/ 2023-04-13 Spark 3.3.0 https://spark.apache.org/docs/3.3.0/ 2022-06-17 Spark 3.3.1 https://spark.apache.org/docs/3.3.1/ 2022-10-25 Spark 3.3.2 https://spark.apache.org/docs/3.3.2/ 2023-02-15 Spark 3.3.3 https://spark.apache.org/docs/3.3.3/ 2023-08-21 Spark 3.4.0 https://spark.apache.org/docs/3.4.0/ 2023-04-13 Spark 3.4.1 https://spark.apache.org/docs/3.4.1/ 2023-06-23 Spark 3.5.0 https://spark.apache.org/docs/3.5.0/ 2023-09-13

RangaReddy · ‎10-05-2023

Hi @hegdemahendra CDP onwards Spark Thrift Server is not supported. You can try the following links maybe it is useful for you: https://stackoverflow.com/questions/29227949/how-to-implement-spark-sql-pagination-query

RangaReddy · ‎10-05-2023

Hi Team, Livy3 with Zeppelin Integration is not yet supported. To use Spark3, you need to install python3 and needs to add the following parameters: PYSPARK3_PYTHON spark.yarn.appMasterEnv.PYSPARK3_PYTHON Reference: https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/running-spark-applications/topics/spark-python-path-variables-livy.html

RangaReddy · ‎10-02-2023

Hi @pranav007 The required setup is little bit complex. You can try copy the core-site.xml, hdfs-site.xml, yarn-site.xml, hive-site.xml, mapred-site.xml, krb_configuration files to resource folder. In the spark code, you need to add a two parameters i.e spark.driver.extraJavaOptions and spark.executor.extraJavaOptionsby specifiing the krb_file location. --conf spark.driver.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \ --conf spark.executor.extraJavaOptions="-Djava.security.krb5.conf=KRB5_PATH" \

RangaReddy · ‎10-02-2023

There are two solutions you can try. 1. Create one more shell operator and perform kinit and after that submit your spark 2. Pass the keytab and principal to the spark-submit

VidyaSargur · ‎09-26-2023

@Emanuel_MXN, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Online	Offline
Last Visited	‎08-29-2024 03:41 AM

Member Since	‎06-02-2020 05:25 AM
Last Visited	‎08-29-2024 03:41 AM
Posts	331
Kudos received	66

Cloudera Community

Re: Icebreg on CDP private cloud 7.1.9

Re: How to set default time zone/local time for Sp...

Re: Load Iceberg Table on PowerBI Desktop

Re: NoClassDefFoundError due to Incompatible Spark...

Re: Creating Iceberg table

Re: Getting Warning: Maximum heap size rounded up ...

Re: Program executing Tez engine instead of Spark

Re: Spark application stuck at ACCEPTED state at s...

Re: How to make Yarn deploy resources to new added...

Feature Releases of Apache Spark 3 minor versions

Re: Spark Thrift server JDBC sql with pagination

Re: livy-on-spark3 interpreter died

Re: Connecting local spark with Hive with kerberos

Re: Issue running spark jobs with Airflow

Re: Delete application logs from spark history spe...