About AKR

AKR · ‎01-05-2020

Hi, This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations. Thanks AKR

AKR · ‎01-05-2020

Hi, Hope this below links helps in deciding the Configurations apart from the previous comments https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/ https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-1/ Thanks AKR

AKR · ‎01-05-2020

Hi, if there are more no of files are present in spark history Server, then FS would not be working as expected. In that case, We may need to write a script to delete the old files that are more then 7 days ( or as per your requirement) from the Spark history server location . Thanks Arun

AKR · ‎01-05-2020

Hi, This seems to be a Kerberos authentication problem. is this issue happens only for a SHS2 UI or does it happens for other URLs too? You could do a1. kinit username 2. kinit -kt service.keytab 3. send keytab file along with the submit command Thanks AKR

AKR · ‎01-05-2020

Hi, Whether those links that was provided helped to solve the issue http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/ http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ if it was fine can you please mark this forum as solved. Thanks AKR

AKR · ‎12-10-2019

Hi, Have you tried submitting applications to other queue? Could you tell us does it worked on other queues? Queue.default has reached 99.7% and this may cause issues. Thanks AK

AKR · ‎12-09-2019

Hi, The correct way to pass the multiple configurations is that it should be passed along with the --conf. Ex: spark-submit --conf org.spark.metadata =false --conf spark.driver.memory=10gb Thanks AK

AKR · ‎12-09-2019

Hi, Hope this document will clarify your doubts. This was a tuning document. https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/ Thanks AK

AKR · ‎12-09-2019

Hi, If you are looking for 2.4 on HDP alone means, as mentioned in previous emails, don't expect any new HDP version anytime before the release of the combined new offering Cloudera Data Platform (CDP) sometime in 2020 or thereafter. As an alternative CDH 6.2+ supports spark 2.4 version. Thanks AK

AKR · ‎11-12-2019

Hi, Have you tried passing the Teradata jdbc Driver jar file that with --jar option and whether the issue got solved? Please share the error messages if you are still facing the issue. Thanks AKR

Online	Offline
Last Visited	‎12-18-2024 09:48 AM

Member Since	‎09-20-2018 10:06 PM
Last Visited	‎12-18-2024 09:48 AM
Posts	354

Cloudera Community

Re: Can't start Kafka Service broker

Re: spark.yarn.executor.memoryOverhead

Re: How to decide spark submit configurations

Re: how to delete all application logs from spark ...

Re: Spark2 history server UI not opening in HDP 3....

Re: How to pick number of executors , cores for ea...

Re: YARN application stuck in ACCEPTED state

Re: Spark submit multiple configurations

Re: Yarn memory allocation & utilization

Re: Which version of HDP supports spark 2.4

Re: Unable to connect Spark 2.0 via jdbc to Terada...