About akv31

akv31 · ‎04-12-2020

Yes, I created a new java.security file with required properties and placed in all nodes in local directory.

akv31 · ‎04-10-2020

Hi Jeremy, Thanks for your response. As suggested I build the jar and issue got resolved. Regards, Arvind

sparkr · ‎10-24-2019

The application might be expecting the log folder to be there in order to generate logs in it. Seems like your problem can be solved by creating the folder in the driver node: /some/path/to/edgeNode/ I hope you also know that you have mentioned the log4j file only for driver program. In order for executors to generate logs you may need to specify the following option in spark-submit "spark.executor.extraJavaOptions=-Dlog4j.configuration=driver_log4j.properties"

_Umesh · ‎06-07-2019

As your intent seems to capture the driver logs in a separate file while executing the app in the cluster mode, make sure that '/some/path/to/edgeNode/' dir is present on all of the NodeManager essentially as in cluster mode the driver will be running in the Yarn app's application master. If you can't make sure that follow a general practice to provide log file path to some pre-existing paths e.g. "/var/log/SparkDriver.log".

Online	Offline
Last Visited	‎04-20-2020 07:25 AM

Member Since	‎11-02-2018 05:18 AM
Last Visited	‎04-20-2020 07:25 AM
Posts	15

Cloudera Community

Re: Enabling SSL/TLSv1 during sqoop import

Re: Envelope error while writing into hive table

Re: Config log4j in Spark

Re: Config log4j in Spark - Driver Logs