About senthiljdpm

senthiljdpm · ‎09-21-2018

Thanks for your reply. What you said is something i tried and it works but i wanted to know how --files argument works by sending the file across the cluster. When I tried by referring json file from home directory, it works when i submitted the job in the cluster on yarn client mode. Can you please let me know how to use --files option to acheive the same thing?

senthiljdpm · ‎09-20-2018

Hi @Damien Saillard, Thanks for your reply. What you said is something i tried and it works but i wanted to know how --files argument works by sending the file across the cluster. When I tried by referring json file from home directory, it works when i submitted the job in the cluster on yarn client mode. Can you please let me know how to use --files option to acheive the same thing?

senthiljdpm · ‎09-19-2018

I am running a PySpark job in Spark 2.3 cluster with the following command. spark-submit --deploy-mode cluster --master yarn --files ETLConfig.json PySpark_ETL_Job_v0.2.py ETLConfig.json has a parameter passed to the PySpark script and I am referring this config json file in the main block as below:- configFilePath = os.path.join(SparkFiles.getRootDirectory(), 'ETLConfig.json') with open(configFilePath, 'r') as configFile: configDict = json.load(configFile) But, the command throws the following error. No such file or directory: u'/tmp/spark-7dbe9acd-8b02-403a-987d-3accfc881a98/userFiles-4df4-5460-bd9c-4946-b289-6433-drgs/ETLConfig.json' May i know what's wrong with my script? I also tried with SparkFiles.get() command but it also didn't work.

Online	Offline
Last Visited	‎09-21-2018 02:06 PM

Member Since	‎09-18-2018 01:52 PM
Last Visited	‎09-21-2018 02:06 PM
Posts	3

Cloudera Community

Re: PySpark spark-submit error when using --files ...

Re: PySpark spark-submit error when using --files ...

PySpark spark-submit error when using --files argu...