question Re: Error while submitting PySpark Application through Livy REST API in Support Questions

question Re: Error while submitting PySpark Application through Livy REST API in Support Questions https://community.cloudera.com/t5/Support-Questions/Error-while-submitting-PySpark-Application-through-Livy-REST/m-p/287581#M213125 <a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/71289">@TVGanesh</a> Isn't the Pyspark file expected in hdfs if using YARN instead of LOCAL? What is the configuartion of your livy.conf if you dont have it in place do the following.<LI-CODE lang="markup">{ "pyFiles": ["/user/tvganesh/test1.py"] }</LI-CODE>Copy the template file is rename it by stripping off .template in livy.conf.template.Then make sure that the following configurations are present in it. Make sure that forward slash is present in the end of the path. py files you should add the test1.py to hdfs and point to the hdfs location instead of from file system level since that won't be present for Livy locally.Go to the the livy conf directorycp /usr/hdp/3.1.0.0-78/etc/livy2/conf.dist/conf then copy livy.conf.template to livy.confCheck the below parameters and set them accordingly# What spark master Livy sessions should use. <LI-CODE lang="markup">livy.spark.master = local</LI-CODE> # What spark deploy mode Livy sessions should use. <LI-CODE lang="markup">livy.spark.deploy-mode =</LI-CODE> # Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected # on user request and then livy server classpath automatically. <LI-CODE lang="markup">livy.repl.enable-hive-context =</LI-CODE> # List of local directories from where files are allowed to be added to user sessions. By # default it's empty, meaning users can only reference remote URIs when starting their # sessions. <LI-CODE lang="markup">livy.file.local-dir-whitelist =</LI-CODE> For local execution <LI-CODE lang="markup">livy.spark.master = local livy.file.local-dir-whitelist =/home/tvganesh/ LOCAL</LI-CODE>  For YARN execution <LI-CODE lang="markup">livy.spark.master = yarn livy.file.local-dir-whitelist =/user/tvganesh/ HDFS</LI-CODE> Please do that and revert   Tue, 14 Jan 2020 09:42:54 GMT Shelton 2020-01-14T09:42:54Z