Hi, I am trying to execute a python file which is stored in hdfs using livy-server. However I am getting an error as Only local python files supported. Host: 10.140.178.24 Port:8999 hadoop fs -ls /hp
-rw-r--r-- 3 root hdfs 1613 2017-03-15 12:44 /hp/pi.py I executed the curl command with the above path for the python file curl -X POST --data '{"file": "/hp/pi.py"}' -H "Content-Type: application/json" 10.140.178.24:8999/batches
{"id":12,"state":"running","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":[]}
However when I look into the logs, i get only local python files are supported. curl 10.140.178.24:8999/batches/12/log | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2486 100 2486 0 0 258k 0 --:--:-- --:--:-- --:--:-- 269k
{
"from": 0,
"id": 13,
"log": [
"Error: Only local python files are supported: Parsed arguments:",
" master local",
" deployMode client",
" executorMemory null",
" executorCores null",
" totalExecutorCores null",
" propertiesFile /usr/hdp/current/spark-thriftserver/conf/spark-defaults.conf",
" driverMemory null",
" driverCores null",
" driverExtraClassPath null",
" driverExtraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64",
" driverExtraJavaOptions null",
" supervise false",
" queue null",
" numExecutors null",
" files null",
" pyFiles null",
" archives null",
" mainClass null",
" primaryResource hdfs://slave0.acme.com:8020/home/ec2-user/livy/pi.py",
" name pi.py",
" childArgs []",
" jars null",
" packages null",
" packagesExclusions null",
" repositories null",
" verbose false",
"",
"Spark properties used, including those specified through",
" --conf and those from the properties file /usr/hdp/current/spark-thriftserver/conf/spark-defaults.conf:",
" spark.yarn.queue -> default",
" spark.history.kerberos.principal -> none",
" spark.executor.extraLibraryPath -> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64",
" spark.yarn.max.executor.failures -> 3",
" spark.driver.extraLibraryPath -> /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64",
" spark.yarn.historyServer.address -> slave0.acme.com:18080",
" spark.eventLog.enabled -> true",
" spark.history.ui.port -> 18080",
" spark.history.provider -> org.apache.spark.deploy.history.FsHistoryProvider",
" spark.history.fs.logDirectory -> hdfs:///spark-history",
" spark.yarn.submit.file.replication -> 3",
" spark.yarn.scheduler.heartbeat.interval-ms -> 5000",
" spark.yarn.executor.memoryOverhead -> 384",
" spark.yarn.containerLauncherMaxThreads -> 25",
" spark.yarn.driver.memoryOverhead -> 384",
" spark.history.kerberos.keytab -> none",
" spark.eventLog.dir -> hdfs:///spark-history",
" spark.yarn.preserve.staging.files -> false",
" spark.master -> local",
"",
" .primaryResource",
"Run with --help for usage help or --verbose for debug output"
],
"total": 52
I am not sure why it is pointing to the local mode.I have read other posts and I have to set spark master as yarn-cluster but I am not sure where and how to set it. Kindly if someone can let me know how to resolve the above issue, it would be great. Any help would be appreciated
... View more
