question Re: PySpark on Zeppelin in sandbox is not loading data in Archives of Support Questions (Read Only)

PySpark on Zeppelin in sandbox is not loading data

azeltov — Wed, 11 Nov 2015 11:46:37 GMT

If I execute this code from zeppelin:

%pyspark base_rdd = sc.textFile("/tmp/philadelphia-crime-data-2015-ytd.csv")
base_rdd.take(10)

I am not getting any results back, if U execute from pyspark CLI same code i get valid data. Note: I am running pyspark in local mode in CLI, not in Yarn mode.

Zeppelin is not returning any errors, and no errors in log files. I am using HDP 2.3.2 sandbox.

Same code works using scala works in zeppelin and in cli

Looks like a bug in zeppelin with pyspark?

Re: PySpark on Zeppelin in sandbox is not loading data

azeltov — Wed, 11 Nov 2015 21:42:11 GMT

@Ali Bajwa @Neeraj have u encountered this issue with Pyspark and zeppelin?

Re: PySpark on Zeppelin in sandbox is not loading data

gbraccialli3 — Thu, 12 Nov 2015 02:08:22 GMT

@azeltov@hortonworks.com

I think issues are:

1- you have to use file:// for local files

2- using pyspark, you have to use print before

see example below (working for me):

%pyspark 
base_rdd = sc.textFile("file:///usr/hdp/current/spark-client/data/mllib/sample_libsvm_data.txt")
print base_rdd.count()
print base_rdd.take(3)

Re: PySpark on Zeppelin in sandbox is not loading data

abajwa — Thu, 12 Nov 2015 02:17:44 GMT

Yup I have done similar with pyspark in Zeppelin as well so should work

Re: PySpark on Zeppelin in sandbox is not loading data

azeltov — Thu, 12 Nov 2015 02:36:59 GMT

I had to give it a fully qualified hdfs:// URI in pyspark for me to work