question Cannot run multiple lines PySpark in Archives of Support Questions (Read Only)

Cannot run multiple lines PySpark

zsokaevakovacs — Fri, 24 Mar 2017 17:08:05 GMT

Hi,

I am trying to get through the HANDS-ON TOUR OF APACHE SPARK IN 5 MINUTES tutorial with the python interpreter, but when I try to run multiple columns like this part:

%pyspark

myLines=sc.textFile('hdfs://sandbox.hortonworks.com/tmp/Hortonworks')

myLinesFiltered=myLines.filter(lambdax:len(x)>0)

count=myLinesFiltered.count()

print count

I got syntax error at the end of the second line. If I run it line by line it works fine, but if I try to run two lines, I always get syntax error, no matter what I run.

Thanks,

Zsoka

Re: Cannot run multiple lines PySpark

namaheshwari — Fri, 24 Mar 2017 23:23:10 GMT

There is an open Apache bug for this:

PySpark Doesn't Support Multi-Line Statements

https://issues.apache.org/jira/browse/ZEPPELIN-84

Re: Cannot run multiple lines PySpark

yvora — Sat, 25 Mar 2017 04:59:30 GMT

https://issues.apache.org/jira/browse/ZEPPELIN-84 is regarding breaking a statement in multiple lines.

@Zsoka Kovacs, you should be able to run below paragraph. Do not give extra \n in between and make sure there are no extra chars copied at the end of the line.

{code}

%pyspark

myLines=sc.textFile('/tmp/Hortonworks')

myLinesFiltered=myLines.filter(lambdax:len(x)>0)

count=myLinesFiltered.count()

print count

{code}