Support Questions
Find answers, ask questions, and share your expertise

Error saveAsTextFile with pyspark

Error saveAsTextFile with pyspark

Super Collaborator

I have this error from pyspark, when I type this, any suggestion?

counts.saveAsTextFile("file:///tmp/JUPYTERR_out7")
Caused by: java.io.IOException: Failed to rename DeprecatedRawLocalFileStatus{path=file:/tmp/JUPYTERR_out7/_temporary/0/task_20181121142733_0009_m_000001/part-00001; isDirectory=false; length=6448; replication=1; blocksize=33554432; modification_time=1542806853000; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false} to file:/tmp/JUPYTERR_out7/part-00001
2 REPLIES 2

Re: Error saveAsTextFile with pyspark

Contributor

Can you try writing it out to hdfs location where you have write permissions ?

Re: Error saveAsTextFile with pyspark

Super Collaborator

its work from pyspark shell but no from jupyter with pyspark kernel.