https://community.cloudera.com/t5/Support-Questions/Spark-error-org-apache-spark-SparkException-Failed-to-get/m-p/230881#M192728 <P><A rel="user" href="https://community.cloudera.com/users/18482/hugofelix.html" nodeid="18482">@Hugo Felix</A></P><P><A rel="user" href="https://community.cloudera.com/users/18482/hugofelix.html" nodeid="18482"></A>It appears you are using Spark version lower than 2.1. </P><P>In your code you have the following line:</P><PRE>val sc =newSparkConf().setMaster("local[2]").setAppName("tweets").set("spark.cleaner.ttl","2000")</PRE><P>spark.cleaner.ttl basically triggers a cleanup after the time ( "2000") that you specify.</P><P>From the official <A href="https://spark.apache.org/docs/1.6.1/configuration.html">Spark 1.6 documentation</A> :</P><P>spark.cleaner.ttl - Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.). Periodic cleanups will ensure that metadata older than this duration will be forgotten. This is useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming applications). Note that any RDD that persists in memory for more than this duration will be cleared as well.</P><P>Default is infinite.</P><P>In your case, it is quite possible that the cleanup is being triggered even before your job finishes.</P><P>Increase the value and try again.</P><P>Refer this <A href="https://issues.apache.org/jira/browse/SPARK-5594">JIRA</A> for an existing discussion to get more insight.</P> Wed, 05 Jul 2017 22:11:17 GMT dineshc 2017-07-05T22:11:17Z