https://community.cloudera.com/t5/Support-Questions/Fetch-distinct-values-of-a-column-in-Dataframe-using-Spark/m-p/174636#M136899 <P>Yes i have 1 fat jar that has all the dependencies. The thing is when i use <STRONG>collect() </STRONG>in my below code it works on yarn-cluster. <STRONG>But using collect() removes parallelism on further operations, so i don want to use collect</STRONG>.</P><P>Without using collect() statement i get "java.lang.NoClassDefFoundError" exception as mentioned above. <STRONG>The below code on local mode works fine without collect statement.</STRONG></P><P>Please help me understand this behavior.</P><PRE> preProcessedDataFrame.registerTempTable("tApplication") preProcessedDataFrame.select(ApplicationId).distinct().<STRONG>collect()</STRONG>.foreach(record =&gt; { val applicationId = record.getAs[String](ApplicationId) val selectedApplicationDataFrame = sqlContext.sql("SELECT * FROM tApplication WHERE ApplicationId = " + applicationId) logger.info("selectedApplicationId: " + applicationId) //DO FURTHER PROCESSSING on selectedApplicationDataFrame..... }) </PRE> Tue, 16 Aug 2016 01:51:37 GMT kaz_narasimhan 2016-08-16T01:51:37Z