question Re: spark issue after ran the job in Archives of Support Questions (Read Only)

spark issue after ran the job

ashneesharma88 — Fri, 15 Jul 2016 20:16:50 GMT

We have 3 node cluster

each node have 32 GB ram.

But still System going in hung stat after running the job.

Job is doing converting dataframe to csv using com.databricks.csv.

Re: spark issue after ran the job

arunak — Sat, 16 Jul 2016 01:10:00 GMT

could you please post little more information on the job, the submit command etc. What is your data source?

Re: spark issue after ran the job

ashneesharma88 — Mon, 18 Jul 2016 15:20:47 GMT

@Arun A K

This is the command :-

We are reading csv files.

java -cp .:spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:commons-csv-1.1.jar:spark-csv_2.10-1.4.0.jar SparkMainPlain xyz

Re: spark issue after ran the job

ashneesharma88 — Mon, 18 Jul 2016 17:15:55 GMT

Issue is resolved after increasing physical ram of the machine. Now it is working fine. I was running the job on 32 GB ram node and I increased the it to 64 GB and ran same code 3-4 times.

Re: spark issue after ran the job

ashneesharma88 — Mon, 18 Jul 2016 18:02:08 GMT

plz suggest if I can tune my cluster for spark.

Re: spark issue after ran the job

ashneesharma88 — Mon, 18 Jul 2016 20:13:30 GMT

can anyone help me to tune the spark to run same job on 32 GB system. Because I my cluster was 32 GB with 3 node, I think 32 GB per node is enough and free memory was always 20 GB on every node.

Re: spark issue after ran the job

arunak — Mon, 18 Jul 2016 20:45:28 GMT

If spark-csv_2.10-1.4.0.jar is your application, please submit it using spark-submit rather than running it as java application. Could you explain little more on what the application is doing? What is the data source? How do you turn your data into a data frame etc...