thanks a lot! ... View more

Thank you Sean for the answer, I actually misspoke and just need to upgrade to Spark 1.3 (I'm using Spark 1.2).    I've been trying to use this guide:   https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf   But I am still only getting Spark 1.2, do you have any suggestions on how I can use this guide to get Spark 1.3? ... View more

Thanks for the explanation. And thanks for tolerating me extending the original question.  This issue can be closed. ... View more

Using the class path precedence is not the correct solution for all cases. A solution that will work in all cases is to use shading for the classes that you have modified versions of (use maven or gradle to do that).   In your case you need to shade the parquet classes that you have modified when you package the jar. Be careful if you change classes like parquet: you could ed up with files that are only readable with your code and force you to keep packaging it with all jobs. That could cause problems later if you decide to use a different method to access the files.   Wilfred ... View more

Can you check the path separator? I would have expected that on windows you would use the \ and not the / can you also explain how you start PySpark: do you use the cmd scripts or under cygwin?   BTW: we do not test windows as a client, so you might see a known issue   Wilfred ... View more

Thank you for response! It solves my troubles. ... View more

Thank you for the explanation! BTW, where do you see the 'Comments'?? I don't have that!   << The comment on the setting in CM should have explained it for you:       Maximum size in bytes for the Java Process heap memory. Passed to Java -Xmx.>>   ... View more