Hi,  The "Requested array size exceeds VM limit" means that your code tries to instantiate an array which has more than 2^31-1 elements (~2 billion) which is the max size of an array in Java. You cannot solve this with adding more memory. You need to split the work between executors and not process data on a single JVM (Driver side).   ... View more

wholeTextFiles is also not a scalable solution. https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.SparkContext.wholeTextFiles.html "Small files are preferred, as each file will be loaded fully in memory." ... View more

Hi @Yosieam , Using "collect" method is not recommended as it needs to collect the data to the Spark driver side and as such it needs to fit the whole dataset into the Driver's memory. Please rewrite your code to avoid the "collect" method. ... View more

HI @shivam0408 , Can you clarify what CDH/HDP/CDP version are you using and what is the datatype of the "DATETIME" column?  What is the desired end result of this command? To drop all the partitions? ... View more

Can you review the whole logfile? The above NPE may be just a side effect of another failure before. ... View more

Hi @PCP2 , can you clarify which HDP/CDH/CDP version are you using? Is this a one-off or an intermittent issue or does it always happen?  Is this affecting only a single job? What kind of an action is Oozie trying to launch?   Thanks, Miklos ... View more

Hi @luckes , Please check if your source code file (test.java) has UTF-8 encoding and how are you compiling the class (for example when using Maven you might need to specify to use utf-8 encoding while compiling the classes. These special characters can be easily lost if somewhere the encoding is not set properly. Alternatively you can use the unicode notation \uXXXX to make sure the character is properly understood by java. For example 张 is: https://www.compart.com/en/unicode/U+5F20 so in source code it looks like  statement.setString(2, "\u5f20\u4e09"); Of course it is rare that one needs to hardcode special characters in the source code, usually it is read from a datafile - where you can specify what encoding to use during reading. ... View more