Hi,  The "Requested array size exceeds VM limit" means that your code tries to instantiate an array which has more than 2^31-1 elements (~2 billion) which is the max size of an array in Java. You cannot solve this with adding more memory. You need to split the work between executors and not process data on a single JVM (Driver side).   ... View more

wholeTextFiles is also not a scalable solution. https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.SparkContext.wholeTextFiles.html "Small files are preferred, as each file will be loaded fully in memory." ... View more

Hi @Yosieam , Using "collect" method is not recommended as it needs to collect the data to the Spark driver side and as such it needs to fit the whole dataset into the Driver's memory. Please rewrite your code to avoid the "collect" method. ... View more

Can you review the whole logfile? The above NPE may be just a side effect of another failure before. ... View more

Hi @PCP2 , can you clarify which HDP/CDH/CDP version are you using? Is this a one-off or an intermittent issue or does it always happen?  Is this affecting only a single job? What kind of an action is Oozie trying to launch?   Thanks, Miklos ... View more

Hi @luckes , Please check if your source code file (test.java) has UTF-8 encoding and how are you compiling the class (for example when using Maven you might need to specify to use utf-8 encoding while compiling the classes. These special characters can be easily lost if somewhere the encoding is not set properly. Alternatively you can use the unicode notation \uXXXX to make sure the character is properly understood by java. For example 张 is: https://www.compart.com/en/unicode/U+5F20 so in source code it looks like  statement.setString(2, "\u5f20\u4e09"); Of course it is rare that one needs to hardcode special characters in the source code, usually it is read from a datafile - where you can specify what encoding to use during reading. ... View more

Hi @luckes , thanks for reporting this. Based on your descriptinon yes, it seems the upsert is replaced everywhere to insert by the driver. Please open a support case through MyCloudera support portal to have this routed to the proper team for enhancement. Other ideas: - have you checked if this behavior can be observed with the latest JDBC driver version too? - please check if the "UseNativeQuery=1" helps in the JDBC connection string - does it work if you avoid the "insert" from the column ("insert_time") names, so for example with a "modification_time" column name?   Thank you  Miklos Szurap, Customer Operations Engineer, Cloudera ... View more

Hi Andrea, Great to see that it has been found now and thanks for marking the post as answered. All the best, Miklos  ... View more