Hi We are running decision tree models in spark+R studio sparklyr and and the model works upto a certain # of decision trees. When we increase the # decision trees spark is not scaling and its failing with various issues. At 1 stage its asking for 164 tasks and spark context is never getting 164 tasks and its just struck there. We are running spark in Stand Alone mode on docker containers. It has 14 workers and 1 master. Below are the spark properties we are using. Please suggest any tuning properties which will help us. The Spark stand alone cluster has available memory. Thanks spark.cores.max=82 spark.driver.memory=10g spark.driver.maxResultSize=10g spark.executor.memory=20g spark.executor.cores=2 spark.network.timeout=800s spark.rpc.askTimeout=800s spark.dynamicAllocation.enabled=true spark.shuffle.service.enabled=true spark.dynamicAllocation.minExecutors=10 spark.dynamicAllocation.maxExecutors=500 spark.driver.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps spark.reducer.maxSizeInFlight=1024m spark.shuffle.file.buffer=256k spark.locality.wait=15s spark.files.maxPartitionBytes=268435456 spark.sql.shuffle.partitions=1000 spark.default.parallelism=1000 spark.broadcast.compress=true spark.io.compression.codec=lz4 spark.rdd.compress=true spark.shuffle.compress=true spark.shuffle.spill.compress=true spark.memory.fraction=0.8 spark.memory.storageFraction=0.4 spark.scheduler.minRegisteredResourcesRatio=0.0 spark.scheduler.maxRegisteredResourcesWaitingTime=30s spark.task.maxFailures=5 spark.shuffle.io.maxRetries=3 spark.shuffle.io.preferDirectBufs=true spark.shuffle.io.retryWait=5s ... View more

HI We have a NIFI flow where we are sourcing the social media surveys from an API and writing them to HDFS via PutHDFS processor in with conflict resolution strategy as "append". This flow works if surveys are coming 1 by 1 with a second or 2 seconds delay. We want to test some 20000 surveys all coming at once and "PutHDFS" processor is failing for this scenario. Error is given below: WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.append: failed to create file XXXXXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXXX because current leaseholder is trying to recreate file. org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user@XXXXXXXXX (auth:KERBEROS) cause:org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file XXXXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXX for client XXXXXXXX because current leaseholder is trying to recreate file. INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.append from XXXXXXXX Call#XXXXX Retry#0: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file XXXXXXXXX for DFSClient_NONMAPREDUCE_XXXXXXXX because current leaseholder is trying to recreate file. With these exception all the records are getting blocked in nifi queue to puthdfs and eventually they are not writing into HDFS. Is there a way to configure Nifi PutHDFS processor to accomodate this use-case? Rt now its configured under scheduling as "Timer Driven", Concurrent tasks as "1" and with run schedule as 0 seconds. Yield duration is 1 second. Please suggest. Thanks Srikaran ... View more

Hi I am using FetchHDFS nifi processor which is running fine to fetch the exact HDFS file. I want to get all HDFS files under a directory hence using GetHDFS by keeping the source file option as "True". But I am getting a Kerberos error saying "ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.hadoop.GetHDFS GetHDFS[id=XXXXXXXXXX] Error retrieving file hdfs://XXXXXXXXXXXXXXXXXXXX.0. from HDFS due to java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt): {} java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:311) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:287) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:287) I am wondering why Same Kerberos credentials are working for "FetchHDFS/ListHDFS" but not "GetHDFS". "GetHDFS" need additional setup? Please suggest. Thanks Srikaran ... View more