Hi, I am using sandbox HDP-3.0.1.0 Ambari Version 2.7.1.0, My hadoop distcp command is stuck, not sure what is happening; [root@sandbox-hdp bin]# hadoop distcp hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/foodmart.db/currency hdfs://sandbox-hdp.hortonworks.com:8020/sandbox/tutorial-f iles/20575/ ERROR: Tools helper /usr/hdp/3.0.1.0-187/hadoop/libexec/tools/hadoop-distcp.sh was not found. 23/09/22 13:37:35 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false , useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0, copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE ], atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/foodmart.db/currency], targetPath=hdfs://s andbox-hdp.hortonworks.com:8020/sandbox/tutorial-files/20575, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192, verboseLog=false}, sourcePaths=[hdfs://sandbox-hdp.hortonworks.com:802 0/warehouse/tablespace/managed/hive/foodmart.db/currency], targetPathExists=true, preserveRawXattrsfalse 23/09/22 13:37:35 INFO client.RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.3:8050 23/09/22 13:37:35 INFO client.AHSProxy: Connecting to Application History server at sandbox-hdp.hortonworks.com/172.18.0.3:10200 23/09/22 13:37:35 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 3; dirCnt = 2 23/09/22 13:37:35 INFO tools.SimpleCopyListing: Build file listing completed. 23/09/22 13:37:35 INFO tools.DistCp: Number of paths in the copy list: 3 23/09/22 13:37:35 INFO tools.DistCp: Number of paths in the copy list: 3 23/09/22 13:37:35 INFO client.RMProxy: Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.18.0.3:8050 23/09/22 13:37:35 INFO client.AHSProxy: Connecting to Application History server at sandbox-hdp.hortonworks.com/172.18.0.3:10200 23/09/22 13:37:35 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1695389098784_0002 23/09/22 13:37:35 INFO mapreduce.JobSubmitter: number of splits:2 23/09/22 13:37:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1695389098784_0002 23/09/22 13:37:36 INFO mapreduce.JobSubmitter: Executing with tokens: [] 23/09/22 13:37:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.0.1.0-187/0/resource-types.xml 23/09/22 13:37:36 INFO impl.YarnClientImpl: Submitted application application_1695389098784_0002 23/09/22 13:37:36 INFO mapreduce.Job: The url to track the job: http://sandbox-hdp.hortonworks.com:8088/proxy/application_1695389098784_0002/                                                   23/09/22 13:37:36 INFO tools.DistCp: DistCp job-id: job_1695389098784_0002 23/09/22 13:37:36 INFO mapreduce.Job: Running job: job_1695389098784_0002 23/09/22 13:42:28 INFO mapreduce.Job: Job job_1695389098784_0002 running in uber mode : false 23/09/22 13:42:28 INFO mapreduce.Job: map 0% reduce 0%     ... View more

Hi All,   I am running spark (livy) from nifi. Currently using HDP 3.0/HDF 3.1 the CDA version.   I was running spark livy through nifi. I could see that nifi flow was completed successfully. But there were several livy-session keep executing in Yarn. I've tried to kill some of them but they were re-appearing in Yarn.  This is causing issue that these livy sessions are consuming CPUs even though I am not running any spark application through livy-nifi.   I am not sure how to resolve this issue. Needed your help here. ... View more