Member since
12-21-2020
91
Posts
8
Kudos Received
13
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1987 | 08-12-2021 05:16 AM | |
| 2210 | 06-29-2021 06:21 AM | |
| 2664 | 06-16-2021 07:15 AM | |
| 1879 | 06-14-2021 12:08 AM | |
| 6239 | 05-14-2021 06:03 AM |
04-27-2021
12:58 AM
1 Kudo
Hi @ryu , It seems that your repo is not correctly configured. Can you paste the output of "yum list webhcat-tar-hive*"? Also, share the contents of /etc/yum.repos.d/<your repo file>.repo Thanks, Megh
... View more
04-26-2021
01:29 AM
Does the solution here not work for you? Thanks, Megh
... View more
04-24-2021
02:53 AM
1 Kudo
Came across this bug and followed the steps given in this comment. Haven't faced this issue after that. Thanks, Megh
... View more
04-22-2021
06:10 AM
Hello Everyone, I have an external table (Text with Gzip compression) partitioned on a date column. I have created another external table with the same structure but with Parquet format with Snappy compression. I want to copy all the data from the text table to parquet table, and I'm using the following query for testing it out for one day. insert into parquet_table partition(asdt) select * from text_table where partition_col='somedate'; The partition contains almost 100M rows. This query runs for almost half an hour and then results into this error: ERROR : Vertex failed, vertexName=Reducer 2, vertexId=vertex_1619005705788_0004_2_01, diagnostics=[Task failed, taskId=task_1619005705788_0004_2_01_000240, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #1
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:306)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Map_1: Shuffle failed with too many fetch failures and insufficient progress!failureCounts=20, pendingInputs=20, fetcherHealthy=false, reducerProgressedEnough=true, reducerStalled=true
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.isShuffleHealthy(ShuffleScheduler.java:1053)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.copyFailed(ShuffleScheduler.java:794)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:318)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57)
... 7 more
, errorMessage=Shuffle Runner Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #1
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:306)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Map_1: Shuffle failed with too many fetch failures and insufficient progress!failureCounts=20, pendingInputs=20, fetcherHealthy=false, reducerProgressedEnough=true, reducerStalled=true
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.isShuffleHealthy(ShuffleScheduler.java:1053)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.copyFailed(ShuffleScheduler.java:794)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:318)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57)
... 7 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #7
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:306)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Map_1: Shuffle failed with too many fetch failures and insufficient progress!failureCounts=24, pendingInputs=24, fetcherHealthy=false, reducerProgressedEnough=true, reducerStalled=true
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.isShuffleHealthy(ShuffleScheduler.java:1053)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.copyFailed(ShuffleScheduler.java:794)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:318)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:182)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:194)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.callInternal(FetcherOrderedGrouped.java:57)
... 7 more
, errorMessage=Shuffle Runner Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: error in shuffle in Fetcher_O {Map_1} #7
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:306)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:288)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) I'm running this in hive on tez in my CDP 7.1.5 I've disabled the default transactional property and one time insert only properties in Hive on Tez config. I think I'm hitting some bug but unable to find any useful info in the hiveserver/Resourcemanager/metastore/nodemanager logs. Has anybody else encountered this? Thanks, Megh
... View more
Labels:
04-22-2021
03:22 AM
Understood. In our case, the culprit turned out to be 50075 port in firewall. I've added as an answer. In any case, thanks for your support. Thanks, Megh
... View more
04-22-2021
01:19 AM
This Issue was resolved after we opened 50075 port (Datanode WebHDFS port) between the source and destination cluster. I think the error org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error is rather misleading. Once I tried to distcp with the active namenode instead of the nameservice, I was able to get the root cause saying "connection timed out" on port 50075. In the official documentation of Cloudera this port is not mentioned as a requirement. Thanks, Megh
... View more
04-20-2021
12:13 AM
Hi @Shelton Do you mean source and destination hostnames and their IPs? If yes, that I've ensured that they're same on both the clusters. If you indeed mean the nameservice, can you please tell me which IP should it resolve to? Generally the format of /etc/hosts entry is as follows: ip hostname alias but in your reply you've given: <nameservice> IP Is it correct? Thanks, Megh
... View more
04-19-2021
08:37 AM
Hi @slambe , Thanks for your reply. I've followed the same documentation. I've configured the nameservice with the steps mentioned in the destination cluster and before kerberization, I was able to execute the same command without any errors (even with the nameservice). Also, I've checked that the failover hasn't happened on the source cluster. The same namenode is active as it was before. Thanks, Megh
... View more
04-19-2021
08:27 AM
Hi @Shelton , Thanks for your response. With the command you've shared, I'm getting the following error: 21/04/19 20:49:36 ERROR tools.DistCp: Exception encountered
java.io.EOFException: End of File Exception between local host is: "<source-hostname>/<source-ip>"; destination host is: "<destination-namenode>":8020; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:789)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:957)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1686)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1745)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1742)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1757)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2088)
at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:443)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1880)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087) Thanks, Megh
... View more
04-19-2021
06:07 AM
Hello Everyone,
I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).
I'm running the distcp command on the secure cluster like follows:
hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder
It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).
When I looked into the task attempt logs, I find this error:
2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2021-04-19 17:00:31,104 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: [Kind: mapreduce.job, Service: job_1618819500492_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@233795b6), Kind: WEBHDFS delegation, Service: ha-webhdfs:<destination-nameservice>, Ident: (token for megh.vidani: WEBHDFS delegation owner=megh.vidani, renewer=yarn, realUser=, issueDate=1618831821242, maxDate=1619436621242, sequenceNumber=322, masterKeyId=30), Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<destination-nameservice>, Ident: (token for megh.vidani: HDFS_DELEGATION_TOKEN owner=megh.vidani@NPCI.ORG.IN, renewer=yarn, realUser=, issueDate=1618831821344, maxDate=1619436621344, sequenceNumber=323, masterKeyId=30)]
2021-04-19 17:00:31,175 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2021-04-19 17:00:31,293 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data10/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data11/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data12/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data13/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data2/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data3/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data4/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data5/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data7/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data8/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data9/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009
2021-04-19 17:00:31,740 INFO [main] org.apache.hadoop.mapred.YarnChild:
/************************************************************
[system properties]
os.name: Linux
os.version: 3.10.0-1160.24.1.el7.x86_64
java.home: /usr/java/jdk1.8.0_281/jre
java.runtime.version: 1.8.0_281-b09
java.vendor: Oracle Corporation
java.version: 1.8.0_281
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.class.path: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002:/etc/hadoop/conf.cloudera.yarn:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/audience-annotations-0.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jul-to-slf4j-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jdom-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-examples-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-api-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-system-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snakeyaml-1.26.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/spark-2.4.0.7.1.5.0-257-yarn-shuffle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang3-3.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-core-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/forbiddenapis-2.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-protobuf.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ehcache-3.3.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-asn1-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.servlet-api-3.1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.xml.bind-api-2.3.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-annotations-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-identity-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archives-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-scala_2.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-qual-2.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aopalliance-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/stax2-api-3.1.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-api-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/xz-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-json-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-xc-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-cli-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-extras-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-log4j12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gson-2.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/fst-2.50.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/tt-instrumentation-7.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-client-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/lz4-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-admin-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/dnsjava-2.1.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-module-jaxb-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-xml-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.inject-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-xdr-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-config-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/mssql-jdbc-6.2.1.jre7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-core-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jettison-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/asm-5.0.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-simple-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.activation-api-1.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-util-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-databind-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-fs2img-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-yarn-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archive-logs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/failureaccess-1.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr311-api-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-crypto-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-pkix-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-api-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-hdfs-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-keyvault-core-1.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-xml-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-console-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/java-util-1.9.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/accessors-smart-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ram-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-core-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/nimbus-jose-jwt-7.9.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/logredactor-2.0.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpcore-4.4.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-servlet-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-web-proxy-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-recipes-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-mapper-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/paranamer-2.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-smart-2.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-templates-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-compress-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/woodstox-core-5.0.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aliyun-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guava-28.1-jre.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-common-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-rumen-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpclient-4.5.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ecs-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-generator.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-security-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-tests-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-plugins-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-server-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-intg-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-json-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcprov-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-datajoin-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-client-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-beanutils-1.9.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-shell.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-codec-1.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-swing-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jcip-annotations-1.0-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kafka-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-shuffle-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-guice-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-auth-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/animal-sniffer-annotations-1.18.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-io-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-format-structures.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-plugin-classloader-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-ozone-filesystem-hadoop3-1.0.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-http-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/re2j-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-server-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-nativetask-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-servlet-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-sls-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-simplekdc-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-groovysh-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-gridmix-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/j2objc-annotations-1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-streaming-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-distcp-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-jackson.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okio-1.6.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-net-3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-1.2.17-cloudera1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-sdk-oss-3.4.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-column.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okhttp-2.7.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration2-2.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-encoding.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsch-0.1.54.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snappy-java-1.1.7.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-client-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-servlet-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/htrace-core4-4.1.0-incubating.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kms-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-api-2.2.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-core-3.4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/metrics-core-3.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/event-publish-7.2.4-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/google-extensions-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-sts-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-nodemanager-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aws-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-registry-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/objenesis-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-uploader-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-compat-qual-2.5.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-data-lake-store-sdk-2.3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-io-2.5.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-all-4.1.51.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-datalake-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-ajax-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/wildfly-openssl-1.0.7.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-json-provider-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-thrift.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration-1.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-openstack-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-webapp-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-storage-7.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-httpfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-resourceestimator-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-base-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-hook-abfs-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcpkix-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-collections-3.2.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/HikariCP-java7-2.4.12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-slf4j-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro-1.8.2.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jna-5.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/swagger-annotations-1.5.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-server-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gcs-connector-2.1.2.7.1.5.0-257-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/token-provider-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsp-api-2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-framework-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/leveldbjni-all-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-i18n.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aws-java-sdk-bundle-1.11.563.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-3.10.6.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ojalgo-43.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-distributedshell-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-io-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-cloud-bindings.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-app-3.1.1.7.1.5.0-257.jar::job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/job.jar
java.io.tmpdir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/tmp
user.dir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002
user.name: megh.vidani
************************************************************/
2021-04-19 17:00:31,741 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,056 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-04-19 17:00:32,235 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: /user/megh.vidani/.staging/_distcp-524677260/fileList.seq:0+500
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,822 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder to webhdfs://<destination-nameservice>/path/to/source/folder
2021-04-19 17:00:32,915 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:00:32,972 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:02:33,218 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:02:33,305 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:03:33,350 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 2 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 1441ms.
2021-04-19 17:03:34,874 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 3 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:04:34,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 4 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 7259ms.
2021-04-19 17:04:42,257 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 5 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:05:42,349 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 6 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 17414ms.
2021-04-19 17:05:59,845 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 7 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:06:59,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 8 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 15922ms.
2021-04-19 17:07:15,921 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 9 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:08:16,013 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 10 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 19586ms.
2021-04-19 17:08:35,740 ERROR [main] org.apache.hadoop.tools.util.RetriableCommand: Failure in Retriable command: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:310)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:256)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:210)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:561)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$900(WebHdfsFileSystem.java:144)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:637)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:675)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:671)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.read(WebHdfsFileSystem.java:2335)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:2152)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:91)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:308)
... 16 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:90)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:522)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$300(WebHdfsFileSystem.java:144)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:723)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.connect(WebHdfsFileSystem.java:2386)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:814)
... 27 more
2021-04-19 17:08:37,073 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,094 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,248 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,333 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
The same thing was working before enabling Kerberos.
Also, on the source cluster end, both active and standby namenode are working properly.
Can anybody point out any other place I should look into?
Thanks,
Megh
... View more