Solved
Distcp is not working after enabling Kerberos
Hello Everyone,
I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).
I'm running the distcp command on the secure cluster like follows:
hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder
It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).
When I looked into the task attempt logs, I find this error:
2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2021-04-19 17:00:31,104 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: [Kind: mapreduce.job, Service: job_1618819500492_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@233795b6), Kind: WEBHDFS delegation, Service: ha-webhdfs:<destination-nameservice>, Ident: (token for megh.vidani: WEBHDFS delegation owner=megh.vidani, renewer=yarn, realUser=, issueDate=1618831821242, maxDate=1619436621242, sequenceNumber=322, masterKeyId=30), Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<destination-nameservice>, Ident: (token for megh.vidani: HDFS_DELEGATION_TOKEN owner=megh.vidani@NPCI.ORG.IN, renewer=yarn, realUser=, issueDate=1618831821344, maxDate=1619436621344, sequenceNumber=323, masterKeyId=30)]
2021-04-19 17:00:31,175 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2021-04-19 17:00:31,293 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data10/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data11/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data12/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data13/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data2/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data3/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data4/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data5/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data7/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data8/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data9/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009
2021-04-19 17:00:31,740 INFO [main] org.apache.hadoop.mapred.YarnChild:
/************************************************************
[system properties]
os.name: Linux
os.version: 3.10.0-1160.24.1.el7.x86_64
java.home: /usr/java/jdk1.8.0_281/jre
java.runtime.version: 1.8.0_281-b09
java.vendor: Oracle Corporation
java.version: 1.8.0_281
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.class.path: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002:/etc/hadoop/conf.cloudera.yarn:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/audience-annotations-0.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jul-to-slf4j-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jdom-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-examples-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-api-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-system-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snakeyaml-1.26.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/spark-2.4.0.7.1.5.0-257-yarn-shuffle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang3-3.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-core-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/forbiddenapis-2.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-protobuf.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ehcache-3.3.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-asn1-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.servlet-api-3.1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.xml.bind-api-2.3.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-annotations-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-identity-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archives-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-scala_2.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-qual-2.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aopalliance-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/stax2-api-3.1.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-api-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/xz-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-json-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-xc-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-cli-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-extras-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-log4j12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gson-2.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/fst-2.50.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/tt-instrumentation-7.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-client-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/lz4-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-admin-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/dnsjava-2.1.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-module-jaxb-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-xml-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.inject-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-xdr-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-config-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/mssql-jdbc-6.2.1.jre7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-core-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jettison-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/asm-5.0.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-simple-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.activation-api-1.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-util-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-databind-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-fs2img-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-yarn-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archive-logs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/failureaccess-1.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr311-api-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-crypto-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-pkix-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-api-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-hdfs-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-keyvault-core-1.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-xml-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-console-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/java-util-1.9.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/accessors-smart-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ram-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-core-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/nimbus-jose-jwt-7.9.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/logredactor-2.0.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpcore-4.4.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-servlet-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-web-proxy-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-recipes-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-mapper-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/paranamer-2.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-smart-2.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-templates-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-compress-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/woodstox-core-5.0.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aliyun-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guava-28.1-jre.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-common-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-rumen-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpclient-4.5.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ecs-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-generator.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-security-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-tests-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-plugins-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-server-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-intg-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-json-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcprov-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-datajoin-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-client-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-beanutils-1.9.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-shell.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-codec-1.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-swing-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jcip-annotations-1.0-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kafka-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-shuffle-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-guice-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-auth-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/animal-sniffer-annotations-1.18.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-io-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-format-structures.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-plugin-classloader-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-ozone-filesystem-hadoop3-1.0.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-http-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/re2j-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-server-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-nativetask-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-servlet-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-sls-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-simplekdc-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-groovysh-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-gridmix-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/j2objc-annotations-1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-streaming-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-distcp-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-jackson.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okio-1.6.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-net-3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-1.2.17-cloudera1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-sdk-oss-3.4.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-column.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okhttp-2.7.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration2-2.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-encoding.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsch-0.1.54.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snappy-java-1.1.7.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-client-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-servlet-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/htrace-core4-4.1.0-incubating.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kms-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-api-2.2.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-core-3.4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/metrics-core-3.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/event-publish-7.2.4-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/google-extensions-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-sts-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-nodemanager-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aws-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-registry-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/objenesis-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-uploader-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-compat-qual-2.5.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-data-lake-store-sdk-2.3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-io-2.5.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-all-4.1.51.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-datalake-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-ajax-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/wildfly-openssl-1.0.7.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-json-provider-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-thrift.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration-1.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-openstack-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-webapp-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-storage-7.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-httpfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-resourceestimator-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-base-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-hook-abfs-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcpkix-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-collections-3.2.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/HikariCP-java7-2.4.12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-slf4j-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro-1.8.2.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jna-5.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/swagger-annotations-1.5.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-server-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gcs-connector-2.1.2.7.1.5.0-257-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/token-provider-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsp-api-2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-framework-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/leveldbjni-all-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-i18n.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aws-java-sdk-bundle-1.11.563.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-3.10.6.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ojalgo-43.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-distributedshell-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-io-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-cloud-bindings.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-app-3.1.1.7.1.5.0-257.jar::job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/job.jar
java.io.tmpdir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/tmp
user.dir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002
user.name: megh.vidani
************************************************************/
2021-04-19 17:00:31,741 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,056 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2021-04-19 17:00:32,235 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: /user/megh.vidani/.staging/_distcp-524677260/fileList.seq:0+500
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,822 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder to webhdfs://<destination-nameservice>/path/to/source/folder
2021-04-19 17:00:32,915 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:00:32,972 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:02:33,218 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:02:33,305 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:03:33,350 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 2 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 1441ms.
2021-04-19 17:03:34,874 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 3 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:04:34,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 4 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 7259ms.
2021-04-19 17:04:42,257 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 5 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:05:42,349 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 6 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 17414ms.
2021-04-19 17:05:59,845 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 7 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:06:59,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 8 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 15922ms.
2021-04-19 17:07:15,921 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 9 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:08:16,013 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 10 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 19586ms.
2021-04-19 17:08:35,740 ERROR [main] org.apache.hadoop.tools.util.RetriableCommand: Failure in Retriable command: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:310)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:256)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:210)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:561)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$900(WebHdfsFileSystem.java:144)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:637)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:675)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:671)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.read(WebHdfsFileSystem.java:2335)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:2152)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:91)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:308)
... 16 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:90)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:522)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$300(WebHdfsFileSystem.java:144)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:723)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.connect(WebHdfsFileSystem.java:2386)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:814)
... 27 more
2021-04-19 17:08:37,073 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,094 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,248 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,333 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
The same thing was working before enabling Kerberos.
Also, on the source cluster end, both active and standby namenode are working properly.
Can anybody point out any other place I should look into?
Thanks,
Megh
