question Re: Distcp is not working after enabling Kerberos in Support Questions

Distcp is not working after enabling Kerberos

vidanimegh — Tue, 21 Apr 2026 09:04:59 GMT

Hello Everyone,

I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).

I'm running the distcp command on the secure cluster like follows:

hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder

It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).

When I looked into the task attempt logs, I find this error:

2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration 2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties 2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2021-04-19 17:00:31,104 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: [Kind: mapreduce.job, Service: job_1618819500492_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@233795b6), Kind: WEBHDFS delegation, Service: ha-webhdfs:<destination-nameservice>, Ident: (token for megh.vidani: WEBHDFS delegation owner=megh.vidani, renewer=yarn, realUser=, issueDate=1618831821242, maxDate=1619436621242, sequenceNumber=322, masterKeyId=30), Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<destination-nameservice>, Ident: (token for megh.vidani: HDFS_DELEGATION_TOKEN owner=megh.vidani@NPCI.ORG.IN, renewer=yarn, realUser=, issueDate=1618831821344, maxDate=1619436621344, sequenceNumber=323, masterKeyId=30)] 2021-04-19 17:00:31,175 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2021-04-19 17:00:31,293 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data10/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data11/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data12/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data13/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data2/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data3/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data4/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data5/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data7/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data8/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data9/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009 2021-04-19 17:00:31,740 INFO [main] org.apache.hadoop.mapred.YarnChild: /************************************************************ [system properties] os.name: Linux os.version: 3.10.0-1160.24.1.el7.x86_64 java.home: /usr/java/jdk1.8.0_281/jre java.runtime.version: 1.8.0_281-b09 java.vendor: Oracle Corporation java.version: 1.8.0_281 java.vm.name: Java HotSpot(TM) 64-Bit Server VM java.class.path: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002:/etc/hadoop/conf.cloudera.yarn:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/audience-annotations-0.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jul-to-slf4j-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jdom-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-examples-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-api-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-system-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snakeyaml-1.26.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/spark-2.4.0.7.1.5.0-257-yarn-shuffle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang3-3.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-core-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/forbiddenapis-2.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-protobuf.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ehcache-3.3.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-asn1-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.servlet-api-3.1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.xml.bind-api-2.3.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-annotations-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-identity-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archives-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-scala_2.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-qual-2.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aopalliance-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/stax2-api-3.1.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-api-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/xz-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-json-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-xc-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-cli-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-extras-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-log4j12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gson-2.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/fst-2.50.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/tt-instrumentation-7.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-client-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/lz4-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-admin-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/dnsjava-2.1.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-module-jaxb-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-xml-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.inject-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-xdr-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-config-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/mssql-jdbc-6.2.1.jre7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-core-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jettison-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/asm-5.0.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-simple-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.activation-api-1.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-util-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-databind-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-fs2img-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-yarn-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archive-logs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/failureaccess-1.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr311-api-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-crypto-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-pkix-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-api-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-hdfs-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-keyvault-core-1.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-xml-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-console-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/java-util-1.9.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/accessors-smart-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ram-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-core-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/nimbus-jose-jwt-7.9.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/logredactor-2.0.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpcore-4.4.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-servlet-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-web-proxy-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-recipes-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-mapper-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/paranamer-2.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-smart-2.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-templates-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-compress-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/woodstox-core-5.0.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aliyun-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guava-28.1-jre.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-common-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-rumen-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpclient-4.5.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ecs-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-generator.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-security-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-tests-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-plugins-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-server-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-intg-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-json-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcprov-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-datajoin-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-client-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-beanutils-1.9.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-shell.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-codec-1.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-swing-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jcip-annotations-1.0-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kafka-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-shuffle-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-guice-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-auth-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/animal-sniffer-annotations-1.18.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-io-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-format-structures.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-plugin-classloader-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-ozone-filesystem-hadoop3-1.0.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-http-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/re2j-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-server-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-nativetask-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-servlet-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-sls-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-simplekdc-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-groovysh-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-gridmix-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/j2objc-annotations-1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-streaming-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-distcp-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-jackson.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okio-1.6.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-net-3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-1.2.17-cloudera1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-sdk-oss-3.4.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-column.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okhttp-2.7.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration2-2.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-encoding.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsch-0.1.54.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snappy-java-1.1.7.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-client-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-servlet-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/htrace-core4-4.1.0-incubating.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kms-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-api-2.2.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-core-3.4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/metrics-core-3.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/event-publish-7.2.4-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/google-extensions-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-sts-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-nodemanager-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aws-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-registry-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/objenesis-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-uploader-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-compat-qual-2.5.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-data-lake-store-sdk-2.3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-io-2.5.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-all-4.1.51.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-datalake-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-ajax-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/wildfly-openssl-1.0.7.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-json-provider-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-thrift.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration-1.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-openstack-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-webapp-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-storage-7.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-httpfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-resourceestimator-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-base-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-hook-abfs-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcpkix-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-collections-3.2.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/HikariCP-java7-2.4.12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-slf4j-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro-1.8.2.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jna-5.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/swagger-annotations-1.5.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-server-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gcs-connector-2.1.2.7.1.5.0-257-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/token-provider-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsp-api-2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-framework-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/leveldbjni-all-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-i18n.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aws-java-sdk-bundle-1.11.563.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-3.10.6.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ojalgo-43.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-distributedshell-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-io-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-cloud-bindings.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-app-3.1.1.7.1.5.0-257.jar::job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/job.jar java.io.tmpdir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/tmp user.dir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002 user.name: megh.vidani ************************************************************/ 2021-04-19 17:00:31,741 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2 2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2021-04-19 17:00:32,056 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2021-04-19 17:00:32,235 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: /user/megh.vidani/.staging/_distcp-524677260/fileList.seq:0+500 2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2 2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2021-04-19 17:00:32,822 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder to webhdfs://<destination-nameservice>/path/to/source/folder 2021-04-19 17:00:32,915 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:00:32,972 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz 2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz 2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0 2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0 2021-04-19 17:02:33,218 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:02:33,305 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:03:33,350 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 2 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 1441ms. 2021-04-19 17:03:34,874 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 3 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:04:34,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 4 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 7259ms. 2021-04-19 17:04:42,257 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 5 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:05:42,349 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 6 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 17414ms. 2021-04-19 17:05:59,845 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 7 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:06:59,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 8 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 15922ms. 2021-04-19 17:07:15,921 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 9 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:08:16,013 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 10 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 19586ms. 2021-04-19 17:08:35,740 ERROR [main] org.apache.hadoop.tools.util.RetriableCommand: Failure in Retriable command: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:310) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:256) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:210) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:561) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$900(WebHdfsFileSystem.java:144) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:637) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:675) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:671) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.read(WebHdfsFileSystem.java:2335) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:2152) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:91) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:308) ... 16 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:90) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:522) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$300(WebHdfsFileSystem.java:144) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:723) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.connect(WebHdfsFileSystem.java:2386) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:814) ... 27 more 2021-04-19 17:08:37,073 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz 2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0 2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0 2021-04-19 17:08:37,094 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:10:37,248 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms. 2021-04-19 17:10:37,333 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.

The same thing was working before enabling Kerberos.

Also, on the source cluster end, both active and standby namenode are working properly.

Can anybody point out any other place I should look into?

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

slambe — Mon, 19 Apr 2021 13:59:51 GMT

Hi Megh,

Looking at the error, it seems that a NN failover happened at the remote cluster when the distcp didn't yet finish. Also, did you mention namespaces instead of hostnames in the distcp command?

I would recommend you review this document on performing distcp between clusters that have Namenode HA enabled.

Thanks,
Saurabh

Re: Distcp is not working after enabling Kerberos

Shelton — Mon, 19 Apr 2021 15:08:00 GMT

@vidanimegh

Can you try it differently like below and let me know?

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://<source-nameservice>/path/to/source/folder hdfs://<destination-nameservice>/path/to/destination/folder

Always try to use nameservice !

Re: Distcp is not working after enabling Kerberos

vidanimegh — Mon, 19 Apr 2021 15:27:11 GMT

Hi @Shelton ,

Thanks for your response.

With the command you've shared, I'm getting the following error:

21/04/19 20:49:36 ERROR tools.DistCp: Exception encountered java.io.EOFException: End of File Exception between local host is: "<source-hostname>/<source-ip>"; destination host is: "<destination-namenode>":8020; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:789) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:957) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1686) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1745) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1742) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1757) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349) at org.apache.hadoop.fs.Globber.glob(Globber.java:202) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2088) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379) at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96) at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193) at org.apache.hadoop.tools.DistCp.run(DistCp.java:155) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:443) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1880) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087)

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

vidanimegh — Mon, 19 Apr 2021 15:37:14 GMT

Hi @slambe ,

Thanks for your reply.

I've followed the same documentation.

I've configured the nameservice with the steps mentioned in the destination cluster and before kerberization, I was able to execute the same command without any errors (even with the nameservice).

Also, I've checked that the failover hasn't happened on the source cluster. The same namenode is active as it was before.

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Shelton — Mon, 19 Apr 2021 20:53:01 GMT

@vidanimegh

What is the /etc/hosts entry remember both hosts /etc/hosts files should have the same entries?

On Source

<source-nameservice> IP <destination-nameservice IP

On destination

<source-nameservice> IP <destination-nameservice IP

Please let me know

Re: Distcp is not working after enabling Kerberos

vidanimegh — Tue, 20 Apr 2021 07:13:17 GMT

Hi @Shelton

Do you mean source and destination hostnames and their IPs? If yes, that I've ensured that they're same on both the clusters.

If you indeed mean the nameservice, can you please tell me which IP should it resolve to?

Generally the format of /etc/hosts entry is as follows:

ip hostname alias

but in your reply you've given:

<nameservice> IP

Is it correct?

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Shelton — Tue, 20 Apr 2021 20:38:14 GMT

@vidanimegh

Ye that s the usual format but I didn't want to complicate the explanation the alias is optionaél anyways.

But you see the logic the 2 hosts must be pingable for eachother.

Happy hadooping

Re: Distcp is not working after enabling Kerberos

vidanimegh — Thu, 22 Apr 2021 08:19:20 GMT

This Issue was resolved after we opened 50075 port (Datanode WebHDFS port) between the source and destination cluster.

I think the error

org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

is rather misleading.

Once I tried to distcp with the active namenode instead of the nameservice, I was able to get the root cause saying "connection timed out" on port 50075.

In the official documentation of Cloudera this port is not mentioned as a requirement.

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

vidanimegh — Thu, 22 Apr 2021 10:22:57 GMT

Understood. In our case, the culprit turned out to be 50075 port in firewall.

I've added as an answer.

In any case, thanks for your support.

Thanks,

Megh