Support Questions
Find answers, ask questions, and share your expertise

Distcp is not working after enabling Kerberos

Solved Go to solution

Distcp is not working after enabling Kerberos

Expert Contributor

Hello Everyone,

 

I'm trying to distcp from an HDP 3.1.5 cluster (non-kerberized) to CDP 7.1.5 cluster (kerberized).

 

I'm running the distcp command on the secure cluster like follows:

hadoop distcp -Ddfs.client.use.datanode.hostname=true -Ddfs.datanode.use.datanode.hostname=true -Dipc.client.fallback-to-simple-auth-allowed=true webhdfs://<source-nameservice>/path/to/source/folder webhdfs://<destination-nameservice>/path/to/destination/folder

 

It starts fine and then fails after 40 mins (Map progress reaches 97% and then gets stuck).

 

When I looked into the task attempt logs, I find this error:

 

 

2021-04-19 17:00:30,916 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2021-04-19 17:00:30,985 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-04-19 17:00:31,038 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2021-04-19 17:00:31,104 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: [Kind: mapreduce.job, Service: job_1618819500492_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@233795b6), Kind: WEBHDFS delegation, Service: ha-webhdfs:<destination-nameservice>, Ident: (token for megh.vidani: WEBHDFS delegation owner=megh.vidani, renewer=yarn, realUser=, issueDate=1618831821242, maxDate=1619436621242, sequenceNumber=322, masterKeyId=30), Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:<destination-nameservice>, Ident: (token for megh.vidani: HDFS_DELEGATION_TOKEN owner=megh.vidani@NPCI.ORG.IN, renewer=yarn, realUser=, issueDate=1618831821344, maxDate=1619436621344, sequenceNumber=323, masterKeyId=30)]
2021-04-19 17:00:31,175 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2021-04-19 17:00:31,293 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data10/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data11/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data12/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data13/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data2/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data3/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data4/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data5/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data7/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data8/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009,/data9/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009
2021-04-19 17:00:31,740 INFO [main] org.apache.hadoop.mapred.YarnChild: 
/************************************************************
[system properties]
os.name: Linux
os.version: 3.10.0-1160.24.1.el7.x86_64
java.home: /usr/java/jdk1.8.0_281/jre
java.runtime.version: 1.8.0_281-b09
java.vendor: Oracle Corporation
java.version: 1.8.0_281
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.class.path: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002:/etc/hadoop/conf.cloudera.yarn:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/audience-annotations-0.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jul-to-slf4j-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jdom-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-examples-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-api-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-system-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snakeyaml-1.26.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/spark-2.4.0.7.1.5.0-257-yarn-shuffle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang3-3.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-core-2.8.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/forbiddenapis-2.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-protobuf.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ehcache-3.3.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-asn1-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.servlet-api-3.1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.xml.bind-api-2.3.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-annotations-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-identity-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archives-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-scala_2.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-qual-2.8.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aopalliance-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/stax2-api-3.1.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-api-1.7.30.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/xz-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-json-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-xc-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-cli-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-extras-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/slf4j-log4j12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gson-2.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/fst-2.50.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/tt-instrumentation-7.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-client-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/lz4-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-admin-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/dnsjava-2.1.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-module-jaxb-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-xml-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.inject-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-xdr-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-config-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/mssql-jdbc-6.2.1.jre7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-core-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jettison-1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/asm-5.0.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-simple-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jakarta.activation-api-1.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-util-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-databind-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-fs2img-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-yarn-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-archive-logs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/failureaccess-1.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsr311-api-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-crypto-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-1.9.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-pkix-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-api-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-hdfs-plugin-shim-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-keyvault-core-1.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-pig.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-xml-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-console-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/java-util-1.9.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/accessors-smart-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ram-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-core-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/nimbus-jose-jwt-7.9.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/logredactor-2.0.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpcore-4.4.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-servlet-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-web-proxy-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-lang-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-recipes-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-mapper-asl-1.9.13-cloudera.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/paranamer-2.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-smart-2.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-templates-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-compress-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/woodstox-core-5.0.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aliyun-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guava-28.1-jre.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-common-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-rumen-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/httpclient-4.5.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-ecs-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-generator.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-unmanaged-am-launcher-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-hadoop-bundle.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-security-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-tests-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-plugins-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-server-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-intg-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-json-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-cascading.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcprov-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-core-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerby-util-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-datajoin-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-client-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-beanutils-1.9.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/javax.activation-1.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-shell.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-codec-1.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-swing-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jcip-annotations-1.0-1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kafka-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-nfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-shuffle-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jersey-guice-1.19.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-auth-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/animal-sniffer-annotations-1.18.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-io-2.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-format-structures.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-plugin-classloader-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-ozone-filesystem-hadoop3-1.0.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-http-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/re2j-1.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-server-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-nativetask-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-servlet-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-sls-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-simplekdc-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/groovy-groovysh-3.0.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-gridmix-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/j2objc-annotations-1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-streaming-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-distcp-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-api-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-jackson.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okio-1.6.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-net-3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-common-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/log4j-1.2.17-cloudera1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-sdk-oss-3.4.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-column.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/okhttp-2.7.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration2-2.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-encoding.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsch-0.1.54.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/snappy-java-1.1.7.7.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-client-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-client-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-servlet-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/htrace-core4-4.1.0-incubating.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-common.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-kms-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jaxb-api-2.2.11.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-core-3.4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/metrics-core-3.2.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-annotations-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-hs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/event-publish-7.2.4-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/google-extensions-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aliyun-java-sdk-sts-3.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-common-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-server-nodemanager-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-aws-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-registry-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/objenesis-1.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-uploader-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/checker-compat-qual-2.5.3.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-data-lake-store-sdk-2.3.6.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/json-io-2.5.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-all-4.1.51.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-azure-datalake-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-ajax-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/zookeeper-jute-3.5.5.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/wildfly-openssl-1.0.7.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-json-provider-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/parquet-thrift.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-configuration-1.10.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-openstack-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-webapp-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/azure-storage-7.0.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-httpfs-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-jobclient-3.1.1.7.1.5.0-257-tests.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-native-client-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-resourceestimator-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jackson-jaxrs-base-2.10.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ranger-raz-hook-abfs-2.1.0.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/bcpkix-jdk15on-1.60.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-collections-3.2.2.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/HikariCP-java7-2.4.12.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/flogger-slf4j-backend-0.5.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/avro-1.8.2.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jna-5.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/swagger-annotations-1.5.4.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/kerb-server-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gcs-connector-2.1.2.7.1.5.0-257-shaded.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/token-provider-1.1.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-services-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-hdfs-rbf-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jsp-api-2.1.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/curator-framework-4.2.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/leveldbjni-all-1.8.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-i18n.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/aws-java-sdk-bundle-1.11.563.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/netty-3.10.6.Final.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/ojalgo-43.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-yarn-applications-distributedshell-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-util-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-core-3.1.1.7.1.5.0-257.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/guice-4.0.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/jetty-io-9.4.31.v20200723.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/gateway-cloud-bindings.jar:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/mr-framework/hadoop-mapreduce-client-app-3.1.1.7.1.5.0-257.jar::job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/job.jar
java.io.tmpdir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002/tmp
user.dir: /data6/yarn/nm/usercache/megh.vidani/appcache/application_1618819500492_0009/container_e31_1618819500492_0009_01_000002
user.name: megh.vidani
************************************************************/
2021-04-19 17:00:31,741 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,042 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,056 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2021-04-19 17:00:32,235 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: /user/megh.vidani/.staging/_distcp-524677260/fileList.seq:0+500
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
2021-04-19 17:00:32,240 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-04-19 17:00:32,822 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder to webhdfs://<destination-nameservice>/path/to/source/folder
2021-04-19 17:00:32,915 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:00:32,972 INFO [main] org.apache.hadoop.tools.mapred.CopyMapper: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:00:32,991 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:02:33,218 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:02:33,305 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:03:33,350 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 2 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 1441ms.
2021-04-19 17:03:34,874 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 3 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:04:34,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 4 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 7259ms.
2021-04-19 17:04:42,257 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 5 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:05:42,349 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 6 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 17414ms.
2021-04-19 17:05:59,845 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 7 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:06:59,916 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 8 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 15922ms.
2021-04-19 17:07:15,921 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 9 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:08:16,013 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 10 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 19586ms.
2021-04-19 17:08:35,740 ERROR [main] org.apache.hadoop.tools.util.RetriableCommand: Failure in Retriable command: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:310)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:256)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:210)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
	at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
	at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:561)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$900(WebHdfsFileSystem.java:144)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:877)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:843)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:637)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:675)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:671)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.read(WebHdfsFileSystem.java:2335)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$WebHdfsInputStream.read(WebHdfsFileSystem.java:2152)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:91)
	at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:308)
	... 16 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
	at org.apache.hadoop.hdfs.web.JsonUtilClient.toRemoteException(JsonUtilClient.java:90)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:522)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$300(WebHdfsFileSystem.java:144)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:723)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.connect(WebHdfsFileSystem.java:2386)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:814)
	... 27 more
2021-04-19 17:08:37,073 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Copying webhdfs://<source-nameservice>/path/to/source/folder/000031_0.gz to webhdfs://<destination-nameservice>/path/to/source/folder/000031_0.gz
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Creating temp file: webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,074 INFO [main] org.apache.hadoop.tools.mapred.RetriableFileCopyCommand: Writing to temporary target file path webhdfs://<destination-nameservice>/path/to/destination/folder/.distcp.tmp.attempt_1618819500492_0009_m_000000_0
2021-04-19 17:08:37,094 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,248 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode2-hostname>/<source-cluster-namenode2-ip>:50070. Already retried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.
2021-04-19 17:10:37,333 INFO [main] org.apache.hadoop.hdfs.web.WebHdfsFileSystem: Retrying connect to namenode: <source-cluster-namenode1-hostname>/<source-cluster-namenode1-ip>:50070. Already retried 1 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@682c1e93, delay 0ms.

 

 

 

The same thing was working before enabling Kerberos.

 

Also, on the source cluster end, both active and standby namenode are working properly.

 

Can anybody point out any other place I should look into?

 

Thanks,

Megh

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Distcp is not working after enabling Kerberos

Expert Contributor

This Issue was resolved after we opened 50075 port (Datanode WebHDFS port) between the source and destination cluster.

 

I think the error

org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

 

is rather misleading.

 

Once I tried to distcp with the active namenode instead of the nameservice, I was able to get the root cause saying "connection timed out" on port 50075.

 

In the official documentation of Cloudera this port is not mentioned as a requirement. 

 

Thanks,

Megh

View solution in original post

9 REPLIES 9

Re: Distcp is not working after enabling Kerberos

Cloudera Employee

Hi Megh,

Looking at the error, it seems that a NN failover happened at the remote cluster when the distcp didn't yet finish. Also, did you mention namespaces instead of hostnames in the distcp command?

I would recommend you review this document on performing distcp between clusters that have Namenode HA enabled.

Thanks,
Saurabh

Re: Distcp is not working after enabling Kerberos

Expert Contributor

Hi @slambe ,

 

Thanks for your reply.

 

I've followed the same documentation.

 

I've configured the nameservice with the steps mentioned in the destination cluster and before kerberization, I was able to execute the same command without any errors (even with the nameservice).

 

Also, I've checked that the failover hasn't happened on the source cluster. The same namenode is active as it was before.

 

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Mentor

@vidanimegh 

What is the /etc/hosts entry  remember both hosts  /etc/hosts files should have the same entries? 

On Source

<source-nameservice> IP
<destination-nameservice IP

On destination

<source-nameservice> IP
<destination-nameservice IP

 

Please let me know 

 

 

Re: Distcp is not working after enabling Kerberos

Expert Contributor

Hi @Shelton 

 

Do you mean source and destination hostnames and their IPs? If yes, that I've ensured that they're same on both the clusters.

 

If you indeed mean the nameservice, can you please tell me which IP should it resolve to?

 

Generally the format of /etc/hosts entry is as follows:

ip hostname alias

 

but in your reply you've given:

<nameservice> IP

 

Is it correct?

 

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Mentor

@vidanimegh 

Ye that s the usual format but I didn't want to complicate the  explanation  the alias is optionaél anyways.

But you see the logic the 2 hosts must be pingable for  eachother.

 

Happy hadooping

Re: Distcp is not working after enabling Kerberos

Expert Contributor

Understood. In our case, the culprit turned out to be 50075 port in firewall.

I've added as an answer.

In any case, thanks for your support.

 

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Mentor

@vidanimegh 

Can you try it differently like below and let me know?

 

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://<source-nameservice>/path/to/source/folder hdfs://<destination-nameservice>/path/to/destination/folder

 

 

Always try to use nameservice !

 

Re: Distcp is not working after enabling Kerberos

Expert Contributor

Hi @Shelton ,

 

Thanks for your response.

 

With the command you've shared, I'm getting the following error:

21/04/19 20:49:36 ERROR tools.DistCp: Exception encountered
java.io.EOFException: End of File Exception between local host is: "<source-hostname>/<source-ip>"; destination host is: "<destination-namenode>":8020; : java.io.EOFException; For more details see:  http://wiki.apache.org/hadoop/EOFException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:789)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
        at org.apache.hadoop.ipc.Client.call(Client.java:1508)
        at org.apache.hadoop.ipc.Client.call(Client.java:1405)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:957)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1686)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1745)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1742)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1757)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
        at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2088)
        at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
        at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
        at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:443)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1880)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087)

 

Thanks,

Megh

Re: Distcp is not working after enabling Kerberos

Expert Contributor

This Issue was resolved after we opened 50075 port (Datanode WebHDFS port) between the source and destination cluster.

 

I think the error

org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException: org.apache.hadoop.ipc.StandbyException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

 

is rather misleading.

 

Once I tried to distcp with the active namenode instead of the nameservice, I was able to get the root cause saying "connection timed out" on port 50075.

 

In the official documentation of Cloudera this port is not mentioned as a requirement. 

 

Thanks,

Megh

View solution in original post