Skip to main content
orlandoteixeira
Gold ⭐️
orlandoteixeira
Gold ⭐️
December 4, 2015

DISTCP errors when copying to AWS in HDP 2.3.x

  • October 5, 2026 at 9:43 AM
  • 0 replies
  • 1281 views
  • Translate All

Problem:

When trying to distcp to AWS, this error is reported:

2015-12-03 09:50:01,132 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:224)

and also:

Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener

Solution:

In ambari, under yarn, configs please add: /usr/hdp/2.3.2.0-2950/hadoop/* and /usr/hdp/2.3.2.0-2950/hadoop/lib* to the yarn.application.classpath

Then under mapred, configs, do the same for mapreduce.application.classpath

restart all affected services and it should work.

    Terms & ConditionsAccessibility statement