jets3t.properties file getting overwritten by CDH

[#1]

Hi,

I'm using CDH 5.4.1 and am trying to setup a s3distcp copy from CDH cluster to S3.  We are using the jets3t framework to do this, which requires the jets3t.properties file exist in /etc/hadoop/conf directory on each node in the cluster.  I have tried manually creating this file, but it looks like it is being periodically overwritten by CDH or CM.  How should I go about getting this file to persist in the /etc/hadoop/conf directory?  Is it possible?  A safety valve will not work in this situation because an entirely different properties file needs to exist in the /etc/hadoop/conf folder outside of the view of Cloudera.

[#2]

The contents of /etc/hadoop/conf will be overwritten every time Client Configuration is deployed. As of now, Cloudera Manager does not have the facility to deploy arbitrary files as part of Client Configuration deployment.

[#3]

I had the same problem and worked with Cloudera support and eventually got a solution. You are right, by DEFAULT it looks in /etc/hadoop/conf for the jets3t properties files, and that is the wrong place to put it since it will get overwritten with each config deployment. The solution is to put it somewhere else, one each node, and make the required changes to the YARN config in CM:

1. Add your file location to yarn.application.classpath
2. Add the following to Gateway Client Environment Advanced Configuration Snippet for hadoop-env.sh (Safety Valve): HADOOP_CLASSPATH=<your file location>:$HADOOP_CLASSPATH

3. Restart YARN

[#4]

Oh, and of course deploy the YARN config