Support Questions
Find answers, ask questions, and share your expertise file getting overwritten by CDH file getting overwritten by CDH




I'm using CDH 5.4.1 and am trying to setup a s3distcp copy from CDH cluster to S3.  We are using the jets3t framework to do this, which requires the file exist in /etc/hadoop/conf directory on each node in the cluster.  I have tried manually creating this file, but it looks like it is being periodically overwritten by CDH or CM.  How should I go about getting this file to persist in the /etc/hadoop/conf directory?  Is it possible?  A safety valve will not work in this situation because an entirely different properties file needs to exist in the /etc/hadoop/conf folder outside of the view of Cloudera.




Re: file getting overwritten by CDH

The contents of /etc/hadoop/conf will be overwritten every time Client Configuration is deployed. As of now, Cloudera Manager does not have the facility to deploy arbitrary files as part of Client Configuration deployment.
Gautam Gopalakrishnan

Re: file getting overwritten by CDH

New Contributor

I had the same problem and worked with Cloudera support and eventually got a solution. You are right, by DEFAULT it looks in /etc/hadoop/conf for the jets3t properties files, and that is the wrong place to put it since it will get overwritten with each config deployment. The solution is to put it somewhere else, one each node, and make the required changes to the YARN config in CM:


1. Add your file location to yarn.application.classpath
2. Add the following to Gateway Client Environment Advanced Configuration Snippet for (Safety Valve): HADOOP_CLASSPATH=<your file location>:$HADOOP_CLASSPATH

3. Restart YARN



Re: file getting overwritten by CDH

New Contributor
Oh, and of course deploy the YARN config