Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

jets3t.properties file getting overwritten by CDH

jets3t.properties file getting overwritten by CDH

Explorer

Hi,

 

I'm using CDH 5.4.1 and am trying to setup a s3distcp copy from CDH cluster to S3.  We are using the jets3t framework to do this, which requires the jets3t.properties file exist in /etc/hadoop/conf directory on each node in the cluster.  I have tried manually creating this file, but it looks like it is being periodically overwritten by CDH or CM.  How should I go about getting this file to persist in the /etc/hadoop/conf directory?  Is it possible?  A safety valve will not work in this situation because an entirely different properties file needs to exist in the /etc/hadoop/conf folder outside of the view of Cloudera.

 

 

3 REPLIES 3

Re: jets3t.properties file getting overwritten by CDH

The contents of /etc/hadoop/conf will be overwritten every time Client Configuration is deployed. As of now, Cloudera Manager does not have the facility to deploy arbitrary files as part of Client Configuration deployment.
Regards,
Gautam Gopalakrishnan

Re: jets3t.properties file getting overwritten by CDH

New Contributor

I had the same problem and worked with Cloudera support and eventually got a solution. You are right, by DEFAULT it looks in /etc/hadoop/conf for the jets3t properties files, and that is the wrong place to put it since it will get overwritten with each config deployment. The solution is to put it somewhere else, one each node, and make the required changes to the YARN config in CM:

 

1. Add your file location to yarn.application.classpath
2. Add the following to Gateway Client Environment Advanced Configuration Snippet for hadoop-env.sh (Safety Valve): HADOOP_CLASSPATH=<your file location>:$HADOOP_CLASSPATH

3. Restart YARN

 

 

Re: jets3t.properties file getting overwritten by CDH

New Contributor
Oh, and of course deploy the YARN config