Support Questions
Find answers, ask questions, and share your expertise

hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

New Contributor

I am new to HDP and have been messing around with ambari / hdp for couple of days so far.

I got a Centos 6.8 box and have successfully installed ambari-server / ambari-agent (2.5.1.0) on it. Everything looks good on from the Ambari web ui. However, I am not able to install HDFS service (together with Zookeeper which is required). It always failed at "HDFS Client Install". There are several things to be noted based on my two-days messing around it:

1. STDERR is empty while no error/waring shows in the STDOUT(one example attached output-20.txt)

2. It halted at "hdfs-site.xml" generation right after the logging message "2017-08-23 19:34:18,905 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml". However, the config file does not show up at the mentioned directory (there are plenty of other files)

3. As most of ambari-server / agent are in python, I've tried to debug in to code and try to understand the logic behind it. In general I think it's quite straightforward. All custom configurations defined by user through the web ui are applied to a config template (jinja2 is the underlying template engine) and the final config file "hdfs-site.xml" should be generated and placed in the right directory. Based on my debugging, it failed during template processing, unfortunately, I did not figure out why.

4. I tried both Ambari 2.4 and 2.5 with hdp 2.5 and 2.6 respectively, both failed with exactly the same issue.

Please kindly advise if anything you know about the issue, thank you !

8 REPLIES 8

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

Expert Contributor

There are a number of things that could have gone wrong in your installation - I am not able to pinpoint the issue from your description but the following document contains step-by-step instructions to setup a new HDP cluster. It could be anything from not having the correct repositories defined for your package sources to a configuration error. Please step through the document and either verify you've done everything correctly or, more preferably, restart the installation from scratch so that you know you are following the document step-by-step.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/ambari_repos...

If you are just getting started with HDP then you may also want to check out the HDP and HDF sandboxes, they provide a pre-built 2-node environment for you to get acquainted with the software/platform. I've included the link below.

https://hortonworks.com/products/sandbox/

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

Super Mentor

@hien chu

Please try reinstalling HDFS client on the mentioned host ... it should be able to generate the hdfs-site.xml properly there.

Or else there may be a symlink issue:

# ls -lart  /usr/hdp/current/hadoop-client/conf
lrwxrwxrwx. 1 root root 25 Jun  9 11:12 /usr/hdp/current/hadoop-client/conf -> /etc/hadoop/2.6.1.0-129/0

.

The "/usr/hdp/current/hadoop-client/conf" is actually a symlink not a directory.

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

Expert Contributor

There are a number of things that could have gone wrong in your installation - I am not able to pinpoint the issue from your description but the following document contains step-by-step instructions to setup a new HDP cluster. It could be anything from not having the correct repositories defined for your package sources to a configuration error. Please step through the document and either verify you've done everything correctly or, more preferably, restart the installation from scratch so that you know you are following the document step-by-step.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/ambari_repos...

If you are just getting started with HDP then you may also want to check out the HDP and HDF sandboxes, they provide a pre-built 2-node environment for you to get acquainted with the software/platform. I've included the link below.

https://hortonworks.com/products/sandbox/

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

New Contributor

That's where I started. I did not try sandbox yet because I don't have VM in hand.

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

New Contributor

@Jay SenSharma

I checked my system and the link "/usr/hdp/current/hadoop-client" is correct and pointed to a actual version. There are also other config files, such as core-site.xml. The only missing file is "hdfs-site.xml". I also tried to manually copied a hdfs-site.xml to the directory from another working hadoop cluster. Suprisingly the file is gone after I retried the installation.

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

@hien chu

The same problem was happened with me. I have tried all the possibilities. Finally I have reinstall the OS and installed HDP 2.6 with ambari 2.5.

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

New Contributor

@Ashnee Sharma
That's really frustrating, I moved to Ubuntu 16 and all works, however, I think Centos 6 is still one of the most popular server os. Which version of Centos are you using?

Re: hdfs-site.xml not generated under /usr/hdp/current/hadoop-client/conf

@hien chu

I was using cent os 6.5