question Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart? in Archives of Support Questions (Read Only)

Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

condla — Sun, 17 Jan 2016 02:41:37 GMT

I played around with rack awareness in HDP 2.3.0 and Ambari 2.1.0

Configuring it solely in Ambari in the Hosts view works fine.

Then I wanted to deploy my own script and my own data file. So I put the script and the data file in in /etc/hadoop/conf/ on all nodes and pointed HDFS to the script via the property net.topology.script.file.name. -> Saved -> restarted.

What happened: The file I deployed was overwritten by the ambari default rack topology script, which points to the default rack data file, which is overwritten after each restart as well.

In terms of consistency, it makes sense to centrally manage this file. On the other hand, there is no option to configure the script and the data file in Ambari.

How would one deploy self-made topology scirpts and data files?

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

nsabharwal — Sun, 17 Jan 2016 02:57:08 GMT

@Stefan Kupstaitis-Dunkler

See this http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_Ambari_Users_Guide/content/ch03s11.html

Using a Custom Topology Script

It is possible to not have Ambari manage the Rack information for hosts. Instead, you can use a custom topology script to provide rack information to HDFS and not use the Amabri-generated topology.py script. If you choose to manage Rack information on your own, you will need to create your own topology script and manage distributing the script to all hosts. Ambari will also not have any knowledge of host Rack information so heatmaps will not display by Rack in Ambari Web.

To manage Rack information on your own, in the Services > HDFS > Configs, modify the net.topology.script.file.name property. Set this property value to your own custom topology script (for example /etc/hadoop/conf/topology.sh ). Distribute that topology script to your hosts and manage the Rack mapping information for your script outside of Ambari.

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

condla — Sun, 17 Jan 2016 03:01:13 GMT

I did exactly this -> But as mentioned, what happend is that Ambari overwrote my custom script with the default one.

EDIT: so I assume it is a bug?

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

nsabharwal — Sun, 17 Jan 2016 03:09:22 GMT

@jeff Tagging Jeff. I believe we can open a jira to track this down @Stefan Kupstaitis-Dunkler

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

nsabharwal — Sun, 17 Jan 2016 03:14:17 GMT

@Stefan Kupstaitis-Dunkler Official doc has the right answer. If it's not working and assuming that you did not miss anything then it's a bug. Not sure if downvote was the right action as answer is based on official docs.

I will keep you posted in case I find more information.

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

jeff1 — Sun, 17 Jan 2016 03:20:00 GMT

There was a bug in Ambari 2.1.0, fixed in Ambari 2.1.1 (and later) where Ambari would overwrite your custom script.

https://issues.apache.org/jira/browse/AMBARI-12662

https://hortonworks.jira.com/browse/BUG-42095

Seems you are using Ambari 2.1.0, so this is fixed in newer releases.

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

condla — Sun, 17 Jan 2016 03:24:06 GMT

@Neeraj Sabharwal

Sorry, but your answer is equivalent to what I explained did not work. And I got this information from the official doc already.

I do not believe to have missed anything: Created a file, deployed it on all nodes, configured the property and restarted. After the restart my topology file had the content of the default one... tried it two times. I will try it in Ambari 2.1.2 if it shows the same behaviour.

Where do I open the jira?

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

nsabharwal — Sun, 17 Jan 2016 03:25:37 GMT

Thanks @jeff

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

nsabharwal — Sun, 17 Jan 2016 03:28:29 GMT

@Stefan Kupstaitis-Dunkler

See this https://issues.apache.org/jira/secure/CreateIssue!default.jspa

Re: Is it intended that a manually created rack topology script and data file is overwritten by Ambari after an HDFS restart?

condla — Sun, 17 Jan 2016 03:29:32 GMT

Thanks....