An easy way to define a prefix for all hdp components ?

[#1]

Hi,

is there a simple way to install all hdp components using a common prefix ? A single path prefix box where I could enter /opt/mydumbo/ ? I do try to accomplish this with linking all I can. I don't mind having a /var/log/hbase which is a link to /opt/mydumbo/hbase/log. Can this be accomplished with a blueprint ?

Thanks,

[#2]

@Peter Bartal everything goes into /usr/hdp/version can you elaborate on your use case?

[#3]

Yes the binaries. But I think some of the ambari-server, ambari-agent and some python scripts end up also in /usr/bin or /usr/sbin and of course all operational logs /var/run /var/log.

[#4]

As far as I know there is no easy way of doing this and personally I would not recommend changing the default location(s) /usr/hdp/..... I have seen similar objectives/projects (e.g. changing default log directories, away from /var/log) and it is somewhat difficult to change all the necessary configurations + running tests to be sure everything works as expected

Could you elaborate why you need/want to change the default location?

[#5]

Changing the log directories actually works it just is a bit of work but changing the installation directories of the components? Pretty sure that will end in tears. I would second Jonas with "Don't do it"

[#6]

The reason is that for HDP there is a dedicated volume where everything should be stored. This is a requirement so the OS stays unaffected by hdp doings. The other possibility would be to size the locations where hdp is storing it's files and create appropriate volumes/mounts. The linking actually works quite well, I have a sudoers file with a dedicated user and all the folders are linked into a single volume and works for several weeks now.

[#7]

But I did it by hand and will have to update it accordingly for every component when added and I do not really want to thing about upgrades. Yes I think it is a PIA but I somehow imagined it will be easier than convincing our security

[#8]

@Peter Bartal

For log directory change..see this

https://community.hortonworks.com/questions/4329/log-file-location-is-there-a-way-to-change-varlog.h...

[#9 · Accepted solution]

I have it now working (moved everything to one volume for now and linked all the directories). The important thing is to keep the rights of the files. Basically the requirement is to isolate the appliacation to a set of volumes so you won't overflow the system tmp/log partitions. Then the outline might look like (several data dirs are used so you can keep several copies of data on different volumes/disks on the same host):

/opt/mydumbo/app/ - binaries, static config like in /etc/...

/opt/mydumbo/data1/ - dynamic data like hdfs/namenode hdfs/datanode...

/opt/mydumbo/data2/ - dynamic data like hdfs/namenode hdfs/datanode...

/opt/mydumbo/data3/ - dynamic data like hdfs/namenode hdfs/datanode...

/opt/mydumbo/log - all the log files

/opt/mydumbo/tmp - temporary data

Ideally I would like to blueprint it. I let it run for a while but hints, pointers and doubts are welcome. Might end in tears as mentioned before but I want to try 🙂 The truth is no kerberos and such is used at the moment so might be easier.

Thanks,

Peter

[#10]

Thanks for sharing! Let us know how it works out or if you run into any trouble.