Installing multiple Regionserver on one host via Ambari

[#1]

I managed to manually get a second RS running on a host with the following changes:

1) Configuration

The configuration of our first RS on this host lies within /usr/hdp/current/hbase-regionserver/conf, which actually links to /etc/hbase/2.5.3.0-37/0 .

In order to have a second configuration I copied the '0' folder to '1' in the same parent folder (same permissions as the '0' folder) and adapted the following parameters:

within hbase-site.xml

hbase.regionserver.info.port from 16030 to 26130
hbase.regionserver.port from 16020 to 26020

within hbase-env.sh

export HBASE_CONF_DIR=${HBASE_CONF_DIR:-/etc/hbase/2.5.3.0-37/1}export HBASE_LOG_DIR=/var/log/hbase/hbase1export HBASE_PID_DIR=/var/run/hbase/hbase1

I changed log and pid directory to a subfolder as the permissions to write are already set correctly

Adapting the WAL directory the RS writes to is not necessary, as the port is contained in the folder, and thus it's unique (I got the information from https://amahanty.wordpress.com/2012/11/04/understanding-hbase-files-and-directories )

2) Execution

Even though there's only one hbase-regionserver "folder" within /usr/hdp/current , this is sufficient, as we can define another configuration folder upon startup

Based on the original command that is logged when restarting a RS via Ambari

Execute['/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-regionserver/conf start regionserver'] {'not_if': 'ambari-sudo.sh  -H -E test -f /var/run/hbase/hbase-hbase-regionserver.pid && ps -p `ambari-sudo.sh  -H -E cat /var/run/hbase/hbase-hbase-regionserver.pid` >/dev/null 2>&1',
'user': 'hbase'}

I executed the following:

/usr/hdp/current/hbase-regionserver/bin/hbase-daemon.sh --config /etc/hbase/2.5.3.0-37/1 start regionserver

3) Finally

Within the hbase-master web UI I saw the new RS, and after some time, the regions were automatically distributed to the new RS.

Within the log file I see

impl.MetricsSystemImpl: Error creating sink 'timeline'org.apache.hadoop.metrics2.impl.MetricsConfigException:
Error creating plugin: org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink

so probably I'm missing some metrics configuration, but I haven't taken a look any further at the moment. Probably ports?

Anyhow, we can't manage the new RS with Ambari, which makes sense, as Ambari doesn't know anything about this service.

Well, the obvious question now is: How can we extend Ambari in order to install further RS on one host?

Open questions:

a) File /usr/hdp/current/hbase-regionserver/conf/regionservers

- When adding an additional RS on a host, would it result in a second entry in the regionservers file?

- What's the meaning of this file?

b) /etc/hbase/2.5.3.0-37/0

- the folder '0', what's the meaning behind this 'zero' folder ?

c) Does the Ambari design support adding additional RS on one host?

I cloned the source from https://git-wip-us.apache.org/repos/asf/ambari.git and browsed through the code, checked out the ambari design document etc. and have a basic overview of the design, really basic (just to underline this)

What I think right now how it works, and please correct me if I'm wrong.

Basically, when I add a new host, the basic file structure for all services is put on this host. Only when assigning a component to this newly added host, some magic happens.

With 'magic' I mean all the operations that need to be done in order to be able to monitor the component with Ambari.

At this point I'm not sure where to add some code in order to define a different configuration folder for the RS. Of course, I also want to be able to restart/stop/start etc the component via Ambari, and when I make any changes to the configuration of a RS or HBase itself, the additional RS should also be considered.

Can you give me any hints?

Thanks.

[#2]

Well, I finally got some time to dig a little bit deeper.

I followed the instructions as shown on the wiki. First,

https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development . I installed everything on my Ubuntu machine and did a maven build. rpm was successful, the one with jdeb failed for some projects, as the plugin jdeb was missing. (ambari-funtest, ambari-logsearch, ambari-metrics-grafana, ambari-metrics-storm-sink)

After fixing the issues the maven build was fine on my machine.

Next I tried

https://cwiki.apache.org/confluence/display/AMBARI/Development+in+Docker . With this file the set-up of the container failed, firstly because of Oracle's JDK download issue (had to place it in the directory and transfer it to the docker image with the COPY command), secondly, the Maven version 3.0.5 was not enough, so I changed to the latest, 3.5.0. Anyhow, the maven build that gets executed at the end of the Dockerfile failed for several times for the ambari-server:

Caused by: java.lang.OutOfMemoryError: PermGen space

I tried with both the trunk and 2.5.0 branch, both times the errors. If there was some error on my side please let me know.

Anyhow, doing something with Docker now is obsolete as it won't compile.

So, this is just some feedback for you. Finally coming to my findings and questions:

Installing HBase is managed by the HDP stack, what I'm not quite sure is, where are the scripts located? The only scripts I see are those located in ambari-server/src/main/resources/common-services - are those the correct ones?

So, as I want to add multiple region servers on one host, would I just add another component referencing new scripts handling the changes I have to make?

Where would I add this component, the ones in the common_scripts folder or do I need to adapt anything in the stack ? If the latter, there is no definition of components, or at least I can't find them.

Thanks for any hints.

[#3 · Accepted solution]

@David Pocivalnik

every time recommendation API call is made, new entry for logs is made at path "/var/run/ambari-server/stack-recommendations/" on ambari-server host. there will be several and if you are not certain about the last entry then please move all historical existing entries to different location and then again go through add service wizard to produce a repro.

Entry will have stackadvisor.err and stackadvisor.out files. Please refer them for further debugging the issue. If it's not clear from logs, please compress the directory and post it on this thread, So that we can get more information for debugging purpose.

[#4]

Thanks for the hint. The log files provided me further details, but in the end I ended up with other problems. see my other questions. Thanks

[#5]

nice post, we tried this and it works like a charm!

we have shared the way we did this here:

https://github.com/linehrr/hbase-multi-hosting

hopefully this can help the others who's trying to achieve similar thing.