question Re: HBASE/FALCON replication in Archives of Support Questions (Read Only)

HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 02:05:27 GMT

I am trying to setup Falcon to setup HBASE replication.

So I tried to create a Falcon cluster as outlined in :

https://hortonworks.com/hadoop-tutorial/create-falcon-cluster/

But after creating the Falcon user and logging into Ambari as Falcon user, I found the screen empty - without the Your Views options as listed in above url under sub heading : Preparing HDFS directories.

But if I log out and then re-login as admin then I can see the views for Yarn, Hive and Tez.

Am I missing something?

Appreciate the feedback.

Re: HBASE/FALCON replication

ylu — Fri, 03 Mar 2017 02:35:28 GMT

First, I think you meant HDFS/Falcon replication. Hbase has its own replication method.

Regarding you see screen empty, that's the permission issue. I think you missed one step in the tutorial that to setup permission (assign to views group in the tutorial) for the newly created falcon user.

Re: HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 02:50:40 GMT

BTW I am using Hortonworks 2.3.6 - I think.

/usr/hdp/2.3.6.0-3796/

So not sure if I am following the right document - appreciate pointers to the right document for configuring Falcon.

Re: HBASE/FALCON replication

ylu — Fri, 03 Mar 2017 02:54:13 GMT

I am copying the part of tutorial below for your reference.

You can see the newly added falcon user. Click on it to assign it a group so that it can access Ambari views. Write "views" and select it in LocalGroupMembership box and then click on tick mark to add a falcon user in the "views" group.

Re: HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 04:41:20 GMT

Right I saw that and tried it. But there was no pre-existing group called views and if I tired to enter the value and click the right button, it would give error. Do we need to create this group ourselves or will that pre-exist?

Appreciate the feedback.

Re: HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 04:53:56 GMT

I am not able to do this step :

Write "views" and select it in LocalGroupMembership box and then click on tick mark to add a falcon user in the "views" group.

Because if I type in Views in the Local Group Membership/Add group, and trying to click on the Tick mark, it gives an error : Cannot add user to group An internal system exception occurred: Group Views doesn't exist

Re: HBASE/FALCON replication

ylu — Fri, 03 Mar 2017 05:22:06 GMT

the views is the group that was pre-defined in a tutorial before this one. So, you need to create a group called views before you can assign the falcon user to this group.

In order to add user/group to access views, you can just follow this path: admin->manage ambari->views->click one of the views->click that view link-> go to permission->add user or group that can access to that view.

Re: HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 05:51:37 GMT

Frank thanks so much!!!

Re: HBASE/FALCON replication

vtpcnk — Fri, 03 Mar 2017 22:19:15 GMT

I am still not seeing the 'Files View' when I open ambari as Falcon. Any idea what I could be missing?

Frank, you had mentioned that I had missed some earlier steps to create group etc - can you point me to the exact documentation I have missed?

Appreciate the feedback.

Re: HBASE/FALCON replication

ylu — Fri, 03 Mar 2017 22:59:24 GMT

have you seen any other views by setting correct permission? If just only the files view, you may follow the steps below:

You need to set up an HDFS proxy user for the Ambari daemon account. For example, if ambari-server daemon is runnng as root, you set up a proxy user for root in core-site by adding and changing properties in HDFS > Configs > Custom core-site:

hadoop.proxyuser.root.groups=*

hadoop.proxyuser.root.hosts=*

Restart the required components as indicated by Ambari.

Re: HBASE/FALCON replication

vtpcnk — Sat, 04 Mar 2017 00:08:35 GMT

It is all good now. Thanks.

Re: HBASE/FALCON replication

vtpcnk — Sat, 04 Mar 2017 01:42:16 GMT

In Creating a Cluster Entity, I am not sure what values to use.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_data_governance/content/section_creating_cluster_entity.html

What are the values to be entered in the Interfaces and Properties sections?

The Hortonworks documentation lacks specific instructions.

Re: HBASE/FALCON replication

ylu — Sat, 04 Mar 2017 02:18:01 GMT

The interfaces are required component. I think when you enter into that page, it will give you sample value, i.e. hftp://sandbox.hortonworks.com:50070, you then need to change the url to you own setting if not on the sandbox. Detail information on these setting, you can find at https://falcon.apache.org/EntitySpecification.html.

The properties are optional, you can define your own properties. You can check in the link I listed above as well.

Re: HBASE/FALCON replication

vtpcnk — Sat, 04 Mar 2017 04:26:46 GMT

Frank thanks. I found all the required values and clicked on Save. It is still giving some error but I am not able to see that the error is. If I click on Notifications I can see some old errors but not able to scroll down to see the latest error. If I click on the scroll bar the Notification window simply closes. Any idea where the physical logs of falcon are held?

Re: HBASE/FALCON replication

vtpcnk — Sat, 04 Mar 2017 04:50:19 GMT

It finally came through after a long long time. I thought there was an error but after more than an hour I got a create cluster success message! Ok, next step is to create the Falcon backup cluster. My HDP backup cluster is actually another separate physical cluster. So should I create the Falcon backup cluster on the primary or the backup cluster. Appreciate the feedback.

Re: HBASE/FALCON replication

vtpcnk — Tue, 07 Mar 2017 01:27:31 GMT

I had setup Falcon mirroring based on this document

https://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falcon/

The one single difference is that unlike the example in the document which seems to be using a single hdfs cluster, I am using two different clusters - each with its set of nodes.

At the outset after running the mirror job it is NOT working for me.

Some initial questions are :

1. After creating a Falcon Cluster, how do we view it(s) definition? I had created two falcon clusters - one for the primary and another for the target. I would like to see the information falcon holds for them.

2. After executing a Falcon job how do we check the logs to see what it did and where it failed?

Appreciate the insights.

Re: HBASE/FALCON replication

vtpcnk — Tue, 07 Mar 2017 22:52:27 GMT

Hello, any feedback on the Falcon issues? I have configured the Falcon clusters and also the job. But replication is not happening. I enabled Ozzie log as shown in the Hortonworks site (including restarting Oozie) but still I can see the log as disabled in the job error log. Not sure why.

Also in Falcon UI if I click on the job I can see one or two instances (?) of the job running, while the rest are WAITING. So to restart the job I just have to Suspend and Resume?

So when will the RUNNING jobs end and when will the WAITING jobs start?

Appreciate the feedback.

Re: HBASE/FALCON replication

ylu — Tue, 07 Mar 2017 22:58:25 GMT

Could you open a new thread, so that somebody else will chime in if I am not available?

Re: HBASE/FALCON replication

bandalora — Wed, 08 Mar 2017 03:14:29 GMT

See if these docs help: http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-views/content/ch_using_falcon_view.html and http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_data-movement-and-integration/content/index.html.