In this article, we will see how to perform the mirroring of Hive data and metadata using Falcon from source cluster to destination cluster. This article is based on HDP 2.5.
- Configure Hive
Configure source and target Hive by clicking “Hive” from the Ambari Services menu, then click “configs” to add following custom properties on Ambari UI by scroll down to “Custom hive-site”, click it and then click “Add Property”.
Add following property name with value:
hive.metastore.event.listeners = org.apache.hive.hcatalog.listener.DbNotificationListener hive.metastore.dml.events = true
Press OK to save the changes, then click Restart all the impacted services.
- Bootstrap Table and DB
Before creating Hive DR mirroring job to replicate Hive data/metadata for a DB or Table, it is required to perform initial bootstrap of Table and DB from source to target cluster.
Table Bootstrap
For bootstrapping table replication, do an EXPORT of the table in question at source cluster, distcp the export directory to the target cluster, and do an IMPORT at target cluster. Export-Import can be seen here :
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
For example create the table global_sales and insert records:
hive > create table global_sales
(customer_id string, item_id string, quantity float, price float, time timestamp)
partitioned by (country string);
hive > insert into table global_sales partition (country = 'us') values ('c1', 'i1', '1', '1', '2001-01-01 01:01:01');Start Bootstrap
## On source cluster : hive > export table global_sales to '/user/ambari-qa/export_sql'; $ hadoop distcp hdfs://machine-1-1.openstacklocal:8020/user/ambari-qa/export_sql hdfs://machine-2-1.openstacklocal:8020/user/ambari-qa/import_sql ## On target cluster : hive > import table global_sales from '/user/ambari-qa/import_sql';
Above steps will set up the target table in sync with source table so that the events on the source cluster that modify the table will then be replicated over.
Database Bootstrap
For bootstrapping DB replication, first target DB must be created. This step is expected because DB replication definitions can be set up only on pre-existing DB by users.
Second, we need to export all tables in the source D
B and import them in the target DB, as described in Table bootstrap.
- Set up source and target cluster staging/working directory
Source cluster:
[root@machine-1-1 ~]# su - falcon
hadoop fs -mkdir -p /apps/falcon/primaryCluster/staging
hadoop fs -mkdir -p /apps/falcon/primaryCluster/working
hadoop fs -chmod 777 /apps/falcon/primaryCluster/staging
Target cluster:
[root@machine-2-1 ~]# su - falcon
hadoop fs -mkdir -p /apps/falcon/backupCluster/staging
hadoop fs -mkdir -p /apps/falcon/backupCluster/working
hadoop fs -chmod 777 /apps/falcon/backupCluster/staging
- Create cluster entity
Navigate to Falcon UI from Ambari services menu and create source cluster entity using Falcon UI by clicking “Create” -> “Cluster”
Save the source cluster entity by clicking “Next”->”Save” .
Create target cluster entity using Falcon UI by clicking “Create” -> “Cluster”
Save the cluster entity by clicking “Next”->”Save” .
- Insert records in source Hive server for replication.
Insert some records in source Hive server to replicate to target Hive server.
hive > insert into table global_sales partition (country = 'uk') values ('c2', 'i2', '2', '2', '2001-01-01 01:01:02');- Prepare and submit Hive DR Mirroring
To submit the Hive DR mirroring job, click “Create”->”Mirror”->”Hive” and then fill the required values.
Click Next -> Save the Hive DR mirror job.
- Submit and Schedule HiveDR
- Check output
Once scheduled Hive DR process completed (checked from Oozie UI), verify the target Hive server for output.
Earlier, we inserted two records at source Hive server and now at target Hive server both records are available.
---
