Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Zookeeper connection with Hbase
Labels:
- Labels:
-
Apache Oozie
Explorer
Created on ‎07-08-2014 01:18 AM - edited ‎09-16-2022 02:01 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi, i'm importing data from oracle db to hbase with sqoop.
I used sqoop line command, and all it's done.
sqoop import --connect jdbc:oracle:thin:@XXX:port/XXX --username XXX --password XXX --table XXX -m 1 --incremental lastmodified --last-value '2014-06-23' --check-column XXX --append --hbase-table XXX --column-family info --hbase-row-key XXX --hbase-bulkload
I tryed with oozie, and i have got this error in loop:
INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
My cluster configuration is with 5 host:
master,slave1,slave2,slave3,
slave4
sqoop 1 client:
slave1,slave2,slave3,slave4
zookeeper server:
master,slave1,slave2
hbase server:
hbaserestserver -> master
hbasethriftserver -> master
master -> master
regionserver -> slave1,slave2,slave3,slave4
I have some doubt on my configuration:
if my configuration is all ok, my question is:
zookeeper server with localhost/127.0.0.1:2181 is correct to start?
because i've got only 3 zookeeper server and i think this is my problem.
sqoop 1 client:
slave1,slave2,slave3,slave4
zookeeper server:
master,slave1,slave2
hbase server:
hbaserestserver -> master
hbasethriftserver -> master
master -> master
regionserver -> slave1,slave2,slave3,slave4
I have some doubt on my configuration:
if my configuration is all ok, my question is:
zookeeper server with localhost/127.0.0.1:2181 is correct to start?
because i've got only 3 zookeeper server and i think this is my problem.
And there is some SQOOP_CLASSPATH or HBASE_CLASSPATH to set? oozie don't initialize everything?
Thanks
Thanks
1 ACCEPTED SOLUTION
Mentor
Created ‎07-20-2014 06:37 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since Oozie lacks knowledge of where your HBase configs lie, you will need to pass the client hbase-site.xml file (placed somewhere on HDFS, by copying from /etc/hbase/conf/hbase-site.xml on any HBase gateway node) via the <job-xml>…</job-xml> option.
Alternatively, try the below command instead (will not be sufficient for secured clusters, which need further properties), replacing zk-host1,zk-host2,zk-host3 with your actual 3 hosts appropriately:
sqoop import -Dhbase.zookeeper.quorum=zk-host1,zk-host2,zk-host3 --connect jdbc:oracle:thin:@XXX:port/XXX --username XXX --password XXX --table XXX -m 1 --incremental lastmodified --last-value '2014-06-23' --check-column XXX --append --hbase-table XXX --column-family info --hbase-row-key XXX --hbase-bulkload
Alternatively, try the below command instead (will not be sufficient for secured clusters, which need further properties), replacing zk-host1,zk-host2,zk-host3 with your actual 3 hosts appropriately:
sqoop import -Dhbase.zookeeper.quorum=zk-host1,zk-host2,zk-host3 --connect jdbc:oracle:thin:@XXX:port/XXX --username XXX --password XXX --table XXX -m 1 --incremental lastmodified --last-value '2014-06-23' --check-column XXX --append --hbase-table XXX --column-family info --hbase-row-key XXX --hbase-bulkload
1 REPLY 1
Mentor
Created ‎07-20-2014 06:37 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since Oozie lacks knowledge of where your HBase configs lie, you will need to pass the client hbase-site.xml file (placed somewhere on HDFS, by copying from /etc/hbase/conf/hbase-site.xml on any HBase gateway node) via the <job-xml>…</job-xml> option.
Alternatively, try the below command instead (will not be sufficient for secured clusters, which need further properties), replacing zk-host1,zk-host2,zk-host3 with your actual 3 hosts appropriately:
sqoop import -Dhbase.zookeeper.quorum=zk-host1,zk-host2,zk-host3 --connect jdbc:oracle:thin:@XXX:port/XXX --username XXX --password XXX --table XXX -m 1 --incremental lastmodified --last-value '2014-06-23' --check-column XXX --append --hbase-table XXX --column-family info --hbase-row-key XXX --hbase-bulkload
Alternatively, try the below command instead (will not be sufficient for secured clusters, which need further properties), replacing zk-host1,zk-host2,zk-host3 with your actual 3 hosts appropriately:
sqoop import -Dhbase.zookeeper.quorum=zk-host1,zk-host2,zk-host3 --connect jdbc:oracle:thin:@XXX:port/XXX --username XXX --password XXX --table XXX -m 1 --incremental lastmodified --last-value '2014-06-23' --check-column XXX --append --hbase-table XXX --column-family info --hbase-row-key XXX --hbase-bulkload
