28787
DISCUSSIONS
102102
MEMBERS
3161
ARTICLES
Created on 09-12-2016 11:26 PM - edited 09-12-2016 11:28 PM
HI
We are working with cdh5.7 with kerberos.
There is our sqoop script, that is sqoop mysql data to hbase directly.
<workflow-app name="My_Workflow" xmlns="uri:oozie:workflow:0.5"> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>root.plarch</value> </property> <property> <name>oozie.launcher.mapred.job.queue.name</name> <value>ploozie</value> </property> </configuration> </global> <credentials> <credential name="hbase" type="hbase"> </credential> </credentials> <start to="sqoop-5cf0"/> <kill name="Kill"> <message>[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <action name="sqoop-5cf0" cred="hbase"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>hbase.zookeeper.quorum</name> <value>arch-od-data03.beta1</value> </property> <property> <name>hbase.zookeeper.client.port</name> <value>2181</value> </property> </configuration> <command> import -Dmapreduce.job.queuename=root.plarch --connect jdbc:mysql://pis01.xxx:3306/PIS -username xxxx -password xxxxxx --table T_SITE --columns "SITE_ID,SITE_NAME,SITE_DOMAIN,REMARK,CRE_TIME,SHORT_NAME,MENU_URL,REGION,IS_SPLIT,ALIAS_NAME" --hbase-table arch_raw:t_site --column-family person --hbase-row-key SITE_ID</command> </sqoop> <ok to="End"/> <error to="Kill"/> </action> <end name="End"/> </workflow-app>
when we put cred="hbase" action here , the sqoop action cannot be start, we got the below error:
org.apache.oozie.action.ActionExecutorException: JA009: Broken pipe at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:454) at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:434) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1033) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1204) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64) at org.apache.oozie.command.XCommand.call(XCommand.java:286) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65)
when I remove the cred="hbase" from sqoop action config,
we got the below exception
Tue Sep 13 13:39:04 CST 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68381: row 'arch_raw:t_site,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=xxxxx,60020,1473403857069, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320) at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295) at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160) at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
my question is, how to sqoop mysql data to hbase in kerberos cluster directly with oozie?
Thanks in advance
BR
Paul