Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

error when sqoop to hbase in kerberos cluster with oozie

avatar
Expert Contributor

HI

We are working with cdh5.7 with kerberos.

There is our sqoop script, that is sqoop mysql data to hbase directly.

 

<workflow-app name="My_Workflow" xmlns="uri:oozie:workflow:0.5">
	<global>
	   <job-tracker>${jobTracker}</job-tracker>
	   <name-node>${nameNode}</name-node>
	   <configuration>
		<property>
			<name>mapred.job.queue.name</name>
			<value>root.plarch</value>
		</property>
		<property>
			<name>oozie.launcher.mapred.job.queue.name</name>
			<value>ploozie</value>
		</property>		
		</configuration>
	</global>
		<credentials>
	<credential name="hbase" type="hbase">
    </credential>

    </credentials>
    <start to="sqoop-5cf0"/>
    <kill name="Kill">
        <message>[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="sqoop-5cf0" cred="hbase">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
            <property>
		<name>hbase.zookeeper.quorum</name>
		<value>arch-od-data03.beta1</value>
	    </property>
            <property>
		<name>hbase.zookeeper.client.port</name>
	  	<value>2181</value>
            </property>  
	    </configuration>
            <command> import -Dmapreduce.job.queuename=root.plarch --connect jdbc:mysql://pis01.xxx:3306/PIS -username xxxx -password xxxxxx --table T_SITE --columns "SITE_ID,SITE_NAME,SITE_DOMAIN,REMARK,CRE_TIME,SHORT_NAME,MENU_URL,REGION,IS_SPLIT,ALIAS_NAME" --hbase-table arch_raw:t_site --column-family person --hbase-row-key SITE_ID</command>
        </sqoop>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

when we put cred="hbase" action here , the sqoop action cannot be start, we got the below error:

org.apache.oozie.action.ActionExecutorException: JA009: Broken pipe
	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:454)
	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:434)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1033)
	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1204)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)

when I remove the cred="hbase" from sqoop action config,

we got the below exception

Tue Sep 13 13:39:04 CST 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68381: row 'arch_raw:t_site,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=xxxxx,60020,1473403857069, seqNum=0

	at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
	at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
	at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
	at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)

my question is, how to sqoop mysql data to hbase in kerberos cluster directly with oozie?

 

Thanks in advance

BR

Paul

 

Who agreed with this topic