Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie sqoop job failed after enable kerberos

oozie sqoop job failed after enable kerberos

Expert Contributor

i have filed a issue list in CM forum area, but the Cloudera supporter told me about oozie issue should send to here, so i paster these to here.

 

prior to paster the issue here, i'd like to introduce my issue of oozie sqoop job.  everything is ok before i enable kerberos, but oozie sqoop job failed after enable kerberos.  

 

 

OOZIE errors while using sqoop import data(this is a very urgent case, since i still didn't resolve issue)

 

the sqoop command: 

import  -connect jdbc:mysql://10.32.87.4:3306/mdmuser -username admin -password xxx  -table  t_phone  -hive-table t_phone -hive-database report  -hive-import  -hive-overwrite  -null-string \\N  -null-non-string \\N  -hive-drop-import-delims -fields-terminated-by \001 -lines-terminated-by \n  -m 1

 

完成 状态 名称 持续时间 提交者 上次修改 Id

Sun, 16 Nov 2014 06:14:32KILLEDSqoop_Chris21m:4shueSun, 16 Nov 2014 06:14:320000000-141116214447513-oozie-oozi-W
Sun, 16 Nov 2014 05:33:53SUCCEEDEDSqoop_Chris56shueSun, 16 Nov 2014 05:33:530000001-141116211525787-oozie-oozi-W

 

i think you will ask me what's errors in container or logs, 

 

the standout like below :

35331 [main] WARN  org.apache.sqoop.hive.TableDefWriter  - Column updated_time had to be cast to a less precise type in Hive
  35341 [main] INFO  org.apache.sqoop.hive.HiveImport  - Loading uploaded data into Hive
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Intercepting System.exit(1)
  

 

the standerr is ;

 

  14/11/16 21:54:27 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
  
  Logging initialized using configuration in jar:file:/yarn/nm/filecache/791/hive-exec-0.13.1-cdh5.2.0.jar!/hive-log4j.properties
  FAILED: IllegalStateException Unxpected Exception thrown: Table not found t_phone
  Intercepting System.exit(1)
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1

 

this is very strange case, since the sqoop command has been finished success, and the data also in the tmp directory(/user/hue), but while load these data into HIVE, then it's awalys Heart beat(i don't know what does oozie doing, maybe find something.)

 

as you can see this job is success before enable kerberos, so don't doubt my table is not exist, if i add -create-hive-table tag, then it will show database not found.

 

about container log, nothing useful i have found.

 

i have checked the warehouse permission, you kow it's no problem. anyway, i didn't find useful information.

9 REPLIES 9

Re: oozie sqoop job failed after enable kerberos

Expert Contributor

hi, Cloudera support, coule you give me some advises, since i have been suspended on this issue for one week. 

 

i don't know why OOZIE sqoop job always heartbeat while kerberos enable.

Re: oozie sqoop job failed after enable kerberos

Expert Contributor
My cdh5.2 Is a new installation, and install cm first then add other service. After that, enable kerberos by CM. Test oozie sqoop function. But failed. Hive ,Impala, hbase etc.. work fine.
I am wating any responses.

Re: oozie sqoop job failed after enable kerberos

Expert Contributor

who is this forum manager?  i know this website does not gurantee to reply every message, but could you give me some advises ?

 

i don't know you guys have tested oozie sqoop function, but i have tested many times, either upgrade from CDH5.1 or new installation of CDH5.2, this issue always happend. and you konw, sometimes HADOOP erros is no useful for us ,  we can't get any direction from these errors.

 

 

hope can pay a little attention on this issue.  thanks.

Re: oozie sqoop job failed after enable kerberos

Expert Contributor

Does the sqoop command work fine without using Oozie on a kerberized cluster?  Just wanted to confirm as I didn't see it mentioned anywhere earlier.

 

Typically we recommend a two phase approach with Sqoop in Oozie, especially when using kerberos.  By this, we mean use Sqoop to dump the Data from your DB to HDFS.  Then run a Hive action that loads the data from HDFS into Hive.  Having said that, I think it might work if you use Hcat credentials in your Sqoop action.  To do this:

 

- Right after the first line of your workflow, add a credentials section like:

 

<workflow-app xmlns="uri:oozie:workflow:0.4" name="hive-wf">
<credentials>
<credential name='hcat_creds' type='hcat'>
<property>
<name>hcat.metastore.uri</name>
<value>thrift://cdh52-1.test.com:8983</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/cdh52-1.test.com@TEST.COM</value>
</property>
</credential>
</credentials>

 

- Then include the hcat credentials in your sqoop action, like:

 

<action name="sqoop-action" cred='hcat_creds'>

 

Can you try this and see if it helps?

Re: oozie sqoop job failed after enable kerberos

Expert Contributor

yes, the  oozie sqoop job is success before enable kerberos. 

 

your suggestion is unacceptable for me,  since sqoop can import data from mysql or oracle to hive directly.  i just wonder, why oozie sqoop job failed after enable kerberos, and prompt the table not found. and if i add -create-hive-table, then prompt database not found.

 

 

about your suggestion i will try, but this is not best solution. 

 

 

Re: oozie sqoop job failed after enable kerberos

Master Guru
In secure mode your action may hang infinitely if it gets told to talk to a Hive MetaStore Server (over thrift), unless you perform the mentioned credentials step above.

Were you able to try the credentials addition prior to moving off of using Hive imports directly (and doing so subsequently via another action)?

Re: oozie sqoop job failed after enable kerberos

Expert Contributor
Truth be told. I have tried this solution , but failed too. I just wonder,have you tried sqoop by oozie in Kerberos mode? It's success?
Highlighted

Re: oozie sqoop job failed after enable kerberos

New Contributor

Today I ran into the exact same issue.

 

After enabling kerberos, if 'Sqoop Action' tries to pull data from rdbms into HDFS and then import into Hive as part of  ONE action, it will FAIL.

 

But as recommended above, if you SPLIT the SqoopAction into two, Sqoop action to pull data into HDFS and the Hive Action to upload data into Hive it works. You MUST use HCAT credentials in oozie workflow as suggested by cloudera above.

 

I think its a bug..

 

Thank you

Raja Thiruvathuru

 

 

 

 

Re: oozie sqoop job failed after enable kerberos

New Contributor

I am facing the same issue. Any resolutons, besides splitting the OOzie task into 2 ( which is not acceptable)?