Reply
Expert Contributor
Posts: 163
Registered: ‎09-29-2014

oozie sqoop job failed after enable kerberos

i have filed a issue list in CM forum area, but the Cloudera supporter told me about oozie issue should send to here, so i paster these to here.

 

prior to paster the issue here, i'd like to introduce my issue of oozie sqoop job.  everything is ok before i enable kerberos, but oozie sqoop job failed after enable kerberos.  

 

 

OOZIE errors while using sqoop import data(this is a very urgent case, since i still didn't resolve issue)

 

the sqoop command: 

import  -connect jdbc:mysql://10.32.87.4:3306/mdmuser -username admin -password xxx  -table  t_phone  -hive-table t_phone -hive-database report  -hive-import  -hive-overwrite  -null-string \\N  -null-non-string \\N  -hive-drop-import-delims -fields-terminated-by \001 -lines-terminated-by \n  -m 1

 

完成 状态 名称 持续时间 提交者 上次修改 Id

Sun, 16 Nov 2014 06:14:32KILLEDSqoop_Chris21m:4shueSun, 16 Nov 2014 06:14:320000000-141116214447513-oozie-oozi-W
Sun, 16 Nov 2014 05:33:53SUCCEEDEDSqoop_Chris56shueSun, 16 Nov 2014 05:33:530000001-141116211525787-oozie-oozi-W

 

i think you will ask me what's errors in container or logs, 

 

the standout like below :

35331 [main] WARN  org.apache.sqoop.hive.TableDefWriter  - Column updated_time had to be cast to a less precise type in Hive
  35341 [main] INFO  org.apache.sqoop.hive.HiveImport  - Loading uploaded data into Hive
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Heart beat
  Intercepting System.exit(1)
  

 

the standerr is ;

 

  14/11/16 21:54:27 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
  
  Logging initialized using configuration in jar:file:/yarn/nm/filecache/791/hive-exec-0.13.1-cdh5.2.0.jar!/hive-log4j.properties
  FAILED: IllegalStateException Unxpected Exception thrown: Table not found t_phone
  Intercepting System.exit(1)
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1

 

this is very strange case, since the sqoop command has been finished success, and the data also in the tmp directory(/user/hue), but while load these data into HIVE, then it's awalys Heart beat(i don't know what does oozie doing, maybe find something.)

 

as you can see this job is success before enable kerberos, so don't doubt my table is not exist, if i add -create-hive-table tag, then it will show database not found.

 

about container log, nothing useful i have found.

 

i have checked the warehouse permission, you kow it's no problem. anyway, i didn't find useful information.

Expert Contributor
Posts: 163
Registered: ‎09-29-2014

Re: oozie sqoop job failed after enable kerberos

hi, Cloudera support, coule you give me some advises, since i have been suspended on this issue for one week. 

 

i don't know why OOZIE sqoop job always heartbeat while kerberos enable.

Expert Contributor
Posts: 163
Registered: ‎09-29-2014

Re: oozie sqoop job failed after enable kerberos

My cdh5.2 Is a new installation, and install cm first then add other service. After that, enable kerberos by CM. Test oozie sqoop function. But failed. Hive ,Impala, hbase etc.. work fine.
I am wating any responses.
Expert Contributor
Posts: 163
Registered: ‎09-29-2014

Re: oozie sqoop job failed after enable kerberos

who is this forum manager?  i know this website does not gurantee to reply every message, but could you give me some advises ?

 

i don't know you guys have tested oozie sqoop function, but i have tested many times, either upgrade from CDH5.1 or new installation of CDH5.2, this issue always happend. and you konw, sometimes HADOOP erros is no useful for us ,  we can't get any direction from these errors.

 

 

hope can pay a little attention on this issue.  thanks.

Cloudera Employee
Posts: 98
Registered: ‎07-31-2013

Re: oozie sqoop job failed after enable kerberos

Does the sqoop command work fine without using Oozie on a kerberized cluster?  Just wanted to confirm as I didn't see it mentioned anywhere earlier.

 

Typically we recommend a two phase approach with Sqoop in Oozie, especially when using kerberos.  By this, we mean use Sqoop to dump the Data from your DB to HDFS.  Then run a Hive action that loads the data from HDFS into Hive.  Having said that, I think it might work if you use Hcat credentials in your Sqoop action.  To do this:

 

- Right after the first line of your workflow, add a credentials section like:

 

<workflow-app xmlns="uri:oozie:workflow:0.4" name="hive-wf">
<credentials>
<credential name='hcat_creds' type='hcat'>
<property>
<name>hcat.metastore.uri</name>
<value>thrift://cdh52-1.test.com:8983</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>hive/cdh52-1.test.com@TEST.COM</value>
</property>
</credential>
</credentials>

 

- Then include the hcat credentials in your sqoop action, like:

 

<action name="sqoop-action" cred='hcat_creds'>

 

Can you try this and see if it helps?

Expert Contributor
Posts: 163
Registered: ‎09-29-2014

Re: oozie sqoop job failed after enable kerberos

yes, the  oozie sqoop job is success before enable kerberos. 

 

your suggestion is unacceptable for me,  since sqoop can import data from mysql or oracle to hive directly.  i just wonder, why oozie sqoop job failed after enable kerberos, and prompt the table not found. and if i add -create-hive-table, then prompt database not found.

 

 

about your suggestion i will try, but this is not best solution. 

 

 

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: oozie sqoop job failed after enable kerberos

In secure mode your action may hang infinitely if it gets told to talk to a Hive MetaStore Server (over thrift), unless you perform the mentioned credentials step above.

Were you able to try the credentials addition prior to moving off of using Hive imports directly (and doing so subsequently via another action)?
Expert Contributor
Posts: 163
Registered: ‎09-29-2014

Re: oozie sqoop job failed after enable kerberos

Truth be told. I have tried this solution , but failed too. I just wonder,have you tried sqoop by oozie in Kerberos mode? It's success?
New Contributor
Posts: 4
Registered: ‎02-09-2015

Re: oozie sqoop job failed after enable kerberos

Today I ran into the exact same issue.

 

After enabling kerberos, if 'Sqoop Action' tries to pull data from rdbms into HDFS and then import into Hive as part of  ONE action, it will FAIL.

 

But as recommended above, if you SPLIT the SqoopAction into two, Sqoop action to pull data into HDFS and the Hive Action to upload data into Hive it works. You MUST use HCAT credentials in oozie workflow as suggested by cloudera above.

 

I think its a bug..

 

Thank you

Raja Thiruvathuru

 

 

 

 

New Contributor
Posts: 1
Registered: ‎04-26-2015

Re: oozie sqoop job failed after enable kerberos

I am facing the same issue. Any resolutons, besides splitting the OOzie task into 2 ( which is not acceptable)?

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.