Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Connecting to sqoop jobs from oozie

Highlighted

Connecting to sqoop jobs from oozie

Expert Contributor

How to execute sqoop jobs through oozie?

I am not connecting to the default meta store and have MySQL configured for it.

I have sqoop-site.xml which will by default connect to Mysql metastore.

But oozie tries to connect to default meta store when meta-connect is not configured.

I can pass meta-connect while executing the sqoop job but how do I specify username and password for the metaconnection here?

I can execute the sqoop jobs from command line but my JDBC connection string looks like

jdbc:mysql://FQDN:3306/sqoop?user=sqoop&pass=sqoop

but this clearly does not work in oozie workflow.xml.

Neither does replacing & with & in the connection string.

2016-05-11 10:21:44,920 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
	at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
	at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
	at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
	at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
	at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

It ofcourse does not take --username and passowrd arguements.

How do people connect to a custom metastore through oozie then?

I seem to have no help regarding this from documentation either.

2 REPLIES 2
Highlighted

Re: Connecting to sqoop jobs from oozie

Mentor

@simran kaur

Have a look at this

Highlighted

Re: Connecting to sqoop jobs from oozie

You mean you tried to use & in your connection string but it didn't work?

There are some sources on the web saying that only HSQLDB is supported as Sqoop metastore. But can you try instructions given here:

  • create Mysql database "sqoop" and user "sqoop", and grant its privileges (I believe you already did this?)
  • create and populate table called SQOOP_ROOT in database "sqoop"
  • Edit sqoop-site.xml, disable autoconnect, create autoconnnect url, username and password
  • Refresh all you Sqoop clients, so that they are aware of sqoop-site updates
  • Upload sqoop-site.xml to Oozie Sqoop share/lib
  • Reload Oozie share/lib (or restart Oozie)
  • Retry your Sqoop action
Don't have an account?
Coming from Hortonworks? Activate your account here