Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.5: Falcon Sqoop Import/Export not working

HDP 2.5: Falcon Sqoop Import/Export not working

New Contributor

I have created read only and write type datasources in Falcon using MySql database Manager.

When I have test the connection in datasource it is success.

Note: I have uploaded the MySQL driver into the Oozie share path as mentioned in the link,

https://falcon.apache.org/0.9/ImportExport.html

11559-sqoop-import.png

I have created the feed to import but Job action got failed in Oozie,

11560-oozie-job-action.png

I have looked into the Yarn log to find the error,

java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:328) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1853) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1653) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.main(Sqoop.java:243) at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:202) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:182) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Found the parameter in YARN log,

Sqoop command arguments : import --connect jdbc:mysql://instance_IP/test --table db_raw_data --username sqoop_user --password sqoop --num-mappers 1 --delete-target-dir --target-dir hdfs://cluster_URI/user/tsldp/falcon/demo/mysql-feed-import-filesystem/2017-01-21-11-00

Driver Jar is not passed as an argument which we have configured for MySQL datasource.

Its looks like,Falcon Sqoop Import and Export won't work from Falcon UI because of this issue.

Thanks,

Uvaraj.S

1 REPLY 1
Highlighted

Re: HDP 2.5: Falcon Sqoop Import/Export not working

New Contributor

@Uvaraj Seerangan Make sure that the oozie sharelib has been updated after adding the MySQL driver

- Update oozie sharelib:

# oozie admin -oozie http://<oozie-server>:11000/oozie -sharelibupdate

- Check that the jar is available for the sqoop action:

# oozie admin -oozie http://<oozie-server>:11000/oozie -shareliblist sqoop*

If there is still an issue, it would be helpful to review the oozie launcher logs and the sqoop action logs to verify the driver was loaded correctly.

Don't have an account?
Coming from Hortonworks? Activate your account here