Custom shared lib for Sqoop

[#1]

Hi,

due to the bug reported here:

https://issues.apache.org/jira/browse/SQOOP-1420

I have created a custom version of Sqoop forcing the timezone to be GMT when converting epoch to String. Now, I need to use the custom jar in Oozie, when launching the Sqoop action.

Looking at this: http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-oozie/ I choose to override the shared lib for the Sqoop action adding:

<configuration>
  <property>
    <name>oozie.action.sharelib.for.sqoop</name>
    <value>sqoopPatched</value>
  </property>  
</configuration>

in the Sqoop action (and creating the "sqoopPatched" folder in the shared lib path "/user/oozie/share/lib/lib_20140625170654/sqoopPatched" with the modified "sqoop-1.4.4-cdh5.0.2.jar"). However, the workflow fails because it doesn't find the main method of Sqoop.

Anyone knows how to solve this? May be that with the new version of the shared lib (http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/) is it not possible anymore to override libs?

Thanks,

Michele

PS: I do not want to change the standard sqoop in the shared lib because I am not sure that the modified jar is right for any kind of timestamp column/database. But I still need it in 10 jobs, so I prefer to create a custom shared lib instead of adding a lib folder in each workflow folder.

[#2 · Accepted solution]

Hi,

You can definately do this.  I'd have to see the actual exception to be sure, but I'm guessing by "it doesn't find the main method of Sqoop." you were getting a ClassNotFoundException on SqoopMain?  SqoopMain is actually an Oozie class and is in the oozie-sharelib-sqoop-4.0.0-cdh5.0.2.jar in the sqoop sharelib.  You'll need to copy this jar into your sqoopPatched sharelib.  You may also need some of the other jars from the Sqoop sharelib.  The safest thing is to copy the "sqoop" sharelib to "sqoopPatched" and then just replace your custom sqoop-1.4.4-cdh5.0.2.jar in there; this way, you'll be sure to have everything.

- Robert

[#3]

Hi Robert,

thanks for the answer. You are right, that one is my exception. I had it even copying the entire "sqoop" dir in "sqoop2" dir without replacing any jar. However, doing tests I had changed permissions and restarted the Oozie server and now it is working.

Thanks,

Michele