Reply
Highlighted
Explorer
Posts: 11
Registered: ‎07-14-2015
Accepted Solution

Error Sqooping data with Oozie, java.lang.NoSuchMethodError

[ Edited ]

Im currently trying to Sqoop some data from an Oracle DB to HDFS using Oozie to schedule the sqoop workflow.

My Sqoop version: Sqoop 1.4.5-cdh5.4.2

My Sqoop code:

import -- connect {JDBCpath} \

--username {Username} \

--password {Password} \

--verbose \

--table {Table} \

--where "{Query}" \

-z \

--compression-codec org.apache.hadoop.io.compress.SnappyCodec \

--as-parquetfile \

--target-dir {TargetDirectory} \

--split-by {columnToSplitBy} \

-m 14

 

The Error I have been getting:

java.lang.NoSuchMethodError: org.kitesdk.data.impl.Accessor.registerDatasetRepository(Lorg/kitesdk/data/spi/URIPattern;Lorg/kitesdk/data/spi/OptionBuilder;)V

 

[Edited to focus on my main problem, rather than the research I've done into it which may or may not be the correct path in solving this. This information will be reposted in a reply.]

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Error Sqooping data with Oozie, java.lang.NoSuchMethodError

Unclear from your description, but given that CDH 5.4 includes Kite 1.0.0 in its default packaging, are you using an older version manually for some other purpose?
Explorer
Posts: 11
Registered: ‎07-14-2015

Re: Error Sqooping data with Oozie, java.lang.NoSuchMethodError

[ Edited ]

[Research I've done: It looks like the method registerDatasetRepository was included in the kite-data-core.jar file in the 0.14.1 version but was removed in version 0.15.0. We are currently using one of the kite-data-core.jar files after version 0.15.0 and I am unable to use the older version since there are other things using that jar on the hdfs.

Is there any way I can tell sqoop to not use kite or add another jar with this method included or any other solution to this error?]

 

We want to use the newest version (and as far as i know, we currently are) but the code that is generated from the sqoop command wants to use the registerDatasetRepository method from kite-data-core.jar. But it looks like that method no longer exists in the newer versions of kite-data-core.jar.

Explorer
Posts: 11
Registered: ‎07-14-2015

Re: Error Sqooping data with Oozie, java.lang.NoSuchMethodError

Would there be any way to tell oozie to not use kite when sqooping to avoid this error entirely?

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Error Sqooping data with Oozie, java.lang.NoSuchMethodError

Sqoop will use Kite if parquet-file output format is mentioned.

However, I am unable to reproduce your issue on a fresh 5.4.2 Oozie install (Sqoop action import done from a MySQL source, but Kite gets used if you specify --as-parquetfile, not dependent on the source).

Was your CDH upgraded from another release, or is it a new cluster installation?

Can you check if your Oozie system share-lib is perhaps using older kite jars? If it does look older, can you run the Oozie -> Actions -> Update ShareLib command from CM, and then retry?
Explorer
Posts: 11
Registered: ‎07-14-2015

Re: Error Sqooping data with Oozie, java.lang.NoSuchMethodError

Updating the SharLib Worked! =D I did have to bounce oozie before it fixed it but this solution works.