Posts: 10
Registered: ‎05-20-2014

Cannot programmatically access S3N data in CDH 5.0.0/5.0.1

An app that I'm writing needs to copy data from an S3N file system to HDFS. However, when I attempt to do this using the hadoop .jar files provided with CDH 5.0.x, I get the following error:


java.lang.VerifyError: Bad type on operand stack
Exception Details:
org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore.copy(Ljava/lang/String;Ljava/lang/String;)V @142: invokevirtual
Type 'org/jets3t/service/model/S3Object' (current frame, stack[4]) is not assignable to 'org/jets3t/service/model/StorageObject'
Current Frame:
bci: @142
flags: { }
locals: { 'org/apache/hadoop/fs/s3native/Jets3tNativeFileSystemStore', 'java/lang/String', 'java/lang/String' }
stack: { 'org/jets3t/service/S3Service', 'java/lang/String', 'java/lang/String', 'java/lang/String', 'org/jets3t/service/model/S3Object', integer }


I've uploaded the output from a Scala interpreter (with the Hadoop conf and lib folders on the path) illustrating the problem here. I've confirmed this on both the CDH 5.0 Virtualbox image and a CDH 5.0.1 that I manually created. Does the version of Hadoop that ships with CDH contain an incompatible version of jets3n? If so, are there any workarounds? 






Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Cannot programmatically access S3N data in CDH 5.0.0/5.0.1

This seems like a problem stemming from something Scala does, from what am seeing.

When I attempt to build the FS via a S3N URI in a regular Java app, the exception is not seen although the construction trace follows the same path as the reported error stack trace in your paste.

Perhaps there's something incorrect, or a bug, within Scala that may be causing this and thats where you need to look?

Our community is getting a little larger. And a lot better.

Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.