Member since
12-07-2014
16
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
46347 | 12-18-2014 04:44 PM |
04-30-2015
12:24 PM
Thanks, I'll upgrade my cluster and try it again
... View more
03-17-2015
03:39 PM
Thank you for telling me this, you really save my day!
... View more
03-16-2015
06:39 PM
After I upgrade from CDH 5.3.1 to CDH 5.3.2, and then run "select" SQL queries, it gives me some errors: select * from my_table limit 10; FAILED: SemanticException Unable to determine if s3n://my-bucket/is encrypted: java.lang.IllegalArgumentException: Wrong FS: s3n://my-bucket/, expected: hdfs://us0701aohc172018080109.tangome.gbl:8020 I tryied to drop my table and re-create it again add partitions to it, then run select queries, still the same error. Before upgrading everything is OK. Does anybody happen to know why?
... View more
Labels:
12-18-2014
05:34 PM
Besides, there is a dirty work you need to do, since CDH 5.2 dosen't have the class org/apache/hadoop/fs/s3native/ProgressableResettableBufferedFileInputStream You need to extract this class from Amazon EMR and repackage it. Or you can compile one from its source code here: https://github.com/libin/s3distcp/blob/master/src/main/java/com/amazon/external/elasticmapreduce/s3distcp/ProgressableResettableBufferedFileInputStream.java
... View more
12-18-2014
04:44 PM
I have resolved this, download aws-java-sdk from http://sdk-for-java.amazonwebservices.com/latest/aws-java-sdk.zip, unzip it, and copy every jar in aws-java-sdk/lib/ and aws-java-sdk/third-party/ to your datanodes' /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop
... View more
12-10-2014
06:16 PM
I used this method to resolve NoClassDefFoundError problems before and it worked everytime, but this time, it didn't 😞
... View more
12-10-2014
06:15 PM
I downloaded the s3distcp jar from s3://elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar, and run it as the following: hadoop jar ~/s3distcp.jar --dest hdfs://resource-manager.localhost:8020/user/ec2-user/2015/ --src s3n://test/2015/ But it failed, the error message was: Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/services/s3/AmazonS3Client at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.createAmazonS3Client(S3DistCp.java:456) at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.createInputFileListS3(S3DistCp.java:405) at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.createInputFileList(S3DistCp.java:380) at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:640) at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:523) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.amazon.external.elasticmapreduce.s3distcp.Main.main(Main.java:13) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) I had already copied aws-java-sdk-s3-1.9.7.jar to /opt/cloudera/parcels/CDH/jars on every nodes, and added "/opt/cloudera/parcels/CDH/jars/*" to mapreduce.application.classpath via the cloudera manager web console and restarted the cluster.
... View more
Labels:
12-10-2014
06:10 PM
Thanks, this solved my problem, I didn't notice that my disk is too small,
... View more
12-08-2014
10:04 PM
When I'm installing a CDH cluster using Cloudera Manager, I got the following errors: CDH 5.2.1-1.cdh5.2.1.p0.12 ui.nErrors Error when distributing to xxxthis is hostname) : Untar failed with return code 2. Error when distributing to xxx: Untar failed with return code 2. Error when distributing to xxx : Untar failed with return code 2. Error when distributing to xxx: Untar failed with return code 2. Error when distributing to xxx : Untar failed with return code 2. Error when distributing to xxx : Untar failed with return code 2. Error when distributing to xxx : Untar failed with return code 2. Error when distributing to xxx: Untar failed with return code 2. Error when distributing to xxx: Untar failed with return code 2. The log of cloudera manager agent is as follows: [08/Dec/2014 21:47:01 +0000] 31154 Thread-13 downloader INFO Starting download of: http://hostname:7180/cmf/parcel/download/CDH-5.2.1-1.cdh5.2.1.p0.12-el6.parcel [08/Dec/2014 21:47:01 +0000] 31154 Thread-13 downloader INFO Completed download of http://hostname:7180/cmf/parcel/download/CDH-5.2.1-1.cdh5.2.1.p0.12-el6.parcel code=200 state=downloaded [08/Dec/2014 21:47:01 +0000] 31154 Thread-13 parcel_cache INFO Unpacking /opt/cloudera/parcel-cache/CDH-5.2.1-1.cdh5.2.1.p0.12-el6.parcel into /opt/cloudera/parcels [08/Dec/2014 21:47:01 +0000] 31154 Thread-13 throttling_logger ERROR Error while unpacking /opt/cloudera/parcel-cache/CDH-5.2.1-1.cdh5.2.1.p0.12-el6.parcel Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/parcel_cache.py", line 179, in cb unpack_parcel(op.path, unpack_dir, self.path) File "/usr/lib64/cmf/agent/src/cmf/parcel_cache.py", line 200, in unpack_parcel raise Exception("Untar failed with return code %d" % (return_code, )) Exception: Untar failed with return code 2 Does anyone happens to know why?
... View more
Labels: