Hi. Thank you for your support on this issue! We will proceed to upgrade our production cluster to 5.7.1 now that we know the details for this log entry. ... View more

This is my testing result at the Hadoop master node which used for namenode and hive server2.   When I executing beeline for local file loading to a table, I met same error. It was the permission issue on the file to the hive user which is the owner of HiveServer2 process.   It was solve when I grant read permission the file including whole path.   Please check the file accessible permission as this. sudo -u hive cat ~/test.txt     ... View more

Hi,   Can you please please suggest, how to add a python script wrapped in a bash script which is located in some other hdfs /apps directory ?   Regards,  Murari ... View more

Glad to hear @AxelJ! Feel free to mark the thread resolved so its easier for others with similar problems to locate your solution. ... View more

Thanks very much Harsh ! Appreciated ! ... View more

Hi, Harsh I configured flume on AWS to write in s3 bucket, while running flume agent it throwing some error My flume sink config: tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@bucketname/ tier1.sinks.sink1.hdfs.filePrefix = Flumedata tier1.sinks.sink1.hdfs.fileType = DataStream tier1.sinks.sink1.hdfs.writeFormat= Text tier1.sinks.sink1.hdfs.batchSize = 100 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollSize = 73060835 tier1.sinks.sink1.hdfs.rollInterval = 0 #tier1.sinks.sink1.hdfs.idleTimeout = 180 #tier1.sinks.sink1.hdfs.closeTries = 0 and error: 2016-06-01 18:17:53,737 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; at amazon.emr.metrics.ClientUtil.getInstanceId(ClientUtil.java:115) at amazon.emr.metrics.MetricsConfig.getInstanceId(MetricsConfig.java:294) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:195) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:182) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:177) at amazon.emr.metrics.MetricsSaver.ensureSingleton(MetricsSaver.java:652) at amazon.emr.metrics.MetricsSaver.addInternal(MetricsSaver.java:332) at amazon.emr.metrics.MetricsSaver.addValue(MetricsSaver.java:178) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1667) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:791) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:780) at org.apache.flume.sink.hdfs.HDFSDataStream.doOpen(HDFSDataStream.java:86) at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:113) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:246) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; at amazon.emr.metrics.ClientUtil.getInstanceId(ClientUtil.java:115) at amazon.emr.metrics.MetricsConfig.getInstanceId(MetricsConfig.java:294) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:195) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:182) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:177) at amazon.emr.metrics.MetricsSaver.ensureSingleton(MetricsSaver.java:652) at amazon.emr.metrics.MetricsSaver.addInternal(MetricsSaver.java:332) at amazon.emr.metrics.MetricsSaver.addValue(MetricsSaver.java:178) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1667) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:791) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:780) at org.apache.flume.sink.hdfs.HDFSDataStream.doOpen(HDFSDataStream.java:86) at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:113) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:246) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) please help if you are familiar with this. Thanks in advance ... View more