In my case the problem was the port number which was incorrect. I ensured that I used the Namenode port.   ... View more

Hi, Harsh I configured flume on AWS to write in s3 bucket, while running flume agent it throwing some error My flume sink config: tier1.sinks.sink1.type = hdfs tier1.sinks.sink1.channel = channel1 tier1.sinks.sink1.hdfs.path = s3n://ACCESS_KEY_ID:SECRET_ACCESS_KEY@bucketname/ tier1.sinks.sink1.hdfs.filePrefix = Flumedata tier1.sinks.sink1.hdfs.fileType = DataStream tier1.sinks.sink1.hdfs.writeFormat= Text tier1.sinks.sink1.hdfs.batchSize = 100 tier1.sinks.sink1.hdfs.rollCount = 0 tier1.sinks.sink1.hdfs.rollSize = 73060835 tier1.sinks.sink1.hdfs.rollInterval = 0 #tier1.sinks.sink1.hdfs.idleTimeout = 180 #tier1.sinks.sink1.hdfs.closeTries = 0 and error: 2016-06-01 18:17:53,737 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; at amazon.emr.metrics.ClientUtil.getInstanceId(ClientUtil.java:115) at amazon.emr.metrics.MetricsConfig.getInstanceId(MetricsConfig.java:294) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:195) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:182) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:177) at amazon.emr.metrics.MetricsSaver.ensureSingleton(MetricsSaver.java:652) at amazon.emr.metrics.MetricsSaver.addInternal(MetricsSaver.java:332) at amazon.emr.metrics.MetricsSaver.addValue(MetricsSaver.java:178) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1667) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:791) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:780) at org.apache.flume.sink.hdfs.HDFSDataStream.doOpen(HDFSDataStream.java:86) at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:113) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:246) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse; at amazon.emr.metrics.ClientUtil.getInstanceId(ClientUtil.java:115) at amazon.emr.metrics.MetricsConfig.getInstanceId(MetricsConfig.java:294) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:195) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:182) at amazon.emr.metrics.MetricsConfig.<init>(MetricsConfig.java:177) at amazon.emr.metrics.MetricsSaver.ensureSingleton(MetricsSaver.java:652) at amazon.emr.metrics.MetricsSaver.addInternal(MetricsSaver.java:332) at amazon.emr.metrics.MetricsSaver.addValue(MetricsSaver.java:178) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1667) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448) at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:894) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:791) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:780) at org.apache.flume.sink.hdfs.HDFSDataStream.doOpen(HDFSDataStream.java:86) at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:113) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:246) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) please help if you are familiar with this. Thanks in advance ... View more

Glad to Hear it!  ... View more

Hi, Thanks for Your reply It's working fine i got it Before but forgot to update my answer. As You said need to remove single quotes and slashes then it's working fine. I used directly means, Instead of this ^\s*\#+|\#+$ I used direct ## to replace with pipe line symbol. ... View more

Hi, As you said i'm using spooldir source it's working fine. But one problem is flume generating more files with less records but i want like one or two files. As i said before, i have 500 records log file i want to populate as one file this is just test case but in real scenario i have lakhs of records in one log file please help . my config file is same as above which i shared with spooldir source ... View more