Hi All, I have a POC cluster on Cloudera 5.9 running on 3 Nodes. However recently I am noticing I get HDFS disk space alert very frequently. My total cluster size is 1.8 TB. I noticed that size of my dfs/dn/current/<blockpool>/current/finalized directory is too high. I am aware the finalized contains blocks that are not being written to by a client and have been completed. However whenever I move some of the subdir to anyother mount, it is replaced very fast in a couple of days with again many files(subdir). I have these questions: Can I delete old subdirs becuase they only contain the info of the files that were written and completed. The auto-generation of so many files in a day means, that the connectivity of that particular node to Namenode is frquently going up and down. Hence creating so many subdirs? Size of dfs/dn/current/<blockpool>/current/finalized on 3 Nodes: [hdfs@MasterNode1 current]$ du -sh finalized/ 639G finalized/ [root@DataNode1 current]# du -sh finalized/ 435G finalized/ [root@DataNode2 current]# du -sh finalized 426G finalized Just for Nov 29 and 30 you can see so many Subdirs created, almost of a size of 800 MB to 3 GB drwxr-xr-x 20 hdfs hdfs 4096 Nov 29 10:07 subdir41 drwxr-xr-x 13 hdfs hdfs 4096 Nov 29 10:09 subdir42 drwxr-xr-x 31 hdfs hdfs 4096 Nov 29 10:12 subdir43 drwxr-xr-x 24 hdfs hdfs 4096 Nov 29 10:17 subdir44 drwxr-xr-x 26 hdfs hdfs 4096 Nov 29 10:20 subdir45 drwxr-xr-x 17 hdfs hdfs 4096 Nov 29 10:24 subdir46 drwxr-xr-x 10 hdfs hdfs 4096 Nov 29 10:25 subdir47 drwxr-xr-x 29 hdfs hdfs 4096 Nov 29 10:32 subdir48 drwxr-xr-x 21 hdfs hdfs 4096 Nov 29 10:40 subdir51 drwxr-xr-x 12 hdfs hdfs 4096 Nov 29 10:40 subdir52 drwxr-xr-x 13 hdfs hdfs 4096 Nov 29 11:30 subdir53 drwxr-xr-x 27 hdfs hdfs 4096 Nov 29 11:30 subdir54 drwxr-xr-x 15 hdfs hdfs 4096 Nov 29 11:32 subdir55 drwxr-xr-x 117 hdfs hdfs 4096 Nov 29 13:48 subdir69 drwxr-xr-x 119 hdfs hdfs 4096 Nov 29 14:36 subdir71 drwxr-xr-x 136 hdfs hdfs 4096 Nov 29 15:18 subdir79 drwxr-xr-x 258 hdfs hdfs 12288 Nov 29 15:46 subdir193 drwxr-xr-x 89 hdfs hdfs 4096 Nov 29 16:06 subdir33 drwxr-xr-x 129 hdfs hdfs 4096 Nov 30 05:34 subdir72 drwxr-xr-x 122 hdfs hdfs 4096 Nov 30 06:21 subdir75 drwxr-xr-x 124 hdfs hdfs 4096 Nov 30 07:55 subdir77 drwxr-xr-x 95 hdfs hdfs 4096 Nov 30 08:32 subdir78 drwxr-xr-x 126 hdfs hdfs 4096 Nov 30 11:32 subdir85 drwxr-xr-x 124 hdfs hdfs 4096 Nov 30 12:08 subdir86 drwxr-xr-x 112 hdfs hdfs 4096 Nov 30 13:25 subdir88 drwxr-xr-x 130 hdfs hdfs 4096 Nov 30 14:25 subdir90 drwxr-xr-x 112 hdfs hdfs 4096 Nov 30 15:00 subdir91 drwxr-xr-x 57 hdfs hdfs 4096 Nov 30 18:23 subdir26 drwxr-xr-x 173 hdfs hdfs 4096 Nov 30 19:01 subdir34 drwxr-xr-x 30 hdfs hdfs 4096 Nov 30 19:03 subdir49 drwxr-xr-x 11 hdfs hdfs 4096 Nov 30 19:03 subdir50 drwxr-xr-x 27 hdfs hdfs 4096 Nov 30 19:06 subdir56 drwxr-xr-x 79 hdfs hdfs 4096 Nov 30 19:08 subdir57 drwxr-xr-x 141 hdfs hdfs 4096 Nov 30 19:49 subdir61 drwxr-xr-x 109 hdfs hdfs 4096 Nov 30 21:53 subdir64 drwxr-xr-x 126 hdfs hdfs 4096 Nov 30 22:08 subdir65 drwxr-xr-x 136 hdfs hdfs 4096 Nov 30 23:08 subdir68 Please advice. Thanks, Shilpa ... View more

Hi, I have 3 node Cloudera 5.9 Cluster running on CentOS 6.7. Recently during any write operation on Hadoop, I am witnessing these errors in Datanode logs. However the write happens but I am concerned why this is happening. 2017-07-29 10:33:04,109 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: <datanodename>:50010:DataXceiver error processing WRITE_BLOCK operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:802) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246) at java.lang.Thread.run(Thread.java:745) 2017-07-29 10:36:06,172 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute DatanodeNetworkCounts of Hadoop:service=DataNode,name=DataNodeInfo threw an except ion javax.management.RuntimeMBeanException: java.lang.NullPointerException at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:346) at org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:324) at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:217) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1296) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.datanode.DataNode.getDatanodeNetworkCounts(DataNode.java:1956) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) ... 31 more 2017-07-29 10:36:06,231 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute NamenodeAddresses of Hadoop:service=DataNode,name=DataNodeInfo threw an exception javax.management.RuntimeMBeanException: java.lang.NullPointerException 2017-07-31 14:49:41,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: <datanodename>:50010:DataXceiver error processing WRITE_BLOCK operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010 java.io.IOException: Not ready to serve the block pool, BP-939287337-X.X.X.X-1484085163925. at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:1284) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1292) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246) at java.lang.Thread.run(Thread.java:745) 2017-07-31 15:04:30,304 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:338ms (threshold=300ms) 2017-07-31 15:42:09,593 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: lnxdatanode2.centralus.cloudapp.azure.com:50010:DataXceiver error processing WRITE_BLOCK operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010 java.io.IOException: Premature EOF from inputStream at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:802) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246) at java.lang.Thread.run(Thread.java:745) Some Important configurations of my cluster: yarn.nodemanager.resource.memory-mb - 12GB yarn.scheduler.maximum-allocation-mb - 16GB mapreduce.map.memory.mb - 4GB mapreduce.reduce.memory.mb - 4GB mapreduce.map.java.opts.max.heap - 3GB mapreduce.reduce.java.opts.max.heap - 3GB namenode_java_heapsize - 6GB secondarynamenode_java_heapsize - 6GB dfs_datanode_max_locked_memory - 3GB dfs blocksize - 128 MB Can anyone please help me? Thanks, Shilpa ... View more

Hi All, I have 3 node Cloudera 5.9 Cluster. I am trying to use Flume to ingest data from Twitter using a keyword. However I am facing 2 issues: 1. File generated has no information related to the keywords used. [hdfs@XXXX ~]$ hadoop fs -cat /user/flume/twitter_data/FlumeData.1496272139910|grep "rosario" [hdfs@XXXX ~]$ 2. The file have non-printable or gibberish characters My Flume.conf is as follow: # Naming the components on the current agent. TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS # Describing/Configuring the source TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource TwitterAgent.sources.Twitter.consumerKey = XXXX TwitterAgent.sources.Twitter.consumerSecret = XXXX TwitterAgent.sources.Twitter.accessToken = XXXX TwitterAgent.sources.Twitter.accessTokenSecret = XXXX TwitterAgent.sources.Twitter.keywords = rosario brindis # Describing/Configuring the sink TwitterAgent.sinks.HDFS.type = hdfs TwitterAgent.sinks.HDFS.hdfs.path = hdfs://X.X.X.X:8020/user/hdfs/twitter_data/ TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000 TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 TwitterAgent.sinks.HDFS.hdfs.callTimeout = 180000 # Describing/Configuring the channel TwitterAgent.channels.MemChannel.type = memory TwitterAgent.channels.MemChannel.capacity = 100000 TwitterAgent.channels.MemChannel.transactionCapacity = 1000 # Binding the source and sink to the channel TwitterAgent.sources.Twitter.channels = MemChannel TwitterAgent.sinks.HDFS.channel = MemChannel Please help as I am not sure what is going wrong. Thanks, Shilpa ... View more

Flume hdfs sink is creating too many multiple files into hdfs for loading source file as small as 200KB. I have used rollCount set to 256MB even then it is not helping. Some file created on HDFS is as small as 1.5KB. Please help me make a single file because this file has to be read by R for filtering of Keywords. My flume.conf is agent.sources = avro-collection-source agent.channels = memoryChannel agent.sinks = hdfs-sink # For each one of the sources, the type is defined agent.sources.avro-collection-source.type = avro agent.sources.avro-collection-source.bind = 10.0.0.6 agent.sources.avro-collection-source.port = 60000 agent.sources.avro-collection-source.interceptors = interceptor1 agent.sources.avro-collection-source.interceptors.interceptor1.type = timestamp # The channel can be defined as follows. agent.sources.avro-collection-source.channels = memoryChannel # Each sink's type must be defined agent.sinks.hdfs-sink.type = hdfs #agent.sinks.hdfs-sink.hdfs.path = hdfs://10.0.10.4:8020/flume/events #agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S agent.sinks.hdfs-sink.useLocalTimeStamp = true agent.sinks.hdfs-sink.hdfs.callTimeout = 180000 #Specify the channel the sink should use agent.sinks.hdfs-sink.channel = memoryChannel # File size to trigger roll, in bytes (256Mb) agent.sinks.hdfs-sink.rollSize = 268435456 # Number of seconds to wait before rolling current file (in seconds) agent.sinks.sink.hdfs.rollInterval = 0 agent.sinks.sink.hdfs.rollCount = 0 # Each channel's type is defined. agent.channels.memoryChannel.type = memory Thanks, Shilpa ... View more

Hi, I have 3 node cluster having Cloudera 5.9 running on CentOS6.7. My Spark is running on Yarn. I have to install R on Spark home directoryso that I can use SparkR. I got the epel RPM, and then tried to install R using YUM however its giving error. I even tried some other RPM however they are giving error too. Using --skip-broken option is also not working. Please help [root@LnxMasterNode01 spark]# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm Retrieving http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm warning: /var/tmp/rpm-tmp.XuRVi8: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:epel-release ########################################### [100%] Last part of the code shows error below: [root@LnxMasterNode01 spark]# yum install R Loaded plugins: fastestmirror, security Setting up Install Process Loading mirror speeds from cached hostfile * epel: ftp.osuosl.org Resolving Dependencies --> Running transaction check ---> Package R.i686 0:2.13.0-2.el6.rf will be updated ---> Package R.x86_64 0:3.3.2-2.el5 will be an update --> Processing Dependency: libRmath-devel = 3.3.2-2.el5 for package: R-3.3.2-2.el5.x86_64 --> Processing Dependency: R-devel = 3.3.2-2.el5 for package: R-3.3.2-2.el5.x86_64 --> Running transaction check . . ---> Package ppl.x86_64 0:0.10.2-11.el6 will be installed ---> Package texlive-texmf-errata-dvips.noarch 0:2007-7.1.el6 will be installed ---> Package texlive-texmf-errata-fonts.noarch 0:2007-7.1.el6 will be installed --> Finished Dependency Resolution Error: Package: R-core-3.3.2-2.el5.x86_64 (epel) Requires: libtk8.4.so()(64bit) Error: Package: R-core-3.3.2-2.el5.x86_64 (epel) Requires: libtcl8.4.so()(64bit) Error: Package: R-core-3.3.2-2.el5.x86_64 (epel) Requires: libgssapi.so.2(libgssapi_CITI_2)(64bit) Error: Package: R-core-3.3.2-2.el5.x86_64 (epel) Requires: libRblas.so()(64bit) Error: Package: libRmath-3.3.2-2.el5.x86_64 (epel) Requires: libgssapi.so.2(libgssapi_CITI_2)(64bit) Error: Package: libRmath-3.3.2-2.el5.x86_64 (epel) Requires: libgssapi.so.2()(64bit) Error: Package: R-core-3.3.2-2.el5.x86_64 (epel) Requires: libgssapi.so.2()(64bit) You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest [root@LnxMasterNode01 spark]# I also checked here http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ and http://www.jason-french.com/blog/2013/03/11/installing-r-in-linux/ these two links also suggest the same thing. Am I doing something wrong. Please suggest. Thanks, Shilpa ... View more

Hi Community, I have 3 node cluster, using latest cloudera parcels for 5.9 version. OS is CentOS 6.7 on all three of them. I am using Flume to get data from a third party website(basically a news website) to hdfs. Using http-source, mem-channel and hdfs-sink. The flume log says that source,channel and sink are started but there is no data streaming to hdfs. There is no further entry in the log after http-source started. I am not sure why nothing is happening. My flume.conf is: tier1.sources = http-source tier1.channels = mem-channel-1 tier1.sinks = hdfs-sink # For each source, channel, and sink, set tier1.sources.http-source.type = org.apache.flume.source.http.HTTPSource tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler tier1.sources.http-source.bind = localhost tier1.sources.http-source.url = http://www.jornada.unam.mx/ultimas tier1.sources.http-source.port = 5440 tier1.sources.http-source.channels = mem-channel-1 tier1.channels.mem-channel-1.type = memory tier1.sinks.hdfs-sink.type = hdfs tier1.sinks.hdfs-sink.channel = mem-channel-1 tier1.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S tier1.channels.mem-channel-1.capacity = 100 Flume-ng.log in debug mode: 2016-12-30 12:20:47,634 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 2016-12-30 12:20:47,649 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider started 2016-12-30 12:20:47,650 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume. conf for changes 2016-12-30 12:20:47,650 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/var/run/cloudera-scm-agent/process/600-flum e-AGENT/flume.conf 2016-12-30 12:20:47,654 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink 2016-12-30 12:20:47,654 DEBUG org.apache.flume.conf.FlumeConfiguration: Created context for hdfs-sink: channel 2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink 2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: tier1 2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink 2016-12-30 12:20:47,655 DEBUG org.apache.flume.conf.FlumeConfiguration: Starting validation of configuration for agent: tier1 2016-12-30 12:20:47,656 INFO org.apache.flume.conf.LogPrivacyUtil: Logging of configuration details is disabled. To see configuration details in the log run the agent w ith -Dorg.apache.flume.log.printconfig=true JVM argument. Please note that this is not recommended in production systems as it may leak private information to the logfi le. 2016-12-30 12:20:47,661 DEBUG org.apache.flume.conf.FlumeConfiguration: Created channel file-channel-1 2016-12-30 12:20:47,667 DEBUG org.apache.flume.conf.FlumeConfiguration: Creating sink: hdfs-sink using HDFS 2016-12-30 12:20:47,669 DEBUG org.apache.flume.conf.FlumeConfiguration: Channels:file-channel-1 2016-12-30 12:20:47,669 DEBUG org.apache.flume.conf.FlumeConfiguration: Sinks hdfs-sink 2016-12-30 12:20:47,670 DEBUG org.apache.flume.conf.FlumeConfiguration: Sources http-source 2016-12-30 12:20:47,670 INFO org.apache.flume.conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [tier1] 2016-12-30 12:20:47,670 INFO org.apache.flume.node.AbstractConfigurationProvider: Creating channels 2016-12-30 12:20:47,677 INFO org.apache.flume.channel.DefaultChannelFactory: Creating instance of channel file-channel-1 type file 2016-12-30 12:20:47,696 INFO org.apache.flume.node.AbstractConfigurationProvider: Created channel file-channel-1 2016-12-30 12:20:47,697 INFO org.apache.flume.source.DefaultSourceFactory: Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource 2016-12-30 12:20:47,697 DEBUG org.apache.flume.source.DefaultSourceFactory: Source type org.apache.flume.source.http.HTTPSource is a custom type 2016-12-30 12:20:47,757 INFO org.apache.flume.sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs 2016-12-30 12:20:47,768 INFO org.apache.flume.node.AbstractConfigurationProvider: Channel file-channel-1 connected to [http-source, hdfs-sink] 2016-12-30 12:20:47,774 INFO org.apache.flume.node.Application: Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flu me.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@f212168 counterGroup:{ name:null counters:{} } }} channels:{file-channel-1=FileChannel file-channel-1 { dataDirs: [/mnt/resource/flume/data] }} } 2016-12-30 12:20:47,783 INFO org.apache.flume.node.Application: Starting Channel file-channel-1 2016-12-30 12:20:47,783 INFO org.apache.flume.channel.file.FileChannel: Starting FileChannel file-channel-1 { dataDirs: [/mnt/resource/flume/data] }... 2016-12-30 12:20:47,793 INFO org.apache.flume.channel.file.Log: Encryption is not enabled2016-12-30 12:20:47,794 INFO org.apache.flume.channel.file.Log: Replay started 2016-12-30 12:20:47,795 INFO org.apache.flume.channel.file.Log: Found NextFileID 0, from [] 2016-12-30 12:20:47,814 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Preallocated /mnt/resource/flume/checkpoint/checkpoint to 8008232 for capacity 1000000 2016-12-30 12:20:47,816 INFO org.apache.flume.channel.file.EventQueueBackingStoreFileV3: Starting up with /mnt/resource/flume/checkpoint/checkpoint and /mnt/resource/flume/checkpoint/checkpoint.meta 2016-12-30 12:20:47,916 INFO org.apache.flume.channel.file.FlumeEventQueue: QueueSet population inserting 0 took 0 2016-12-30 12:20:47,919 INFO org.apache.flume.channel.file.Log: Last Checkpoint Fri Dec 30 12:20:47 CST 2016, queue depth = 0 2016-12-30 12:20:47,922 INFO org.apache.flume.channel.file.Log: Replaying logs with v2 replay logic 2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.ReplayHandler: Starting replay of [] 2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0 2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.FlumeEventQueue: Search Count = 0, Search Time = 0, Copy Count = 0, Copy Time = 0 2016-12-30 12:20:47,932 INFO org.apache.flume.channel.file.Log: Rolling /mnt/resource/flume/data 2016-12-30 12:20:47,932 INFO org.apache.flume.channel.file.Log: Roll start /mnt/resource/flume/data 2016-12-30 12:20:47,941 INFO org.apache.flume.tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) 2016-12-30 12:20:47,944 INFO org.apache.flume.tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 503316480, Remaining = 503316480 2016-12-30 12:20:47,972 INFO org.apache.flume.channel.file.LogFile: Opened /mnt/resource/flume/data/log-1 2016-12-30 12:20:47,981 INFO org.apache.flume.channel.file.Log: Roll end 2016-12-30 12:20:47,981 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Start checkpoint for /mnt/resource/flume/checkpoint/checkpoint, elements to sync = 0 2016-12-30 12:20:47,985 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1483122047933, queueSize: 0, queueHead: 0 2016-12-30 12:20:48,001 INFO org.apache.flume.channel.file.Log: Updated checkpoint for file: /mnt/resource/flume/data/log-1 position: 0 logWriteOrderID: 1483122047933 2016-12-30 12:20:48,001 INFO org.apache.flume.channel.file.FileChannel: Queue Size after replay: 0 [channel=file-channel-1] 2016-12-30 12:20:48,037 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: file-channel-1: Successfully registered new MBean. 2016-12-30 12:20:48,037 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: file-channel-1 started 2016-12-30 12:20:48,037 INFO org.apache.flume.node.Application: Starting Sink hdfs-sink 2016-12-30 12:20:48,037 INFO org.apache.flume.node.Application: Starting Source http-source 2016-12-30 12:20:48,039 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs-sink: Successfully registered new MBean. 2016-12-30 12:20:48,039 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started 2016-12-30 12:20:48,040 DEBUG org.apache.flume.SinkRunner: Polling sink runner starting 2016-12-30 12:20:48,062 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-12-30 12:20:48,064 INFO org.mortbay.log: jetty-6.1.26.cloudera.42016-12-30 12:20:48,135 INFO org.mortbay.log: Started SelectChannelConnector@localhost:5440 2016-12-30 12:20:48,135 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: http-source: Successfully registered new MBean. 2016-12-30 12:20:48,136 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: http-source started 2016-12-30 12:21:17,794 DEBUG org.apache.flume.channel.file.FlumeEventQueue: Checkpoint not required 2016-12-30 12:21:18,088 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume.conf for changes 2016-12-30 12:21:47,794 DEBUG org.apache.flume.channel.file.FlumeEventQueue: Checkpoint not required 2016-12-30 12:21:48,089 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume.conf for changes Please help me find a solution. I even tried replacing mem-channel with File-channel, even then it is stopping at the same point with no data streaming. My project is at critical stage. Please guide me. Thanks, Shilpa ... View more