Member since
12-09-2015
34
Posts
5
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
691 | 05-05-2017 05:19 PM |
12-11-2017
09:07 PM
Hi All, I have a POC cluster on Cloudera 5.9 running on 3 Nodes. However recently I am noticing I get HDFS disk space alert very frequently. My total cluster size is 1.8 TB. I noticed that size of my dfs/dn/current/<blockpool>/current/finalized directory is too high. I am aware the finalized contains blocks that are not being written to by a client and have been completed. However whenever I move some of the subdir to anyother mount, it is replaced very fast in a couple of days with again many files(subdir). I have these questions:
Can I delete old subdirs becuase they only contain the info of the files that were written and completed. The auto-generation of so many files in a day means, that the connectivity of that particular node to Namenode is frquently going up and down. Hence creating so many subdirs? Size of dfs/dn/current/<blockpool>/current/finalized on 3 Nodes: [hdfs@MasterNode1 current]$ du -sh finalized/
639G finalized/
[root@DataNode1 current]# du -sh finalized/
435G finalized/
[root@DataNode2 current]# du -sh finalized
426G finalized Just for Nov 29 and 30 you can see so many Subdirs created, almost of a size of 800 MB to 3 GB drwxr-xr-x 20 hdfs hdfs 4096 Nov 29 10:07 subdir41
drwxr-xr-x 13 hdfs hdfs 4096 Nov 29 10:09 subdir42
drwxr-xr-x 31 hdfs hdfs 4096 Nov 29 10:12 subdir43
drwxr-xr-x 24 hdfs hdfs 4096 Nov 29 10:17 subdir44
drwxr-xr-x 26 hdfs hdfs 4096 Nov 29 10:20 subdir45
drwxr-xr-x 17 hdfs hdfs 4096 Nov 29 10:24 subdir46
drwxr-xr-x 10 hdfs hdfs 4096 Nov 29 10:25 subdir47
drwxr-xr-x 29 hdfs hdfs 4096 Nov 29 10:32 subdir48
drwxr-xr-x 21 hdfs hdfs 4096 Nov 29 10:40 subdir51
drwxr-xr-x 12 hdfs hdfs 4096 Nov 29 10:40 subdir52
drwxr-xr-x 13 hdfs hdfs 4096 Nov 29 11:30 subdir53
drwxr-xr-x 27 hdfs hdfs 4096 Nov 29 11:30 subdir54
drwxr-xr-x 15 hdfs hdfs 4096 Nov 29 11:32 subdir55
drwxr-xr-x 117 hdfs hdfs 4096 Nov 29 13:48 subdir69
drwxr-xr-x 119 hdfs hdfs 4096 Nov 29 14:36 subdir71
drwxr-xr-x 136 hdfs hdfs 4096 Nov 29 15:18 subdir79
drwxr-xr-x 258 hdfs hdfs 12288 Nov 29 15:46 subdir193
drwxr-xr-x 89 hdfs hdfs 4096 Nov 29 16:06 subdir33
drwxr-xr-x 129 hdfs hdfs 4096 Nov 30 05:34 subdir72
drwxr-xr-x 122 hdfs hdfs 4096 Nov 30 06:21 subdir75
drwxr-xr-x 124 hdfs hdfs 4096 Nov 30 07:55 subdir77
drwxr-xr-x 95 hdfs hdfs 4096 Nov 30 08:32 subdir78
drwxr-xr-x 126 hdfs hdfs 4096 Nov 30 11:32 subdir85
drwxr-xr-x 124 hdfs hdfs 4096 Nov 30 12:08 subdir86
drwxr-xr-x 112 hdfs hdfs 4096 Nov 30 13:25 subdir88
drwxr-xr-x 130 hdfs hdfs 4096 Nov 30 14:25 subdir90
drwxr-xr-x 112 hdfs hdfs 4096 Nov 30 15:00 subdir91
drwxr-xr-x 57 hdfs hdfs 4096 Nov 30 18:23 subdir26
drwxr-xr-x 173 hdfs hdfs 4096 Nov 30 19:01 subdir34
drwxr-xr-x 30 hdfs hdfs 4096 Nov 30 19:03 subdir49
drwxr-xr-x 11 hdfs hdfs 4096 Nov 30 19:03 subdir50
drwxr-xr-x 27 hdfs hdfs 4096 Nov 30 19:06 subdir56
drwxr-xr-x 79 hdfs hdfs 4096 Nov 30 19:08 subdir57
drwxr-xr-x 141 hdfs hdfs 4096 Nov 30 19:49 subdir61
drwxr-xr-x 109 hdfs hdfs 4096 Nov 30 21:53 subdir64
drwxr-xr-x 126 hdfs hdfs 4096 Nov 30 22:08 subdir65
drwxr-xr-x 136 hdfs hdfs 4096 Nov 30 23:08 subdir68 Please advice. Thanks, Shilpa
... View more
Labels:
08-01-2017
01:02 AM
Hi, I have 3 node Cloudera 5.9 Cluster running on CentOS 6.7. Recently during any write operation on Hadoop, I am witnessing these errors in Datanode logs. However the write happens but I am concerned why this is happening. 2017-07-29 10:33:04,109 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: <datanodename>:50010:DataXceiver error processing WRITE_BLOCK
operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:802)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
at java.lang.Thread.run(Thread.java:745)
2017-07-29 10:36:06,172 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute DatanodeNetworkCounts of Hadoop:service=DataNode,name=DataNodeInfo threw an except
ion
javax.management.RuntimeMBeanException: java.lang.NullPointerException
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:346)
at org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:324)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1296)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.DataNode.getDatanodeNetworkCounts(DataNode.java:1956)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
... 31 more
2017-07-29 10:36:06,231 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting attribute NamenodeAddresses of Hadoop:service=DataNode,name=DataNodeInfo threw an exception
javax.management.RuntimeMBeanException: java.lang.NullPointerException
2017-07-31 14:49:41,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: <datanodename>:50010:DataXceiver error processing WRITE_BLOCK operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010 java.io.IOException: Not ready to serve the block pool, BP-939287337-X.X.X.X-1484085163925.
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:1284)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.java:1292)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:624)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
at java.lang.Thread.run(Thread.java:745)
2017-07-31 15:04:30,304 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:338ms (threshold=300ms)
2017-07-31 15:42:09,593 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: lnxdatanode2.centralus.cloudapp.azure.com:50010:DataXceiver error processing WRITE_BLOCK operation src: /Y.Y.Y.Y:43298 dst: /X.X.X.X:50010 java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:802)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
at java.lang.Thread.run(Thread.java:745) Some Important configurations of my cluster:
yarn.nodemanager.resource.memory-mb - 12GB
yarn.scheduler.maximum-allocation-mb - 16GB
mapreduce.map.memory.mb - 4GB
mapreduce.reduce.memory.mb - 4GB
mapreduce.map.java.opts.max.heap - 3GB
mapreduce.reduce.java.opts.max.heap - 3GB
namenode_java_heapsize - 6GB
secondarynamenode_java_heapsize - 6GB
dfs_datanode_max_locked_memory - 3GB dfs blocksize - 128 MB
Can anyone please help me? Thanks, Shilpa
... View more
Labels:
05-31-2017
11:24 PM
Hi All, I have 3 node Cloudera 5.9 Cluster. I am trying to use Flume to ingest data from Twitter using a keyword. However I am facing 2 issues: 1. File generated has no information related to the keywords used. [hdfs@XXXX ~]$ hadoop fs -cat /user/flume/twitter_data/FlumeData.1496272139910|grep "rosario"
[hdfs@XXXX ~]$
2. The file have non-printable or gibberish characters My Flume.conf is as follow: # Naming the components on the current agent.
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
# Describing/Configuring the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = XXXX
TwitterAgent.sources.Twitter.consumerSecret = XXXX
TwitterAgent.sources.Twitter.accessToken = XXXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXXX
TwitterAgent.sources.Twitter.keywords = rosario brindis
# Describing/Configuring the sink
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://X.X.X.X:8020/user/hdfs/twitter_data/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.callTimeout = 180000
# Describing/Configuring the channel
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 100000
TwitterAgent.channels.MemChannel.transactionCapacity = 1000
# Binding the source and sink to the channel
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel
Please help as I am not sure what is going wrong. Thanks, Shilpa
... View more
Labels:
05-05-2017
05:19 PM
1 Kudo
Hi All, I found the Heap size of Datanode Role of my NameNode was low (1 GB) hence increased it to 3GB and hprof files are now not getting generated. I changed the heap dump path back to /tmp to verify for 24 hours ago to verify. HDFS > configuration > DataNode DefaultGroup > Resource Management > Java Heap Size of DataNode in Bytes in Cloudera Manager. Thanks, Shilpa
... View more
05-03-2017
09:30 PM
Hmm.. actually I thought about it once but didnt do it. But till I find a resolution i need a workaround. So i have changed the HeapDump Path to /dev/null but only for Datanodes. Thanks @Ward Bekker 🙂
... View more
05-03-2017
05:35 PM
Hi All, I have 3 node cluster running on CentOS 6.7 having cloudera 5.9. Namenode has been facing issue of .hprof files in /tmp directory leading to 100% disk usage on / mount. The owner of these files are hdfs:hadoop. I know hprof is created when we have a heap dump of the process at the time of the failure. This is typically seen in scenarios with "java.lang.OutOfMemoryError". Hence I increased the RAM of my NN to 112GB from 56GB. My configs are: yarn.nodemanager.resource.memory-mb - 12GB yarn.scheduler.maximum-allocation-mb - 16GB mapreduce.map.memory.mb - 4GB mapreduce.reduce.memory.mb - 4GB mapreduce.map.java.opts.max.heap - 3GB mapreduce.reduce.java.opts.max.heap - 3GB namenode_java_heapsize - 6GB secondarynamenode_java_heapsize - 6GB dfs_datanode_max_locked_memory - 3GB dfs blocksize - 128 MB The datanode log on NN has below error but they are also present on other DN (on all 3 nodes basically): 2017-05-03 10:03:17,914 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode{data=FSDataset{dirpath='[/bigdata/dfs/dn/current]'}, localName='XXXX.azure.com:50010', datanodeUuid='4ea75665-b223-4456-9308-1defcad54c89', xmitsInProgress=0}:Exception transfering block BP-939287337-X.X.X.4-148408516
3925:blk_1077604623_3864267 to mirror X.X.X.5:50010: java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready for read. ch : java.ni
o.channels.SocketChannel[connected local=/X.X.X.4:43801 remote=X.X.X.5:50010]
2017-05-03 10:03:17,922 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: XXXX.azure.com:50010:DataXceiver error processing WRITE_BLO
CK operation src: /X.X.X.4:53902 dst: /X.X.X.4:50010
java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/X.X.X.4:438
01 remote=/X.X.X.5:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2241)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:743)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
at java.lang.Thread.run(Thread.java:745)
2017-05-03 10:04:52,371 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: XXXX.azure.com:50010:DataXceiver error processing WRITE_BLO
CK operation src: /X.X.X.4:54258 dst: /X.X.X.4:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:201)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:500)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:896) This log is getting such errors even during night time or early morning when nothing is running. My cluster is used to getting webpage info using wget and then processing the data using SparkR. Apart from this, I am also getting Block count more than threshold, for which I have another thread. http://community.cloudera.com/t5/Storage-Random-Access-HDFS/Datanodes-report-block-count-more-than-t... Please help!. Cluster configs(after recent upgrades) - NN: RAM- 112GB, Core 16, Disk : 500GB DN1: RAM- 56GB, Core 8, Disk: 400GB DN2: RAM- 28GB, Core 4, Disk: 400GB Thanks, Shilpa
... View more
Labels:
01-24-2017
12:02 AM
Can anyone help please.
... View more
01-21-2017
01:47 AM
I set log4jproperty to debug however I didnt get anything suspicious. what to do
... View more
01-20-2017
10:39 PM
Thanks @Michael M . I am already running flume in debug mode using flume-ng agent -n agent -c conf -f /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng/conf/flume.conf -Dflume.root.logger=DEBUG There is nothing suspicious in the log: 17/01/20 16:28:35 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memoryChannel: Successfully registered new MBean.
17/01/20 16:28:35 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memoryChannel started
17/01/20 16:28:35 INFO node.Application: Starting Sink hdfs-sink
17/01/20 16:28:35 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs-sink: Successfully registered new MBean.
17/01/20 16:28:35 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started
17/01/20 16:28:35 INFO node.Application: Starting Source avro-collection-source
17/01/20 16:28:35 INFO source.AvroSource: Starting Avro source avro-collection-source: { bindAddress: 10.*.*.*, port: 60000 }...
17/01/20 16:28:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: avro-collection-source: Successfully registered new MBean.
17/01/20 16:28:36 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: avro-collection-source started
17/01/20 16:28:36 INFO source.AvroSource: Avro source avro-collection-source started.
17/01/20 16:28:51 INFO ipc.NettyServer: [id: 0x9444dc65, /10.*.*.*:42268 => /*.*.*.*:60000] OPEN
17/01/20 16:28:51 INFO ipc.NettyServer: [id: 0x9444dc65, /10.*.*.*:42268 => /*.*.*.*:60000] BOUND: /*.*.*.*:60000
17/01/20 16:28:51 INFO ipc.NettyServer: [id: 0x9444dc65, /10.*.*.*:42268 => /*.*.*.*:60000]
CONNECTED: /*.*.*.*:42268
17/01/20 16:28:51 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawLocalFileSystem = false
17/01/20 16:28:52 INFO hdfs.BucketWriter: Creating hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331600.tmp
17/01/20 16:28:55 INFO hdfs.BucketWriter: Closing hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331600.tmp
17/01/20 16:28:55 INFO hdfs.BucketWriter: Renaming hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331600.tmp to hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331600
17/01/20 16:28:55 INFO hdfs.BucketWriter: Creating hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331601.tmp
17/01/20 16:28:55 INFO hdfs.BucketWriter: Closing hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331601.tmp
17/01/20 16:28:55 INFO hdfs.BucketWriter: Renaming hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331601.tmp to hdfs://*.*.*.*:8020/user/hdfs/flume/17-01-20/1628/FlumeData.1484951331601
... View more
01-20-2017
10:01 PM
@Michael Mi tried both but it didn't help. It is still craeting multiple tiny files # File size to trigger roll, in bytes (256Mb)
agent.sinks.hdfs-sink.rollSize = 65536
# Number of seconds to wait before rolling current file (in seconds)
agent.sinks.hdfs-sink.rollInterval = 3600
#agent.sinks.hdfs-sink.rollCount = 0
or even with this # File size to trigger roll, in bytes (256Mb)
agent.sinks.hdfs-sink.rollSize = 65536
# Number of seconds to wait before rolling current file (in seconds)
agent.sinks.hdfs-sink.rollInterval = 0
#agent.sinks.hdfs-sink.rollCount = 0
... View more
01-20-2017
09:05 PM
Also, these tiny files doesn't have anything readable. Are they even useful. I am so worried: [hdfs@LnxMasterNode01 www.jornada.unam.mx]$ hadoop fs -text "/user/hdfs/flume/17-01-20/1320/FlumeData.1484940053671"
1484940053689 64 65 73 63 72 69 70 74 69 6f 6e 3a 20 41 6d 69 67 75 69 73
1484940053689
1484940053689 54 69 74 6c 65 3a 20 52 6f 63 68 61
1484940053689 4c 69 6e 6b 3a 20 68 74 74 70 3a 2f 2f 77 77 77 2e 6a 6f 72 6e 61 64 61 2e 75 6e 61 6d 2e 6d 78 2f 32 30 31 37 2f 30 31 2f 32 30 2f 69 6e 64 65 78 2e 70 68 70 3f 73 65 63 74 69 6f 6e 3d 63 61 72 74 6f 6e 65 73 26 69 64 3d 33 26 70 61 72 74 6e 65 72 3d 72 73 73
1484940053689 50 75 62 6c 69 73 68 20 44 61 74 65 3a 20 46 72 69 2c 20 32 30 20 4a 61 6e 20 32 30 31 37 20 30 38 3a 30 37 3a 33 32 20 47 4d 54
1484940053689 61 75 74 68 6f 72 3a 20
1484940057285 63 6f 6d 6d 65 6e 74 73 3a 20
1484940057285 64 65 73 63 72 69 70 74 69 6f 6e 3a 20 45 6c 20 76 65 63 69 6e 6f 20 64 65 20 61 72 72 69 62 61
1484940057285
[hdfs@LnxMasterNode01 www.jornada.unam.mx]$ hadoop fs -cat "/user/hdfs/flume/17-01-20/1320/FlumeData.1484940053671"
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable▒E▒X̭a▒▒▒▒7'▒HY▒S
▒description: Amiguis
Y▒S
Y▒S
▒
Title: RochaY▒S
▒WLink: http://www.jornada.unam.mx/2017/01/20/index.php?section=cartones&id=3&partner=rssY▒S
▒+Publish Date: Fri, 20 Jan 2017 08:07:32 GMY▒S
author: ▒▒▒▒▒E▒X̭a▒▒▒▒7'▒Y▒S▒
comments: Y▒S▒ description: El vecino de arriba
Y▒S▒▒▒▒▒▒E▒X̭a▒▒▒▒7'▒
... View more
01-20-2017
07:31 PM
I did this, it helped a little. The only thing after deleting %M/%S is that flume is not creating some sub-directories under /user/hdfs/flume/17-01-20/ directory like earlier hadoop fs -ls /user/hdfs/flume/17-01-20/
Found 6 items
drwxr-xr-x - hdfs supergroup 0 2017-01-20 12:24 /user/hdfs/flume/17-01-20/1224 <- This and 2 below directories and more sub/directories were created earlier under them
drwxr-xr-x - hdfs supergroup 0 2017-01-20 12:25 /user/hdfs/flume/17-01-20/1225
drwxr-xr-x - hdfs supergroup 0 2017-01-20 12:26 /user/hdfs/flume/17-01-20/1226
drwxr-xr-x - hdfs supergroup 0 2017-01-20 13:23 /user/hdfs/flume/17-01-20/13 <- without %M%S
drwxr-xr-x - hdfs supergroup 0 2017-01-20 13:21 /user/hdfs/flume/17-01-20/1320 <- without %S
-rw-r--r-- 2 hdfs supergroup 0 2017-01-20 13:10 /user/hdfs/flume/17-01-20/jornada
However still, There are multiple tiny files getting created under <date>/<hour> directory Like these [root@LnxMasterNode01 RSS]# hadoop fs -ls /user/hdfs/flume/17-01-20/13
Found 96 items
-rw-r--r-- 2 hdfs supergroup 1138 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151434
-rw-r--r-- 2 hdfs supergroup 1069 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151435
-rw-r--r-- 2 hdfs supergroup 1122 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151436
-rw-r--r-- 2 hdfs supergroup 594 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151437
-rw-r--r-- 2 hdfs supergroup 1131 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151438
-rw-r--r-- 2 hdfs supergroup 1203 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151439
-rw-r--r-- 2 hdfs supergroup 1509 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151440
-rw-r--r-- 2 hdfs supergroup 963 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151441
-rw-r--r-- 2 hdfs supergroup 865 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151442
-rw-r--r-- 2 hdfs supergroup 1273 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151443
-rw-r--r-- 2 hdfs supergroup 961 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151444
-rw-r--r-- 2 hdfs supergroup 915 2017-01-20 13:22 /user/hdfs/flume/17-01-20/13/FlumeData.1484940151445
-rw-r--r-- 2 hdfs
... View more
01-20-2017
06:19 PM
Hi @mqureshi @Deepesh Can you help please.
... View more
01-20-2017
01:05 AM
As of now the file which my exec source is listening to is of 64KB. I tried to set agent.sinks.hdfs-sink.rollSize to 65536 and also agent.sinks.hdfs-sink.rollInterval to 300 (so that it doesnt roll the file until it is 5mins). But it didn't work together or separately (i.e. using only one of the property). But nothing is working My flume.conf agent.sources = avro-collection-source
agent.channels = memoryChannel
agent.sinks = hdfs-sink
# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type = avro
agent.sources.avro-collection-source.bind = 10.0.0.6
agent.sources.avro-collection-source.port = 60000
agent.sources.avro-collection-source.interceptors = interceptor1
agent.sources.avro-collection-source.interceptors.interceptor1.type = timestamp
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.hdfs-sink.type = hdfs
#agent.sinks.hdfs-sink.hdfs.path = hdfs://10.0.10.4:8020/flume/events
#agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume
agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S
agent.sinks.hdfs-sink.useLocalTimeStamp = true
agent.sinks.hdfs-sink.hdfs.callTimeout = 180000
#Specify the channel the sink should use
agent.sinks.hdfs-sink.channel = memoryChannel
# File size to trigger roll, in bytes
agent.sinks.hdfs-sink.rollSize = 65536
# Number of seconds to wait before rolling current file (in seconds)
agent.sinks.hdfs-sink.rollInterval = 300
#agent.sinks.hdfs-sink.rollCount = 0
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000
... View more
01-20-2017
12:59 AM
Yes, of course i restarted all agents
... View more
01-19-2017
11:18 PM
I corrected that typo however I still face the same issue. 😞 now, my flume.conf has following entry: # File size to trigger roll, in bytes (256Mb)
agent.sinks.hdfs-sink.rollSize = 268435456
# Number of seconds to wait before rolling current file (in seconds)
agent.sinks.hdfs-sink.rollInterval = 0
agent.sinks.hdfs-sink.rollCount = 0
Flume.log shows multiple entry like this 17/01/19 17:13:51 INFO hdfs.BucketWriter: Creating hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/34/FlumeData.1484867616377.tmp
17/01/19 17:13:51 INFO hdfs.BucketWriter: Closing hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/34/FlumeData.1484867616377.tmp
17/01/19 17:13:51 INFO hdfs.BucketWriter: Renaming hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/34/FlumeData.1484867616377.tmp to hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/34/FlumeData.1484867616377
17/01/19 17:13:51 INFO hdfs.BucketWriter: Creating hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/34/FlumeData.1484867616378.tmp
17/01/19 17:13:51 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawL
ocalFileSystem = false
17/01/19 17:13:51 INFO hdfs.BucketWriter: Creating hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/37/FlumeData.1484867631947.tmp
17/01/19 17:13:52 INFO hdfs.BucketWriter: Closing hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/37/FlumeData.1484867631947.tmp
17/01/19 17:13:52 INFO hdfs.BucketWriter: Renaming hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/37/FlumeData.1484867631947.tmp to hdfs://40.122.210.251:8020/user/hdfs/flume/17-01-19/1713/37/FlumeData.1484867631947
... View more
01-19-2017
10:34 PM
Flume hdfs sink is creating too many multiple files into hdfs for loading source file as small as 200KB. I have used rollCount set to 256MB even then it is not helping. Some file created on HDFS is as small as 1.5KB. Please help me make a single file because this file has to be read by R for filtering of Keywords. My flume.conf is agent.sources = avro-collection-source
agent.channels = memoryChannel
agent.sinks = hdfs-sink
# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type = avro
agent.sources.avro-collection-source.bind = 10.0.0.6
agent.sources.avro-collection-source.port = 60000
agent.sources.avro-collection-source.interceptors = interceptor1
agent.sources.avro-collection-source.interceptors.interceptor1.type = timestamp
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.hdfs-sink.type = hdfs
#agent.sinks.hdfs-sink.hdfs.path = hdfs://10.0.10.4:8020/flume/events
#agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume
agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S
agent.sinks.hdfs-sink.useLocalTimeStamp = true
agent.sinks.hdfs-sink.hdfs.callTimeout = 180000
#Specify the channel the sink should use
agent.sinks.hdfs-sink.channel = memoryChannel
# File size to trigger roll, in bytes (256Mb)
agent.sinks.hdfs-sink.rollSize = 268435456
# Number of seconds to wait before rolling current file (in seconds)
agent.sinks.sink.hdfs.rollInterval = 0
agent.sinks.sink.hdfs.rollCount = 0
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory Thanks, Shilpa
... View more
Labels:
01-06-2017
11:53 PM
The problem was, the IP for hdfs sink, I gave was private IP and once I changed it to public IP. It started streaming the data. agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume So, the issue is resolved. 🙂
... View more
01-06-2017
11:52 PM
Hi @Kuldeep Kulkarni @Deepesh The problem was, the IP for hdfs sink, I gave was private IP and once I changed it to public IP. It started streaming the data. So, the issue is resolved. 🙂
... View more
01-06-2017
05:44 PM
@Kuldeep Kulkarni if possible can you please check too. Thanks,Shilpa
... View more
01-05-2017
05:39 PM
Hi @Deepesh, Even on Github I can see codes for RSS feed streaming. https://github.com/dgkris/RSSpipe. However I tried to follow each step mentioned in link https://www.ibm.com/developerworks/library/bd-flumews/ .
my java code compiled using the RSS feed of the news website i want to stream and is able to pull feed data configured flume.conf on each node as per the link. However, I am still not getting files on hdfs. the node on which avro source is defined to collect the data from other 2 nodes and push it to HDFS sink is giving below exception 17/01/05 18:11:00 INFO source.AvroSource: Avro source avro-collection-source started.
17/01/05 18:11:18 INFO ipc.NettyServer: [id: 0x1576aaa8, /10.0.0.4:50034 => /10.0.0.6:60000] OPEN
17/01/05 18:11:18 INFO ipc.NettyServer: [id: 0x1576aaa8, /10.0.0.4:50034 => /10.0.0.6:60000] BOUND: /10.0.0.6:60000
17/01/05 18:11:18 INFO ipc.NettyServer: [id: 0x1576aaa8, /10.0.0.4:50034 => /10.0.0.6:60000] CONNECTED: /10.0.0.4:50034
17/01/05 18:11:21 INFO hdfs.HDFSSequenceFile: writeFormat = Writable, UseRawLocalFileSystem = false
17/01/05 18:11:22 INFO hdfs.BucketWriter: Creating hdfs://10.0.10.4:8020/flume/events/FlumeData.1483661481514.tmp
17/01/05 18:11:45 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 0 time(s); maxRetries=45
17/01/05 18:12:05 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 1 time(s); maxRetries=45
17/01/05 18:12:25 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 2 time(s); maxRetries=45
17/01/05 18:12:45 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 3 time(s); maxRetries=45
17/01/05 18:13:05 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 4 time(s); maxRetries=45
17/01/05 18:13:25 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 5 time(s); maxRetries=45
17/01/05 18:13:45 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 6 time(s); maxRetries=45
17/01/05 18:14:05 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 7 time(s); maxRetries=45
17/01/05 18:14:22 WARN hdfs.HDFSEventSink: HDFS IO error
java.io.IOException: Callable timed out after 180000 ms on file: hdfs://10.0.10.4:8020/flume/events/FlumeData.1483661481514.tmp
at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:693)
at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:235)
at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:514)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:686)
... 6 more
17/01/05 18:14:27 INFO hdfs.BucketWriter: Creating hdfs://10.0.10.4:8020/flume/events/FlumeData.1483661481515.tmp
17/01/05 18:14:47 INFO ipc.Client: Retrying connect to server: 10.0.10.4/10.0.10.4:8020. Already tried 0 time(s); maxRetries=45 I have googled the error, I also increased testAgent.sinks.testSink.hdfs.callTimeout = 180000, as suggested by https://issues.apache.org/jira/browse/FLUME-2429 I even increased the value of these 2 HDFS properties, dfs.socket.timeout and dfs.datanode.socket.write.timeout to 30000 from default value 3000. my flume.conf on that node is agent.sources = avro-collection-source
agent.channels = memoryChannel
agent.sinks = hdfs-sink
# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type = avro
agent.sources.avro-collection-source.bind = 10.0.0.6
agent.sources.avro-collection-source.port = 60000
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://10.0.10.4:8020/flume/events
agent.sinks.hdfs-sink.hdfs.callTimeout = 180000
#Specify the channel the sink should use
agent.sinks.hdfs-sink.channel = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000 Please guide me to fix this. Thanks, Shilpa
... View more
01-05-2017
05:34 PM
Hi @Deepesh Thanks so much for your response. I thought the HTTP GET would help me stream the data from the webpage. Was I wrong? As of now I have written a shell script to curl the webpage and thinking of using spooldir flume source. However I need to go at least 3 levels deep in the website, curl can't do that and wget is taking 1.5 hours. I am not that good in Python or Java. Can you help me if possible please. Please advise and help Thanks, Shilpa
... View more
01-05-2017
05:01 PM
Hi @nyadav, Remember I am using cloudera distrbution. And here we dont have SparkR by default. I have installed R on my node but still can not see sparkR under $spark_home/bin directory. However can go to R console, See this and let me know what do you think: [root@LnxMasterNode01 spark]# ll
total 36276
drwxr-xr-x 3 root root 4096 Oct 21 05:00 assembly
drwxr-xr-x 2 root root 4096 Oct 21 05:00 bin
drwxr-xr-x 2 root root 4096 Oct 21 05:00 cloudera
lrwxrwxrwx 1 root root 15 Nov 25 16:01 conf -> /etc/spark/conf
-rw-r--r-- 1 root root 12232 Jan 4 16:20 epel-release-5-4.noarch.rpm
drwxr-xr-x 3 root root 4096 Oct 21 05:00 examples
drwxr-xr-x 2 root root 4096 Oct 21 05:08 lib
-rw-r--r-- 1 root root 17352 Oct 21 05:00 LICENSE
drwxr-xr-x 2 root root 4096 Jan 2 18:09 logs
-rw-r--r-- 1 root root 23529 Oct 21 05:00 NOTICE
drwxr-xr-x 6 root root 4096 Oct 21 05:00 python
-rw-r--r-- 1 root root 37053596 Jan 4 17:16 R-2.13.0-2.el6.rf.i686.rpm
-rw-r--r-- 1 root root 0 Oct 21 05:00 RELEASE
drwxr-xr-x 2 root root 4096 Oct 21 05:00 sbin
lrwxrwxrwx 1 root root 19 Nov 25 16:01 work -> /var/run/spark/work
[root@LnxMasterNode01 spark]# cd bin
[root@LnxMasterNode01 bin]# ll
total 24
-rwxr-xr-x 1 root root 2857 Oct 21 05:00 load-spark-env.sh
-rwxr-xr-x 1 root root 3459 Oct 21 05:00 pyspark
-rwxr-xr-x 1 root root 2384 Oct 21 05:00 run-example
-rwxr-xr-x 1 root root 2858 Oct 21 05:00 spark-class
-rwxr-xr-x 1 root root 3026 Oct 21 05:00 spark-shell
-rwxr-xr-x 1 root root 1050 Oct 21 05:00 spark-submit
[root@LnxMasterNode01 bin]#[root@LnxMasterNode01 bin]# R
R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
>q()
Save workspace image? [y/n/c]: n
[root@LnxMasterNode01 bin]# ./sparkR
-bash: ./sparkR: No such file or directory
[root@LnxMasterNode01 bin]#
... View more
01-05-2017
12:41 AM
3 Kudos
this issue is resolved. I guess I previously install
the wrong EPEL release package on this machine. So to resolve it, I did: [root@LnxMasterNode01
spark]# yum clean all [root@LnxMasterNode01
spark]# yum install epel-release [root@LnxMasterNode01
spark]# yum install R Now, I am able to run 'R' however I cannot see it in my Spark home directory nor spark/bin has sparkR. [root@LnxMasterNode01 spark]# ll
total 36276
drwxr-xr-x 3 root root 4096 Oct 21 05:00 assembly
drwxr-xr-x 2 root root 4096 Oct 21 05:00 bin
drwxr-xr-x 2 root root 4096 Oct 21 05:00 cloudera
lrwxrwxrwx 1 root root 15 Nov 25 16:01 conf -> /etc/spark/conf
-rw-r--r-- 1 root root 12232 Jan 4 16:20 epel-release-5-4.noarch.rpm
drwxr-xr-x 3 root root 4096 Oct 21 05:00 examples
drwxr-xr-x 2 root root 4096 Oct 21 05:08 lib
-rw-r--r-- 1 root root 17352 Oct 21 05:00 LICENSE
drwxr-xr-x 2 root root 4096 Jan 2 18:09 logs
-rw-r--r-- 1 root root 23529 Oct 21 05:00 NOTICE
drwxr-xr-x 6 root root 4096 Oct 21 05:00 python
-rw-r--r-- 1 root root 37053596 Jan 4 17:16 R-2.13.0-2.el6.rf.i686.rpm
-rw-r--r-- 1 root root 0 Oct 21 05:00 RELEASE
drwxr-xr-x 2 root root 4096 Oct 21 05:00 sbin
lrwxrwxrwx 1 root root 19 Nov 25 16:01 work -> /var/run/spark/work
[root@LnxMasterNode01 spark]# Is it same as SparkR? Please guide [root@LnxMasterNode01 ~]# R
R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions. .
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> q()
Save workspace image? [y/n/c]: n
[root@LnxMasterNode01 ~]# Thanks, Shilpa
... View more
01-04-2017
11:47 PM
Hi, I have 3 node cluster having Cloudera 5.9 running on CentOS6.7. My Spark is running on Yarn. I have to install R on Spark home directoryso that I can use SparkR. I got the epel RPM, and then tried to install R using YUM however its giving error. I even tried some other RPM however they are giving error too. Using --skip-broken option is also not working. Please help [root@LnxMasterNode01
spark]# rpm -Uvh
http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm Retrieving
http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm warning:
/var/tmp/rpm-tmp.XuRVi8: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing...
########################################### [100%] 1:epel-release
########################################### [100%] Last part of the code shows error below: [root@LnxMasterNode01 spark]# yum install R
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
* epel: ftp.osuosl.org
Resolving Dependencies
--> Running transaction check
---> Package R.i686 0:2.13.0-2.el6.rf will be updated
---> Package R.x86_64 0:3.3.2-2.el5 will be an update
--> Processing Dependency: libRmath-devel = 3.3.2-2.el5 for package: R-3.3.2-2.el5.x86_64
--> Processing Dependency: R-devel = 3.3.2-2.el5 for package: R-3.3.2-2.el5.x86_64
--> Running transaction check . . ---> Package ppl.x86_64 0:0.10.2-11.el6 will be installed
---> Package texlive-texmf-errata-dvips.noarch 0:2007-7.1.el6 will be installed
---> Package texlive-texmf-errata-fonts.noarch 0:2007-7.1.el6 will be installed
--> Finished Dependency Resolution
Error: Package: R-core-3.3.2-2.el5.x86_64 (epel)
Requires: libtk8.4.so()(64bit)
Error: Package: R-core-3.3.2-2.el5.x86_64 (epel)
Requires: libtcl8.4.so()(64bit)
Error: Package: R-core-3.3.2-2.el5.x86_64 (epel)
Requires: libgssapi.so.2(libgssapi_CITI_2)(64bit)
Error: Package: R-core-3.3.2-2.el5.x86_64 (epel)
Requires: libRblas.so()(64bit)
Error: Package: libRmath-3.3.2-2.el5.x86_64 (epel)
Requires: libgssapi.so.2(libgssapi_CITI_2)(64bit)
Error: Package: libRmath-3.3.2-2.el5.x86_64 (epel)
Requires: libgssapi.so.2()(64bit)
Error: Package: R-core-3.3.2-2.el5.x86_64 (epel)
Requires: libgssapi.so.2()(64bit)
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
[root@LnxMasterNode01 spark]# I also checked here http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/ and http://www.jason-french.com/blog/2013/03/11/installing-r-in-linux/ these two links also suggest the same thing. Am I doing something wrong. Please suggest. Thanks, Shilpa
... View more
01-04-2017
07:15 PM
Hi Community, I have 3 node cluster, using latest cloudera parcels for 5.9 version. OS is CentOS 6.7 on all three of them. I am using Flume to get data from a third party website(basically a news website) to hdfs. Using http-source, mem-channel and hdfs-sink. The flume log says that source,channel and sink are started but there is no data streaming to hdfs. There is no further entry in the log after http-source started. I am not sure why nothing is happening. My flume.conf is: tier1.sources = http-source tier1.channels = mem-channel-1 tier1.sinks = hdfs-sink # For each source,
channel, and sink, set tier1.sources.http-source.type = org.apache.flume.source.http.HTTPSource tier1.sources.http-source.handler
= org.apache.flume.source.http.JSONHandler tier1.sources.http-source.bind = localhost tier1.sources.http-source.url
= http://www.jornada.unam.mx/ultimas tier1.sources.http-source.port = 5440 tier1.sources.http-source.channels
= mem-channel-1 tier1.channels.mem-channel-1.type = memory tier1.sinks.hdfs-sink.type = hdfs tier1.sinks.hdfs-sink.channel = mem-channel-1 tier1.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S tier1.channels.mem-channel-1.capacity
= 100 Flume-ng.log in debug mode: 2016-12-30 12:20:47,634 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
2016-12-30 12:20:47,649 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Configuration provider started
2016-12-30 12:20:47,650 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume.
conf for changes
2016-12-30 12:20:47,650 INFO org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/var/run/cloudera-scm-agent/process/600-flum
e-AGENT/flume.conf
2016-12-30 12:20:47,654 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink
2016-12-30 12:20:47,654 DEBUG org.apache.flume.conf.FlumeConfiguration: Created context for hdfs-sink: channel
2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink
2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: tier1
2016-12-30 12:20:47,655 INFO org.apache.flume.conf.FlumeConfiguration: Processing:hdfs-sink
2016-12-30 12:20:47,655 DEBUG org.apache.flume.conf.FlumeConfiguration: Starting validation of configuration for agent: tier1
2016-12-30 12:20:47,656 INFO org.apache.flume.conf.LogPrivacyUtil: Logging of configuration details is disabled. To see configuration details in the log run the agent w
ith -Dorg.apache.flume.log.printconfig=true JVM argument. Please note that this is not recommended in production systems as it may leak private information to the logfi
le.
2016-12-30 12:20:47,661 DEBUG org.apache.flume.conf.FlumeConfiguration: Created channel file-channel-1
2016-12-30 12:20:47,667 DEBUG org.apache.flume.conf.FlumeConfiguration: Creating sink: hdfs-sink using HDFS
2016-12-30 12:20:47,669 DEBUG org.apache.flume.conf.FlumeConfiguration: Channels:file-channel-1
2016-12-30 12:20:47,669 DEBUG org.apache.flume.conf.FlumeConfiguration: Sinks hdfs-sink
2016-12-30 12:20:47,670 DEBUG org.apache.flume.conf.FlumeConfiguration: Sources http-source
2016-12-30 12:20:47,670 INFO org.apache.flume.conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [tier1]
2016-12-30 12:20:47,670 INFO org.apache.flume.node.AbstractConfigurationProvider: Creating channels
2016-12-30 12:20:47,677 INFO org.apache.flume.channel.DefaultChannelFactory: Creating instance of channel file-channel-1 type file
2016-12-30 12:20:47,696 INFO org.apache.flume.node.AbstractConfigurationProvider: Created channel file-channel-1
2016-12-30 12:20:47,697 INFO org.apache.flume.source.DefaultSourceFactory: Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
2016-12-30 12:20:47,697 DEBUG org.apache.flume.source.DefaultSourceFactory: Source type org.apache.flume.source.http.HTTPSource is a custom type
2016-12-30 12:20:47,757 INFO org.apache.flume.sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs
2016-12-30 12:20:47,768 INFO org.apache.flume.node.AbstractConfigurationProvider: Channel file-channel-1 connected to [http-source, hdfs-sink]
2016-12-30 12:20:47,774 INFO org.apache.flume.node.Application: Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flu
me.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@f212168 counterGroup:{
name:null counters:{} } }} channels:{file-channel-1=FileChannel file-channel-1 { dataDirs: [/mnt/resource/flume/data] }} }
2016-12-30 12:20:47,783 INFO org.apache.flume.node.Application: Starting Channel file-channel-1
2016-12-30 12:20:47,783 INFO org.apache.flume.channel.file.FileChannel: Starting FileChannel file-channel-1 { dataDirs: [/mnt/resource/flume/data] }...
2016-12-30 12:20:47,793 INFO org.apache.flume.channel.file.Log: Encryption is not enabled2016-12-30 12:20:47,794 INFO org.apache.flume.channel.file.Log: Replay started
2016-12-30 12:20:47,795 INFO org.apache.flume.channel.file.Log: Found NextFileID 0, from []
2016-12-30 12:20:47,814 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Preallocated /mnt/resource/flume/checkpoint/checkpoint to 8008232 for capacity 1000000
2016-12-30 12:20:47,816 INFO org.apache.flume.channel.file.EventQueueBackingStoreFileV3: Starting up with /mnt/resource/flume/checkpoint/checkpoint and /mnt/resource/flume/checkpoint/checkpoint.meta
2016-12-30 12:20:47,916 INFO org.apache.flume.channel.file.FlumeEventQueue: QueueSet population inserting 0 took 0
2016-12-30 12:20:47,919 INFO org.apache.flume.channel.file.Log: Last Checkpoint Fri Dec 30 12:20:47 CST 2016, queue depth = 0
2016-12-30 12:20:47,922 INFO org.apache.flume.channel.file.Log: Replaying logs with v2 replay logic
2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.ReplayHandler: Starting replay of []
2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0
2016-12-30 12:20:47,927 INFO org.apache.flume.channel.file.FlumeEventQueue: Search Count = 0, Search Time = 0, Copy Count = 0, Copy Time = 0
2016-12-30 12:20:47,932 INFO org.apache.flume.channel.file.Log: Rolling /mnt/resource/flume/data
2016-12-30 12:20:47,932 INFO org.apache.flume.channel.file.Log: Roll start /mnt/resource/flume/data
2016-12-30 12:20:47,941 INFO org.apache.flume.tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
2016-12-30 12:20:47,944 INFO org.apache.flume.tools.DirectMemoryUtils: Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 503316480, Remaining = 503316480
2016-12-30 12:20:47,972 INFO org.apache.flume.channel.file.LogFile: Opened /mnt/resource/flume/data/log-1
2016-12-30 12:20:47,981 INFO org.apache.flume.channel.file.Log: Roll end
2016-12-30 12:20:47,981 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Start checkpoint for /mnt/resource/flume/checkpoint/checkpoint, elements to sync = 0
2016-12-30 12:20:47,985 INFO org.apache.flume.channel.file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1483122047933, queueSize: 0, queueHead: 0
2016-12-30 12:20:48,001 INFO org.apache.flume.channel.file.Log: Updated checkpoint for file: /mnt/resource/flume/data/log-1 position: 0 logWriteOrderID: 1483122047933
2016-12-30 12:20:48,001 INFO org.apache.flume.channel.file.FileChannel: Queue Size after replay: 0 [channel=file-channel-1]
2016-12-30 12:20:48,037 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: file-channel-1: Successfully registered new MBean.
2016-12-30 12:20:48,037 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: file-channel-1 started
2016-12-30 12:20:48,037 INFO org.apache.flume.node.Application: Starting Sink hdfs-sink
2016-12-30 12:20:48,037 INFO org.apache.flume.node.Application: Starting Source http-source
2016-12-30 12:20:48,039 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfs-sink: Successfully registered new MBean.
2016-12-30 12:20:48,039 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started
2016-12-30 12:20:48,040 DEBUG org.apache.flume.SinkRunner: Polling sink runner starting
2016-12-30 12:20:48,062 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-12-30 12:20:48,064 INFO org.mortbay.log: jetty-6.1.26.cloudera.42016-12-30 12:20:48,135 INFO org.mortbay.log: Started SelectChannelConnector@localhost:5440
2016-12-30 12:20:48,135 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: http-source: Successfully registered new MBean.
2016-12-30 12:20:48,136 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: http-source started
2016-12-30 12:21:17,794 DEBUG org.apache.flume.channel.file.FlumeEventQueue: Checkpoint not required
2016-12-30 12:21:18,088 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume.conf for changes
2016-12-30 12:21:47,794 DEBUG org.apache.flume.channel.file.FlumeEventQueue: Checkpoint not required
2016-12-30 12:21:48,089 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/600-flume-AGENT/flume.conf for changes
Please help me find a solution. I even tried replacing mem-channel with File-channel, even then it is stopping at the same point with no data streaming. My project is at critical stage. Please guide me. Thanks, Shilpa
... View more
Labels: