question Re: Use Flume to get a webpage data. How to configure, how to use it to stream data in Support Questions

Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Fri, 16 Sep 2022 10:49:54 GMT

Hi,

I have 3 node cluster, using latest cloudera parcels for 5.9 version. OS is CentOS 6.7 on all three of them.

I am using Flume for the 1st time. I have just used 'add service' option on CLoudera GUI to add Flume.

My purpose is to get the data from a webpage to hdfs/hbase.

Can you please help me how can I do it? what else do I need to make the data streaming from a webpage possible.

Also, I have a seen an example on net for Twitter, there we need to make token on twitter page to get the data. However the webpage I am referring is a 3rd party one and I am not sure how to configure Flume to get the data on my cluster. I guess it would be over http.

Please help me to get this done.

Thanks in Advance.

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

csguna — Tue, 29 Nov 2016 13:42:33 GMT

Since you are going to use third party webpage I assuming that you wont be able to integrate or deploy flume sdk.if the webpage is ok in sending data via HTTP rather than using Flume's RPC , then I think HTTP source would be a good fit. From a client point of view HTTP source will act like a web server that accepts flume event.Either you can write your own Handler or use HTTPSourceXMLHandler in your configuration , the default Handler accepts Json format .

The format which HTTPSourceXMLHandler accept is state below

<events>
<event 1 2 3 ..>
<headers>
<header 1 2 3 ..>
</header>
<body> </body>
</event..>
</events>

The handler will parse the XML into flume events and pass it on to the HTTP Source. Which will then pass on to Channel and goes to Sink or Another agent depends on the flow.

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Tue, 29 Nov 2016 20:11:56 GMT

Thanks so much for the reply. It really helps me to understand how can it work.

However as I told you i am using Flume for the 1st time, i have no idea how to change its configurations to make it work. my configs are currently set to default values.

I will really appreciate it if you could help me with them. PFB

I look forward to your reply.

Thanks in Advance!

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Wed, 30 Nov 2016 22:57:32 GMT

Please help me with the Flume configurations. I am running out of time.

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Fri, 02 Dec 2016 17:26:36 GMT

Hey guys!

I am still wating for the reply. Please help me with those configs or point me to a document.

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

pdvorak — Fri, 02 Dec 2016 20:05:22 GMT

Flume doesn't have the ability to poll an http service, however it can act as an http service itself (http://flume.apache.org/FlumeUserGuide.html#http-source) that you can post json data to (or other formats).

I would suggest reviewing the documentation here: http://flume.apache.org/FlumeUserGuide.html

To see some examples and different configuration options.

In Cloudera Manager, you will be editing Configuration file section, and that is the configuration that is read when flume starts up.

-pd

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Fri, 02 Dec 2016 20:51:41 GMT

Thanks pd. 🙂

I will go through the links and will get back If I am stuck anywhere.

-Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Fri, 16 Dec 2016 13:36:40 GMT

Hey @pdvorak,

My flume.conf look like below. However I have a question as i want to get data from a third party website, in my case a news website.

I dont know which port to use in the configuration, please see highlighted part below. I thought of using 8080 but

it is not open.

it i

[root@LnxMasterNode01 ~]# nc -l 132.247.1.32 8080
nc: Cannot assign requested address

Can you tell me how to deal with this part.

Also, i am looking for special keywords in this website such as fashion, sports and i want the data only related to them. Can you tell me if this can be done using multiple sink for each header such as:

# list the sources, sinks and channels in the agent

agent_foo.sources = avro-AppSrv-source1

agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2

agent_foo.channels = mem-channel-1 mem-channel-2

# set channels for source

agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1 mem-channel-2

# set channel for sinks

agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1 mem-channel-2

# channel selector configuration

agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing

agent_foo.sources.avro-AppSrv-source1.selector.header = Category

agent_foo.sources.avro-AppSrv-source1.selector.mapping.Fashion = mem-channel-1

agent_foo.sources.avro-AppSrv-source1.selector.mapping.Baseball = mem-channel-2

agent_foo.sources.avro-AppSrv-source1.selector.mapping.Basketball = mem-channel-1 mem-channel-2

agent_foo.sources.avro-AppSrv-source1.selector.default = mem-channel-1

Thanks in Advance!

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Fri, 16 Dec 2016 17:49:13 GMT

Hi @pdvorak/ All,

Please help. I am not sure how to deal with it.

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Mon, 19 Dec 2016 22:26:57 GMT

Hi @hshreedharan / @pdvorak / All,

My project is in crisis really need help.

my flume.conf is as follows:

tier1.sources = http-source

tier1.channels = mem-channel-1

tier1.sinks = hdfs-sink

# For each source, channel, and sink, set

# standard properties.

tier1.sources.http-source.type = http

tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler

tier1.sources.http-source.bind = 132.247.1.32

tier1.sources.http-source.port = 21

tier1.sources.http-source.channels = mem-channel-1

tier1.channels.mem-channel-1.type = memory

tier1.sinks.hdfs-sink.type = hdfs

tier1.sinks.hdfs-sink.channel = mem-channel-1

tier1.sinks.hdfs-sink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S

# Other properties are specific to each type of

# source, channel, or sink. In this case, we

# specify the capacity of the memory channel.

tier1.channels.mem-channel-1.capacity = 100

The error I am getting while starting flume-ng:

2016-12-19 15:52:27,356 WARN org.mortbay.log: failed SelectChannelConnector@132.247.1.32:21: java.net.BindException: Cannot assign requested address
2016-12-19 15:52:27,356 WARN org.mortbay.log: failed Server@9439eee: java.net.BindException: Cannot assign requested address
2016-12-19 15:52:27,357 ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSource. Exception follows.
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:207)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-19 15:52:27,367 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} } - Exception follows.
java.lang.RuntimeException: java.net.BindException: Cannot assign requested address
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:211)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

I guess, the source is not able to connect to IP and Port I provided. This is because the IP I provided is of 3rd Party, a news webpage. And I dont know which port is open to connect.

Please guide me how can I can enable data streaming from a 3rd Party webpage to HDFS using Flume.

Also, I cannot see my plugins.d for Flume. Can this be one of the reason? Do i have to download it jars separetly? I have installed Flume from Cloudera Manager UI.

[root@LnxMasterNode01 CDH-5.9.0-1.cdh5.9.0.p0.23]# cd /var/lib/flume-ng/
[root@LnxMasterNode01 flume-ng]# ll
total 0
[root@LnxMasterNode01 flume-ng]# cd /
[root@LnxMasterNode01 /]# find . -name plugins.d
./etc/audisp/plugins.d
[root@LnxMasterNode01 /]#

@hshreedharan I saw your post on Flume on a seperate forum hence looped you. Please help.

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Mon, 19 Dec 2016 22:56:41 GMT

@pdvorak / @hshreedharan

I ran a curl on the IP and saw it is using port 80 to connect to the news webpage. Even Telnet is working on the port 80.

[root@LnxMasterNode01 /]# telnet 132.247.1.32 80
Trying 132.247.1.32...
Connected to 132.247.1.32.
Escape character is '^]'.
^CConnection closed by foreign host.

However when restarting flume, I am getting the same error as earlier. Can this be ONLY related to absence of plugins.d(see the previous post)

2016-12-19 16:45:00,353 WARN org.mortbay.log: failed SelectChannelConnector@132.247.1.32:80: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 WARN org.mortbay.log: failed Server@36772002: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSource. Exception follows.
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:207)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-19 16:45:00,364 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{
name:http-source,state:IDLE} } - Exception follows.
java.lang.RuntimeException: java.net.BindException: Cannot assign requested address

Please help me resolve this issue.

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Mon, 26 Dec 2016 19:31:29 GMT

I have edited my flume.conf to

# Please paste flume.conf here.

# Sources, channels, and sinks are defined per
# agent name, in this case 'tier1'.
tier1.sources = http-source
tier1.channels = mem-channel-1
tier1.sinks = hdfs-sink
# For each source, channel, and sink, set
# standard properties.
tier1.sources.http-source.type = http
tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler
tier1.sources.http-source.bind = localhost
tier1.sources.http-source.url = http://www.jornada.unam.mx/ultimas
tier1.sources.http-source.port = 5440
tier1.sources.http-source.channels = mem-channel-1
tier1.channels.mem-channel-1.type = memory
tier1.sinks.hdfs-sink.type = hdfs
tier1.sinks.hdfs-sink.channel = mem-channel-1
tier1.sinks.hdfs-sink.hdfs.path = hdfs://lnxmasternode01.centralus.cloudapp.azure.com/flume/events/%y-%m-%d/%H%M/%S
# Other properties are specific to each type of
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
tier1.channels.mem-channel-1.capacity = 100

Now, I can see http-source as started in flume logs.

However, no data is getting streamed in hdfs path i mentioned in the config. Can anyone suggest now what to do?

-bash-4.1$ hadoop fs -ls /flume
Found 1 items
drwxr-xr-x - flume hdfs 0 2016-12-23 11:49 /flume/events
-bash-4.1$ hadoop fs -ls /flume/events
-bash-4.1$

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Tue, 27 Dec 2016 00:56:13 GMT

I checked the flume jars for source, I can find only these with cloudera bundle:

[hadoop@LnxMasterNode01 jars]$ ll flume*source*
-rw-r--r-- 1 root root 20586 Oct 21 04:58 flume-avro-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 26893 Oct 21 04:58 flume-jms-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 22843 Oct 21 04:58 flume-kafka-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 61447 Oct 21 04:58 flume-scribe-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 34830 Oct 21 04:58 flume-taildir-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 80709 Oct 21 04:58 flume-thrift-source-1.6.0-cdh5.9.0.jar
-rw-r--r-- 1 root root 14540 Oct 21 04:58 flume-twitter-source-1.6.0-cdh5.9.0.jar

Can this be the reason why http source is not working i.e. data streaming not happening despite no error in flume.log and it says http-source started?

How do I get jars related to http-source?

Thanks,

Shilpa

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

pdvorak — Fri, 06 Jan 2017 20:13:19 GMT

As I stated before, flume can't consume from a remote http server. You would need to have something that could consume from the remote server and then post to flume.

-pd

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

ShilpaSinha — Sat, 07 Jan 2017 00:05:05 GMT

@pdvorak thanks!

Yes, i wrote a java code to pull RSS feed and used Exec source and Avro Sink on 2 nodes and Avro Source as collector and HDFS sink on the 3rd node.

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

aparnak — Mon, 06 Feb 2017 23:21:37 GMT

Hi Shilpa,

Were you able to get webpage data to HDFS via flume? Please let me know what all you did.

Thanks.

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

Dboudart — Tue, 06 Jun 2017 14:32:29 GMT

Hi @ShilpaSinha,

can you share how you get that java code to pull the RSS feed?

Regards,

David

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

pdvorak — Tue, 20 Jun 2017 16:52:00 GMT

Here is an example for creating a simple java RSS reader and setting flume up to read the output:

http://www.ibm.com/developerworks/library/bd-flumews/

-pd

Re: Use Flume to get a webpage data. How to configure, how to use it to stream data

Dboudart — Tue, 20 Jun 2017 16:58:53 GMT

Hi @pdvorak,

thanks a lot for you answer, I've already checked that page and it helped.

Thanks again.