Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

Explorer

While setting an http-source for a flume agent, i used the port 8989. and the agent starts fine but the same agent fails on port 8080 and gives error

03 Jan 2017 22:03:35,232 ERROR [lifecycleSupervisor-1-1] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} } - Exception follows.
java.lang.IllegalStateException: Running HTTP Server found in source: http-source before I started one.Will not attempt to start.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:189)
        at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

My next point is about making a get and post requests to the http-source. Since the http-source of flume listens to the post requests. What i tried is somthing like this .

>>> r=requests.post('http://hdp.localdomain:8080',data={'k':'v'})
>>> r.status_code
200
>>> r=requests.post('http://hdp.localdomain:8989',data={'k':'v'})
>>> r.status_code
400

Now the post works perfect for the 8080 port as the status is 200 though the http-source is not listening on that port so i dont expect to get anything in the hdfs sink from there but i dont understand why i am get 400 for 8989 and i dont get anything in the hdfs sink. Any genius who has the guts to show here ;)

4 REPLIES 4

Re: The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

Master Collaborator

Since port 8080 is already open, can you check which application is responsible for it. You can use something like

netstat -ntlp | grep 8080

The above will give you a PID that you can search by doing a "ps auwwx | grep <pid>" listing thereby helping you identify the exact application using it.

Re: The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

Explorer

root@host-192-130-3-218:~# netstat -ntlp |grep 8080 tcp6 0 0 :::8080 :::* LISTEN 1511/java

root@host-192-130-3-218:~# ps auwwx| grep 1511
root      1511  1.8 10.9 7910448 891328 ?      Sl    2016 1044:43 /usr/jdk64/jdk1.8.0_40/bin/java -server -XX:NewRatio=3 -XX:+UseConcM                                  arkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava                                  .security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.                                  useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.s                                  erver.controller.AmbariServer
root     17206  0.0  0.0  10464   936 pts/0    S+   08:57   0:00 grep --color=auto 1511

Yes the ambari server is listening on 8080.

root@host-192-130-3-218:~# netstat -ntlp | grep 8989
tcp6       0      0 :::8989                 :::*                    LISTEN      22462/java
root@host-192-130-3-218:~# ps auwwx| grep 22462
root     18454  0.0  0.0  10460   936 pts/0    S+   08:59   0:00 grep --color=auto 22462
flume    22462  0.1  2.5 3849856 209216 ?      Sl   Jan03   0:56 /usr/jdk64/jdk1.8.0_40/bin/java -Xmx20m -Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.F                     lumeTimelineMetricsSink -Dflume.monitoring.node=hadoop.localdomain:6188 -cp /usr/hdp/current/flume-server/conf/HttpAgent:/usr/hdp/2.3.6.0-3796/flume/lib/*::/usr/lib/flu                     me/lib/ambari-metrics-flume-sink.jar:/usr/hdp/2.3.6.0-3796/hadoop/conf:/usr/hdp/2.3.6.0-3796/hadoop/lib/activation-1.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/apacheds-i18                     n-2.0.0-M15.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.3.6.0-3796                     /hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/asm-3.2.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/aws-jav                     a-sdk-1.7.4.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/c                     ommons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.3.6.0-3796/hadoop                     /lib/commons-collections-3.2.2.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2                     .3.6.0-3796/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-io-2.4.jar:/usr/hdp                     /2.3.6.0-3796/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/commons-math3-3.1.1.jar:/usr/h                     dp/2.3.6.0-3796/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/curator-framework-2.7.1.jar:/u                     sr/hdp/2.3.6.0-3796/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.3.                     6.0-3796/hadoop/lib/hadoop-lzo-0.6.0.2.3.6.0-3796.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/hadoop-lzo-0.6.0.2.3.6.0-3796-javadoc.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/hadoop                     -lzo-0.6.0.2.3.6.0-3796-sources.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.                     3.6.0-3796/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2                     .3.6.0-3796/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-databind-2.2.3.jar:/                     usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jackson-xc-1.9.                     13.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jaxb-impl-2.2.3-1.j                     ar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jersey-server-1.9.jar:/usr                     /hdp/2.3.6.0-3796/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.3.6.0-3                     796/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.6.0-3796/hadoop/                     lib/jsr305-3.0.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/junit-4.11.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/microsoft-window                     sazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/native:/usr/hdp/2.3.6.0-3796/hadoop/lib/netty-3.6.2                     .Final.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/ojdbc6.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.6.0-3796/hadoop/lib/protobuf-java-2.5.0.jar:/usr/                     hdp/2.3.6.0-3796/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.3.6.0-3796/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.3.6.0-3796.j
root@host-192-130-3-218:~#

Here 8989 is listening to flume. So one thing is clear. 8080 is for ambari. But 8989 is listening but why does it give 400 status code for any requests.

Re: The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

Super Mentor

@irfan aziz

You are making a POST request there (Not GET) . Http Response 400 code means Bad request, which means the Request Payload "{'k':'v'}" is not good or suitable for the endpoint (services running on)

http://hdp.localdomain:8989

.

Highlighted

Re: The flume agent http-source starts without error on port 8989 but fails with error on port 8080?

Explorer

@Jay SenSharma Thank you Jay but this http://hdp.localdomain:8989 is my http-source url. So i thought i could just post anything to it and it should ingest and deliver to the sink in my case hdfs. How will i know what type of data i can post.

Don't have an account?
Coming from Hortonworks? Activate your account here