Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi is not loading data from twitter

Highlighted

Nifi is not loading data from twitter

Requirement: I'm trying to consume twitter feed from https://dev.twitter.com/streaming/overview

What I did:

1) Created 3 node HDP Cluster in Cloud ( SSH set up is working b/w 3 nodes)

2) Created 1 node HDF Cluster in Cloud ( SSH is working here as well)

3) NIFI is up and running and created an account here https://apps.twitter.com/app/ and got access key and secret key

4) WF is running but I'm not getting data from twitter to HDFS ( in HDP cluster)

56485-nifihcc.png

5) Get Twitter Processor details

6) Put HDFS ( here I added HDP cluster core-site.xml and hdfs-site.xml in mentioned location)

Log Message:

[root@hdf01 nifi]# tail -f nifi-app.log

2018-01-28 15:03:01,344 INFO [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 2a61ffef-cc6c-4a49-9dfa-cd89f73a34d1 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:01,347 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:01,339 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:01,347; send took 7 millis 2018-01-28 15:03:03,264 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 5710 nanos 2018-01-28 15:03:06,353 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request e3976619-9c0f-42f7-b44a-d716230d4b2f (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:06,357 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:06,348 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:06,357; send took 9 millis 2018-01-28 15:03:08,266 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6277 nanos 2018-01-28 15:03:08,524 INFO [Timer-Driven Process Thread-8] o.a.n.r.ambari.AmbariReportingTask AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] Successfully sent metrics to Ambari in 0 ms 2018-01-28 15:03:11,361 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 8d15dc5f-1acb-4348-b9f7-afbc93c623ea (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:11,363 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:11,357 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:11,363; send took 6 millis 2018-01-28 15:03:13,268 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 5870 nanos 2018-01-28 15:03:16,369 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 21e0c538-e335-4982-85b7-f660787cd20e (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:16,372 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:16,364 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:16,372; send took 8 millis 2018-01-28 15:03:18,270 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6248 nanos 2018-01-28 15:03:21,379 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 385415b7-f1cd-4557-a974-2086fa422e00 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:21,386 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:21,373 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:21,386; send took 13 millis 2018-01-28 15:03:23,272 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 8775 nanos 2018-01-28 15:03:26,393 INFO [Process Cluster Protocol Request-3] o.a.n.c.p.impl.SocketProtocolListener Finished processing request b4cfd8f8-4ac6-4588-8353-9824a502030e (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:26,395 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:26,387 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:26,395; send took 7 millis 2018-01-28 15:03:28,274 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6547 nanos 2018-01-28 15:03:31,401 INFO [Process Cluster Protocol Request-4] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 5aa63aad-b8e6-4e0f-8d8f-9b489e9c7783 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:31,405 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:31,396 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:31,405; send took 8 millis

56488-nifihcc3.png

56486-nifihcc2.png

1 REPLY 1
Highlighted

Re: Nifi is not loading data from twitter

Does the "Sample Endpoint" work?

Don't have an account?
Coming from Hortonworks? Activate your account here