Requirement: I'm trying to consume twitter feed from https://dev.twitter.com/streaming/overview
What I did:
1) Created 3 node HDP Cluster in Cloud ( SSH set up is working b/w 3 nodes)
2) Created 1 node HDF Cluster in Cloud ( SSH is working here as well)
3) NIFI is up and running and created an account here https://apps.twitter.com/app/ and got access key and secret key
4) WF is running but I'm not getting data from twitter to HDFS ( in HDP cluster)
5) Get Twitter Processor details
6) Put HDFS ( here I added HDP cluster core-site.xml and hdfs-site.xml in mentioned location)
Log Message:
[root@hdf01 nifi]# tail -f nifi-app.log
2018-01-28 15:03:01,344 INFO [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 2a61ffef-cc6c-4a49-9dfa-cd89f73a34d1 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:01,347 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:01,339 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:01,347; send took 7 millis 2018-01-28 15:03:03,264 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 5710 nanos 2018-01-28 15:03:06,353 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request e3976619-9c0f-42f7-b44a-d716230d4b2f (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:06,357 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:06,348 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:06,357; send took 9 millis 2018-01-28 15:03:08,266 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6277 nanos 2018-01-28 15:03:08,524 INFO [Timer-Driven Process Thread-8] o.a.n.r.ambari.AmbariReportingTask AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] Successfully sent metrics to Ambari in 0 ms 2018-01-28 15:03:11,361 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 8d15dc5f-1acb-4348-b9f7-afbc93c623ea (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:11,363 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:11,357 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:11,363; send took 6 millis 2018-01-28 15:03:13,268 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 5870 nanos 2018-01-28 15:03:16,369 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 21e0c538-e335-4982-85b7-f660787cd20e (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 2 millis 2018-01-28 15:03:16,372 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:16,364 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:16,372; send took 8 millis 2018-01-28 15:03:18,270 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6248 nanos 2018-01-28 15:03:21,379 INFO [Process Cluster Protocol Request-1] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 385415b7-f1cd-4557-a974-2086fa422e00 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:21,386 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:21,373 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:21,386; send took 13 millis 2018-01-28 15:03:23,272 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 8775 nanos 2018-01-28 15:03:26,393 INFO [Process Cluster Protocol Request-3] o.a.n.c.p.impl.SocketProtocolListener Finished processing request b4cfd8f8-4ac6-4588-8353-9824a502030e (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:26,395 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:26,387 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:26,395; send took 7 millis 2018-01-28 15:03:28,274 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 6547 nanos 2018-01-28 15:03:31,401 INFO [Process Cluster Protocol Request-4] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 5aa63aad-b8e6-4e0f-8d8f-9b489e9c7783 (type=HEARTBEAT, length=1832 bytes) from hdf01.hdf.com:9090 in 3 millis 2018-01-28 15:03:31,405 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-01-28 15:03:31,396 and sent to hdf01.hdf.com:9088 at 2018-01-28 15:03:31,405; send took 8 millis
