nifi gettwitter works intermittently

[#1]

Get Twitter is working some but not all times. If I connect to the sample endpoint, it retrieves data. If I connect to the filter endpoint with a valid filter that contains popular topics, it retrieves nothing. If I go back to the sample endpoint, it eventually stops getting any data. If I regenerate my twitter keys, it seems to work for a while and then stops getting data. I saw the issue regarding server clock synchronization and I have made sure that nntp is running properly and the clock is right. not sure what else to adjust.

[#2 · Accepted solution]

Perhaps you are being rate-limited by the Twitter API? How often is GetTwitter executing? You can check this on the Scheduling tab of the processor configuration dialog. If "Run Schedule" is set to zero seconds, then the processor will run as fast as possible, which could certainly cause throttling / rate-limiting and thus an eventual lack of data.

[#3]

I've just finished reading up on twitter's API and perhaps rate-limiting is the issue.

I'm using a template pulled from HWX and it looks like it had the scheduling set to zero seconds. I have increased it to 30 seconds and still it does not work. Will reset my keys and see if that fixes it. perhaps twitter cuts off access for those identifiers.

What is a "reasonable" schedule time that won't run into the rate limit?

Here are some log messages:

2016-07-27 05:38:49,352 INFO [hosebird-client-io-thread-0] com.twitter.hbc.httpclient.ClientBase GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Establishing a connection 2016-07-27 05:38:50,837 WARN [hosebird-client-io-thread-0] com.twitter.hbc.httpclient.ClientBase GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Error connecting w/ status code - 420, reason - Enhance Your Calm 2016-07-27 05:39:07,949 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.twitter.GetTwitter GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Received error HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt to reconnect

[#4]

generally working now. except that every time it runs, it returns one and exactly one result .json file. i find it hard to believe that there is exactly one tweet meeting my criteria ever 30 seconds.