Support Questions
Find answers, ask questions, and share your expertise

nifi gettwitter works intermittently

Solved Go to solution
Highlighted

nifi gettwitter works intermittently

Contributor

Get Twitter is working some but not all times. If I connect to the sample endpoint, it retrieves data. If I connect to the filter endpoint with a valid filter that contains popular topics, it retrieves nothing. If I go back to the sample endpoint, it eventually stops getting any data. If I regenerate my twitter keys, it seems to work for a while and then stops getting data. I saw the issue regarding server clock synchronization and I have made sure that nntp is running properly and the clock is right. not sure what else to adjust.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: nifi gettwitter works intermittently

Super Guru

Perhaps you are being rate-limited by the Twitter API? How often is GetTwitter executing? You can check this on the Scheduling tab of the processor configuration dialog. If "Run Schedule" is set to zero seconds, then the processor will run as fast as possible, which could certainly cause throttling / rate-limiting and thus an eventual lack of data.

View solution in original post

3 REPLIES 3
Highlighted

Re: nifi gettwitter works intermittently

Super Guru

Perhaps you are being rate-limited by the Twitter API? How often is GetTwitter executing? You can check this on the Scheduling tab of the processor configuration dialog. If "Run Schedule" is set to zero seconds, then the processor will run as fast as possible, which could certainly cause throttling / rate-limiting and thus an eventual lack of data.

View solution in original post

Re: nifi gettwitter works intermittently

Contributor

I've just finished reading up on twitter's API and perhaps rate-limiting is the issue.

I'm using a template pulled from HWX and it looks like it had the scheduling set to zero seconds. I have increased it to 30 seconds and still it does not work. Will reset my keys and see if that fixes it. perhaps twitter cuts off access for those identifiers.

What is a "reasonable" schedule time that won't run into the rate limit?

Here are some log messages:

2016-07-27 05:38:49,352 INFO [hosebird-client-io-thread-0] com.twitter.hbc.httpclient.ClientBase GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Establishing a connection 2016-07-27 05:38:50,837 WARN [hosebird-client-io-thread-0] com.twitter.hbc.httpclient.ClientBase GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Error connecting w/ status code - 420, reason - Enhance Your Calm 2016-07-27 05:39:07,949 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.twitter.GetTwitter GetTwitter[id=b2e83358-2e38-42c5-88a7-9f8a3f8307ba] Received error HTTP_ERROR: HTTP/1.1 420 Enhance Your Calm. Will attempt to reconnect

Highlighted

Re: nifi gettwitter works intermittently

Contributor

generally working now. except that every time it runs, it returns one and exactly one result .json file. i find it hard to believe that there is exactly one tweet meeting my criteria ever 30 seconds.