Support Questions
Find answers, ask questions, and share your expertise

NiFi unable to connect Twitter

Re: NiFi unable to connect Twitter

@Kishore Kumar

Try and connect to the Sample Endpoint of Twitter. Does that work or do you get the same error?

Re: NiFi unable to connect Twitter

Contributor

Try that as well. Same error .

d4734] IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/sample.json?delimited=length&stall_warnings=true 2017-08-30 00:37:06,092 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.twitter.GetTwitter GetTwitter[id=17f6d19f-6a6f-46df-a98c-5059921d4734] Received error CONNECTION_ERROR: connect timed out. Will attempt to reconnect 2017-

Re: NiFi unable to connect Twitter

Expert Contributor

Check your network firewall and local firewall, disable iptables to test.

Re: NiFi unable to connect Twitter

Contributor

@Joe Here is the output

# service iptables status

iptables: Firewall is not running.

Re: NiFi unable to connect Twitter

Mentor

@Kishore Kumar

You will have to give this last try I downloaded a VM image a couple of hours ago. And below is what I exactly did.

1. Network I used Bridged Adapter

2. Memory 12 GB

3. Booted the sandbox

4. on the initial login <ALt + F5> 5. password root/hadoop

6. At the prompt as root

7. switched to the docker image

# ssh -p2222 root@127.0.0.1 

Enter password: hadoop

# ambari-admin-password-reset 

entered admin and confirmed

7. The above step initialized the ambari server and successfully

8. Now check the hosts

cat /etc/hosts you should see something like this

127.0.0.1 localhost 
::1 localhost ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters 
172.17.0.2 sandbox.hortonworks.com sandbox 

8. Could access ambari UI through http://sandbox.hortonworks:8080

9. Setup new tweet account

Application Settings

Keep the "Consumer Secret" a secret. This key should never be human-readable in your application.
Consumer Key (API Key)	yyFCPXXvfXXXXXTmM
Consumer Secret (API Secret)	f29yXXXXXXXXXAGalLLDwQ
Access Level	Read and write (modify app permissions)
Owner jaaaaaslton
Owner I999999986118144

Your Access Token

This access token can be used to make API requests on your own account's behalf. Do not share your access tokensecret with anyone. 
Access Token 902XXX566X6XFEPtWL0oKY 
Access Token Secret  K2CdXXXXXXXXXDNhB8 
Access LevelRead and write Owner jaaaaaslton 
Owner I999999986118144

Then downloaded the Twitter_JSON_Flow.xml

And followed the instructions all went smoothly. see attached screenshots and next bunch


solr04.jpgsolr02.jpgtwitter.jpgsolr03.jpgtweet-flowing.jpg

Re: NiFi unable to connect Twitter

Mentor

@Kishore Kumar

My guess is your tweet setup did you select all the processors

Now that all processors are selected, go to the actions toolbar and click the start button play_signal. You can see your workflow running. ???

If you followed the tutorial well you should encounter an issue except make sure you HDP version matches that used in creating the hive tables and views.

Good luck


solr.jpgbanana-dashboard.jpghive-queryjpg.jpghive20-view.jpgnifi.jpg

Re: NiFi unable to connect Twitter

Contributor

@Geoffrey Shelton Okot

I am selecting all then running he processors . It fails at the Twitter processor itself . .

Re: NiFi unable to connect Twitter

Mentor

@Kishore Kumar

Do you mean the Grab Garden Hose? If you Can you download a new Twitter_JSON_Flow.xml , and make sure is a valid xml format by checking it with a free online formatter

Is there no firewall on your network ? Can you access your twitter account https://stream.twitter.com/1.1/statuses/sample.json?delimited=length&stall_warnings=true

Re: NiFi unable to connect Twitter

Contributor

@Geoffrey Shelton Okot.

Yeah "Grab Garden Hose" . Using the curl from NiFi server, i am able to access that end point .

curl https://stream.twitter.com/1.1/statuses/sample.json?delimited=length&stall_warnings=true

[1] 5432 [root@c9t21841 nifi]# <html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title> </head> <body> <h2>HTTP ERROR: 401</h2> <p>Problem accessing '/1.1/statuses/sample.json?delimited=length'. Reason: <pre> Unauthorized</pre> </body> </html>

Re: NiFi unable to connect Twitter

Contributor

Here is more details out put of curl command

curl --get 'https://stream.twitter.com/1.1/statuses/filter.json?delimited=length&stall_warnings=true ' --data 'count=2&screen_name=twitterapi' --header 'Authorization: OAuth oauth_consumer_key="ZCY8L9UGJ4GO1c3fV4vw3jPCJ", oauth_nonce="BBBBBBBBBBBBBBBBBBBBBBB", oauth_signature="CCCCCCCCCCCCCCCCCCCCCCCCCCC", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1471672391", oauth_token="75463767-XhrwuoEAmHaeFD6RHWyvJXLphcsU8PA0oaCeCNVQD", oauth_version="1.0"' --verbose

* About to connect() to proxy port 8080 (#0) * Trying 16.85.88.10... connected *

Connected to (16.85.88.10) port 8080 (#0) * Establish HTTP proxy tunnel to stream.twitter.com:443 > CONNECT stream.twitter.com:443 HTTP/1.1 > Host: stream.twitter.com:443 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Proxy-Connection: Keep-Alive > Authorization: OAuth oauth_consumer_key="ZCY8L9UGJ4GO1c3fV4vw3jPCJ", oauth_nonce="BBBBBBBBBBBBBBBBBBBBBBB", oauth_signature="CCCCCCCCCCCCCCCCCCCCCCCCCCC", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1471672391", oauth_token="75463767-XhrwuoEAmHaeFD6RHWyvJXLphcsU8PA0oaCeCNVQD", oauth_version="1.0" > < HTTP/1.1

200 Connection established