Hi, has anyone managed to stream twitter data into NIFI using GNIP's API Endpoint. I managed to stream using keys (GetTwitter processor), but i'm having difficulty streaming using their enterprise endpoint e.g. https://gnip-stream.twitter.com/stream/powertrack/accounts/{account_name}/publishers/twitter/dev.json The problem appears to be authentication. I attempted this using both an invokehttp and gethttp processor but both returned 401. The formatting matches the CURL request i'm sending, including encoding my credentials. I've instead resorted to producing to kafka, but interested to know if anyone managed to do this using InvokeHTTP. Thanks, Mike curl --compressed -v -u{username} "https://gnip-stream.twitter.com/stream/powertrack/accounts/{accountname}/publishers/twitter/dev.json" ... View more

I'm attempting to pull JSON data from a streaming API using an endpoint like below: https://{username}.streetside.io/stream/{source}_main/subscription/main/part/1/data.json The endpoint requires basic auth, which I have attempted to pass by using InvokeHTTP's digest auth, as well as passing the string in a flowfile, and I also tried base64 encoding the string. But all returned 401 errors. I had a similar issue with basic auth when trying to connect to a GNIP stream. I also tried with a GetHTTP processor, however this returned a 403 error. I can pull the data with the following python code. BUFFERSIZE=1024 PAUSESECONDS=10 user='myuser' password = 'mypassword' auth_header = 'Basic %s' % (":".join([user,password]).encode('Base64').strip('\r\n')) conn = httplib.HTTPSConnection("..com") while True: try: conn.request("GET", "/stream/_/subscription/part/1/data.json", {'Authorization':auth_header}) response = conn.getresponse() API endpoint data example: { "id": 1809300141351467, "text": { "type": "status", "event": "add", "status": { "isLongText": false, "in_reply_to_status_id": "", "created_at": "Sun Sep 30 21:51:02 +0800 2018", "mid": "4290068346945048", "annotations": [ { "mapi_request": true } ], "source": "iPhone 6", "retweeted_status": { "sync_mblog": true, "isLongText": true, "photoTag": "[]", "in_reply_to_status_id": "", "page_type": 96, "created_at": "Sun Sep 30 21:31:03 +0800 2018", "mid": "4290063317758790", "annotations": [ { "client_mblogid": "55e20d7b-b423-4f98-bc25-a1b1556fa676" }, { "mapi_request": true } Any ideas? Cheers, Mike ... View more