Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi InvokeHTTP Basic Auth 401

Highlighted

Nifi InvokeHTTP Basic Auth 401

New Contributor

I'm attempting to pull JSON data from a streaming API using an endpoint like below:

https://{username}.streetside.io/stream/{source}_main/subscription/main/part/1/data.json 

The endpoint requires basic auth, which I have attempted to pass by using InvokeHTTP's digest auth, as well as passing the string in a flowfile, and I also tried base64 encoding the string. But all returned 401 errors. I had a similar issue with basic auth when trying to connect to a GNIP stream.

I also tried with a GetHTTP processor, however this returned a 403 error.

I can pull the data with the following python code.

BUFFERSIZE=1024
PAUSESECONDS=10
user='myuser'
password = 'mypassword'
auth_header = 'Basic %s' % (":".join([user,password]).encode('Base64').strip('\r\n'))
conn = httplib.HTTPSConnection("..com")

while True:

try:

 conn.request("GET", "/stream/_/subscription/part/1/data.json",
{'Authorization':auth_header})
 response = conn.getresponse()

API endpoint data example:

{
  "id": 1809300141351467,
  "text": {
    "type": "status",
    "event": "add",
    "status": {
      "isLongText": false,
      "in_reply_to_status_id": "",
      "created_at": "Sun Sep 30 21:51:02 +0800 2018",
      "mid": "4290068346945048",
      "annotations": [
        {
          "mapi_request": true
        }
      ],
      "source": "iPhone 6",
      "retweeted_status": {
        "sync_mblog": true,
        "isLongText": true,
        "photoTag": "[]",
        "in_reply_to_status_id": "",
        "page_type": 96,
        "created_at": "Sun Sep 30 21:31:03 +0800 2018",
        "mid": "4290063317758790",
        "annotations": [
          {
            "client_mblogid": "55e20d7b-b423-4f98-bc25-a1b1556fa676"
          },
          {
            "mapi_request": true
          }

Any ideas?

Cheers,

Mike


socialgistflow.pngscreenshot-from-2018-09-30-15-08-30.png
Don't have an account?
Coming from Hortonworks? Activate your account here