Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to send headers to InvokeHTTP process.?

Highlighted

How to send headers to InvokeHTTP process.?

Super Collaborator

Hi,

i am trying to download an object from google cloud storage using OAuth API. i got the access token.

i can see the objects metadata when i use the Remote URL in InvokeHTTP as as

https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token={access_token}

{
 "kind": "storage#object",
 "id": "tarapareddy/test2/1522956104046841",
 "selfLink": "https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2",
 "name": "test2",
 "bucket": "tarapareddy",
 "generation": "1522956104046841",
 "metageneration": "1",
 "contentType": "application/octet-stream",
 "timeCreated": "2018-04-05T19:21:44.033Z",
 "updated": "2018-04-05T19:21:44.033Z",
 "storageClass": "MULTI_REGIONAL",
 "timeStorageClassUpdated": "2018-04-05T19:21:44.033Z",
 "size": "59225004",
 "md5Hash": "BG1rEe0zy8oe6U/Udcg0NQ==",
 "mediaLink": "https://www.googleapis.com/download/storage/v1/b/tarapareddy/o/test2?generation=1522956104046841&alt=media",
 "crc32c": "w1oBgw==",
 "etag": "CPmx3Mzto9oCEAE="
}

but i want the object downloaded into my flow..their documentation shows to use CURL in the format below. how do i set the -o .?? i tried setting it in Attributes to send , but it didnt work.

curl -X GET \
    -H "Authorization: Bearer [OAUTH2_TOKEN]" \
    -o "[SAVE_TO_LOCATION]" \
    "https://www.googleapis.com/storage/v1/b/[BUCKET_NAME]/o/[OBJECT_NAME]?alt=media"
6 REPLIES 6
Highlighted

Re: How to send headers to InvokeHTTP process.?

Super Guru

I think you should be able to provide "Authorization" in Attributes to Send, and have an attribute "Authorization" with value "Bearer [OAUTH2_TOKEN]", with that value replaced with the token (you can use UpdateAttribute with Expression Language to set all that up). Then you should be able to use the URL that works with curl.

If that doesn't work, try an attribute with any name (say, "auth") with value "Authorization: Bearer [OAUTH2_TOKEN]" and include "auth" in the Attributes to Send property.

Highlighted

Re: How to send headers to InvokeHTTP process.?

New Contributor

Hi @mburgess 

 

I tried adding attribute name “auth” in Attributes to Send property and had this “auth” as a custom property in my invokeHTTP processor with value 'Authorization: Bearer ${access_token}' where the value of variable access_token gets populated from EvaluateJsonPath processor in front of my invokeHTTP, but somehow I am still getting unauthorized as the response.

Is there anything else that I’m missing for token to get passed on as a header in this call?

Highlighted

Re: How to send headers to InvokeHTTP process.?

Super Collaborator

@Matt Burgess,

Thanks for the reply , I am more concerned about the -o "[SAVE_TO_LOCATION]" , should I be sending it also in

Attributes to Send. can I send both (as comma separated or how).?? the CURL command shows options -H and -o ,can I add it to the attributes.?

Regards,

Sai

Highlighted

Re: How to send headers to InvokeHTTP process.?

Super Guru

You don't need to specify that parameter, in curl the -o lets you pick the destination of the output. In InvokeHttp the response will be in the content of the outgoing flow file. If you want to put that content into a file, you can use PutFile after InvokeHttp.

Highlighted

Re: How to send headers to InvokeHTTP process.?

Super Collaborator

@Matt Burgess ,

I am getting this error when I tried to send the Authorization from Attributes To Send..any idea why.??

InvokeHTTP[id=e6af6693-2428-3261-5418-a9cd3fc4c76b] Routing to Failure due to exception: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A

Re: How to send headers to InvokeHTTP process.?

Super Collaborator

Hi @Matt Burgess ,

i replaced the space with %20 after Bearer , but now getting this response..

{ "error": { "errors": [ { "domain": "global", "reason": "authError", "message": "Invalid Credentials", "locationType": "header", "location": "Authorization" } ], "code": 401, "message": "Invalid Credentials" } }

where as if i try the url directly using , its working..any help.??

https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token=${access_token}

Don't have an account?
Coming from Hortonworks? Activate your account here