Member since
07-08-2016
260
Posts
44
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2430 | 05-02-2018 06:03 PM | |
4900 | 10-18-2017 04:02 PM | |
1571 | 08-25-2017 08:59 PM | |
2152 | 07-21-2017 08:13 PM | |
8625 | 04-06-2017 09:54 PM |
04-06-2018
02:30 PM
@Matt Burgess, Thanks for the reply , I am more concerned about the -o "[SAVE_TO_LOCATION]" , should I be sending it also in Attributes to Send. can I send both (as comma separated or how).?? the CURL command shows options -H and -o ,can I add it to the attributes.? Regards, Sai
... View more
04-05-2018
09:00 PM
Hi, i am trying to download an object from google cloud storage using OAuth API. i got the access token. i can see the objects metadata when i use the Remote URL in InvokeHTTP as as https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token={access_token} {
"kind": "storage#object",
"id": "tarapareddy/test2/1522956104046841",
"selfLink": "https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2",
"name": "test2",
"bucket": "tarapareddy",
"generation": "1522956104046841",
"metageneration": "1",
"contentType": "application/octet-stream",
"timeCreated": "2018-04-05T19:21:44.033Z",
"updated": "2018-04-05T19:21:44.033Z",
"storageClass": "MULTI_REGIONAL",
"timeStorageClassUpdated": "2018-04-05T19:21:44.033Z",
"size": "59225004",
"md5Hash": "BG1rEe0zy8oe6U/Udcg0NQ==",
"mediaLink": "https://www.googleapis.com/download/storage/v1/b/tarapareddy/o/test2?generation=1522956104046841&alt=media",
"crc32c": "w1oBgw==",
"etag": "CPmx3Mzto9oCEAE="
} but i want the object downloaded into my flow..their documentation shows to use CURL in the format below. how do i set the -o .?? i tried setting it in Attributes to send , but it didnt work. curl -X GET \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-o "[SAVE_TO_LOCATION]" \
"https://www.googleapis.com/storage/v1/b/[BUCKET_NAME]/o/[OBJECT_NAME]?alt=media"
... View more
Labels:
- Labels:
-
Apache NiFi
04-02-2018
04:30 PM
@Constantin Stanca, My file names will change everyday like datafile_yyyymmdd.json , so we still need to create a scheduled job using Nifi and their RESTAPI to move the file to GCS.? Or is there a way that we can separate this .? Regards, Sai
... View more
03-28-2018
09:13 PM
3 Kudos
Hi @Constantin Stanca Can we use their REST APIs to download bigquery tables? Hi @Timothy Spann , we cannot use any other processors which are not supported. Regards, Sai
... View more
03-28-2018
03:31 AM
3 Kudos
Hi, i do not see any processors to connect to Google Big Query and execute queries. is there a way where i can get results from google big query using NiFi.?
... View more
Labels:
- Labels:
-
Apache NiFi
03-21-2018
06:31 PM
@Shu , @Rahul Soni makes sense , my JSON is nested arrays so its not flattening it out..thank you. Regards, Sai
... View more
03-21-2018
05:57 PM
@Shu, i am still not able to use the template now getting the error (screen shot attached). but here is a test json..basically all the arrays can repeat multiple times..how to flatten those..?? {
"Customer_Id": 11111,
"dsc": [
{
"Brand_Nm": "test",
"qa": [
{
"Assignment_Id": 222,
"addr": [
{
"AddressLine_1": "stl",
"City": "stl"
}
]
},{
"Assignment_Id": 223,
"addr": [
{
"AddressLine_1": "stc",
"City": "stc"
}
]
}
]
}, {
"Brand_Nm": "test2",
"qa": [
{
"Assignment_Id": 333,
"addr": [
{
"AddressLine_1": "ofl",
"City": "ofl"
}
]
},{
"Assignment_Id": 334,
"addr": [
{
"AddressLine_1": "estl",
"City": "estl"
}, {
"AddressLine_1": "estl1",
"City": "estl1"
}
]
}
]
}
]
} i like it break it as 3 files like Customer_Id,Brand_Nm 11111,test 11111,test2 Customer_Id,qa_Assignment_Id 11111,222 11111,223 11111,333 11111,334 Customer_Id,Assignment_Id,AddressLine_1,City 11111,222,stc,stc 11111,223,ofl,ofl 11111,333,estl,estl 11111,334,estl,estl 11111,334,estl1,estl1 and also is it possible to flatten the entire structure.. Customer_Id,Brand_Nm,assignment_id,addressline_1,city
1 1111,test,222,stl,sttl 11111,test,223,stc,stc 11111,test2,333,ofl,ofl 11111,test2,334,estl,estl 11111,test2,334,estl1,estl1
... View more
03-21-2018
04:45 PM
Hi, Can anyone help me on how to use NiFi FlattenJSON processor.? i tried to send a JSON as input to the processor and it didnt change on the output. basically it output the same file..where as i was expecting it to flatten all the nested arrays,lists etc.. Regards, Sai
... View more
Labels:
- Labels:
-
Apache NiFi
03-21-2018
03:16 PM
@Shu, I am not able to use your template as it gave me this error Invalid UUID string: 03701c5912a831d30000000000000000. anyway will it work if the incoming file have many records (330K in this case) , at the end it will split in to 3 files.? That would be awesome. also I can have multiple addresses for a customer , if we hard code with 0 will it get all the rows .? Regards, Sai
... View more
03-20-2018
08:57 PM
@Shu , Thank you. It helped. i was able to supply nulls and it came out like below.. one another question along the same lines. what is the best way to split the this JSON in to multiple files based on some columns so that i can create Hive tables on them. for example files with Customer_Id,Brand_Nm,BarCode_No,Offer_Ds,Offer_Expire_Dt Customer_Id,Assignment_id,Assign_Dt,Offer_Nm Customer_Id,Address_Line_1,Address_Line_2,City,State_Cd,Postal_Cd,Vendor_Desc,Offer,Source
... View more