Member since
07-08-2016
260
Posts
43
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
616 | 05-02-2018 06:03 PM | |
1016 | 10-18-2017 04:02 PM | |
372 | 08-25-2017 08:59 PM | |
517 | 07-21-2017 08:13 PM | |
4440 | 04-06-2017 09:54 PM |
04-06-2018
04:20 PM
Hi @Matt Burgess , i replaced the space with %20 after Bearer , but now getting this response.. { "error": { "errors": [ { "domain": "global", "reason": "authError", "message": "Invalid Credentials", "locationType": "header", "location": "Authorization" } ], "code": 401, "message": "Invalid Credentials" } } where as if i try the url directly using , its working..any help.?? https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token=${access_token}
... View more
04-06-2018
03:57 PM
@Matt Burgess , I am getting this error when I tried to send the Authorization from Attributes To Send..any idea why.?? InvokeHTTP[id=e6af6693-2428-3261-5418-a9cd3fc4c76b] Routing to Failure due to exception: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A
... View more
04-06-2018
02:32 PM
Hi, has anyone accessed GCS buckets thru NiFi.? if so what am I doing wrong.? Regards, Sai
... View more
04-06-2018
02:30 PM
@Matt Burgess, Thanks for the reply , I am more concerned about the -o "[SAVE_TO_LOCATION]" , should I be sending it also in Attributes to Send. can I send both (as comma separated or how).?? the CURL command shows options -H and -o ,can I add it to the attributes.? Regards, Sai
... View more
04-05-2018
09:26 PM
Hi, I am trying to access a GCSBucket using ListGCSBucket. I created a GCPCredentialsControllerService using service account JSON that i downloaded from Google console. the service account is an Owner of the project and also Admin on the cloud storage. anything that i am doing wrong.? Regards, Sai
... View more
Labels:
04-05-2018
09:00 PM
Hi, i am trying to download an object from google cloud storage using OAuth API. i got the access token. i can see the objects metadata when i use the Remote URL in InvokeHTTP as as https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token={access_token} {
"kind": "storage#object",
"id": "tarapareddy/test2/1522956104046841",
"selfLink": "https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2",
"name": "test2",
"bucket": "tarapareddy",
"generation": "1522956104046841",
"metageneration": "1",
"contentType": "application/octet-stream",
"timeCreated": "2018-04-05T19:21:44.033Z",
"updated": "2018-04-05T19:21:44.033Z",
"storageClass": "MULTI_REGIONAL",
"timeStorageClassUpdated": "2018-04-05T19:21:44.033Z",
"size": "59225004",
"md5Hash": "BG1rEe0zy8oe6U/Udcg0NQ==",
"mediaLink": "https://www.googleapis.com/download/storage/v1/b/tarapareddy/o/test2?generation=1522956104046841&alt=media",
"crc32c": "w1oBgw==",
"etag": "CPmx3Mzto9oCEAE="
} but i want the object downloaded into my flow..their documentation shows to use CURL in the format below. how do i set the -o .?? i tried setting it in Attributes to send , but it didnt work. curl -X GET \
-H "Authorization: Bearer [OAUTH2_TOKEN]" \
-o "[SAVE_TO_LOCATION]" \
"https://www.googleapis.com/storage/v1/b/[BUCKET_NAME]/o/[OBJECT_NAME]?alt=media"
... View more
Labels:
04-02-2018
04:30 PM
@Constantin Stanca, My file names will change everyday like datafile_yyyymmdd.json , so we still need to create a scheduled job using Nifi and their RESTAPI to move the file to GCS.? Or is there a way that we can separate this .? Regards, Sai
... View more
03-28-2018
09:13 PM
3 Kudos
Hi @Constantin Stanca Can we use their REST APIs to download bigquery tables? Hi @Timothy Spann , we cannot use any other processors which are not supported. Regards, Sai
... View more
03-28-2018
03:31 AM
3 Kudos
Hi, i do not see any processors to connect to Google Big Query and execute queries. is there a way where i can get results from google big query using NiFi.?
... View more
Labels:
03-27-2018
07:43 PM
@Matt Burgess,
i was able to use SplitRecord and split the huge JSON file into multiple files with 10K records in each.
the reason i need to do splitJSON is becasue i have nested JSON
in the example below i have 6 nested arrays brnd,qa,adr,ibcn,cq,ofr for one customer.
only way i know to get those are by doing splitJSON-->EvaluateJSON 6 times..
each time getting values of the elements at that level and getting array jSON and using it on next splitJSON and so on..i have attached my flow..but if there is any better way please let me know.
[
{
"Customer_Id": 1111111,
"brnd": [
{
"Brand_Nm": "Test",
"qa": [
{
"Assignment_Id": 1116211,
"Assign_Dt": null,
"adr": [
{
"AddressLine_1": null,
"AddressLine_2": null,
"City": null,
"State_Cd": null,
"Postal_Cd": "11111 ",
"ibcn": [
{
"BarCode_No": "162117",
"cq": [
{
"Vendor_Desc": "Coupons Inc",
"ofr": [
{
"Offer_Nm": "General_5DollarDryCatFood_EM_2016",
"Offer_Ds": "On (1) bag of Purina Beyond brand dry cat food, any size, any variety.",
"Offer_Expire_Dt": "2017-12-31T00:00:00",
"offer_channel_desc": "EM OFFER",
"SourceFeed_Ds": "A"
}
]
}
]
}
]
}
]
}
]
}
]
}
]
... View more
03-27-2018
04:12 PM
1 Kudo
Hi, I have a huge nested array JSON file 1 GB and i need to flatten it . i am using SplitJSON as my first processor and its erroring with out of memory errors. I have 16gb reserved for java heap. dont know why it errors out. i am sure will have more than a million JSON records. how do i achieve this.? for text files of bigger size i have used split text and then did my processing and merged before i pushed it to destination. looks like splitJSON is waiting until it splits the whole file. How do i achieve that functionality with JSON files.?? if i use splitContent or splitText it will mess up the json format . here is how my test json of 2 records look.. [{"Review_Id":111111111,"Brand_Id":"test","Product_Ds":"testprod1","Email_Id":"test@aol.com","Customer_Id":690,"Rating_No":5,"Recommend_Fg":true,"Review_Nm":"Great tasting!","Review_Ds":"I feed this to the picky dogs at my kennel. They love it!","ReviewStatus_Cd":"A","Review_Dt":"2015-05-01T17:37:28","Reviews_Answers":[{"Answer_Id":655108,"Review_Id":119205458,"Question_Ds":"Age","Answer_Ds":"35to44","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655109,"Review_Id":119205458,"Question_Ds":"Employee","Answer_Ds":"No","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655110,"Review_Id":119205458,"Question_Ds":"Taste my pet enjoys","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655111,"Review_Id":119205458,"Question_Ds":"Gender","Answer_Ds":"Female","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655112,"Review_Id":119205458,"Question_Ds":"Number of dogs","Answer_Ds":"3","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655113,"Review_Id":119205458,"Question_Ds":"Quality","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655114,"Review_Id":119205458,"Question_Ds":"Sample Product","Answer_Ds":"No","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655115,"Review_Id":119205458,"Question_Ds":"Value of Product","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"}]},
{"Review_Id":222222222,"Brand_Id":"test2","Product_Ds":"testprod2","Email_Id":"test@msn.com","Customer_Id":831,"Rating_No":5,"Recommend_Fg":true,"Review_Nm":"My dogs love the tender and crunchy pieces.","Review_Ds":"I have been buying this dog food for quite sometime, I have large and very small dogs and the size of the food fits for them all. They love it.","ReviewStatus_Cd":"A","Review_Dt":"2017-06-27T09:45:19","Reviews_Answers":[{"Answer_Id":1276571,"Review_Id":181705560,"Question_Ds":"*Received free food and\/or goods","Answer_Ds":"Yes","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276572,"Review_Id":181705560,"Question_Ds":"Food purchased","Answer_Ds":"5","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276573,"Review_Id":181705560,"Question_Ds":"Number of dogs","Answer_Ds":"5OrMore","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276574,"Review_Id":181705560,"Question_Ds":"*Entered as part of a promotion","Answer_Ds":"False","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276575,"Review_Id":181705560,"Question_Ds":"Would like to receive emails from","Answer_Ds":"Yes","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276576,"Review_Id":181705560,"Question_Ds":"Employee","Answer_Ds":"No","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276577,"Review_Id":181705560,"Question_Ds":"Quality","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276578,"Review_Id":181705560,"Question_Ds":"Taste my pet enjoys","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276579,"Review_Id":181705560,"Question_Ds":"Value for the money","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"}]}]
... View more
Labels:
03-21-2018
06:31 PM
@Shu , @Rahul Soni makes sense , my JSON is nested arrays so its not flattening it out..thank you. Regards, Sai
... View more
03-21-2018
05:57 PM
@Shu, i am still not able to use the template now getting the error (screen shot attached). but here is a test json..basically all the arrays can repeat multiple times..how to flatten those..?? {
"Customer_Id": 11111,
"dsc": [
{
"Brand_Nm": "test",
"qa": [
{
"Assignment_Id": 222,
"addr": [
{
"AddressLine_1": "stl",
"City": "stl"
}
]
},{
"Assignment_Id": 223,
"addr": [
{
"AddressLine_1": "stc",
"City": "stc"
}
]
}
]
}, {
"Brand_Nm": "test2",
"qa": [
{
"Assignment_Id": 333,
"addr": [
{
"AddressLine_1": "ofl",
"City": "ofl"
}
]
},{
"Assignment_Id": 334,
"addr": [
{
"AddressLine_1": "estl",
"City": "estl"
}, {
"AddressLine_1": "estl1",
"City": "estl1"
}
]
}
]
}
]
} i like it break it as 3 files like Customer_Id,Brand_Nm 11111,test 11111,test2 Customer_Id,qa_Assignment_Id 11111,222 11111,223 11111,333 11111,334 Customer_Id,Assignment_Id,AddressLine_1,City 11111,222,stc,stc 11111,223,ofl,ofl 11111,333,estl,estl 11111,334,estl,estl 11111,334,estl1,estl1 and also is it possible to flatten the entire structure.. Customer_Id,Brand_Nm,assignment_id,addressline_1,city
1 1111,test,222,stl,sttl 11111,test,223,stc,stc 11111,test2,333,ofl,ofl 11111,test2,334,estl,estl 11111,test2,334,estl1,estl1
... View more
03-21-2018
04:45 PM
Hi, Can anyone help me on how to use NiFi FlattenJSON processor.? i tried to send a JSON as input to the processor and it didnt change on the output. basically it output the same file..where as i was expecting it to flatten all the nested arrays,lists etc.. Regards, Sai
... View more
Labels:
03-21-2018
03:16 PM
@Shu, I am not able to use your template as it gave me this error Invalid UUID string: 03701c5912a831d30000000000000000. anyway will it work if the incoming file have many records (330K in this case) , at the end it will split in to 3 files.? That would be awesome. also I can have multiple addresses for a customer , if we hard code with 0 will it get all the rows .? Regards, Sai
... View more
03-20-2018
08:57 PM
@Shu , Thank you. It helped. i was able to supply nulls and it came out like below.. one another question along the same lines. what is the best way to split the this JSON in to multiple files based on some columns so that i can create Hive tables on them. for example files with Customer_Id,Brand_Nm,BarCode_No,Offer_Ds,Offer_Expire_Dt Customer_Id,Assignment_id,Assign_Dt,Offer_Nm Customer_Id,Address_Line_1,Address_Line_2,City,State_Cd,Postal_Cd,Vendor_Desc,Offer,Source
... View more
03-20-2018
06:16 PM
any one has any comment on this.?
... View more
03-20-2018
03:47 PM
1 Kudo
Hi, I am reading a JSON file , and splitting in to rows by using SplitJSON processor and then i am trying to convert then to AVRO using ConvertRecord or ConvertJSONToAvro. I gave the JSON schema with all the possible columns. It is converting the records when it contains all the fields\elements. But if it misses an element its routing to failures. how do you solve this.?? all the incoming records wont have all the fields in schema.
... View more
Labels:
03-19-2018
06:43 PM
Hi , I haven't done any upgrades on my HDF stand alone instance on Windows server yet. We are on 3.0 and want to upgrade to 3.1 Is there any documentation for upgrading on Windows server.? Please share. Regards, Sai
... View more
Labels:
03-15-2018
03:01 PM
@Scott Shaw, thank you , i will look in to those. i am also looking to break down nested JSON files in to multiple files based on tags and then create hive tables on top of each file. Our source of these huge JSON files is traditional RDBS where they are joining multiple files and sending us as one JSON. i like to split the JSON one for each file. will that be a good approach.? Regards, Sai
... View more
03-14-2018
05:36 PM
1 Kudo
Hi, We have huge 4 gb schema less nested JSON files , we have a need to create Hive tables on top of those to enable reporting on top of it. what is the best way to do this.? Can we create Hive tables as it is on JSON files and can use that for reporting? or Do we need to flatten those and create tables.? or Can we read line by line and insert in to hive , will it work on huge files with millions of rows. or Can it be done in any better ways.? Regards, Sai
... View more
Labels:
03-13-2018
08:13 PM
Hi @Alireza Sadeghi , did you able to solve the issue.?? if so how.?? i am running in to the same issue , it is working if i use SplitJSON--> ConvertJSONToAvro processers . but ruuning in to the same issue as you when i directly use CovertJSONToAvro with Record Schema property set.
... View more
03-09-2018
08:24 PM
Hi, We are trying secure our NiFi instance by using our company signed certificates instead of self signed. The instance is working fine with self-signed certs. after adding the new company CA certs to keystore and truststores , we changed the nifi.cluster.node.address and nifi.web.https.host to match the host from certificate and restarted NiFi. It looks like NiFi has restarted successfully but i am not able to log in . it shows Success you are already logged in. The hostname of the server is different than the host name of the certificate.
... View more
Labels:
03-01-2018
08:56 PM
@Bryan Bende looks like I have an option to use SegmentContent and MergeContent after I use ReplaceText on each segment. I tried it with 10mb segment size on a 120 mb file and it worked..now will try on the bigger file.
... View more
03-01-2018
07:56 PM
but even if it is treating as one line, I have 16gb for java heap and the file is only 7.5 gb.it seems like I am getting that error after ReplaceText reads 2gb.(if you look at the screen shot above)
... View more
03-01-2018
07:49 PM
@Bryan Bende I think that is what happening there. what options do I have now.?
... View more
03-01-2018
05:38 PM
@Bryan Bende It looks like ReplaceText is not streaming , I am getting out of memory errors. I am trying this with a 7.5 gb file.. 11:33:19 CSTERRORddf8b120-0161-1000-a2c4-c410a98382c1 ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] failed to process due to java.lang.OutOfMemoryError; rolling back session: java.lang.OutOfMemoryError 11:33:19 CSTERRORddf8b120-0161-1000-a2c4-c410a98382c1 ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] failed to process session due to java.lang.OutOfMemoryError: java.lang.OutOfMemoryError
... View more
03-01-2018
05:13 PM
@Bryan Bende i will try and see how it goes. But Is there a better approach.? in order for me to use json record processors I am thinking it will fail to read the json records where it has issue with rows like the one mentioned above.
... View more