About saikrishna_tara

saikrishna_tara · ‎04-06-2018

Hi @Matt Burgess , i replaced the space with %20 after Bearer , but now getting this response.. { "error": { "errors": [ { "domain": "global", "reason": "authError", "message": "Invalid Credentials", "locationType": "header", "location": "Authorization" } ], "code": 401, "message": "Invalid Credentials" } } where as if i try the url directly using , its working..any help.?? https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token=${access_token}

saikrishna_tara · ‎04-06-2018

@Matt Burgess , I am getting this error when I tried to send the Authorization from Attributes To Send..any idea why.?? InvokeHTTP[id=e6af6693-2428-3261-5418-a9cd3fc4c76b] Routing to Failure due to exception: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A: java.lang.IllegalArgumentException: Unexpected char 0xa0 at 14 in header value: Authorization: Bearer ya29.GlyVBYcjf5qiEVyoPHJzDjf3ZLiXacV5Oaoaj817Tl-FzChWs9TSRc3vVNO8aMEj_HGHQMStSSJOwzQ3-WOygapsx8n4qmM-IVWFstZFtMa5Ld-47ZmKSClWEKFW5A

saikrishna_tara · ‎04-06-2018

Hi, has anyone accessed GCS buckets thru NiFi.? if so what am I doing wrong.? Regards, Sai

saikrishna_tara · ‎04-06-2018

@Matt Burgess, Thanks for the reply , I am more concerned about the -o "[SAVE_TO_LOCATION]" , should I be sending it also in Attributes to Send. can I send both (as comma separated or how).?? the CURL command shows options -H and -o ,can I add it to the attributes.? Regards, Sai

saikrishna_tara · ‎04-05-2018

Hi, I am trying to access a GCSBucket using ListGCSBucket. I created a GCPCredentialsControllerService using service account JSON that i downloaded from Google console. the service account is an Owner of the project and also Admin on the cloud storage. anything that i am doing wrong.? Regards, Sai

saikrishna_tara · ‎04-05-2018

Hi, i am trying to download an object from google cloud storage using OAuth API. i got the access token. i can see the objects metadata when i use the Remote URL in InvokeHTTP as as https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2?access_token={access_token} { "kind": "storage#object", "id": "tarapareddy/test2/1522956104046841", "selfLink": "https://www.googleapis.com/storage/v1/b/tarapareddy/o/test2", "name": "test2", "bucket": "tarapareddy", "generation": "1522956104046841", "metageneration": "1", "contentType": "application/octet-stream", "timeCreated": "2018-04-05T19:21:44.033Z", "updated": "2018-04-05T19:21:44.033Z", "storageClass": "MULTI_REGIONAL", "timeStorageClassUpdated": "2018-04-05T19:21:44.033Z", "size": "59225004", "md5Hash": "BG1rEe0zy8oe6U/Udcg0NQ==", "mediaLink": "https://www.googleapis.com/download/storage/v1/b/tarapareddy/o/test2?generation=1522956104046841&alt=media", "crc32c": "w1oBgw==", "etag": "CPmx3Mzto9oCEAE=" } but i want the object downloaded into my flow..their documentation shows to use CURL in the format below. how do i set the -o .?? i tried setting it in Attributes to send , but it didnt work. curl -X GET \ -H "Authorization: Bearer [OAUTH2_TOKEN]" \ -o "[SAVE_TO_LOCATION]" \ "https://www.googleapis.com/storage/v1/b/[BUCKET_NAME]/o/[OBJECT_NAME]?alt=media"

saikrishna_tara · ‎04-02-2018

@Constantin Stanca, My file names will change everyday like datafile_yyyymmdd.json , so we still need to create a scheduled job using Nifi and their RESTAPI to move the file to GCS.? Or is there a way that we can separate this .? Regards, Sai

saikrishna_tara · ‎03-28-2018

Hi @Constantin Stanca Can we use their REST APIs to download bigquery tables? Hi @Timothy Spann , we cannot use any other processors which are not supported. Regards, Sai

saikrishna_tara · ‎03-28-2018

Hi, i do not see any processors to connect to Google Big Query and execute queries. is there a way where i can get results from google big query using NiFi.?

saikrishna_tara · ‎03-27-2018

@Matt Burgess , Here are the screen shots..

saikrishna_tara · ‎03-27-2018

@Matt Burgess, i was able to use SplitRecord and split the huge JSON file into multiple files with 10K records in each. the reason i need to do splitJSON is becasue i have nested JSON in the example below i have 6 nested arrays brnd,qa,adr,ibcn,cq,ofr for one customer. only way i know to get those are by doing splitJSON-->EvaluateJSON 6 times.. each time getting values of the elements at that level and getting array jSON and using it on next splitJSON and so on..i have attached my flow..but if there is any better way please let me know. [ { "Customer_Id": 1111111, "brnd": [ { "Brand_Nm": "Test", "qa": [ { "Assignment_Id": 1116211, "Assign_Dt": null, "adr": [ { "AddressLine_1": null, "AddressLine_2": null, "City": null, "State_Cd": null, "Postal_Cd": "11111 ", "ibcn": [ { "BarCode_No": "162117", "cq": [ { "Vendor_Desc": "Coupons Inc", "ofr": [ { "Offer_Nm": "General_5DollarDryCatFood_EM_2016", "Offer_Ds": "On (1) bag of Purina Beyond brand dry cat food, any size, any variety.", "Offer_Expire_Dt": "2017-12-31T00:00:00", "offer_channel_desc": "EM OFFER", "SourceFeed_Ds": "A" } ] } ] } ] } ] } ] } ] } ]

saikrishna_tara · ‎03-27-2018

Hi, I have a huge nested array JSON file 1 GB and i need to flatten it . i am using SplitJSON as my first processor and its erroring with out of memory errors. I have 16gb reserved for java heap. dont know why it errors out. i am sure will have more than a million JSON records. how do i achieve this.? for text files of bigger size i have used split text and then did my processing and merged before i pushed it to destination. looks like splitJSON is waiting until it splits the whole file. How do i achieve that functionality with JSON files.?? if i use splitContent or splitText it will mess up the json format . here is how my test json of 2 records look.. [{"Review_Id":111111111,"Brand_Id":"test","Product_Ds":"testprod1","Email_Id":"test@aol.com","Customer_Id":690,"Rating_No":5,"Recommend_Fg":true,"Review_Nm":"Great tasting!","Review_Ds":"I feed this to the picky dogs at my kennel. They love it!","ReviewStatus_Cd":"A","Review_Dt":"2015-05-01T17:37:28","Reviews_Answers":[{"Answer_Id":655108,"Review_Id":119205458,"Question_Ds":"Age","Answer_Ds":"35to44","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655109,"Review_Id":119205458,"Question_Ds":"Employee","Answer_Ds":"No","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655110,"Review_Id":119205458,"Question_Ds":"Taste my pet enjoys","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655111,"Review_Id":119205458,"Question_Ds":"Gender","Answer_Ds":"Female","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655112,"Review_Id":119205458,"Question_Ds":"Number of dogs","Answer_Ds":"3","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655113,"Review_Id":119205458,"Question_Ds":"Quality","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655114,"Review_Id":119205458,"Question_Ds":"Sample Product","Answer_Ds":"No","Created_Dt":"2017-04-04T14:29:28"},{"Answer_Id":655115,"Review_Id":119205458,"Question_Ds":"Value of Product","Rating_No":5,"Created_Dt":"2017-04-04T14:29:28"}]}, {"Review_Id":222222222,"Brand_Id":"test2","Product_Ds":"testprod2","Email_Id":"test@msn.com","Customer_Id":831,"Rating_No":5,"Recommend_Fg":true,"Review_Nm":"My dogs love the tender and crunchy pieces.","Review_Ds":"I have been buying this dog food for quite sometime, I have large and very small dogs and the size of the food fits for them all. They love it.","ReviewStatus_Cd":"A","Review_Dt":"2017-06-27T09:45:19","Reviews_Answers":[{"Answer_Id":1276571,"Review_Id":181705560,"Question_Ds":"*Received free food and\/or goods","Answer_Ds":"Yes","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276572,"Review_Id":181705560,"Question_Ds":"Food purchased","Answer_Ds":"5","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276573,"Review_Id":181705560,"Question_Ds":"Number of dogs","Answer_Ds":"5OrMore","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276574,"Review_Id":181705560,"Question_Ds":"*Entered as part of a promotion","Answer_Ds":"False","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276575,"Review_Id":181705560,"Question_Ds":"Would like to receive emails from","Answer_Ds":"Yes","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276576,"Review_Id":181705560,"Question_Ds":"Employee","Answer_Ds":"No","Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276577,"Review_Id":181705560,"Question_Ds":"Quality","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276578,"Review_Id":181705560,"Question_Ds":"Taste my pet enjoys","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"},{"Answer_Id":1276579,"Review_Id":181705560,"Question_Ds":"Value for the money","Rating_No":5,"Created_Dt":"2017-07-04T10:34:34"}]}]

saikrishna_tara · ‎03-21-2018

@Shu , @Rahul Soni makes sense , my JSON is nested arrays so its not flattening it out..thank you. Regards, Sai

saikrishna_tara · ‎03-21-2018

@Shu, i am still not able to use the template now getting the error (screen shot attached). but here is a test json..basically all the arrays can repeat multiple times..how to flatten those..?? { "Customer_Id": 11111, "dsc": [ { "Brand_Nm": "test", "qa": [ { "Assignment_Id": 222, "addr": [ { "AddressLine_1": "stl", "City": "stl" } ] },{ "Assignment_Id": 223, "addr": [ { "AddressLine_1": "stc", "City": "stc" } ] } ] }, { "Brand_Nm": "test2", "qa": [ { "Assignment_Id": 333, "addr": [ { "AddressLine_1": "ofl", "City": "ofl" } ] },{ "Assignment_Id": 334, "addr": [ { "AddressLine_1": "estl", "City": "estl" }, { "AddressLine_1": "estl1", "City": "estl1" } ] } ] } ] } i like it break it as 3 files like Customer_Id,Brand_Nm 11111,test 11111,test2 Customer_Id,qa_Assignment_Id 11111,222 11111,223 11111,333 11111,334 Customer_Id,Assignment_Id,AddressLine_1,City 11111,222,stc,stc 11111,223,ofl,ofl 11111,333,estl,estl 11111,334,estl,estl 11111,334,estl1,estl1 and also is it possible to flatten the entire structure.. Customer_Id,Brand_Nm,assignment_id,addressline_1,city 1 1111,test,222,stl,sttl 11111,test,223,stc,stc 11111,test2,333,ofl,ofl 11111,test2,334,estl,estl 11111,test2,334,estl1,estl1

saikrishna_tara · ‎03-21-2018

Hi, Can anyone help me on how to use NiFi FlattenJSON processor.? i tried to send a JSON as input to the processor and it didnt change on the output. basically it output the same file..where as i was expecting it to flatten all the nested arrays,lists etc.. Regards, Sai

saikrishna_tara · ‎03-21-2018

@Shu, I am not able to use your template as it gave me this error Invalid UUID string: 03701c5912a831d30000000000000000. anyway will it work if the incoming file have many records (330K in this case) , at the end it will split in to 3 files.? That would be awesome. also I can have multiple addresses for a customer , if we hard code with 0 will it get all the rows .? Regards, Sai

saikrishna_tara · ‎03-20-2018

@Shu , Thank you. It helped. i was able to supply nulls and it came out like below.. one another question along the same lines. what is the best way to split the this JSON in to multiple files based on some columns so that i can create Hive tables on them. for example files with Customer_Id,Brand_Nm,BarCode_No,Offer_Ds,Offer_Expire_Dt Customer_Id,Assignment_id,Assign_Dt,Offer_Nm Customer_Id,Address_Line_1,Address_Line_2,City,State_Cd,Postal_Cd,Vendor_Desc,Offer,Source

saikrishna_tara · ‎03-20-2018

any one has any comment on this.?

saikrishna_tara · ‎03-20-2018

Hi, I am reading a JSON file , and splitting in to rows by using SplitJSON processor and then i am trying to convert then to AVRO using ConvertRecord or ConvertJSONToAvro. I gave the JSON schema with all the possible columns. It is converting the records when it contains all the fields\elements. But if it misses an element its routing to failures. how do you solve this.?? all the incoming records wont have all the fields in schema.

saikrishna_tara · ‎03-19-2018

Hi , I haven't done any upgrades on my HDF stand alone instance on Windows server yet. We are on 3.0 and want to upgrade to 3.1 Is there any documentation for upgrading on Windows server.? Please share. Regards, Sai

saikrishna_tara · ‎03-15-2018

@Scott Shaw, thank you , i will look in to those. i am also looking to break down nested JSON files in to multiple files based on tags and then create hive tables on top of each file. Our source of these huge JSON files is traditional RDBS where they are joining multiple files and sending us as one JSON. i like to split the JSON one for each file. will that be a good approach.? Regards, Sai

saikrishna_tara · ‎03-14-2018

Hi, We have huge 4 gb schema less nested JSON files , we have a need to create Hive tables on top of those to enable reporting on top of it. what is the best way to do this.? Can we create Hive tables as it is on JSON files and can use that for reporting? or Do we need to flatten those and create tables.? or Can we read line by line and insert in to hive , will it work on huge files with millions of rows. or Can it be done in any better ways.? Regards, Sai

saikrishna_tara · ‎03-13-2018

Hi @Alireza Sadeghi , did you able to solve the issue.?? if so how.?? i am running in to the same issue , it is working if i use SplitJSON--> ConvertJSONToAvro processers . but ruuning in to the same issue as you when i directly use CovertJSONToAvro with Record Schema property set.

saikrishna_tara · ‎03-09-2018

Hi, We are trying secure our NiFi instance by using our company signed certificates instead of self signed. The instance is working fine with self-signed certs. after adding the new company CA certs to keystore and truststores , we changed the nifi.cluster.node.address and nifi.web.https.host to match the host from certificate and restarted NiFi. It looks like NiFi has restarted successfully but i am not able to log in . it shows Success you are already logged in. The hostname of the server is different than the host name of the certificate.

saikrishna_tara · ‎03-01-2018

@Bryan Bende looks like I have an option to use SegmentContent and MergeContent after I use ReplaceText on each segment. I tried it with 10mb segment size on a 120 mb file and it worked..now will try on the bigger file.

saikrishna_tara · ‎03-01-2018

but even if it is treating as one line, I have 16gb for java heap and the file is only 7.5 gb.it seems like I am getting that error after ReplaceText reads 2gb.(if you look at the screen shot above)

saikrishna_tara · ‎03-01-2018

@Bryan Bende I think that is what happening there. what options do I have now.?

saikrishna_tara · ‎03-01-2018

@Bryan Bende yes , exactly the same.

saikrishna_tara · ‎03-01-2018

@Bryan Bende It looks like ReplaceText is not streaming , I am getting out of memory errors. I am trying this with a 7.5 gb file.. 11:33:19 CSTERRORddf8b120-0161-1000-a2c4-c410a98382c1 ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] failed to process due to java.lang.OutOfMemoryError; rolling back session: java.lang.OutOfMemoryError 11:33:19 CSTERRORddf8b120-0161-1000-a2c4-c410a98382c1 ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] ReplaceText[id=ddf8b120-0161-1000-a2c4-c410a98382c1] failed to process session due to java.lang.OutOfMemoryError: java.lang.OutOfMemoryError

saikrishna_tara · ‎03-01-2018

@Bryan Bende i will try and see how it goes. But Is there a better approach.? in order for me to use json record processors I am thinking it will fail to read the json records where it has issue with rows like the one mentioned above.

Online	Offline
Last Visited	‎10-26-2018 04:07 PM

Date Registered	‎07-08-2016 07:25 PM
Last Visited	‎10-26-2018 04:07 PM
Total Messages Posted	260
Total Kudos Received	43

Re: I am using unpack processor in NIFI to unzip a...

Re: Changing NiFi Templates

Re: Hive Connection Pool Kerberos Issue

Re: NiFi user getting Tez exception

Re: Help with Hive Regex extract.

Re: How to send headers to InvokeHTTP process.?

Re: How to send headers to InvokeHTTP process.?

Re: NiFi ListGCSBucket Error

Re: How to send headers to InvokeHTTP process.?

NiFi ListGCSBucket Error

How to send headers to InvokeHTTP process.?

Re: How to execute google big query from NiFi.?

Re: How to execute google big query from NiFi.?

How to execute google big query from NiFi.?

Re: Split JSON issues

Re: Split JSON issues

Split JSON issues

Re: NiFi FlattenJSON is not working.

Re: Issue with ConvertRecord or ConvertJSONToAvro ...

NiFi FlattenJSON is not working.

Re: Issue with ConvertRecord or ConvertJSONToAvro ...

Re: Issue with ConvertRecord or ConvertJSONToAvro ...

Re: Upgrading HDF 3.0 to 3.1 On Windows Server.

Issue with ConvertRecord or ConvertJSONToAvro when...

Upgrading HDF 3.0 to 3.1 On Windows Server.

Re: Help Creating Hive Tables on nested JSON files

Help Creating Hive Tables on nested JSON files

Re: Array of JSON records to Avro with Nifi

Cant log in to NiFi after changing certificates an...

Re: NiFi JVM settings for large files.

Re: NiFi JVM settings for large files.

Re: NiFi JVM settings for large files.

Re: NiFi JVM settings for large files.

Re: NiFi JVM settings for large files.

Re: NiFi JVM settings for large files.