About Siraj

Siraj · ‎12-29-2019

@ask_bill_brooks , Sorry, i am not seeing the accept as solution option in my screen. Thanks.

Siraj · ‎12-24-2019

Hello All, The below pipeline solved the issue. The naming convention between the table and the Property/value in Nifi has to be set same. If not, it cannot recognize and either it will fail or NULL gets loaded. It is not as simple as expanding the the JSON using SQLServer query!!! Note: Value data in the EvoluteJSONPath is setup based on the FlattenJSON output. I had the "Seperator" as '_' and "Flattenmode" as 'dot notation' in the FlattenJSON setup.

Siraj · ‎12-24-2019

@Shu_ashu / All, I have a nested JSON (not an array). I am able to read it using "EvaluateJsonPath" for all the direct attributes (IMEI or Counter in my case). The nested attributes($.Events.EvenType.Name or $.Events.Properties.Latitude) is not recognized and they are getting populated as NULL in my destination table. Can you please advise? Data Pipeline is as: GetFile --EvaluateJSONPath -- ConvertJSONtoSQL -- PutSQL EvaluateJSONPath Property (as attached): Sample JSON Data: { "IMEI":"77777777777777", "Events": { "EventType":{"Name":"EventName","Time":1577184434,"ReceievedTime":1577184434,"MsgId":1493134167}, "Properties":{"ThingUnitId":"779000000000062","Latitude":"24","Longitude":"46" },"Counter":0 } Thanks

Siraj · ‎12-23-2019

@MattWho , Thank you for the details. I can understand your point that same file cannot be accessed across cluster nodes. But, I am using Nifi is single node (without cluster) and was thinking that this should work. Yes, i do use "Update Attribute" with the below file name conventions. This generates separate flow file for every message. I am trying to have this as one single file per node. ${filename:replace(${filename},"FileName_"):append(${now():format("yyyy-MM-dd-HH-mm-ss")})}.Json Thank you

Siraj · ‎12-23-2019

Hello All, I do face a similar situation. I have tried using MergeContent before the PutFile. It is ending up with the Error:file with the same name already exists. My Data flow is: ConsumeMQTT (Reads Source) -- MergeContent (MergeFile) -- PutFile (Writes File) Note: If i setup the "Conflict Resolution Strategy" in the PutFile to "replace", this flow is working as only the latest value gets stored in the file. I cannot append data in the same file using the data flow as above. Kindly your inputs.

Siraj · ‎12-22-2019

Hello All, I am reading data from a messaging queue (as text message) and the data pipeline is working fine if i store every message in a separate text file. The next step is to Merge all the messages in to a single file (txt) and i placed MergeContent Processor before the PutFile. It endsup with the error: Error: file with the same name already exists. The Data pipeline is as : 1) ConsumeMQTT (Reads Source) -- UpdateAttribute (SetsFileName and txt type) -- MergeContent (MergeFile) -- PutFile (Writes File) 2) ConsumeMQTT (Reads Source) -- MergeContent (MergeFile) -- PutFile (Writes File) Note: If i setup the "Conflict Resolution Strategy" in the PutFile to "replace", this flow is working as only the latest value gets stored in the file. I cannot append data in the same file using the data flow as above.

Online	Offline
Last Visited	‎01-05-2020 09:06 AM

Member Since	‎12-10-2019 06:14 AM
Last Visited	‎01-05-2020 09:06 AM
Posts	8

Cloudera Community

Re: Composing Json in Nifi

Re: Composing Json in Nifi

Re: Composing Json in Nifi

Re: Append multiple tweets into single file using ...

Re: Append multiple tweets into single file using ...

Nifi MergeContent processor with PutFile (Error: f...