Member since
06-08-2017
1049
Posts
518
Kudos Received
312
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 11202 | 04-15-2020 05:01 PM | |
| 7101 | 10-15-2019 08:12 PM | |
| 3096 | 10-12-2019 08:29 PM | |
| 11437 | 09-21-2019 10:04 AM | |
| 4321 | 09-19-2019 07:11 AM |
10-25-2017
06:47 PM
1 Kudo
@Shu Thanks I just did that before I saw this answer, it worked. Appreciate it. Using the website you refered yesterday http://www.cronmaker.com/ Thanks again
... View more
10-25-2017
03:41 AM
@PJ Yeah, it might be that case.Because if you are having large number of records then it will take a lot of time to convert ORC data to csv format and if you compare these two process executing query with insert overwrite directory will perform much faster with no issues and also we can keep what ever delimiter we need and we don't need to worry about size of the data.
... View more
10-25-2017
07:52 AM
Thanks Adrian Oprea
... View more
10-20-2017
07:53 PM
@Shu @dhieru singh By default their is no guaranteed order in which FlowFiles are pulled from he queue feeding any given processor. This is because NiFi favor performance over order. If you want enforce some sort of order in which FlowFiles are pulled from a inbound queue, you must add a "Prioritizer" to the inbound connection. By default, no prioritizers are added. To apply a prioritizer, simply drag the desired prioritizer(s) to the "Selected Prioritizers" box. Regardless of strategy used in your DistributeLoad processor (round Robin or next available), There will not be a continuos order to the FlowFiles queued to either MergeContent processor. Thanks, Matt
... View more
10-20-2017
02:55 PM
1 Kudo
In Apache NiFi 1.5.0 (not yet released at the time of this writing), SelectHiveQL (via NIFI-4473) will have a property to Normalize Names for Avro, so you won't have to do the alias.
... View more
10-18-2017
07:54 PM
Hi @Shu Thanks for the reply, Yes you are absouletly correct about the error message being a generic one complaining about the config issue. Here my udp port was already occupied by some other app. Chnaged the post number and worked like a charm Thanks again appreciate it Dhiren
... View more
10-18-2017
03:03 PM
Hi @rakesh chow, as ExecuteScripts are experimental in NiFi but you can acheive same results by using another processors as follows. 1.EvaluateJsonPath //Extract all the contents to attributes
2.UpdateAttribute //Update time stamps attributes with new format(yyyy-mm-dd hh...etc)
3.AttributesToJson //Recreate Json content by specifying all the attributes EvaluateJsonPath configs:- My input sample content is {
"ID":"1",
"CID":"1",
"DiscoveredTime":"Mon Sep 11 19:56:13 IST 2017",
"LastDiscoveredTime":"Mon Oct 09 23:38:55 IST 2017"
} in eval json processor extract all the contents of the flowfile to the attributes of flowfile by changing Destination property to flowfile-attribute and add new properties by clicking + sign at right corner. ID $.ID DiscoveredTime $.DiscoveredTime CId
$.CId LastDiscoveredTime
$.LastDiscoveredTime so i have extracted all the content of ff as attributes of ff, in your case you need to add all your json content keys of contents here and this processor keeps all the values of content to attributes. Config Screenshot:- UpdateAttribute Processor Configs:- This processor will helps to update the attributes on the fly we are changing our DiscoveredTime,LastDiscoveredTime attributes on the fly. add new properties 1.DiscoveredTime ${DiscoveredTime:toDate('EEE MMM dd HH:mm:ss ZZZ yyyy'):toNumber():plus(21600000):format("yyyy-MM-dd HH:mm:ss.SSS")} in this expression we are checking which timezone it is and converting that to number and adding 6 hrs(i think you don't need to do :plus(21600000)), after that i'm converting to yyyy-MM-dd HH:mm:ss.SSS format Your expression probably will be ${DiscoveredTime:toDate('EEE MMM dd HH:mm:ss ZZZ yyyy'):toNumber():format("yyyy-MM-dd HH:mm:ss.SSS")} 2. add new property for LastDiscoveredTime ${LastDiscoveredTime:toDate('EEE MMM dd HH:mm:ss ZZZ yyyy'):toNumber():plus(21600000):format("yyyy-MM-dd HH:mm:ss.SSS")} Your expression:- ${LastDiscoveredTime:toDate('EEE MMM dd HH:mm:ss ZZZ yyyy'):toNumber():format("yyyy-MM-dd HH:mm:ss.SSS")} Configs:- AttributesToJson Processor:- in Attributes list property add all the available attributes it will prepare a json messages with the attributes that you have mentioned here. ID,CID,DiscoveredTime,LastDiscoveredTime Change Destination property to flowfile-content Configs:- Input:- {
"ID":"1",
"CID":"1",
"DiscoveredTime":"Mon Sep 11 19:56:13 IST 2017",
"LastDiscoveredTime":"Mon Oct 09 23:38:55 IST 2017"
} Output:- {
"ID" : "1",
"LastDiscoveredTime" : "2017-10-09 23:38:55.000",
"DiscoveredTime" : "2017-09-11 19:56:13.000",
"CID" : ""
} In your case you need to mention all your attributes in attributes to json processor and it will prepare new json content with all your attributes, if you left this property as empty then all the attributes associated with the ff will be part of your json content. Flow Screenshot:-
... View more
10-17-2017
05:41 PM
@Shu Thanks a lot for quick reply. Much appreciated. This solved the issue. The problem was once I set the expression language for creating directory structure. The putFile was looking for the directory structure hence the error. So I had to delete the PutFile processor and the use your EL and it worked like a charm. Thanks again
... View more
10-24-2017
06:39 AM
Thanks @Abdelkrim Hadjidj, It works well. It is interesting when just use $.* to capture content. I understood more about JsonPath Expression. Thank you again, :d
... View more
10-19-2017
01:47 PM
yes it was helpful thank you
... View more