Member since
01-11-2016
355
Posts
230
Kudos Received
74
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8191 | 06-19-2018 08:52 AM | |
3148 | 06-13-2018 07:54 AM | |
3575 | 06-02-2018 06:27 PM | |
3888 | 05-01-2018 12:28 PM | |
5408 | 04-24-2018 11:38 AM |
04-27-2018
08:40 AM
@Ok , I managed to get this fixed. In oracle I can see it works a little differently. I used the following and this worked.I passed the table name followed by a "." and that seemed to do the trick. @Shu GIM.GIDB_GC_SKILL.*,'${now():toNumber():format('yyyy-MM-dd HH:mm:ss')}' AS LOAD_TMS
... View more
11-01-2017
06:44 PM
@Anishkumar Valsalam Great. Yes you can use the same approach. To have load balancing you need lot of files. If there's only few files, NiFi will not balance because it optimizes with batching. Read this article to understand how it works : https://community.hortonworks.com/articles/109629/how-to-achieve-better-load-balancing-using-nifis-s.html Thanks
... View more
11-09-2017
11:17 AM
@Abdelkrim Hadjidj Thanks a lot for your inputs into this, i will try this, i think this will surely solve my issue.
... View more
11-01-2017
03:14 PM
Hi @Abdelkrim Hadjidj for now i will implement it with GetFTP, Nifi is in service provider network and i cannot upgrade at will 😞 Do you maybe know a way to tell GetFTP not to download files that have already been downloaded in the past to avoid unneccesary buffers?
... View more
10-26-2017
01:12 PM
@Abdelkrim Hadjidj @Matt Burgess The use case here is getting the files from Mainframe, once a each day, however I got to know that there is no concept directory structure in Mainframes ( no idea how mainframes work), so it is not able to list the files (GetFTP as well as ListFTP and FetchFTP) Is there any other way to get around? I read some blogs and answers which suggested to use syncsort or Informatica powercenter. We tried using our current way of running shell script which goes and fetch the files. We can run the script using executeProcess and save it on one of the nodes (primary nodes), however primary node keeps changing. yesterday it was different primary today it is different one. In addition, if we mount it to share the directory across the node, it will be against the policies (too much admin work) Any help, thoughts
... View more
10-23-2017
05:06 PM
1 Kudo
You can use the Run Schedule property on the Scheduling tab of the processor to set the interval at which it will be scheduled to run, so for 10k events per second you can set it to "100 nanos".
... View more
10-20-2017
02:54 PM
@Andrew Lim thanks for clarifying further.
... View more
10-19-2017
08:53 PM
@Abdelkrim Hadjidj Perfect! Much clearer now. Thanks.
... View more
10-24-2017
06:39 AM
Thanks @Abdelkrim Hadjidj, It works well. It is interesting when just use $.* to capture content. I understood more about JsonPath Expression. Thank you again, :d
... View more
10-17-2017
01:51 AM
Hi @xav webmaster, i think this answer will help you for sure, Sample Flow :- GenerateFlowFile--> ExtractText-->UpdateAttribute-->PublishKafka GenerateFlowFile:- As for testing purpose i'm using this but in your case you are having some other processors ExtractText Processor:- in this processor i'm extracting the contents of flowfile as attribute. Ex:- adult,dog,bulldog,9,23,male,brown,4,etc The above content of flowfile by adding new property to ExtractText i'm going to extract the content and keeping that as attribute of the flowfile cnt_attr as (.*) //capture everything and add to ff as cnt_attr Configs:- Output of this processor:- Every flowfile will associated with the attribute called cnt_attr to it, we can use this attribute in UpdateAttribute Processor:- To dynamically change the topic names based on the cnt_attr attribute, for this case we need to use Advanced Usage of Update Attribute processor. Right Click on UpdateAttribute processor and click on Advanced Button in the lower right corner. Steps:- open above screenshot in new tab to see 1,2,3,4 steps and refer them with below steps 1. As mentioned in the above screenshot click on FlowFile Policy change to UseOriginal 2. Click on + sign at Rules and give name as adult_dog 3. Click on + sign at Conditions and give our check condition in it ${cnt_attrt:matches('.*adult.*dog.*')} 4. Click on + sign at Actions and give the attribute name as kafka_topic Value for the kafka_topic attribute as adult_dog New Rule:- for cat_dog conditions check is ${cnt_attr:matches('.*cat.*dog.*')} and Actions add attribute name as kafka_topic and value is cat_dog same as 2,3,4 steps above. summarize all the steps:- step 1 we are using original flowfile and
step2 we are creating a rule and
step3 adding conditions to check if the cnt_attr attribute satisfies or not
step4 if it satisfies then adding kafka_topic attribute with desired name to it. like this way we can add as many rules as we want in same UpdateAttribute Processor as you can see in my screenshot i have added 2 Rules(adult_dog,cat_dog). This processor checks which Rule has satisfied and updates kafka_topic attribute with the mentioned name in it. PublishKafka:- use the kafka_topic attribute in Topic Name property of processor ${kafka_topic} Flow Screenshot:- In this way we can use only one UpdateAttribute to dynamically change the value of kafka_topic based on update attribute processor and use same kafka_topic attribute to publish messages to respective topics.
... View more