Member since
03-05-2018
23
Posts
0
Kudos Received
0
Solutions
08-27-2018
08:53 PM
and please suggest how i can set batch_ts value in Update record processor .
... View more
08-27-2018
08:34 PM
Hi @Matt Burgess, We are using PutHDFS processor and further we have use Replace text processor to execute load data in path commond . I can see we have UpdateRecord processor so as per your suggestion i can use this processor after convertAvroto Orc processor ?. below is the flow detail . Regards, Shantanu
... View more
08-27-2018
04:05 PM
Hello , I am facing one issue while extracting data from Oracle Db through QueryDatabase Table processor . We have defined batch_ts column as Timestamp in the HIVE and we are trying to insert data from oracle by using QueryDatabase Table processor but it is inserting NULL value for the batch_ts column although we have defined parameter for this column .Below is the design. In the Update attribute processor we can see the batch_ts value . Let me know if you guys have some suggestion on it. Regards, Shantanu.
... View more
Labels:
- Labels:
-
Apache Hive
06-03-2018
07:22 PM
Hello @William Brooks i tired your options but looks like it not working.It has executed on the same time and 2 flow file inserted in to down stream .
... View more
06-03-2018
02:30 AM
Hello @Shu .I think you are giving me a reply based on subject line but now problem is as below. In above screen shot you can see the setting for Monitor activity with this setting my jobs are running in productions everyday but problem occurred when Querydatabase table processor triggered in different instance and monitor activity processor triggered on different instance and in this case since data is there so at 9 am Querydatabase table processor produce one flow file and its loaded to table and now at 10 Am monitor activity also gets executed and its generate one flow file and again my table got loaded with 0 count data .Can we restrict both processor can run on same instance or any other solution you have for this issue. Monitor activity is generating flow file in both case when querydatabses table processor is generatng flow file in cases of delta data and if querydatabses table processor is not generatngflow file in cases of delta data. We need flow file in case of when querydatabses table processor is not generatngflow file . Regards, Shantanu
... View more
06-02-2018
08:50 PM
Thanks you @William Brooks I will apply your concept and let you know the result. Thanks, Shantanu
... View more
06-01-2018
06:19 PM
Hello @Shu I am facing some issue with Monitor activity . If you do not mind can you help me on it. In above screen shot you can see the setting for Monitor activity with this setting my jobs are running in productions everyday but problem occurred when Querydatabase table processor triggered in different instance and monitor activity processor triggered on different instance and in this case since data is there so at 9 am Querydatabase table processor produce one flow file and its loaded to table and now at 10 Am monitor activity also gets executed and its generate one flow file and again my table got loaded with 0 count data .Can we restrict both processor can run on same instance or any other solution you have for this issue. Regards, Shantanu.
... View more
05-31-2018
09:30 PM
Thank you We are already using Monitoring Scope = cluster but still the problem is there.
... View more
05-31-2018
02:50 PM
Hello Guys, I am facing some issue with Monitor Activity processor . We are extracting data by using QueryDatabaseTable processor its mean we are extracting only delta records but some time delta is not coming so we have used Monitor Activity processor in this scenario to generate one dummy flow file for keep moving my down stream flow but the problem is our Nifi is running on 3 server in production and we have scheduled QueryDatabaseTable processor at 9 AM UTC and Monitor Activity processor at 10 AM UTC and when the flow started at 9 am QueryDatabaseTable processor runs at different server and it has extracted data and it send to down stream and loaded to hive now at 10 am Monitor Activity processor also got executed and generated a flow file because its has executed on different server is there any way to run both processor on same server or any setting inside Monotor activity processor? @shu Regards, Shantanu.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache NiFi
05-22-2018
08:06 PM
Thanks for the reply @shu and @Dukool SHarma . I am not writing any Map and reduce job. On the query level i.e in hive query can we call this LazyOutputFormat?. My table formate is ORC. Regards, Shantanu
... View more
05-18-2018
03:49 PM
Can you please explain how we can implement this concept? Regards, Shantanu.
... View more
03-26-2018
04:52 PM
If i will set Continually Send Messages is true then everyday if data will not available then i can get flow file? right now it has in false stat.
... View more
03-26-2018
01:09 PM
Another thing wanted to add here .For copy Attributes we need to set True i have given false .You can look in to below screen shot. Regards, Shantanu
... View more
03-26-2018
12:51 PM
Thanks Shu, I have used above Cron scheduled but today the behaviour of Monitor activity is very strange, yesterday it ran successfully when in QueryData base processor was not getting any delta record it has generated flow file and sent to execute sql but today again there was not data available for query databases but today Monitor activity not triggered. I do not know what I am missing here flow is same as above just in scheduling i have given 0 0-1 10 1/1 * ? * with threshold value 1 min. Could you please suggest here .I am attaching flow setting for Monitory activity. I have given Execution on Promary node is there any thing with this and Automatically Terminate Relationships i have given activity.restored and success while connecting to QueryDatabases and inactive while connecting to Execute Sql. Please have a look in below screen shot for more detail Regards, Shantanu.
... View more
03-24-2018
10:26 PM
Hello Shu. Sorry to bother you again.just I am curious for above Cron schedule 0 0-1 10 1/1 * ? *,will work on daily base or only for one day because my jobs scheduled to run daily.could you please give me a reply on it Regards Shantanu
... View more
03-24-2018
04:28 PM
Thanks Shu. what is 1/1 in your corn scheduling ? will it impact for next run if i will schedule on 0 0-1 10 1/1 * ? *, Regards, Shnatanu
... View more
03-24-2018
02:38 PM
Thank you Shu, If i will change the threshold value 5 min to 1 min then in this cases i need to change scheduling of Monitor Activity processor? Regards, Shantanu
... View more
03-24-2018
12:13 PM
Hello Guys, I have used Query Databases processor for extracting delta record from table but in cases if delta is not there Querydatabases processor will not generate flow file and my down stream job depends on flow file .So i have used monitor activity and designed a flow and when was the first run its working fine in cases of Zero delta record and it has generated flow file but now in day 2 same situation happened and in table there was not delta record and now monitor activity processor is not generating flow file. Can any one suggest me what can be a problem? Query data bases processor i have given a CRON Driven at 9 am UTC and Monitor Activity at 10 am UTC with Thresold Value 5 min. Regards, Shantanu flow-nifi.png flow-nifi-2.png
... View more
Labels:
- Labels:
-
Apache NiFi
03-23-2018
01:14 PM
Thank you for your quick response . Here in my scenario lettle different I have a .txt file with delimited '|' and we have some 10 columns and one of the column is STORE_ID So i wanted to split that file in to 2 part based on STORE ID value .ex: STORE ID 1 to 10 will go to different and 10 to 20 will go to different file.In your above solution how can i achieve this value range ? Please suggest me . Regards, Shantanu
... View more
03-23-2018
04:19 AM
Hello Guys, I wanted to create on flow in Nifi for splitting single file into 2 file based on one column values.Is there any way to split file on to 2 file .Input file format is a .txt file. Please provide your approach. Regards, Shantanu. @Shu
... View more
- Tags:
- splittext
Labels:
- Labels:
-
Apache NiFi
03-07-2018
06:29 PM
@shu Can we do this by using UpdateAttribute processor with Advance options if possilbe please explain it.
... View more
03-05-2018
09:49 PM
Hi All, Need help to sort-out on one scenario . I am using QueryDatabase processor for extracting delta record from the source and then i am using update attribute processor and finally ConvertAvroToOrc processor and output of these i am going to inject in our data lake that is hive. When the nifi trigger first time it extracted all the data and injected in dlake successfully but in day 2 when there were no record available in terms of new insert or delta it did not picked any record that is correct but its has not updated any value in the data Provenance section and my down stream is depend on it so if it will not update any thing then down stream will not get any flow file and down stream jobs are getting failed. So if any one can suggest me how to handle this type scenario where no records are available in the source for that day and we can generate one flow file to move further with zero count or something like that..Thanks you !!
... View more
- Tags:
- querydatabasetable
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi