Member since
07-25-2018
174
Posts
29
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4917 | 03-19-2020 03:18 AM | |
3033 | 01-31-2020 01:08 AM | |
1231 | 01-30-2020 05:45 AM | |
2317 | 06-01-2016 12:56 PM | |
2695 | 05-23-2016 08:46 AM |
01-29-2020
09:28 AM
Hi All,
I have used three process groups in nifi (pg1,pg2,pg3). There is an output port called as pg1_Out coming out from pg1. This port is connected to pg2_in and pg3_in input ports.
pg1_out------>pg2_in
|
|
|
pg3_in
Problem : when i trigger these process groups then flowfiles are passing to only pg3_in but not to pg2_in. Ideally , the flow files should get passed to both input port.
I do not understand why am i seeing such behaviour in nifi.
Does anyone has faced such problem before? Please respose to me your thoughts on this. I would be really appreciated to your help.
... View more
Labels:
- Labels:
-
Apache NiFi
01-21-2020
09:52 AM
Hi , Thanks for your replay. Yes ,we can use wait and notify processor. But how would i ensure that previous has been completed or failed. What is that criteria or condition? Let me explain you current nifi flow,we have 3 layers and in/out for each layer always requires 4 flowfiles (number of flowfiles depend on input parameters). How can i integrate wait/notify pattern by keeping this in mind? Do you have any nifi example where it's already implemented? Could you please share nifi template xml with me.
... View more
01-20-2020
08:42 PM
Hi @DennisJaheruddi Thank you for writing to me. There are some situation or places in NiFi flow ,where i require to execute SQL INSERT for each flowfilea and at some places need to execute SQL INSERT only once irrespective on number of flowfiles. Could you please point me in right direction for the same?
... View more
01-20-2020
08:33 PM
Hi @DennisJaheruddi Thanks for you reply. Don't you think, if we use wait and Notify processor then pipeline will get proceeded ahead for next execution after the expiration of Expiration Duration attributes value of wait processor. The situation may occur like the execution is not completed for previous layer but NiFi flow get triggered due expiration of above attribute. It seems wrong behavior to me.
... View more
01-19-2020
05:36 AM
Hi All,
I am tyring to execute update query through putSQL processor. I want to update number of processed file count column of one of the transactional table in mysql.
The issue is that whenever multiple flowfiles coming as input to putSQL processor at a time,the processed file count is getting incremented and updated only once. Ideally,the behaviour should be like increment the counter one by one for each flowfile and then update the processed file count column in MySQL table.
NiFi flow:
Fetch processed file count using select query->updateattribute : processed file count+1->putSQL:update query.
Thanks in advance
Please share you response on it.
... View more
Labels:
- Labels:
-
Apache NiFi
01-16-2020
09:48 AM
Hello all,
Problem statement: We have 3 layers in our project,for each layer we have nifi flow. Currently, the layers or nifi flow for each layer is getting executed one after another (sequentially).
Execution flow: layer1->layer2->layer3-> so on.
The trigger point or input parameters to the flow is nothing but number of source files. Which will be processed further in every layer one by one.
I am looking for such solution ,where my next nifi flow for next layer should not get triggered until completion of previous one for previous layer and for all files.
And if current layer got succeeded for all the files then only next one will kick off.
Please suggest me your thought process on the same. I will be very appreciated on your response.
Thank you in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
06-08-2019
09:27 AM
Hi All, I want to create the custom hashmap accumulator in spark for one of my use case. I have already referred and implemented Accumulator as per code is given on below links but not found an end-to-end example for the same. I am trying to do it in java. https://www.brosinski.com/post/extending-spark-accumulators/ Problem : I want to know, How Can I increment value for each key in the same accumulator within foreach() transformation?
... View more
Labels:
- Labels:
-
Apache Spark
01-07-2017
03:22 PM
Thank you Rguruvannagari. This solution really worked for me.
... View more
01-07-2017
11:00 AM
Hi, I have installed below Hadoop services and all services are running fine services.png The problem is that when i am executing below query from Hive cli I am getting the exception in terminal as. Query : select count(*) from mytables; Exception: short exception description: Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://<nanodeHostIp>:8020/hdp/apps/2.2.9.0-3393/mapreduce/mapreduce.tar.gz) please find attached exception file exception.txt and I have also checked path on HDFS "/hdp" which is not completely present on file system so my question is, As have done fresh hadoop installation through Apache ambari then why ambari did not created files or directories under "/hdp" directory automatically. What is the solution for above problem? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Hadoop
01-03-2017
12:05 PM
Hi Ayub, Here is the link for above same question, https://community.hortonworks.com/questions/75818/issue-regarding-apache-atlas-rest-api-to-create-hi.html
... View more