Member since
03-26-2018
108
Posts
10
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
724 | 06-24-2020 09:21 PM | |
650 | 05-20-2020 06:12 AM | |
529 | 02-25-2019 12:39 AM | |
1164 | 02-21-2019 11:33 AM | |
400 | 02-21-2019 11:16 AM |
07-29-2020
08:36 PM
hey Guys , quick one. With the infer schema approach now being available , do we need to have the avro registry ?
... View more
07-29-2020
04:00 PM
Did you get a solution to this . I am also getting a communication error . My Nifi Instance and Mysql are on the same linux server .
... View more
07-08-2020
10:35 PM
HI Team , We have been trying to push data into AWS S3 Bucket . However we are getting a forbidden issue :403 error. Amazon S3 due to Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null): com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden We are able to access the bucket via Postman and all the credentials are right. Is it something to do with the network firewalls in my Organization that is creating this issue.
... View more
Labels:
07-07-2020
06:25 PM
1 Kudo
Thanks @stevenmatison for your response on this
... View more
07-07-2020
05:01 AM
Hi Friends , I am trying to start a process group from Unix for a secured Nifi instance. I do have my trust store and key store files available but I am not sure how to reference them in the curl command. I read somewhere that I can give the -k option with curl if I want to do away with specifying the certs. So I tried running the following command curl -k -X PUT -H 'Content-Type: application/json' -d '{"id":"e3d11d1d-327b-1b5b-96a7-3cb59c48df17","state":"RUNNING"}' https://XXXX:8443/nifi-api/flow/process-groups/e3d11d1d-327b-1b5b-96a7-3cb59c48df17 However I am getting the following error. curl: (56) Received HTTP code 403 from proxy after CONNECT I guess I will need to reference my trust store and keystore , can someone help me here on how to do this ? @Shu_ashu sir - can you please help me here @Matt
... View more
Labels:
07-07-2020
02:42 AM
Hi Friends I tried running the curl from my Linux session on which Nifi installed. This is a secured nifi instance . I got the forbidden error. Please see the error below -bash-4.2$ curl -i -X PUT -H 'Content-Type: application/json' -d '{"id":"e3d11d1d-327b-1b5b-96a7-3cb59c48df17","state":"RUNNING"}' https://XXXXX:8443/nifi//nifi-api/flow/process-groups/e3d11d1d-327b-1b5b-96a7-3cb59c48df17; HTTP/1.1 403 Forbidden Server: squid/3.5.20 Mime-Version: 1.0 Date: Tue, 07 Jul 2020 09:34:57 GMT Content-Type: text/html;charset=utf-8 Content-Length: 3347 X-Squid-Error: ERR_ACCESS_DENIED 0 Vary: Accept-Language Content-Language: en X-Cache: MISS from XXXXX X-Cache-Lookup: NONE from XXXX Via: 1.1 XXXXXsquid/3.5.20) Connection: keep-alive curl: (56) Received HTTP code 403 from proxy after CONNECT @Shu_ashu - can you please help me here sir. Is it that I need to specify the certs somewhere ?
... View more
06-24-2020
09:21 PM
Hi All , I fixed this by changing the source URL to jdbc:mysql://127.0.0.1:3306/source . I still need to understand the reason as to why it works this way when both the applications are installed on the same linux box . Also to note that in my case I have this linux installed on a VM on Google Cloud Platform I will work on it and post the reasoning for reference. Thanks
... View more
06-24-2020
06:24 PM
Hi Team , I am having issues while connecting to mysql database using apache nifi. The error is as follows Cannot create PoolableConnectionFactory Please note that Apache Nifi and MySQL is installed on the same Linux server and hence there shouldn't be any problems with the 2 applications talking to each other. The database connection pool settings is provided below The error when I try extracting the records is provided below This is something very urgent as we are doing a POC and if this succeeds our Projects will be based on this. Any help in this regard will be much appreciated. @Matt @Shu
... View more
Labels:
06-01-2020
04:24 AM
Hi Alim , thanks for this very wonderful post on using Nifi as CDC. Is there a provision in Nifi to use something similar for Oracle Database ?
... View more
05-20-2020
06:12 AM
Thanks , enabling the GIGC has helped along with reducing the Heap JVM settings in the bootstrap conf file. Thanks again
... View more
05-16-2020
11:20 PM
Hi Team , We are using the Nifi Open source version . However we are seeing a lot of work coming our way as far as Nifi is concerned and for that I wanted to know if our team can start contacting Cloudera for the much needed Paid Support. Whom do I need to contact for this and does Cloudera offer stand alone support for Open Source Nifi Installations at a cost or do we need to go ahead and purchase the Cloudera suit ?
... View more
Labels:
05-16-2020
11:13 PM
Hi Team , I need some advice on how to tune our Nifi Cluster. We have a 3 Node Nifi Cluster that processes events in a real time streaming fashion which means our flows are running 24 by 7 . Each of the Nifi Linux servers have a RAM of 31GB. We have set the JVM memory setting in the bootstrap conf file to 28 G. #JVM memory Settings java.arg.2=-Xms28G java.arg.3=-Xms28G However when I run a free -h command on the servers this is what I get which clearly shows that the used Memory is 30 G. As a result of this , we keep getting incidents raised from the Linux team stating that the Nifi servers are always hovering over the 95 % mark and on top of that if any processor in the flow starts misbehaving , it shoots up to 100 percent and the server goes unresponsive. So my question is , even after setting the threshold to 28 G in the bootstrap conf file , why do we still see Nifi taking up almost all the memory allocated to the server ? @Matt
... View more
Labels:
05-04-2020
02:34 PM
Hi all , This is just a random non technical question but none the less important. With many moving into cloud and On Premise Hadoop Offerings on the decline , how do we see the future of a tool like Apache Nifi . Personally , I still see this being used . In our organization , we had the complete big data stack however we made a shift to the cloud last year . We hardly use the on premise hadoop file system now and it is slowly getting decommissioned with us using more and more of cloud storage now. However we continue to use Apache Nifi in a robust way as it is still a very good tool to manipulate the data in motion . We now use Apache Nifi to stream data onto our Cloud storage area. I wanted to hear from you experts on your thoughts on the future of Apache Nifi and how do you perceive its growth in times to come .
... View more
Labels:
04-30-2020
02:07 PM
Thanks for the revert mate. Much Appreciated.
... View more
04-29-2020
08:47 PM
Hi , Thanks for the revert . I just did the same last night by taking up the NAR file from the previous Nifi version and placing it in the lib directory of the current Nifi version . I then went for the restart and then could see the processors. However , this I believe this isn't right and every new version should be backward compatible . Yes , there can be a new flavor of a processor i.e. a few more properties here and there but not having the processor at all in the new release is actually not acceptable as it can break your existing flows . It has caused us a lot of headache as our process runs in real time . Because of this there was an outage of almost a day as it took us some time to figure out the cause and then come up with a solution . Is there a tech support where this can be put up so that subsequent releases should have backward compatibility .
... View more
04-28-2020
09:04 PM
Hi Experts. We recently upgraded our Nifi Version from 1.9.2 to 1.11.4 however many of our flows arent working as some processors arent available in the new version. One of them being the "Infer Avro Schema" . I am not sure why this should be the case as every new release should be backward compatible. Is there a way to restore this processor onto the latest version as our code is impacted. We can also see that this has been an issue and raised separately as well. Please find the link here http://apache-nifi-users-list.2361937.n4.nabble.com/GRPC-processors-missing-after-upgrade-to-1-11-3-td9161.html#a9166 cheers Abhinav
... View more
Labels:
03-09-2020
03:47 AM
hi , I just want to access the following link provided in the responses above https://community.hortonworks.com/content/supportkb/167107/unable-to-perform-http-connections-throug... ) I tried clicking on the link above however , I didnt get any result . The page was blank . Can someone provide me the correct link here so that I can get to know how to configure the proxy in Nifi config.
... View more
03-09-2020
01:43 AM
Hi Guys, I have stumbled on the same issue however I am not sure where to add the proxy in the nifi config. I also tried to click on the link suggested however couldnt get anything there. Can someone please let me know where / how to add the proxy on the nifi server so that we can then connect to snowflake. Its a big show stopper for us in our project . Really need some help here !!! Many thanks in Advance
... View more
03-05-2020
03:24 PM
Can someone please point to me how and where to add the proxy in the nifi.properties file so that I can get Nifi connected to Snowflake.
... View more
04-18-2019
01:28 PM
Thanks Matt . We edited the flow.xml file and removed the reference to 1.4 Update Attribute bundle from it and used the edited flow file to start Nifi and the issue got fixed. No more ghost Update Attribute Processors. Thanks alot for all your help and detailed explanation on this,
... View more
04-10-2019
11:38 PM
Thanks Matt. I had a look at the flow.xml file and could see some multiple bundled up versions of the update attribute processor . There was no mention of the custom processor though however we had multiple versions like 1.4 and 1.8. Is there a possibility that we could tweak the flow.xml file and remove these old instances and get the flow.xml uploaded ?
... View more
04-08-2019
11:54 PM
HI Matt , Thanks for your answer. I am attaching the screen shot below . You can see how just the " Update Attribute" seems to have those dashed lines and is in "invalid" state. I doubt think our developers made any custom change to the update attribute processor. Let me know what you think about this meanwhile I have looked at the app.log and can see this error You were bang on when you said about the 'Ghost' Implementation. What do you think should be the next steps -bash-4.2$ cat nifi-app-*.log |grep "Unable to find bundle for coordinate" Caused by: java.lang.IllegalStateException: Unable to find bundle for coordinate org.apache.nifi:nifi-update-attribute-nar:1.8.0 ERROR [main] o.a.nifi.controller.ExtensionBuilder Could not create Processor of type org.apache.nifi.processors.attributes.UpdateAttribute for ID 629d355f-0167-1000-ffff-ffffbc497b67; creating "Ghost" implementation
... View more
04-08-2019
07:57 PM
Hi Matt , for the existing flow , the "update attribute" processor was in an inconfigurable state i.e it had this dotted line all over it and when I clicked on it it wasnt showing anything. This has happened for all the "update attribute" processors in our flow. For now , we have had to delete this processor and add a new one and add configur it. We also realize that the new "update attribute" processor has an extra feature with version 1.9 which wasnt available in 1.8. However this should be taken care of automatically with an upgrade.
... View more
04-08-2019
04:30 AM
Update Attribute processor Not working after upgrade from version 1.8 to version 1.9. We recently upgraded our Nifi Cluster from version 1.8 to version 1.9 and we were surprised to see the update attribute processor all going invalid for our existing flows. We had to manually fix the existing flow but this ideally should have been handled with the upgrade . Is there a way to overcome this ?
... View more
Labels:
02-25-2019
12:39 AM
1 Kudo
For now we set the nifi.queue.swap.threshold to 150000 and this seems to have done the trick. Loads are performing well and there is no build up of the queue. thanks all
... View more
02-24-2019
11:20 PM
Hi All, We have just productionized our Nifi flows which is a real time streaming application . We have a cluster of 2 Nodes and each node has 6 processors and a high memory of 28 GB. On the Nifi side the memory has been set to 24 GB. Moreover , we tend to see a queue size reaching to around 90,000 or more. For this I have set the property nifi.queue.swap.threshold to 90000 . We also have a disk space of 200 GB. Despite having such a powerful configuration , we still get flow file repository failed to update. I understand that initially the JVM memory is uitilized for the flow file processing and if that gets overused , it spills over to disk space . Our disk space is also pretty high at 200 GB however we still keep on seeing this error. Any pointers or suggestion in this space will be of great help.
... View more
- Tags:
- flow
- nifi-streaming
Labels:
02-24-2019
06:54 PM
Hi Rishabh , did you try shutting down the nifi instances and start them one at a time ?
... View more
02-24-2019
06:53 PM
Hi All , Apologies , I was looking at the wrong property. The property that I should had looked at is nifi.cluster.protocol.heartbeat.interval . I have now set this to 60 seconds and this has fixed my issue. However , I need to check if this has any issues to the overall performance
... View more
02-24-2019
10:08 AM
Hi All , I am getting the error of the node getting disconnected as there was no heatbeat from the node in 42 seconds. However I then changed the nifi.zookeeper.connect.timeout property to 60 seconds and restarted the Nifi server. The problem is that despite this change I am still getting the disconnection error. Is there any other property that I need to set to fix this apart from the zookeeper.conect.timeout propery ??
... View more
Labels: