About ankit_jain

ankit_jain · ‎10-05-2016

Thanks Bryan. This was really helpful. I have few questions - > how can I measure performance of any flow. I'm posting one million messages on Kafka topic(partition=8) later my nifi data flow(4 node cluster as described above) picks all the messages and store in Hdfs. I can verify the count as currently I'm sending one message/flowfile so there would be one million files in Hdfs. > How/Where can I check the ingestion speed ? If there is no direct way then will it be a good idea to write a custom processor to capture start and end time of process?

ankit_jain · ‎10-04-2016

@Bryan Bende I have created 4 node Nifi cluster and now trying to test performance of following data flow - Read data from Kafka topic and write on HDFS. 1. I have two broker on my hadoop cluster. 2. I have created a kafka topic with 3 partition and 1 replication factor. 3. I have written a Kafka processor which post 100k message of 3-4 byte in a single run. 4. I'm using 4 consumeKafka processors to read data from this topic with same group Id and later passing this data to 4 putHdfs processors which write data to 4 different location in HDFS.kafka-to-hdfs-flow.png Idea behind this flow is to achieve parallel processing. but when i run this flow, it takes very long time to ingest data in HDFS. Questions - 1. Is this a correct usecase? 2. What else i could try to achieve maximum performance? i've already gone through this post - https://community.hortonworks.com/articles/16120/how-do-i-distribute-data-across-a-nifi-cluster.html 3. My usecase is similar to Pulling data usecase described in above post. But as there is no NCM in Nifi 1.0. will it work the same way as described? 4. How can i verify if this flow has ingested all the messages successfully and how long it took to ingest those messages? is there any direct way to verify on Nifi UI?

ankit_jain · ‎09-30-2016

i tried deleting the connection flow, it didn't work as well. later i restarted NIFI and processor came out of indefinite block state.

ankit_jain · ‎09-29-2016

Please check attachment i''m not able to delete the consumeKafka processor. i also wanted to know what does the small icon on right-upper corner of processor with some number mean. i guessing this must show some kind of warning messages but nothing happens on mouse hover.nifi-processor-error.png

ankit_jain · ‎09-27-2016

Thanks for letting me this know. But i am not able to view Nifi UI in any other browser. i have generated certs using tls toolkit as mentioned in your post and used following command - ./tls-toolkit.sh standalone -n 'host1,host2' -C 'CN=admin, OU=my cert' -o '/nifi-1.0.0/cert-target-dir' later imported the .p12 file in browser. except IE everywhere i'm getting secure connection failed response. please check the attached screen shot of IE and firefox nifi-security-error.png

ankit_jain · ‎09-27-2016

I'm using IE. surprisingly the certificates i generated using tls toolkit does not working in other browser. For key icon issue i checked in browser console there comes a javascript error when i click on key icon. attached is screenshot of same. nifi-ui-error.png please check if you can help.

ankit_jain · ‎09-26-2016

Thanks for this suggestion. i want to tell you that cluster is up now. I rechecked everything. configuration was fine everywhere it just that i was setting this cluster on Azure and default nifi port 9443 was only open on one of the node hence other nodes were not coming up. I have setup a 3 node cluster now and able to see the UI on all three nodes. Now I am going as per your post and trying to set Access Policies. As per your post " we can create a policy for the root process group by clicking the key icon in the operate palette on the left" but this key icon does not work. its not grey but it doesn't open any popup. is there any catch behind this? i checked user logs as well but find nothing.

ankit_jain · ‎09-23-2016

HeyBryan, I tried the way you mentioned, still getting same error/exception. one thing i didn't understand is that users.xml and authozations.xml files are auto generated files. although i tried coping these two files from my working node to other non-working nodes as you suggested however when i let them self generate authorization.xml remain same across all nodes but inside users.xml value of identity tag differs and always set to respective node host names. please check the screenshot. do not know if this is correct behavior.nifi-users-xml.png

ankit_jain · ‎09-22-2016

here it is all log files. PFA . I can see 1/3 node on UI. although icons are not accessible because other two nodes are down nodes. i have also notice this exception in logs "org.apache.nifi.controller.UninheritableFlowException: Proposed Authorizer is not inheritable by the flow controller because of Authorizer differences: Proposed Authorizations do not match current Authorizations" hence also attaching authorizers.xml file from all three node.nifi-cluster-logs.zip

ankit_jain · ‎09-22-2016

I just tried this approach - first node went up but other nodes are throwing same error. PFA screenshot attached.

Online	Offline
Last Visited	‎06-20-2017 09:13 AM

Member Since	‎06-23-2016 03:15 PM
Last Visited	‎06-20-2017 09:13 AM
Posts	43
Kudos received	3

Cloudera Community

Re: Best way to check nifi cluster performance.

Best way to check nifi cluster performance.

Re: unable to delete consumeKafka processor on can...

unable to delete consumeKafka processor on canvas

Re: unable to set up nifi cluster using NIFI 1.0.0

Re: unable to set up nifi cluster using NIFI 1.0.0

Re: unable to set up nifi cluster using NIFI 1.0.0

Re: unable to set up nifi cluster using NIFI 1.0.0

Re: unable to set up nifi cluster using NIFI 1.0.0

Re: unable to set up nifi cluster using NIFI 1.0.0