@vincent yernaux There is no user configuration that needs to be done to get NiFi to create the nifi.pid file. Do you see any errors in the nifi-bootstrap.log about being unable to create the nifi.pid file? - I would suggest shutting down all 6 nodes. Then verify that the NiFi process is in fact stopped on all nodes. (ps -ef|grep nifi) Then restart your NiFi cluster again and monitor both the nifi-bootstrap.log and nifi-app.log for any errors/warns that may be related to pid file generation. - Thanks, Matt ... View more

@Veerendra Nath Jasthi Where did you get the keystore files you are using on each of your nodes from? I suggest performing a verbose listing on your keystore ( keytool -v -list -keysrtore <keystore,jks file> ) That listing should show a single "PrivateKeyEntry" and that should then show a "Owner" and "Issuer" as below exmaple does: Alias name: nifi-key Creation date: Apr 19, 2018 Entry type: PrivateKeyEntry Certificate chain length: 2 Certificate[1]: Owner: CN=nifi-sme-15.openstacklocal, OU=NIFI Issuer: CN=nifi-sme-26.openstacklocal, OU=NIFI Serial number: 162df02fcaf00000000 Valid from: Thu Apr 19 17:45:37 UTC 2018 until: Sun Apr 18 17:45:37 UTC 2021 Certificate fingerprints: MD5: B2:B3:A8:D0:DC:E4:98:1F:53:30:A6:B4:E0:79:41:1A SHA1: 04:D9:3A:84:7B:75:AE:90:DD:C9:41:D3:83:1C:4F:BB:3C:18:EC:FA SHA256: AD:69:23:80:A1:06:1A:6C:32:A4:4C:95:B5:0E:5F:0E:AA:12:BE:DF:05:84:B8:53:27:F3:D9:46:DD:89:03:7A Signature algorithm name: SHA256withRSA Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: A0 9D B4 20 80 B3 6D 31 70 2E 73 B0 7E E0 17 F9 ... ..m1p.s..... 0010: 3D 31 A1 B4 =1.. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:false PathLen: undefined ] #3: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth serverAuth ] #4: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Non_repudiation Key_Encipherment Data_Encipherment Key_Agreement ] #5: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: nifi-sme-15.openstacklocal DNSName: nifi-sme-15.openstacklocal ] #6: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 9B 43 A1 B1 3D 9B AF B4 1B 1B 8F DA 31 D2 14 88 .C..=.......1... 0010: 4E 3E 93 A9 N>.. ] ] - The "Owner" DN form each of your Nodes keystores should match the node identities you entered in your configs (case sensitive). - Also note that you have names every one of your entries as "Node Identity 1". You should instead have a unique number for each node identity. - My guess here is that maybe your keystore contains more then one "PrivateKeyEntry". Did you create a user certificate "CN=nifiadmin, OU=NIFIrsdevhdf2.medassurant.local, OU=NIFI" and import in to each of your nodes keystores? NiFi has no way to be configured to select a specific "PrivateKeyEntry" when multiple exist in same keystore. - The keystore should contain only 1 "PrivateKeyEntry". It may contain many "trustedCertEntry" entries. - Commonly your keystore.jks will contain only the single PrivateKeyEntry and your truststore.jks will contain 1 to many "TrustedCertEntry". - Once you have made the necessary corrections to your keystore.jks file and/or node identity configurations, you will need to delete the users.xml and authorizations.xml files that NiFi created as they are only created once. If they already exist, they will not be updated by changes you make to node identity configurations or initial admin identities. Once you can successfully access the secured NIFi UI as your initial admin, you will add the rest of your users and se their policies directly from within the UI. - Thanks, Matt - If you found this answer addressed your question, please take a moment to login to the forum and click "accept" on the answer. ... View more

@Gillu Varghese The imbedded ZK was implemented to make setting up new NiFi Clusters quick and easy for POC and development type activities. The errors you are seeing are expected and not much can be done about that. It is recommended that an external ZK is used by NiFi instead of the internal. Especially fro a production type environment. Thank you, Matt ... View more

@Rahul Soni @Gillu Varghese The GenerateFlowFile processor will create 1 GB of content for each FlowFile it creates. The FlowFile content does not live in heap memory space. - Each generated FlowFile will have a core set of FlowFile Attributes created. For example: -------------------------------------------------- Standard FlowFile Attributes Key: 'entryDate' Value: 'Thu Apr 26 14:52:12 UTC 2018' Key: 'lineageStartDate' Value: 'Thu Apr 26 14:52:12 UTC 2018' Key: 'fileSize' Value: '67' FlowFile Attribute Map Content Key: 'filename' Value: '7409235136254821' Key: 'path' Value: './' Key: 'uuid' Value: '119b16a1-7cb2-40ff-b92e-77bc733389e6' -------------------------------------------------- - You can however define attributes on attributes on each generated FlowFile by adding attributes via custom properties in GenerateFlowFile processor: FileSize in the case of heap does not matter when it comes to heap usage of queued FlowFiles. In this way you can create FlowFiles with as many attributes as you want: -------------------------------------------------- Standard FlowFile Attributes Key: 'entryDate' Value: 'Thu Apr 26 14:55:54 UTC 2018' Key: 'lineageStartDate' Value: 'Thu Apr 26 14:55:54 UTC 2018' Key: 'fileSize' Value: '0' FlowFile Attribute Map Content Key: 'attr1' Value: 'This is a test' Key: 'attr2' Value: 'This is a test' Key: 'attr3' Value: 'This is a test' Key: 'attr4' Value: 'This is a test' Key: 'filename' Value: '7409457340083769' Key: 'path' Value: './' Key: 'uuid' Value: 'f6254149-be47-46f3-a659-c5126ae80481' -------------------------------------------------- - You can adjust the run schedule and batch setting to control the number of new FlowFiles generated over a specific time period. - For example: Setting run Schedule to 5 sec and Batch Size to 1000, Every 5 seconds this procesor will produce 1000 new FlowFiles. - Thanks, Matt ... View more

@aman mittal The URL you are trying to reach is not a valid end-point so the "The specified resource could not be found" is the expected response. If you trying to access the NiFi UI, the proper URL would be http://localhost:8081/nifi - If you are trying to get a response from one of NiFi's api end-points, you need to be very specific about which end-point you are trying to hit. The following link provides the syntax for NiFI rest-api endpoints that will return: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html - Thank you, Matt - *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Gillu Varghese In order to build a dataflow like this you just need to know which FlowFile's attributes/metadata are held in heap versus which FlowFile's attributes are swapped to disk. To understand that take a look at this HCC article: - https://community.hortonworks.com/articles/184990/dissecting-the-nifi-connection-heap-usage-and-perf.html - So any connection with queued data with have some amount of heap footprint. You can use an UpdateAttribute processor to add additional Attributes to a FlowFile(s) - But you also have to understand that heap space is not only used by queued FlowFiles. - Thanks, Matt ... View more